Stone surface defect detection method based on mask and semantic collaborative optimization diffusion model
Through the coordinated optimization of the diffusion model between mask and semantics, the problems of low accuracy and low efficiency of traditional stone defect detection methods are solved, and the functions of high-precision detection of stone surface defects and visualization of defect areas are realized.
Patent Information
- Application Number
- CN202510067481.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-16
- Publication Date
- 2025-04-22
- Estimated Expiration
- 2045-01-16
AI Technical Summary
Traditional stone defect detection methods have low detection accuracy and low efficiency, and the types and forms of stone surface defects are diverse, and the visual characteristics are not significant, which increases the detection difficulty.
The diffusion model is optimized in a coordinated manner by obtaining the data set formed by positive sample pictures, negative sample pictures and mask pictures, and diffusion forward propagation, mask characterization distillation, semantic characterization generation, diffusion reverse denoising, abnormal score calculation and abnormal sorting mechanism visualization, so as to achieve high-precision detection of stone surface defects.
It effectively improves the detection accuracy and efficiency of stone surface defects. Through the coordinated optimization of mask and semantic information, the ability to identify different defect types is improved. The generated reconstruction pictures and input pictures calculate abnormal scores, filter out defect areas with high abnormality, and provide visual defect area information.
Smart Images
Figure CN119477922B_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the field of defect detection, and in particular relates to a stone surface defect detection method based on a mask and semantic collaborative optimization diffusion model. Background Art
[0002] With the expansion of stone application areas, the demand for stone surface defect detection is growing. Traditional stone defect detection methods mostly rely on manual inspection or simple machine vision technology, which have problems such as low detection accuracy and low efficiency. In addition, the types and forms of stone surface defects vary, such as cracks, spots, stains, etc. These defects are often not obvious in visual features, which increases the difficulty of the detection task. Summary of the invention
[0003] The purpose of the present invention is to propose a stone surface defect detection method based on mask and semantic collaborative optimization diffusion model, which can effectively improve the detection accuracy of defects.
[0004] The present invention is achieved through the following technical solutions:
[0005] The stone surface defect detection method based on mask and semantic collaborative optimization diffusion model includes the following steps:
[0006] Step S1.1, obtaining a stone surface image containing defects, taking the area without defects in the stone surface image as a positive sample image, and the target area containing defects as a negative sample image, and obtaining mask images corresponding to the positive sample image and the negative sample image, and the positive sample image, the negative sample image and the mask image form a data set;
[0007] Step S1.2, diffusion forward propagation step: respectively input the positive sample images and negative sample images in the data set as input images into the encoder to obtain input representations, and perform Gaussian noise forward diffusion on the input representations to make them gradually approach completely random noise, thereby obtaining full noise representations;
[0008] Step S1.3, mask representation distillation step: after dot-multiplying the input representation obtained from each input image with the mask image corresponding to the corresponding input image, the input is input into the mask-guided knowledge distillation network, so that the student model of the knowledge distillation network learns the mask teacher model to generate the mask representation;
[0009] Step S1.4, semantic representation generation step: input the input representation into the semantic guidance enhancement network containing the multi-dimensional feature pyramid, and the semantic guidance enhancement network gradually and accurately locates the defect area to obtain the semantic representation;
[0010] Step S1.5, diffusion reverse denoising step: after splicing the full noise representation, mask representation and semantic representation, reverse diffusion is performed to gradually remove noise, and decoding is performed to generate a reconstructed image;
[0011] Step S1.6, abnormality score calculation step: input the input image and its corresponding reconstructed image in the data set into the feature extraction network to obtain feature maps of different scales, and perform cosine similarity calculation on the feature maps of different scales to obtain the abnormality score;
[0012] Step S1.7, visualization step of the anomaly sorting mechanism: sort the anomaly scores corresponding to each input image, form an anomaly score list based on the anomaly scores greater than the set anomaly threshold, and feed back the heat map of the input image corresponding to the anomaly score list to the user.
[0013] Furthermore, in step S1.2, according to the formula Get full noise characterization ,in, represents the input representation, represents the encoder, X represents the input image, represents the average anti-noise strength, Indicates the anti-noise intensity corresponding to the i-th pixel of the input image, represents the noise intensity corresponding to the i-th pixel of the input image, T represents the number of pixels in the input image, represents a continuous multiplication operation, represents random Gaussian noise that follows a [0,1] normal distribution.
[0014] Furthermore, the step S1.3 specifically includes the following steps:
[0015] Step S1.31: According to the formula Get the potential representation M0 of the mask, where Mask represents the mask image corresponding to the input image X;
[0016] Step S1.32: Input the potential representation M0 of the mask into the mask teacher model and student model of the knowledge distillation network respectively, and design the overall loss function Train the knowledge distillation network to obtain the student model parameters after knowledge distillation , and then generate the mask representation , where the overall loss function Expressed as , is the mean square error, represents the Euclidean distance, represents the teacher distribution obtained after the latent representation M0 is input into the mask teacher model, represents the student distribution obtained after the potential representation M0 is input into the mask student model, represents the mask teacher model, represents the student model, represents the Kullback-Leibler divergence loss function, represents the Softmax function, is the temperature parameter, , is the weight coefficient.
[0017] Furthermore, in step S1.4, according to the formula Get semantic representation ,in, Represents an SGDB block, represents the spatial-aware feature fusion block, Indicates the SGEB block, represents a broadcast operation, Represents the fused feature map obtained by the multi-dimensional feature pyramid, , Represents the input The high-resolution feature map obtained by upsampling, Represents the input The low-resolution feature map obtained by downsampling, and represents the learnable weight coefficient.
[0018] Furthermore, in step S1.5, according to the formula Decode and generate reconstructed pictures ,in, Represents a decoder, represents the output representation, Represents an SDDB block, Represents the SDM block, , , , Represents a SDEB block, represents the SGM block, Represents a SGEB block.
[0019] Furthermore, in step S1.6, according to the formula Calculate the anomaly score , where N represents the number of feature extraction layers used in the feature extraction network, represents the upsampling factor of the nth feature extraction layer, represents the abnormal graph obtained by the nth feature extraction layer, represents the feature map obtained by the nth feature extraction layer, represents the nth feature extraction layer of the pre-trained feature extraction network, n=1,2,…,N, represents the matrix transpose operation, represents the dot product, represents the cross product; Represents the L2 norm operation.
[0020] Furthermore, the feature extraction network adopts the ResNet50 network architecture.
[0021] Furthermore, in step S1.7, according to the formula Get anomaly score list ,in, Represents the exception sorting mechanism, Indicates the anomaly threshold.
[0022] The present invention has the following beneficial effects:
[0023] 1. The present invention first obtains a data set formed by a positive sample image, a negative sample image and a mask image, and then sequentially performs a diffusion forward propagation step, a mask representation distillation step, a semantic representation generation step, a diffusion reverse denoising step, an anomaly score calculation step and an anomaly sorting mechanism visualization step to realize the detection of defective areas. In this process, the region of interest in the image is accurately marked by a mask mechanism, and the introduction of semantic information helps to improve the recognition ability of different defect types. The two synergistically optimize the reverse denoising process of the diffusion model, which can effectively improve the positioning detection accuracy and reconstruction effect of the defective area, calculate the anomaly score of the generated reconstructed image and the input image, and then use the anomaly sorting mechanism to screen the stone surface images with high anomaly, and feed back the heat maps corresponding to these images to the user, so as to provide the user with a visualized defective area, thereby improving the detection accuracy and efficiency of the defects. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] The present invention is further described in detail below with reference to the accompanying drawings.
[0025] Figure 1 It is a flow chart of the present invention.
[0026] Figure 2 Schematic diagram of the training process of the present invention.
[0027] Figure 3 It is a process principle block diagram of the present invention.
[0028] Figure 4 It is a schematic diagram of each process picture of the present invention. DETAILED DESCRIPTION
[0029] like Figures 1 to 3 As shown, the stone surface defect detection method based on mask and semantic collaborative optimization diffusion model includes the following steps:
[0030] Step S1.1, obtaining a stone surface image containing defects, taking the area without defects in the stone surface image as a positive sample image, and the target area containing defects as a negative sample image, and obtaining mask images corresponding to the positive sample image and the negative sample image, and the positive sample image, the negative sample image and the mask image form a data set;
[0031] In this embodiment, the positive sample image is a background area or a normal area on the stone surface image that does not contain defects, and the negative sample image is obtained by marking on the stone surface image. During the training process, the positive sample images and the negative sample images help the model learn to distinguish between defects and non-defective areas. The mask image is a binary image, in which the defective area is marked as 1 and other areas are marked as 0. The mask image is used for pixel-level supervised learning to guide the model to make classification judgments on each pixel of the image. In the defect location task, the mask image defines the precise boundary of the defect and provides fine-grained label information for the model.
[0032] Step S1.2, diffusion forward propagation step: respectively input the positive sample images and negative sample images in the data set as input images into the encoder to obtain input representations, and diffuse Gaussian noise forward on the input representations to make them gradually approach completely random noise, and obtain full noise representations;
[0033] Specifically, according to the formula Get full noise characterization ,in, represents the input representation, , w, h, c represent the width, height and number of channels of the corresponding image respectively, represents the encoder, represents the input image, W and H represent the width and height of the input image X respectively. The input image X is a color image with three channels, RGB, i.e. 3 represents the number of channels. The input image is generally larger than the feature map in the middle, i.e. W>w. represents the average anti-noise strength, Indicates the anti-noise intensity corresponding to the i-th pixel of the input image, represents the noise intensity corresponding to the i-th pixel of the input image, T represents the number of pixels in the input image, represents a continuous multiplication operation, represents random Gaussian noise that follows a [0,1] normal distribution, The input image includes a positive sample image and a negative sample image. Each positive sample image and each negative sample image performs steps S1.2 to S1.7. Figure 4 In the figure, a negative sample image is taken as an input image as an example to assist in explaining steps S1.2 to S1.7.
[0034] Step S1.3, mask representation distillation step: after dot-multiplying the input representation obtained from each input image with the mask image corresponding to the corresponding input image, the input is input into the mask-guided knowledge distillation network, so that the student model of the knowledge distillation network learns the mask teacher model to generate the mask representation;
[0035] The specific steps include:
[0036] Step S1.31: According to the formula Get the potential representation M0 of the mask, where Represents the mask image corresponding to the input image X, Represents pixel-level point-by-point product operation;
[0037] Step S1.32: Input the potential representation M0 of the mask into the mask teacher model and student model of the knowledge distillation network respectively, and design the overall loss function Train the knowledge distillation network to obtain the student model parameters after knowledge distillation , and then generate the mask representation , where the overall loss function of the knowledge distillation network is Expressed as , is the mean square error, represents the Euclidean distance, represents the teacher distribution obtained after the latent representation M0 is input into the mask teacher model, represents the student distribution obtained after the potential representation M0 is input into the mask student model, represents the mask teacher model, represents the student model, n represents the number of defect categories set, and represents the output of the mask teacher model and the student model on category i, represents the accumulation operation, represents the Kullback-Leibler divergence loss function, Represents the student distribution to simulate the teacher distribution, so that students can get as close to teachers as possible. represents the Softmax function, is the temperature parameter, which is used to adjust the "softening" degree of Softmax. , is the weight coefficient.
[0038] Step S1.4, semantic representation generation step: input the input representation into the semantic guidance enhancement network containing the multi-dimensional feature pyramid, and the semantic guidance enhancement network gradually and accurately locates the defect area to obtain the semantic representation;
[0039] Specifically, the BPNet (Bidirectional feature Pyramid Network) architecture is used to implement multi-dimensional feature fusion of the input image. In this embodiment, , , , Represents the fused feature map obtained by the multi-dimensional feature pyramid, where Represents the input The high-resolution feature map obtained by upsampling, Represents the input The low-resolution feature map obtained by downsampling, represents the upsampling operation, represents the downsampling operation, represents the fusion operation of feature maps at the same layer, and represents the learnable weight coefficient;
[0040] The semantic-guided network SG (Semantic-Guided) is used to gradually and accurately locate the defect area, that is, Get semantic representation ,in, represents the SGDB block (SG Decoder Block, the decoder block of the semantic guidance network), represents the spatial-aware feature fusion block, represents SGEB blocks (SG Encoder Blocks, encoder blocks of semantically guided networks), Represents a broadcast operation.
[0041] Step S1.5, diffusion reverse denoising step: after splicing the full noise representation, mask representation and semantic representation, reverse diffusion is performed to gradually remove noise, and decoding is performed to generate a reconstructed image;
[0042] Specifically, according to the formula Decode and generate reconstructed pictures ,in, Represents a decoder, represents the output representation, Represents SDDB block (SD Decoder Block, decoder block of the reverse denoising process), Represents SDM blocks (SD Middleblocks, the middle blocks of the reverse denoising process), , , , Represents SDEB blocks (SD Encoder Blocks, encoder blocks for the reverse denoising process), represents SGM blocks (SG Middle blocks, the middle blocks of semantically guided networks), Represents SGEB blocks (SG Encoder Blocks, encoder blocks of semantic guidance network). The reverse denoising process is specifically the stable diffusion reverse denoising process (SD: Stable Diffusion reverse denoising).
[0043] Step S1.6, abnormality score calculation step: input the input image and its corresponding reconstructed image in the data set into the feature extraction network to obtain feature maps of different scales, and perform cosine similarity calculation on the feature maps of different scales to obtain the abnormality score;
[0044] According to the formula Calculating anomaly score , where N=3 represents the number of feature extraction layers used in the feature extraction network. represents the upsampling factor of the nth feature extraction layer, represents the abnormal graph obtained by the nth feature extraction layer, represents the feature map obtained by the nth feature extraction layer, represents the nth feature extraction layer of the pre-trained feature extraction network, n=1,2,3, represents the matrix transpose operation, represents the dot product, represents the cross product; In this embodiment, the feature extraction network adopts the ResNet50 network architecture.
[0045] Step S1.7, visualization step of the anomaly sorting mechanism: sort the anomaly scores corresponding to each input image, form an anomaly score list based on the anomaly scores greater than the set anomaly threshold, and feed back the heat map of the input image corresponding to the anomaly score list to the user;
[0046] Specifically, according to the formula Get anomaly score list ,in, Represents the exception sorting mechanism, Indicates the set anomaly threshold. For positive samples, their anomaly scores are for the anomaly threshold, so the anomaly score list only contains the anomaly scores corresponding to the negative samples as input images. In the stored information, the negative sample images corresponding to the anomaly score list are retrieved, and the negative sample images, the reconstructed images corresponding to the negative sample images, the anomaly scores, and the heat map are fed back to the user. The user can intuitively see the defective area based on the heat map. The heat map is derived from the negative sample images, and this process is a prior art.
[0047] To verify the performance of the present invention, the present invention uses ResNet50 as the feature extraction network and selects n∈{1,2,3} as the feature extraction layer. On a single training NVIDIA Ampere A40 GPU, AutoDl can be run 1000 times on 48GB with a batch size of 12. The Adam optimizer is used and the learning rate is set to 1×10 −5 .
[0048] Six performance indicators, including AUROCseg, APseg, F1maxseg, AUROCcls, APcls, and F1maxcls, are set to evaluate the performance of defect localization and detection tasks. Specifically, "seg" represents pixel-level defect localization, while "cls" refers to image-level defect detection. The present invention mainly focuses on the location of defect edges, and the segmentation metric (i.e., AUROCseg) is considered the most important indicator. Table 1 compares the performance indicators of the present invention and the comparative method. It can be seen that for AUROCseg, the present invention (MESD) performs best.
[0049] Table 1
[0050]
[0051] The above description is only a preferred embodiment of the present invention, and therefore cannot be used to limit the scope of implementation of the present invention. That is, equivalent changes and modifications made according to the scope of the patent application of the present invention and the contents of the specification should still fall within the scope covered by the patent of the present invention.
Claims
1. A stone surface defect detection method based on mask and semantic collaborative optimization diffusion model, characterized by: The steps include: Step S1.1, obtaining a stone surface image containing defects, taking the area without defects in the stone surface image as a positive sample image, and the target area containing defects as a negative sample image, and obtaining mask images corresponding to the positive sample image and the negative sample image, and the positive sample image, the negative sample image and the mask image form a data set; Step S1.2, diffusion forward propagation step: respectively input the positive sample images and negative sample images in the data set as input images into the encoder to obtain input representations, and perform Gaussian noise forward diffusion on the input representations to make them gradually approach completely random noise, thereby obtaining full noise representations; Step S1.3, mask representation distillation step: after dot-multiplying the input representation obtained from each input image with the mask image corresponding to the corresponding input image, the input is input into the mask-guided knowledge distillation network, so that the student model of the knowledge distillation network learns the mask teacher model to generate the mask representation; Step S1.4, semantic representation generation step: input the input representation into the semantic guidance enhancement network containing the multi-dimensional feature pyramid, and the semantic guidance enhancement network gradually and accurately locates the defect area to obtain the semantic representation; Step S1.5, diffusion reverse denoising step: after splicing the full noise representation, mask representation and semantic representation, reverse diffusion is performed to gradually remove noise, and decoding is performed to generate a reconstructed image; Step S1.6, abnormality score calculation step: input the input image and its corresponding reconstructed image in the data set into the feature extraction network to obtain feature maps of different scales, and perform cosine similarity calculation on the feature maps of different scales to obtain the abnormality score; Step S1.7, visualization step of the anomaly sorting mechanism: sort the anomaly scores corresponding to each input image, form an anomaly score list based on the anomaly scores greater than the set anomaly threshold, and feed back the heat map of the input image corresponding to the anomaly score list to the user; In step S1.2, according to the formula Get full noise characterization ,in, represents the input representation, represents the encoder, X Represents the input image, represents the average anti-noise strength, Indicates the first i The anti-noise intensity corresponding to the pixel point is Indicates the first i The noise intensity corresponding to the pixel point is T Represents the number of pixels in the input image. represents a continuous multiplication operation, represents random Gaussian noise that follows a [0,1] normal distribution; The step S1.3 specifically includes the following steps: Step S1.31, according to the formula Get the potential representation of the mask M 0, where Mask Represents the input image X The corresponding mask image; Step S1.32: The potential representation of the mask M 0 Input the mask teacher model and student model of the knowledge distillation network respectively, and design the overall loss function Train the knowledge distillation network to obtain the student model parameters after knowledge distillation , and then generate the mask representation , where the overall loss function Expressed as , is the mean square error, represents the Euclidean distance, Representing potential representation M 0 The teacher distribution obtained after inputting the mask teacher model, Representing potential representation M 0 is the student distribution obtained after inputting the masked student model, represents the mask teacher model, represents the student model, represents the Kullback-Leibler divergence loss function, represents the Softmax function, is the temperature parameter, , is the weight coefficient.
2. The stone surface defect detection method based on mask and semantic collaborative optimization diffusion model according to claim 1 is characterized in that: In step S1.4, according to the formula Get semantic representation ,in, Represents an SGDB block, represents the spatial-aware feature fusion block, Indicates the SGEB block, represents a broadcast operation, Represents the fused feature map obtained by the multi-dimensional feature pyramid, , Represents the input The high-resolution feature map obtained by upsampling, Represents the input The low-resolution feature map obtained by downsampling, and represents the learnable weight coefficient.
3. The stone surface defect detection method based on mask and semantic collaborative optimization diffusion model according to claim 2 is characterized in that: In step S1.5, according to the formula Decode and generate reconstructed pictures ,in, Represents a decoder, represents the output representation, Represents an SDDB block, Represents the SDM block, , , , Represents a SDEB block, represents the SGM block, Represents a SGEB block.
4. The stone surface defect detection method based on mask and semantic collaborative optimization diffusion model according to claim 3 is characterized in that: In step S1.6, according to the formula Calculate the anomaly score ,in, N Represents the number of feature extraction layers used in the feature extraction network, Indicates n The upsampling factor of the feature extraction layer, Indicates n The abnormal map obtained by the feature extraction layer, Indicates n The feature map obtained by the layer feature extraction layer, represents the first n Layer feature extraction layer, n =1,2,…, N , represents the matrix transpose operation, represents the dot product, represents the cross product; Represents the L2 norm operation.
5. The stone surface defect detection method based on mask and semantic collaborative optimization diffusion model according to claim 4 is characterized in that: The feature extraction network adopts the ResNet50 network architecture.
6. The stone surface defect detection method based on mask and semantic collaborative optimization diffusion model according to claim 5 is characterized in that: In step S1.7, according to the formula Get anomaly score list ,in, Represents the exception sorting mechanism, Indicates the anomaly threshold.