High-reflection object surface defect detection method based on semantic tag guidance and multi-source image fusion

Abstract

The invention provides a high-reflection object surface defect detection method based on semantic tag guidance and multi-source image fusion. According to the method, an image fusion module is used for fusing a phase image and a sine fringe reflection image. Meanwhile, a semantic fusion label is designed and is used for guiding the phase image to be fused into the features of more deformation defects in the projected sine fringe reflection image in the image fusion process, and the fused image has the advantages of both the phase image and the projected sine fringe reflection image. Compared with an existing method, the method has the advantages that the requirement for hardware is low, meanwhile, the number of image processing can be greatly reduced, and in addition, the problem that the effect of phase detection in deformation defect detection is poor is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of highly reflective object surface defect detection, and in particular to a highly reflective object surface defect detection method based on semantic label guidance and multi-source image fusion. Background Art

[0002] During the production and processing of highly reflective objects, the surface is often affected by the environment and process, and some minor defects may appear, such as scratches, stains, particles, pits, etc. Traditional manual inspection is inefficient and easily interfered by subjective factors. With the development of a new generation of artificial intelligence technology, inspection technologies represented by deep learning are widely used in defect detection tasks on the surfaces of highly reflective objects, such as mirrors, glass, stainless steel, electronic screens and other highly reflective surfaces.

[0003] Since the surface of an object is highly reflective, the current detection scheme is to project a series of grating images onto the surface of a highly reflective object through an LCD screen, use a camera to collect the grating images reflected by the surface, and then analyze the collected images to detect potential defects. According to the type of projected image, the detection methods can be divided into two categories. One is the "stripe detection method", which projects a series of black and white stripe light images onto the surface of a highly reflective object, uses the stripe light to "amplify" the surface defects, detects potential defects, and combines the oversampling method to ensure detection accuracy; the other is the "phase detection method", which projects a set of structured light images onto the surface of the object to be tested, restores the surface phase information through the projected structured light images, and finally detects the phase image. Both methods have some shortcomings. The fringe detection method has high requirements for hardware. The frame rate of the camera and the screen needs to be above 200Hz to meet the production rhythm requirements of the production line. In addition, dozens of images need to be detected at a single point, and the data transmission pressure is relatively high. The essence of the phase detection method is to detect the change of the surface curvature of the object to be tested. For some deformation defects, such as tiny pits and inclusions on the surface, due to the relatively gentle change of curvature, the defects are often not obvious enough in the phase image, and the detection effect is relatively poor.

[0004] The phase detection method can be divided into two steps: the first step is to project structured light onto the surface of a highly reflective object, and use a camera to collect the structured light reflected by the surface to obtain a structured light reflection image; the second step is to use the collected structured light reflection image to restore the phase information of the surface of the object to be tested, convert the phase information into a "phase image", and then detect the phase image. In the first step, the projected structured light includes stripe light with sinusoidal information (hereinafter referred to as "sinusoidal stripe light"), and the image obtained by the surface of the object to be tested after the sinusoidal stripe light is reflected (hereinafter referred to as "sinusoidal stripe reflection image") has a high sensitivity to deformation defects and contains a large amount of feature information related to deformation defects. This feature can just make up for the shortcomings of the phase detection method. From this perspective, the present invention proposes a method for detecting surface defects of highly reflective objects based on semantic label guidance and multi-source image fusion. The phase image and the sinusoidal stripe reflection image are fused using an image fusion module. At the same time, a semantic fusion label is designed to guide the phase image to integrate more deformation defect features in the sinusoidal stripe reflection image during the image fusion process, and the fused image has the advantages of both. Compared with existing methods, the method of the present invention has lower hardware requirements and can greatly reduce the amount of image processing. It also overcomes the problem that phase detection is not effective in deformation defect detection and can improve detection accuracy. Summary of the invention

[0005] The purpose of the present invention is to solve the problem that the fringe detection method and the phase detection method perform poorly in the task of detecting highly reflective objects, and propose a method for detecting surface defects of highly reflective objects based on semantic label guidance and multi-source image fusion. This method makes the defects on the surface of the object to be tested more prominent in the phase image, and does not require an increase in hardware costs, which is conducive to improving the detection speed and the accuracy of the detection of surface defects of highly reflective objects, especially the detection of deformation defects.

[0006] The present invention is implemented by the following technical scheme. The present invention proposes a method for detecting surface defects of highly reflective objects based on semantic label guidance and multi-source image fusion. The method comprises the following steps:

[0007] S1: preprocessing the collected structured light image to generate a phase map Φ(x, y) of the surface of the object to be measured;

[0008] S2: The phase image Φ and N sinusoidal fringe reflection images S n They are sent to the feature encoding module Encoder respectively, and after feature fusion, the fused feature F is obtained. f ;

[0009] S3: The fused feature F f Send it to the decoding module Decoder to get the fused image I f ;

[0010] S4: Construct semantic fusion labels to improve the feature representation ability of the fused image on deformation defects;

[0011] S5: Using the constructed semantic fusion labels, calculate the loss function of image fusion and update the training image fusion model;

[0012] S6: In the inference stage, the trained image fusion model is used to input the phase image and the sinusoidal fringe reflection image to obtain a fused image, and the fused image is sent to the downstream detector to obtain the defect detection result of the surface of the highly reflective object.

[0013] Furthermore, the collected structured light image includes two parts. One part is the image collected by the camera after the projected sinusoidal fringe structured light is reflected by the surface of the object to be measured, which is called the sinusoidal fringe reflection image and is denoted by S n The other part is the image collected by the camera after the projected Gray code structured light is reflected by the surface of the object to be measured, which is called the Gray code reflection image and is represented by G n express.

[0014] Furthermore, the specific steps of step S1 are:

[0015] S101: First, the sinusoidal fringe reflection image is used to calculate the wrapping phase of the surface of the object to be measured using the least squares method:

[0016]

[0017] In the formula, Represents the wrapped phase of the pixel (x, y) in the wrapped phase image, S n (x, y) represents the light intensity of the pixel point (x, y) in the sinusoidal fringe reflection image, which is equal to the grayscale level of the point. n represents the nth sinusoidal fringe reflection image, and N represents the total number of sinusoidal fringe reflection images. Δφ n Represents the phase value of the projected sinusoidal structured light;

[0018] S102: Binarize the collected Gray code reflection image to obtain a binary Gray code reflection image B i (x, y), the binary formula is:

[0019]

[0020] Where G i represents the i-th Gray code reflection image collected, Th i Represents the i-th threshold image, and the calculation formula is:

[0021]

[0022] Where w i and bi is an empirical coefficient used to adapt to external lighting conditions, and offset is a constant;

[0023] S103: Reflect the image B according to the binary Gray code i , get the stripe level K:

[0024]

[0025] Where i represents the i-th binarized Gray code reflection image, M represents the total number of Gray code reflection images, represents the exclusive-or operation;

[0026] S104: Wrap phase Add it to the fringe order K(x, y) to obtain the final phase map Φ(x, y) of the surface of the object to be measured;

[0027]

[0028] Furthermore, the specific steps of step S2 are:

[0029] S201: The feature encoding module is composed of a deformable convolution module and two gradient residual dense blocks connected in series;

[0030] S202: The deformable convolution module includes a convolution layer with a kernel size of 3×3, a batch normalization layer, and a leaky ReLU activation function; the outer layer of the gradient residual module is a residual structure. The mainstream uses the idea of ​​dense connection to connect two deformable convolution modules. The residual stream uses the Sobel operator to extract gradient information from the feature map. Before adding element by element, two 1×1 ordinary convolution layers are applied to each stream to keep the dimension consistent;

[0031] S203: The original images are respectively encoded by feature encoding to obtain encoding features F Φ and The encoding feature F Φ and Splicing is performed along the channel direction to obtain the fusion feature F of the phase image and the sinusoidal fringe reflection image. f .

[0032] Further, in step S3, the decoding module includes four 3*3 deformable convolution blocks, each of which uses the LeakyReLU module as the activation function, which are connected in series in sequence, and finally a 1*1 convolution and a Tanh activation function are used to adjust the image to the same number of channels as the original input to obtain the final fused image I f .

[0033] Furthermore, the specific steps of step S4 are:

[0034] S401: Generate a mask M for each sinusoidal fringe reflection image n (x, y); First, according to the annotation information, the area containing deformation defects is extracted from the N sinusoidal fringe reflection images, denoted as D n (x, y); then, D n (x, y) is divided into J equal parts along its center line; these parts are sequentially divided into two groups of equal blocks, thereby generating J / 2 different grouping methods; the two image blocks in the jth (j = 1, 2, ..., J / 2) group of the nth sinusoidal fringe reflection image are defined as and

[0035] For the nth sinusoidal fringe reflection image, the metric ρ is calculated using the following formula n

[0036]

[0037] In the formula, · represents the absolute value; for each deformation defect, the one with the largest ρ is selected n The sinusoidal fringe reflection image of the value is then converted into the corresponding mask M n The area within the bounding box of the deformation class defect in (x, y) is set to 1;

[0038] S402: Based on the mask image M n (x, y), define the following fusion label:

[0039]

[0040] Where Φ(x, y) represents the phase image, S n (x, y) represents the sinusoidal fringe reflection image, N represents the total number of sinusoidal fringe reflection images, Represents the product of corresponding elements.

[0041] Further, in step S5, the joint loss function guiding the fusion network training is The definition is as follows:

[0042]

[0043] In the formula, represents the semantic loss, represents the content loss, and λ1 is used to balance the two parts of the loss.

[0044] Furthermore, the specific steps of step S5 are:

[0045] S501: Semantic Loss Using fused image I f and fusion tag I lThe structural similarity index SSIM between them is measured and defined as follows:

[0046] SSIM(I f , I l )=[l(I f , I l )] α ·[c(I f , I l )] β ·[s(I f , I l )] γ

[0047] Where l(I f , I l )、c(I f , I l )、s(I f , I l ) are brightness, contrast, and structure comparison functions respectively; parameters α, β, and γ are used to adjust the importance of each comparison function; the definitions of the three comparison functions are as follows:

[0048]

[0049]

[0050]

[0051] in and and is the mean intensity and standard deviation of the fused image and fused label, represents the covariance between corresponding regions of two images, C1, C2 and C3 are constants;

[0052] S502: Content loss Including strength loss and texture loss Two parts, defined as follows:

[0053]

[0054] In the formula, λ2 is used to balance the texture loss and intensity loss. and texture loss Defined as:

[0055]

[0056]

[0057] Where H and W represent the height and width of the image, max(·) represents the maximum value of the element, represents the Sobel gradient operator used to measure texture information, |·| represents the element-wise absolute value operation, and ||·||1 represents the calculation of the first-order norm of the internal vector or matrix.

[0058] The present invention also proposes an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of a method for detecting surface defects of highly reflective objects based on semantic label guidance and multi-source image fusion are implemented.

[0059] The present invention also proposes a computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the steps of a method for detecting surface defects of highly reflective objects based on semantic label guidance and multi-source image fusion.

[0060] Compared with the prior art, the present invention has the following beneficial effects:

[0061] Existing methods for detecting surface defects of highly reflective objects mainly include "stripe detection method" and "phase detection method". The former has high requirements on hardware cost, and the latter has poor detection effect on tiny deformation defects. The present invention proposes a method for detecting surface defects of highly reflective objects based on semantic label guidance and multi-source image fusion. The phase image and the sinusoidal stripe reflection image are fused using an image fusion module. At the same time, a semantic fusion label is designed to guide the phase image to integrate more deformation defect features in the projected sinusoidal stripe reflection image during the image fusion process. The fused image has the advantages of both. Compared with existing methods, the method described in the present invention has lower hardware requirements and can greatly reduce the amount of image processing. In addition, it overcomes the problem that phase detection is not effective in detecting deformation defects.

[0062] Based on the above advantages, the present invention provides a new detection scheme for the task of detecting defects on the surface of highly reflective objects, which helps to improve the accuracy of the detection of defects on the surface of highly reflective objects, and improves the detection speed without increasing the hardware cost. The present invention can be migrated and used in various tasks of detecting the surface of highly reflective objects, including but not limited to mirrors, glass, metal surfaces, car paint, car interiors, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0064] Figure 1 This is the overall flow chart of the surface defect detection method for highly reflective objects based on semantic label guidance and multi-source image fusion.

[0065] Figure 2 This is a diagram of the semantic label guidance and multi-source image fusion structure.

[0066] Figure 3 The figure shows the process and effect of obtaining semantic fusion labels.

[0067] Figure 4 The defect effect diagram before and after fusion.

[0068] Figure 5 This is a diagram showing the ability of the present invention to detect surface defects of actual highly reflective objects. DETAILED DESCRIPTION

[0069] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0070] The present invention proposes a method for detecting defects on the surface of highly reflective objects based on semantic label guidance and multi-source image fusion. The method first preprocesses all structured light images collected by a camera to generate a phase image of the surface of the object to be tested; then, an Encoder module is used to extract and fuse features of the phase image and the sinusoidal fringe reflection image; the fused features are then sent to a Decoder module to generate a fused image; then, through the designed semantic fusion label, the fused image is guided to pay more attention to the deformation defect part in the sinusoidal fringe reflection image, and the model is updated by calculating the loss function of the image fusion; finally, in the inference stage, the fused image is sent to a downstream detector to obtain the defect detection result of the object surface.

[0071] Specifically, combined Figure 1-Figure 5 The present invention proposes a method for detecting surface defects of highly reflective objects based on semantic label guidance and multi-source image fusion, the method comprising the following steps:

[0072] S1: pre-process the collected structured light image to generate a phase map Φ(x,y) of the surface of the object to be measured;

[0073] The collected structured light image includes two parts. One part is the image collected by the camera after the projected sinusoidal fringe structured light is reflected by the surface of the object to be measured, which is called the sinusoidal fringe reflection image and is denoted by S. nThe other part is the image collected by the camera after the projected Gray code structured light is reflected by the surface of the object to be measured, which is called the Gray code reflection image and is represented by G n express.

[0074] The specific steps of step S1 are:

[0075] S101: First, the sinusoidal fringe reflection image is used to calculate the wrapping phase of the surface of the object to be measured using the least squares method:

[0076]

[0077] In the formula, Represents the wrapped phase of the pixel (x, y) in the wrapped phase image, S n (x, y) represents the light intensity of the pixel point (x, y) in the sinusoidal fringe reflection image, which is equal to the grayscale level of the point. n represents the nth sinusoidal fringe reflection image, and N represents the total number of sinusoidal fringe reflection images. Δφ n Represents the phase value of the projected sinusoidal structured light;

[0078] S102: Binarize the collected Gray code reflection image to obtain a binary Gray code reflection image B i (x, y), the binary formula is:

[0079]

[0080] Where G i represents the i-th Gray code reflection image collected, Th i Represents the i-th threshold image, and the calculation formula is:

[0081]

[0082] Where w i and b i is an empirical coefficient used to adapt to external lighting conditions, and offset is a constant used to convert b i Control within a smaller range;

[0083] S103: Reflect the image B according to the binary Gray code i , get the stripe level K:

[0084]

[0085] Where i represents the i-th binarized Gray code reflection image, M represents the total number of Gray code reflection images, represents the exclusive-or operation;

[0086] S104: Wrap phase Add it to the fringe order K(x, y) to obtain the final phase map Φ(x, y) of the surface of the object to be measured;

[0087]

[0088] S2: The phase image Φ and N sinusoidal fringe reflection images S n They are sent to the feature encoding module Encoder respectively, and after feature fusion, the fused feature F is obtained. f ;

[0089] The specific steps of step S2 are:

[0090] S201: The feature encoding module is composed of a deformable convolution module and two gradient residual dense blocks connected in series;

[0091] S202: The deformable convolution module includes a convolution layer with a kernel size of 3×3, a batch normalization layer, and a Leaky Rectified Linear Unit (Leaky ReLU) activation function; the outer layer of the gradient residual module is a residual structure. The mainstream uses the idea of ​​dense connection to connect two deformable convolution modules. The residual stream uses the Sobel operator to extract gradient information from the feature map. Before adding element by element, two 1×1 ordinary convolution layers are applied to each stream to keep the dimension consistent;

[0092] S203: The original images are respectively encoded by feature encoding to obtain encoding features F Φ and The encoding feature F Φ and Splicing is performed along the channel direction to obtain the fusion feature F of the phase image and the sinusoidal fringe reflection image. f .

[0093] S3: The fused feature F f Send it to the decoding module Decoder to get the fused image I f ;

[0094] In step S3, the decoding module includes four 3*3 deformable convolution blocks, each of which uses the Leaky ReLU module as the activation function, which are connected in series in sequence. Finally, a 1*1 convolution and a Tanh activation function are used to adjust the image to the same number of channels as the original input to obtain the final fused image I f .

[0095] S4: Constructing semantic fusion labels to improve the feature expression ability of the fused image on deformation defects; In order to specifically improve the feature expression ability of the fused image on deformation defects, the present invention designs a set of semantic fusion labels I l(x, y), which is used to guide the fused image to pay more attention to the deformation defect part in the sinusoidal fringe reflection image.

[0096] The specific steps of step S4 are:

[0097] S401: Generate a mask M for each sinusoidal fringe reflection image n (x, y); First, according to the annotation information, the area containing deformation defects is extracted from the N sinusoidal fringe reflection images, denoted as D n (x, y); then, D n (x, y) is divided into J equal parts along its center line; these parts are sequentially divided into two groups of equal blocks, thereby generating J / 2 different grouping methods; the two image blocks in the jth (j = 1, 2, ..., J / 2) group of the nth sinusoidal fringe reflection image are defined as and

[0098] For the nth sinusoidal fringe reflection image, the metric ρ is calculated using the following formula n

[0099]

[0100] In the formula, |·| represents the absolute value; for each deformation defect, the one with the largest ρ is selected n The sinusoidal fringe reflection image of the value is then converted into the corresponding mask M n The area within the bounding box of the deformation class defect in (x, y) is set to 1;

[0101] S402: Based on the mask image M n (x, y), define the following fusion label:

[0102]

[0103] Where Φ(x, y) represents the phase image, S n (x, y) represents the sinusoidal fringe reflection image, N represents the total number of sinusoidal fringe reflection images, Represents the product of corresponding elements.

[0104] S5: Using the constructed semantic fusion labels, calculate the loss function of image fusion and update the training image fusion model;

[0105] In step S5, the joint loss function that guides the training of the fusion network is The definition is as follows:

[0106]

[0107] In the formula, represents the semantic loss, represents the content loss, and λ1 is used to balance the two parts of the loss.

[0108] The specific steps of step S5 are:

[0109] S501: Semantic Loss Using fused image I f and fusion tag I l The structural similarity index SSIM between them is measured and defined as follows:

[0110] SSIM(I f , I l )=[l(I f , I l )] α ·[c(I f , I l )] β ·[s(I f , I l )] γ

[0111] Where l(I f , I l )、c(I f , I l )、s(I f , I l ) are brightness, contrast, and structure comparison functions respectively; parameters α, β, and γ are used to adjust the importance of each comparison function; the definitions of the three comparison functions are as follows:

[0112]

[0113]

[0114]

[0115] in and and is the mean intensity and standard deviation of the fused image and fused label, represents the covariance between corresponding regions of two images, C1, C2 and C3 are small constants to prevent division by 0;

[0116] S502: Content loss Including strength loss and texture loss Two parts, defined as follows:

[0117]

[0118] In the formula, λ2 is used to balance the texture loss and intensity loss. and texture loss Defined as:

[0119]

[0120]

[0121] Where H and W represent the height and width of the image, max(·) represents the maximum value of the element, represents the Sobel gradient operator used to measure texture information, |·| represents the element-wise absolute value operation, and ||·||1 represents the calculation of the first-order norm of the internal vector or matrix.

[0122] S6: In the inference stage, the trained image fusion model is used to input the fused image obtained by the phase image and the sinusoidal fringe reflection image, and the fused image is sent to the downstream detector to obtain the defect detection result of the surface of the highly reflective object.

[0123] The specific implementation of the present invention is described below in conjunction with the embodiments and drawings.

[0124] In this embodiment, an LCD screen is used as a structured light source to project 4 sinusoidal structured light images and 5 Gray code structured light images onto a mirror surface to be tested, and a black and white camera is used to collect the Gray code image reflected by the surface of the test object. The system structure used in this embodiment is as follows: Figure 2 The image resolution is 2048*2432 and the depth is 8 bits. The method of the present invention is used to detect defects on the surface of the object to be tested.

[0125] Execute step 1: pre-process the 9 structured light images taken to obtain a phase map Φ(x, y) of the surface of the object to be tested.

[0126] First, the four sinusoidal fringe reflection images S n The least squares method is used to calculate the wrapping phase of the surface of the object to be measured:

[0127]

[0128] In the formula, Represents the wrapped phase of the pixel (x, y) in the wrapped phase image, S n (x, y) represents the light intensity of the pixel point (x, y) in the sinusoidal fringe reflection image, which is equal to the grayscale level of the point, n represents the nth sinusoidal fringe reflection image, and N represents the total number of projected sinusoidal fringe structured lights, which is 4 in this embodiment. Δφ n Represents the phase value of the projected sinusoidal structured light.

[0129] Then, the five collected Gray code reflection images are binarized to obtain the binarized Gray code reflection image B i(x, y), the binary formula is:

[0130]

[0131] In the formula, G i represents the i-th Gray code reflection image collected, Th i Represents the i-th threshold image, and the calculation formula is:

[0132]

[0133] In this embodiment, the empirical coefficient w i and b i The values ​​are set to 1 and 1.2 respectively to overcome the interference of light, and the offset is set to 10.

[0134] Then, according to the binary Gray code reflection image B i , get the stripe level K:

[0135]

[0136] Where i represents the i-th binarized Gray code reflection image, M represents the total number of Gray code reflection images, which is 5 in this embodiment. represents the XOR operation. Finally, the phase Added to the fringe order K(x, y), the final phase map Φ(x, y) of the surface of the object to be measured is obtained.

[0137]

[0138] Execute step 2: In this embodiment, the phase image Φ and the four sinusoidal fringe reflection images S n The extracted features are concatenated along the channel direction to obtain the fusion feature F f The whole process can be described as:

[0139] F f =cat(Encoder(Φ),Encoder(S n ))

[0140] The feature encoding module Encoder is composed of a deformable convolution module and two gradient residual dense blocks connected in series.

[0141] The deformable convolution module consists of a deformable convolution, a batch normalization layer, and a LeakyRectified Linear Unit (Leaky ReLU) activation function in series, where the padding and stride of the deformable convolution layer are set to 1, the convolution kernel size is 3*3, and the number is 16.

[0142] The outer layer of the gradient residual module is a residual structure. The mainstream uses the idea of ​​dense connection to connect two deformable convolution modules. The parameters are consistent with the above modules. The residual stream uses the Sobel operator to extract gradient information from the feature map. Before adding element by element, two 1×1 ordinary convolution layers are applied to each stream to keep the dimensions consistent. After two gradient residual modules, the number of feature channels of each image becomes 32 and 48 respectively. Since the padding and stride of the convolution module used are both 1, the width and height of the image remain unchanged and are still maintained at (2048, 2432).

[0143] Phase image Φ and 4 sinusoidal fringe reflection images S n The encoding features F are obtained by feature encoding respectively Φ and The number of channels of the former is 48, and the number of channels of the latter is 4×48=192. Splicing is performed along the channel direction to obtain the fusion feature F of the phase image and the sinusoidal fringe reflection image. f , the number of channels is 240, and the width and height are (2048, 2432).

[0144] Execute step three: merge the fused features F f Send it to the decoding module Decoder to get the fused image I f The whole process is described as follows: f =Decoder(F f ).

[0145] The decoding module includes three deformable convolution modules, all of which use the Leaky ReLU module as the activation function. The convolution kernel size, step size, and padding remain the same as in step 2. The number of convolution kernels used is 96, 32, and 16. The three deformable convolution modules are connected in series, and finally a 1*1 ordinary convolution and a Tanh activation function are used to adjust the image to a single channel, which is consistent with the original input, to obtain the final fused image I f .

[0146] Execute step 4: Build semantic fusion labels.

[0147] This embodiment takes the common deformation defect pit as an example. First, a mask M is generated for each sinusoidal fringe reflection image. n First, according to the annotation information, the area containing the pit defect is extracted from the N sinusoidal fringe reflection images, which is recorded as D n (x, y). Then, D n(x, y) is divided into 8 equal parts along its center line. These parts are divided into two groups of equal blocks in sequence, resulting in 4 different grouping methods. The two image blocks in the (j=1, 2, ..., J / 2)th group of the nth sinusoidal fringe reflection image are defined as and

[0148] For the nth sinusoidal fringe reflection image, the metric ρ is calculated using the following formula n :

[0149]

[0150] In the formula, |·| represents the absolute value. For each pit defect, select the one with the largest ρ n Then the corresponding mask M n The area within the defect bounding box in (x, y) is set to 1. The process and effect of obtaining semantic fusion labels are shown in Figure 3 shown.

[0151] According to the mask image M n (x, y), define the following fusion label:

[0152]

[0153] Where Φ(x, y) represents the phase image, M n (x, y) represents the mask image, S n (x, y) represents the sinusoidal fringe light reflection image, N represents the total number of sinusoidal fringe light reflection images, Represents the product of corresponding elements.

[0154] Execute step 5: In this embodiment, the joint loss function guiding the fusion network training The definition is as follows:

[0155]

[0156] In the formula, represents the semantic loss, represents content loss, λ1 is used to balance the two losses, and is set to 1 in this embodiment.

[0157] Semantic Loss Using fused image I f and fusion tag I l The structural similarity index SSIM between them is measured and defined as follows:

[0158] SSIM(I f , I l )=[l(I f , I l)] α ·[c(I f , I l )] β ·[s(I f , I l )] γ

[0159] Where l(I f , I l )、c(I f , I l )、s(I f , I l ) are brightness, contrast, and structure comparison functions, respectively. Parameters α, β, and γ are used to adjust the importance of each comparison function, and in this embodiment, they are all set to 1. The definitions of the three comparison functions are as follows:

[0160]

[0161]

[0162]

[0163] in and and is the mean intensity and standard deviation of the fused image and fused label, represents the covariance between corresponding regions of two images, C1, C2, and C3 are small constants to prevent division by zero.

[0164] Content loss Including strength loss and texture loss Two parts, defined as follows:

[0165]

[0166] In the formula, λ2 is used to balance texture loss and intensity loss, and is set to 0.5 in this embodiment. Intensity loss and texture loss Defined as:

[0167]

[0168]

[0169] Where H and W represent the height and width of the image, max(·) represents the maximum value of the element, represents the Sobel gradient operator used to measure texture information, |·| represents the element-wise absolute value operation, and ||·||1 represents the calculation of the first-order norm of the internal vector or matrix.

[0170] Execute step 6: After the image fusion model is trained using the above method, the phase image and four sinusoidal fringe reflection images are input to obtain the fused image I f The comparison between the fused image and the phase image is Figure 4 As shown. In the inference stage, I f The defect detection result of the surface of the object to be tested is sent to the downstream detector. This process is described as: Res = Detector (I f ).

[0171] In this embodiment, Yolov11s is selected as the downstream detector, and the detection results obtained in the embodiment are tested below. There are 2502 defect detection data sets, of which 2002 images are used as training sets and the remaining 500 images are used as test sets. The detection effects of the fringe detection method, the phase detection method and the method used in this embodiment on the surface pit defects of the object to be tested are as follows (the larger the number in the table, the better the detection effect):

[0172]

[0173] The results show that the surface defect detection method for highly reflective objects based on semantic label guidance and multi-source image fusion proposed in the present invention effectively improves the detection effect of the phase detection method on tiny deformation defects. Figure 5 The final defect detection effect of the method of the present invention is demonstrated. It can be found that for the vast majority of defects, the method proposed in the present invention can achieve relatively accurate prediction, which is of great significance for the detection of high-brightness and high-reflective object surfaces such as glass and mirrors.

[0174] The present invention also proposes an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of a method for detecting surface defects of highly reflective objects based on semantic label guidance and multi-source image fusion are implemented.

[0175] The present invention also proposes a computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the steps of a method for detecting surface defects of highly reflective objects based on semantic label guidance and multi-source image fusion.

[0176] The memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus RAM (DRRAM). It should be noted that the memory of the method described in the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0177] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a high-density digital video disc (DVD)), or a semiconductor medium (eg, a solid state disc (SSD)).

[0178] In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in a processor or an instruction in the form of software. The steps of the method disclosed in conjunction with the embodiment of the present application can be directly embodied as a hardware processor for execution, or a combination of hardware and software modules in a processor for execution. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in conjunction with its hardware. To avoid repetition, it is not described in detail here.

[0179] It should be noted that the processor in the embodiment of the present application can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method embodiment can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or the hardware and software modules in the decoding processor can be combined and performed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0180] The above is a detailed introduction to the method for detecting surface defects of highly reflective objects based on semantic label guidance and multi-source image fusion proposed in the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for detecting surface defects of highly reflective objects based on semantic label guidance and multi-source image fusion, characterized in that: The method comprises the following steps: S1: pre-process the collected structured light image to generate a phase map Φ(x,y) of the surface of the object to be measured; S2: The phase image Φ and N sinusoidal fringe reflection images S n They are sent to the feature encoding module Encoder respectively, and after feature fusion, the fused feature F is obtained. f ; S3: The fused feature F f Send it to the decoding module Decoder to get the fused image I f ; S4: Construct semantic fusion labels to improve the feature representation ability of the fused image in deformation defects; S5: Using the constructed semantic fusion labels, calculate the loss function of image fusion and update the training image fusion model; S6: In the inference stage, the trained image fusion model is used to input the phase image and the sinusoidal fringe reflection image to obtain a fused image, and the fused image is sent to the downstream detector to obtain the defect detection result of the surface of the highly reflective object.

2. The method according to claim 1, characterized in that The collected structured light image includes two parts. One part is the image collected by the camera after the projected sinusoidal fringe structured light is reflected by the surface of the object to be measured, which is called the sinusoidal fringe reflection image and is denoted by S. n The other part is the image collected by the camera after the projected Gray code structured light is reflected by the surface of the object to be measured, which is called the Gray code reflection image and is represented by G n express.

3. The method according to claim 2, characterized in that The specific steps of step S1 are: S101: First, the sinusoidal fringe reflection image is used to calculate the wrapping phase of the surface of the object to be measured using the least squares method: In the formula, Represents the wrapped phase of the pixel (x, y) in the wrapped phase image, S n (x, y) represents the light intensity of the pixel point (x, y) in the sinusoidal fringe reflection image, which is equal to the grayscale level of the point. n represents the nth sinusoidal fringe reflection image, and N represents the total number of sinusoidal fringe reflection images. Δφ n Represents the phase value of the projected sinusoidal structured light; S102: Binarize the collected Gray code reflection image to obtain a binary Gray code reflection image B i (x,y), the binary formula is: Where G i represents the i-th Gray code reflection image collected, Th i Represents the i-th threshold image, and the calculation formula is: Where w i and b i is an empirical coefficient used to adapt to external lighting conditions, and offset is a constant; S103: Reflect the image B according to the binary Gray code i , get the stripe level K: Where i represents the i-th binarized Gray code reflection image, M represents the total number of Gray code reflection images, and ⊕ represents an exclusive-OR operation; S104: Wrap phase Add it to the fringe order K(x,y) to get the final phase map Φ(x,y) of the surface of the object to be measured; 4. The method according to claim 3, characterized in that The specific steps of step S2 are: S201: The feature encoding module is composed of a deformable convolution module and two gradient residual dense blocks connected in series; S202: The deformable convolution module includes a convolution layer with a kernel size of 3×3, a batch normalization layer, and a leaky ReLU activation function; the outer layer of the gradient residual module is a residual structure. The mainstream uses the idea of ​​dense connection to connect two deformable convolution modules. The residual stream uses the Sobel operator to extract gradient information from the feature map. Before adding element by element, two 1×1 ordinary convolution layers are applied to each stream to keep the dimension consistent; S203: The original images are respectively encoded by feature encoding to obtain encoding features F Φ and The encoding feature F Φ and Splicing is performed along the channel direction to obtain the fusion feature F of the phase image and the sinusoidal fringe reflection image f .

5. The method according to claim 4, characterized in that In step S3, the decoding module includes four 3*3 deformable convolution blocks, each of which uses the Leaky ReLU module as the activation function, which are connected in series in sequence. Finally, a 1*1 convolution and a Tanh activation function are used to adjust the image to the same number of channels as the original input to obtain the final fused image I f .

6. The method according to claim 5, characterized in that The specific steps of step S4 are: S401: Generate a mask M for each sinusoidal fringe reflection image n (x, y); First, according to the annotation information, the area containing deformation defects is extracted from the N sinusoidal fringe reflection images, denoted as D n (x,y); then, D n (x, y) is divided into J equal parts along its center line; these parts are sequentially divided into two groups of equal blocks, resulting in J / 2 different grouping methods; the two image blocks in the jth (j = 1, 2, ..., J / 2) group of the nth sinusoidal fringe reflection image are defined as and For the nth sinusoidal fringe reflection image, the metric ρ is calculated using the following formula n In the formula, |·| represents the absolute value; for each deformation defect, the one with the largest ρ is selected n The sinusoidal fringe reflection image of the value is then converted into the corresponding mask M n The area within the bounding box of the deformation defect in (x, y) is set to 1; S402: Based on the mask image M n (x,y), define the following fusion label: Where Φ(x,y) represents the phase image, S n (x, y) represents the sinusoidal fringe reflection image, N represents the total number of sinusoidal fringe reflection images, and ° represents the product of the corresponding elements.

7. The method according to claim 6, characterized in that In step S5, the joint loss function that guides the training of the fusion network is The definition is as follows: In the formula, represents the semantic loss, represents the content loss, and λ1 is used to balance the two parts of the loss.

8. The method according to claim 7, characterized in that The specific steps of step S5 are: S501: Semantic Loss Using fused image I f and fusion tag I l The structural similarity index SSIM between them is measured and defined as follows: SSIM(I f ,I l )=[l(I f ,I l )] α ·[c(I f ,I l )] β ·[s(I f ,I l )] γ Where l(I f ,I l )、c(I f ,I l )、s(I f ,I l ) are brightness, contrast, and structure comparison functions respectively; parameters α, β, and γ are used to adjust the importance of each comparison function; the definitions of the three comparison functions are as follows: in and and is the mean intensity and standard deviation of the fused image and fused label, represents the covariance between corresponding regions of two images, C1, C2 and C3 are constants; S502: Content loss Including strength loss and texture loss Two parts, defined as follows: In the formula, λ2 is used to balance the texture loss and intensity loss. and texture loss Defined as: Where H and W represent the height and width of the image, max(·) represents the maximum value of the element, represents the Sobel gradient operator used to measure texture information, |·| represents the element-wise absolute value operation, and ||·||1 represents the calculation of the first-order norm of the internal vector or matrix.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium for storing computer instructions, characterized in that: When the computer instructions are executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images