A multi-scale spatial joint model and its visual detection method

Abstract

The present invention discloses a multi-scale space joint model and a visual detection method thereof, which belongs to the technical field of machine vision defect detection, and includes an FCOS basic network, wherein the FCOS basic network is connected with a classification fusion pyramid and a regression fusion pyramid, and the classification fusion pyramid is connected to the regression fusion pyramid via a class space regression similarity module. The present application can extract multi-scale class space regression quality information and multi-scale regression space feature information of image data through the FCOS basic network, and respectively perform data fusion processing on the multi-scale class space regression quality information and multi-scale regression space feature information through the classification fusion pyramid and the regression fusion pyramid, so as to enhance defect features and thus improve the accuracy of defect detection; the present application inputs the category feature information into the regression fusion pyramid to guide the information fusion of the regression space, which can reduce the situation where the classification and regression results are inconsistent during the model training and the model is used for defect detection, and ensure the accuracy of defect detection.

Description

Technical Field

[0001] The present invention relates to the technical field of machine vision defect detection, and in particular to a multi-scale space joint model and a visual detection method thereof. Background Art

[0002] In the process of industrial production, product defects are often inevitable due to manufacturing processes, usage intensity, working environment and other reasons. Inspection of product surface is one of the important links to ensure the quality of finished products. Manual inspection is one of the traditional detection solutions. Although this method does not require high technical skills of relevant personnel, it requires training workers to identify complex surface defects. Compared with machine recognition, manual inspection is time-consuming and inefficient. In addition, it is easily affected by the external environment and subjective judgment during the detection process, resulting in unstable detection accuracy and prone to problems such as missed detection and false detection.

[0003] Defect detection methods based on machine vision have solved the above problems of low inspection efficiency and inconsistent evaluation standards to a certain extent. However, in the classic machine vision defect method, the features of the defects must be manually designed to adapt to specific fields. Due to the limitations of this type of classifier, it is necessary to use various filter groups, histograms, wavelet transforms, morphological operations and other technologies to manually design the corresponding features. Therefore, manually designed features are particularly important in the classic method. However, there are many types of products and the product upgrade cycle is short. The defect features need to be redesigned for different products, which leads to a longer machine vision defect development cycle and increased production costs.

[0004] As a fast and effective data analysis method, deep learning can mine the deep structure of data and automatically learn the representative features of data. It has developed rapidly in recent years and has been widely used in the field of image recognition. However, it faces the following challenges in the field of defect detection:

[0005] 1. The multi-scale detection capability of defect detection algorithms needs to be strengthened. Traditional target detection network algorithms perform well on medium-sized and large-sized targets, but due to the different shapes and sizes of defective objects, and even the possibility of objects of extreme shapes and sizes, it is extremely difficult to accurately identify and locate the target.

[0006] 2. Compared with traditional data, defect data has a more complex target background. In traditional target detection such as vehicle detection, the feature information of vehicles, pedestrians, houses, roads, etc. is quite different, and the semantic information gap is obvious. The surface of the test piece often has stains, scratches, bumps, etc., which are very similar to defects and easily cause false detection.

[0007] 3. Defect detection algorithms require higher single-frame detection efficiency. Traditional target detection models perform offline detection on image data, but in the actual application of defect detection, the model deployed on the production line needs to achieve online detection.

[0008] In summary, based on the existing target detection algorithm, a defect detection method is proposed, which can combine the features of data at multiple scales and automatically identify the location of defects, which is of great significance to actual detection tasks. Summary of the invention

[0009] The purpose of the present invention is to overcome the problems of low detection accuracy and positioning precision in the prior art, and to provide a multi-scale spatial joint model and a visual detection method thereof.

[0010] The objective of the present invention is achieved through the following technical solutions: a multi-scale spatial joint model, which includes an FCOS basic network for extracting multi-scale category feature information and multi-scale regression feature information of an input image, wherein the multi-scale category feature information output end of the FCOS basic network is connected to a classification fusion pyramid, and the multi-scale regression feature information output end of the FCOS basic network is connected to a regression fusion pyramid;

[0011] The classification fusion pyramid is connected to the regression fusion pyramid via the class space regression similarity module; the class space regression similarity module is used to extract the output feature fusion information C of the classification fusion pyramid. i 'Class space regression quality information D i , and regress the class space quality information D i Input regression fusion pyramid; classification fusion pyramid is based on multi-scale category feature information C i Further extract feature fusion information C i ', based on feature fusion information C i 'Defect category judgment; regression fusion pyramid based on multi-scale regression feature information R i , Class space regression quality information D i Further extract feature fusion information R i ', based on feature fusion information R i 'Perform defect location.

[0012] In one example, the classification fusion pyramid includes a third category feature extraction layer, a fourth category feature extraction layer, a fifth category feature extraction layer, a sixth category feature extraction layer, and a seventh category feature extraction layer. The third category feature extraction layer, the fourth category feature extraction layer, and the fifth category feature extraction layer perform feature fusion from bottom to top or from top to bottom, and the fifth category feature extraction layer, the sixth category feature extraction layer, and the seventh category feature extraction layer perform feature fusion from top to bottom or from bottom to top.

[0013] In one example, the regression fusion pyramid includes a third regression feature extraction layer, a fourth regression feature extraction layer, a fifth regression feature extraction layer, a sixth regression feature extraction layer, and a seventh regression feature extraction layer. The third regression feature extraction layer, the fourth regression feature extraction layer, and the fifth regression feature extraction layer perform feature fusion from bottom to top or from top to bottom, and the fifth regression feature extraction layer, the sixth regression feature extraction layer, and the seventh regression feature extraction layer perform feature fusion from top to bottom or from bottom to top.

[0014] In one example, the class space regression similarity module includes a plurality of sequentially connected convolutional layers.

[0015] It should be further explained that the technical features corresponding to the above-mentioned model examples can be combined or replaced with each other to form a new technical solution.

[0016] The present application includes a visual detection method of a multi-scale spatial joint model, which is implemented based on one or more of the multi-scale spatial joint models described in the above examples, and includes the following steps:

[0017] S1: Build a multi-scale spatial joint model;

[0018] S2: training a multi-scale spatial joint model;

[0019] S21: Extract training sample features based on the basic network to obtain multi-scale category feature information C i and multi-scale regression feature information R i ;

[0020] S22: C i Input into the classification fusion pyramid to perform feature fusion calculation to obtain feature fusion information C i ', and based on C i Train the classification fusion pyramid;

[0021] S23: Extract C i 'Class space regression quality information D i ;

[0022] S24: Based on R i , D i Train the regression fusion pyramid;

[0023] S25: Repeat steps S21-S24 until the multi-scale spatial joint model converges to obtain a trained multi-scale spatial joint model;

[0024] S3: Defect detection of video stream data of the test piece to be inspected is realized based on the multi-scale spatial joint model.

[0025] In one example, building a multi-scale spatial joint model includes:

[0026] Adopt FCOS as the basic network;

[0027] Based on the Head output results of the FCOS basic network, a classification fusion pyramid and a regression fusion pyramid are built respectively;

[0028] Connect the classification fusion pyramid to the regression fusion pyramid through the class space regression similarity module.

[0029] In one example, the fused pyramid includes:

[0030] C i Input into the classification fusion pyramid to perform neighbor-level category similarity calculation and neighbor-level information fusion calculation, and further extract class space regression quality information D based on the fusion information i ; and / or, R i , D i Input into the regression fusion pyramid and perform neighbor-level regression similarity calculation and neighbor-level information fusion calculation.

[0031] In one example, the similarity calculation and neighbor level information fusion calculation include:

[0032] Based on the class space neighboring level similarity calculation, the class space similarity index of the current level and the adjacent level at the same target detection position is obtained. Similarity index of class space based on sigmoid function Calculate and obtain the weight used to represent the degree of tendency between hierarchical information Then, when calculating the fusion of neighboring information in the class space, the proportion of the fusion information of the current layer and the adjacent layer is controlled; based on the Gaussian kernel function, the similarity index of the class space is Calculate and obtain the fuse selection used to represent the degree of difference between the hierarchical information Then, the ratio of fused information to original information is controlled during the fusion calculation of neighboring information in class space; and the class space regression quality information D is further extracted based on the fused information. i , and / or,

[0033] Based on the regression space neighboring similarity calculation, the regression similarity index of the current layer and the adjacent layer at the same target detection position is obtained. Regression similarity index based on sigmoid function and class space regression quality information D i Calculate and weight to get the weight used to represent the degree of tendency between hierarchical information Then, the proportion of fusion information between the current layer and the adjacent layer is controlled when calculating the fusion of neighboring information in the regression space; the regression similarity index is calculated based on the Gaussian kernel function. Calculate and obtain the fuse selection used to represent the degree of difference between the hierarchical information Then, the ratio of fused information to original information is controlled during the fusion calculation of neighboring information in the regression space.

[0034] In one example, the defect detection includes:

[0035] Inputting the video stream data of the test piece to be inspected into the multi-scale spatial joint model;

[0036] Multi-scale spatial joint model extracts the category feature fusion information of the video stream data of the test piece to be inspected i ', regression feature fusion information R i ';

[0037] Traversing C i 'For each point in the feature map, compare the data value of the point with the defect judgment threshold. If it is greater than, it is judged as a defect. i 'The 4D data of the corresponding position is obtained by locating the feature map, and the defect target frame of the point is drawn based on it to realize defect location.

[0038] In one example, the method further comprises a preprocessing step:

[0039] Convert a single channel image to an RGB channel image; and / or,

[0040] Calculate the pixel mean and standard deviation of the image and standardize the image so that it satisfies the normal distribution.

[0041] It should be further explained that the technical features corresponding to the above detection method examples can be combined or replaced with each other to form a new technical solution.

[0042] The present invention also includes a storage medium on which computer instructions are stored. When the computer instructions are executed, the steps of the visual detection method of the multi-scale space joint model formed by any one of the above examples or multiple examples are executed.

[0043] The present invention also includes a terminal, including a memory and a processor, wherein the memory stores computer instructions that can be executed on the processor, and when the processor runs the computer instructions, it executes the steps of the visual detection method of the multi-scale space joint model formed by any example or multiple examples.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] 1. In one example, the FCOS basic network can extract multi-scale class space regression quality information and multi-scale regression space feature information of image data, and further perform data fusion processing on the multi-scale class space regression quality information and multi-scale regression space feature information through the classification fusion pyramid and regression fusion pyramid, so as to enhance the defect characteristics, suppress background noise, and thus improve the accuracy of defect detection; this application inputs the category feature information into the regression fusion pyramid to guide the information fusion of the regression space, which can realize end-to-end model training and defect detection, and ensure the accuracy of defect detection.

[0046] 2. In one example, the similarity index of the current layer and the adjacent layer at the same target detection position is obtained by calculating the similarity index; the weight is calculated by calculating the similarity index to indicate the degree of tendency between the layer information, and then the proportion of the fusion information of the current layer and the adjacent layer is adjusted to enhance the feature information of the current layer; the fuse selection is calculated by calculating the similarity index to indicate the degree of difference between the layer information, and the proportion of the fusion information and the original information is adjusted to weaken the fusion of high-frequency noise. This improves the accuracy of defect detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The specific implementation methods of the present invention are further described in detail below in conjunction with the accompanying drawings. The accompanying drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The same reference numerals are used in these drawings to represent the same or similar parts. The schematic embodiments of the present application and their descriptions are used to explain the present application and do not constitute improper limitations on the present application.

[0048] Figure 1 A structural diagram of a multi-scale spatial joint model in an example of the present invention;

[0049] Figure 2 A diagram of a classification fusion pyramid structure in an example of the present invention;

[0050] Figure 3 A regression fusion pyramid structure diagram in an example of the present invention;

[0051] Figure 4 is a structural diagram of a class space regression similarity module in an example of the present invention;

[0052] Figure 5 is a flow chart of a visual inspection method in an example of the present invention;

[0053] Figure 6 A flowchart of a visual inspection method according to a preferred embodiment of the present invention;

[0054] Figure 7 This is a schematic diagram of the detection results of the RetinaNet model in an example of the present invention;

[0055] Figure 8 This is a schematic diagram of the detection results of the CenterNet model in an example of the present invention;

[0056] Fig. 9 This is a schematic diagram of the FCOS model detection results in an example of the present invention;

[0057] Fig.10 This is a schematic diagram of the ATSS model detection results in an example of the present invention;

[0058] Fig.11 This is a schematic diagram of the detection results of the multi-scale space joint model in an example of the present invention. DETAILED DESCRIPTION

[0059] The technical solution of the present invention is described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0060] In the description of the present invention, it should be noted that the directions or positional relationships indicated by "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside", etc. are directions or positional relationships based on the drawings, which are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0061] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, "installation", "connection" and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0062] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0063] In one example, the present invention provides a multi-scale space joint model, which realizes defect detection (determination and positioning) by extracting and fusing category feature information and regression feature information in different scale spaces. Figure 1As shown, the multi-scale space joint model includes a FCOS basic network for extracting multi-scale category feature information and multi-scale regression feature information of the input image, and the multi-scale category feature information output end of the FCOS basic network is connected to a classification fusion pyramid, and the multi-scale regression feature information output end of the FCOS basic network is connected to a regression fusion pyramid. Among them, the FCOS network is an existing neural network model, including a sequentially connected backbone architecture convolutional neural network Backbone, FPN and Head, and the Head includes two branches of classification (Classification) + regression (Regression), respectively building a classification fusion pyramid and a regression fusion pyramid, and taking the regression quality evaluation as the calculation output of the class space regression similarity module.

[0064] Furthermore, the classification fusion pyramid is connected to the regression fusion pyramid via the class space regression similarity module. Specifically, the fusion feature information of each level output by the classification fusion pyramid is further extracted by the class space regression similarity module and then input to the corresponding level of the regression fusion pyramid. That is, each output layer of the classification fusion pyramid is connected to each input layer of the regression fusion pyramid via the class space regression similarity module. The class space regression similarity module is used to extract the output feature fusion information C of the classification fusion pyramid. i 'Class space regression quality information D i , and regress the class space quality information D i Input regression fusion pyramid; among them, the class space regression quality information D i , which is used to characterize the quality of the regression frame, that is, the degree of deviation between the point position and the target center. The classification fusion pyramid is based on multi-scale category feature information C i Further extract feature fusion information C i ', based on feature fusion information C i 'Defect type judgment; regression fusion pyramid based on multi-scale regression feature information R i , Class space regression quality information D i Further extract feature fusion information R i ', that is, mapping the feature information of the class space to the regression space, thereby guiding the information fusion of the regression space, and then realizing the end-to-end model training and defect detection, ensuring the accuracy of defect detection and achieving accurate defect location.

[0065] In this example, the FCOS basic network can extract the multi-scale class space regression quality information and multi-scale regression space feature information of the image data, and further perform data fusion processing on the multi-scale class space regression quality information and multi-scale regression space feature information through the classification fusion pyramid and regression fusion pyramid, so as to enhance the defect characteristics, suppress background noise, and improve the accuracy of defect detection.

[0066] In one example, if Figure 1 As shown, the classification fusion pyramid includes a third category feature extraction layer, a fourth category feature extraction layer, a fifth category feature extraction layer, a sixth category feature extraction layer, and a seventh category feature extraction layer. The third category feature extraction layer, the fourth category feature extraction layer, and the fifth category feature extraction layer perform feature fusion from bottom to top, and the fifth category feature extraction layer, the sixth category feature extraction layer, and the seventh category feature extraction layer perform feature fusion from top to bottom. Specifically, the category feature extraction layer includes a deconvolution layer and / or a convolution layer and / or a connection layer. Each layer can be optionally skip-connected, and the connection layer is used for information fusion connection. More specifically, as Figure 2 As shown, the third category feature extraction layer includes the first convolution layer a; the fourth category feature extraction layer includes the first connection layer a and the second convolution layer a connected in sequence; the fifth category feature extraction layer includes the second connection layer a and the third connection layer a in parallel, and also includes the fourth connection layer a, and the output ends of the second connection layer a and the third connection layer a are both connected to the fourth connection layer a; the sixth category feature extraction layer includes the fifth connection layer a and the first deconvolution layer a connected in sequence; the seventh category feature extraction layer includes the second deconvolution layer a. Furthermore, the output end of the first convolution layer a is connected to the first connection layer a, the output end of the second convolution layer a is connected to the second connection layer a, the output end of the first deconvolution layer a is connected to the third connection layer a, and the output end of the second deconvolution layer a is connected to the fifth connection layer a. The feature extraction process of the above levels is: C i A bottom-up fusion method is adopted between i=3, 4, and 5. The first convolution layer a and the second convolution layer a are convolved with 3×3×3 respectively, and initialized with a normal distribution with a mean of 0.11 and a standard deviation of 0.01. A top-down fusion method is adopted between i=5, 6, and 7. The first deconvolution layer a and the second deconvolution layer a are deconvolved with 3×4×4 respectively, and initialized with a normal distribution with a mean of 0.25 and a standard deviation of 0.01. In this way, the fusion processing of neighboring feature information in the class space is realized, and rich feature information in the class space is obtained.

[0067] Furthermore, when performing information fusion processing on the feature extraction layers of each category in the classification fusion pyramid, it is necessary to obtain the class space similarity index of the current layer and the adjacent layer at the same target detection position based on the class space neighboring similarity calculation; the class space similarity index is calculated based on the sigmoid function to obtain the weight used to represent the degree of tendency between the hierarchical information, and then the proportion of the fused information of the current layer and the adjacent layer is controlled in the class space neighboring information fusion calculation; the class space similarity index is calculated based on the Gaussian kernel function to obtain the fuse selection used to represent the degree of difference between the hierarchical information, and then the proportion of the fused information and the original information is controlled in the class space neighboring information fusion calculation; and the class space regression quality information is further extracted based on the fused information.

[0068] In one example, if Figure 1 As shown, the regression fusion pyramid includes a third regression feature extraction layer, a fourth regression feature extraction layer, a fifth regression feature extraction layer, a sixth regression feature extraction layer, and a seventh regression feature extraction layer. The third regression feature extraction layer, the fourth regression feature extraction layer, and the fifth regression feature extraction layer perform feature fusion from bottom to top, and the fifth regression feature extraction layer, the sixth regression feature extraction layer, and the seventh regression feature extraction layer perform feature fusion from top to bottom. Specifically, the regression feature extraction layer includes a deconvolution layer and / or a convolution layer and / or a connection layer. Each layer can be optionally skip-connected, and the connection layer is used for information fusion connection. More specifically, as Figure 3 As shown, the third regression feature extraction layer includes the first convolution layer b; the fourth regression feature extraction layer includes the first connection layer b and the second convolution layer b connected in sequence; the fifth regression feature extraction layer includes the second connection layer b and the third connection layer b in parallel, and also includes the fourth connection layer b, and the output ends of the second connection layer b and the third connection layer b are both connected to the fourth connection layer b; the sixth regression feature extraction layer includes the fifth connection layer b and the first deconvolution layer b connected in sequence; the seventh regression feature extraction layer includes the second deconvolution layer b. Furthermore, the output end of the first convolution layer b is connected to the first connection layer b, the output end of the second convolution layer b is connected to the second connection layer b, the output end of the first deconvolution layer b is connected to the third connection layer b, and the output end of the second deconvolution layer b is connected to the fifth connection layer b. The feature extraction process of the above levels is: R i A bottom-up fusion method is adopted between i=3,4,5, and the first convolution layer a and the second convolution layer a are used for convolution processing with 4×3×3 respectively; a top-down fusion method is adopted between i=5,6,7, and the first convolution layer a and the second convolution layer a are used for deconvolution processing with 4×4×4 respectively, and both are initialized with a normal distribution with a mean of 0 and a standard deviation of 0.01, so as to realize the fusion processing of neighboring feature information in the regression space and obtain rich feature information in the regression space.

[0069] Furthermore, when each regression feature extraction layer in the regression fusion pyramid performs information fusion processing, it is necessary to obtain the regression similarity index of the current layer and the adjacent layer at the same target detection position based on the regression space neighboring similarity calculation; the regression similarity index and the class space regression quality information are calculated and weighted based on the sigmoid function to obtain the weight used to represent the degree of tendency between the hierarchical information, and then control the proportion of the fused information of the current layer and the adjacent layer when calculating the fusion of the regression space neighboring information; the regression similarity index is calculated based on the Gaussian kernel function to obtain the fuse selection used to represent the degree of difference between the hierarchical information, and then control the proportion of the fused information and the original information when calculating the fusion of the regression space neighboring information.

[0070] In one example, the class space regression similarity module includes several sequentially connected convolutional layers. Specifically, Figure 4As shown in the figure, the class space regression similarity module includes the third convolution layer, the fourth convolution layer and the fifth convolution layer connected in sequence. i When inputting the class space regression similarity module, the class space feature extraction is performed in sequence through the third convolution layer with a convolution kernel of 3×3×3, the fourth convolution layer with a convolution kernel of 3×3×1, and the fifth convolution layer with a convolution kernel of 1×1×1, and the output D of b×1×w×h is obtained. i . Furthermore, the above convolutional layers are initialized with a normal distribution with a mean of 0 and a standard deviation of 0.01. To ensure initial convergence, back propagation at the input needs to be cut off in the first 100 iterations. In this example, the regression box quality features in the class space are further extracted through the class space regression similarity module, thereby weakening the detection of regression boxes far from the target center and low-quality regression boxes, thereby improving the classification and regression accuracy of the model.

[0071] The present invention also includes a visual detection method of a multi-scale space joint model, which has the same inventive concept as the above multi-scale space joint model, and the method is implemented based on the multi-scale space joint model formed by any one of the above examples or a combination of multiple examples, such as Figure 5 As shown, the following steps are included:

[0072] S1: Build a multi-scale spatial joint model;

[0073] S2: training a multi-scale spatial joint model;

[0074] S21: Extract training sample features based on the basic network to obtain multi-scale category feature information C i and multi-scale regression feature information R i ;

[0075] S22: C i Input into the classification fusion pyramid to perform feature fusion calculation to obtain feature fusion information C i ', and based on C i Train the classification fusion pyramid;

[0076] S23: Extract C i 'Class space regression quality information to obtain class space regression quality information D i ;

[0077] S24: Based on R i , D i Train the regression fusion pyramid;

[0078] S25: Repeat steps S21-S24 until the multi-scale spatial joint model converges to obtain a trained multi-scale spatial joint model;

[0079] S3: Defect detection of video stream data of the test piece to be inspected is realized based on the multi-scale spatial joint model.

[0080] In one example, building a multi-scale spatial joint model includes:

[0081] S11: uses FCOS as the basic network;

[0082] S12: Based on the output results of the Head of the FCOS basic network, a classification fusion pyramid and a regression fusion pyramid are built respectively; among them, the output result of the Head is the output result of the FCOS basic network, and the Head has two convolution branches, classification and regression. Then, the classification fusion pyramid C is built respectively based on the output results of the Head. i b×m×w×h and regression fusion pyramid R i b×4×w×h , where i∈{3,4,5,6,7}, corresponds to the level of the FPN feature pyramid in the FCOS network, and m is the number of categories.

[0083] S13: Connecting the classification fusion pyramid to the regression fusion pyramid through the class space regression similarity module. Specifically, the fusion feature information of each level output by the classification fusion pyramid is further extracted by the class space regression similarity module and then input into the corresponding level of the regression fusion pyramid.

[0084] In one example, step S21 obtains multi-scale category feature information C i and multi-scale regression feature information R i Specifically include:

[0085] When the specimen video stream data is input into the FCOS network, the Backbone in the basic network extracts features in sequence through the internal convolution modules. The feature map output by the i-th convolution module is denoted as B i b×n×w×h . i , i=3,4,5 to extract FPN features and get P i b×n×w×h , i=3,4,5. Downsample P5 to get P6, and then downsample P6 to get P7; P i , i=3,4,5,6,7, each layer obtains multi-scale category feature information C through weight sharing. i and multi-scale regression feature information R i .

[0086] Furthermore, based on C i , D i Training the classification fusion pyramid based on R i , D iThe same model training scheme is used to train the regression fusion pyramid, that is, firstly, the samples are divided into a training sample set (C i , R i and D i ) and the test sample set (C i and R i ), input the samples in the training set into the corresponding fusion pyramid, when the classification or regression error of the fusion pyramid is less than the threshold, the model converges, and then the training of the fusion pyramid model is completed. Of course, the fusion pyramid model can also be trained by weight change, when the weight change is less than the set threshold, the model converges and the training stops; or the fusion pyramid model can be trained by setting the maximum number of iterations, and the training stops when the number of iterations exceeds the maximum number of iterations.

[0087] In one example, the classification fusion pyramid for feature extraction in step S22 includes:

[0088] C i Input to the classification fusion pyramid to perform neighboring category similarity calculation and neighboring information fusion calculation; extract neighboring information to enhance the current level information, first calculate the similarity between the layer and the adjacent layer at the same position, where the positive and negative are only used to indicate the bias towards the layer or the adjacent layer, and the value size is used to indicate the degree of tendency. Further, the similarity calculation and neighboring information fusion calculation include:

[0089] Based on the class space neighboring level similarity calculation, the class space similarity index of the current level and the adjacent level at the same target detection position is obtained. Similarity index of class space based on sigmoid function Calculate and obtain the weight used to represent the degree of tendency between hierarchical information Then, when calculating the fusion of neighboring information in the class space, the proportion of the fusion information of the current layer and the adjacent layer is controlled; based on the Gaussian kernel function, the similarity index of the class space is Calculate and obtain the fuse selection used to represent the degree of difference between the hierarchical information Then, the ratio of fused information to original information is controlled during the fusion calculation of neighboring information in class space; and the class space regression quality information D is further extracted based on the fused information. i .

[0090] Specifically, the feature extraction process of the classification fusion pyramid in step S22 is as follows:

[0091] C i As the input of the classification fusion pyramid, the final result C is obtained i ' b×m×w× , where C i '=C i,i=3,7. When i=4,5,6, it is necessary to perform fusion calculation of adjacent level information, and when i=5, it is necessary to perform two different calculations and then fuse them again.

[0092] Specifically, when i=4,5, a bottom-up fusion method is adopted, and the category similarity between multiple levels is calculated as follows:

[0093]

[0094] Where f(·) is a convolution operation that downsamples the lower-layer feature map to the size of the upper-layer feature map.

[0095] When i=5,6, a top-down fusion method is used, and the category similarity between multiple levels is calculated as follows:

[0096]

[0097] Where g(·) is a deconvolution operation that upsamples the upper feature map to the size of the lower feature map.

[0098] The similarity results To perform z-score standardization, the calculation is as follows:

[0099]

[0100] in, They are Mean and standard deviation at each i, m, b layer.

[0101] The standardized similarity results are calculated using the sigmoid function, the symbolic features of the similarity are discarded, and a differentiable weight is obtained.

[0102]

[0103] in, Used to indicate the degree to which the fusion information tends to this layer.

[0104] The standardized similarity results are calculated using the Gaussian kernel function to obtain a differentiable fuse selection

[0105]

[0106] Among them, select When the difference between neighboring information is large, a smaller Used to weaken fusion strength.

[0107] The information fusion calculation when i=4,5 is as follows:

[0108]

[0109] in, Represents the class space regression quality information provided by the current layer, represents the class space regression quality information provided by the adjacent upper layer, where and Used to control the fusion strength, that is, the ratio of fusion information to original information in the class space.

[0110] The information fusion calculation when i=5,6 is as follows:

[0111]

[0112] in, Indicates information provided by the adjacent lower layer.

[0113] Finally, C i =C' i , where the two C′5 obtained through the above calculation adopt the following information fusion method:

[0114]

[0115] C′5=w C5 C′ u5 +(1-w C5 )C' d5

[0116] in, and C′ u5 They represent the fusion information obtained by top-down calculation when i=5 and 6 respectively. and C′5; and C' d5 They represent the fusion information obtained by bottom-up calculation when i=4 and 5 respectively. and C′5.

[0117] In one example, in step S23, C is extracted i 'Class space regression quality information to obtain class space regression quality information D i, is to further map the feature information of the class space to the regression space, and then use centerness as the training target output by the class space regression similarity module, so as to keep the model training and model prediction (defect detection) processes consistent. Specifically, the general detection of defects based on neural networks is to first realize the judgment of defects through the classification branch, that is, to judge whether a defect occurs. If a defect occurs, the defect position is further located through the regression branch. There is a certain connection between classification and regression. In the training stage, classification and regression are trained with two independent branches. Then the traditional model structure has the problem of inconsistency between training and defect detection. The present invention guides the fusion of regression space through the information of the class space, and reduces the inconsistency between the classification score (classification score, corresponding to classification accuracy) and the regression IOU / centerness score (center position score, corresponding to regression accuracy) in the training and defect detection stages. Among them, IOU is the intersection-union ratio, which represents the ratio of the intersection and union of the "predicted bounding box" and the "real bounding box" output by the regression fusion pyramid. More specifically, for the class space, D i Using centerness as the training target of the class space regression similarity module output gives the classification score of the classification another meaning. The higher the score, the closer it is to the target center, that is, the higher the classification score, the closer it is to the correct scale level, avoiding inconsistencies in training and reasoning. For example, a sample with a low classification score has an extremely high IOU / centerness score, which makes it mistakenly ranked in front of samples with high classification scores and low IOU / centerness scores. For the regression space, the output of the class space regression similarity module is used as a control factor for regression fusion. The classification score of the classification can be expressed in a coupled manner of the distance from the target center and the distance from the correct scale level, thereby controlling the strength of regression fusion. For example, a smaller output indicates that the position is too far away from the center or is not at the correct scale level. In the fusion stage, after the similarity between layers is calculated, if it is positive, it means that the position and level in this layer are more accurate, and the fusion information should be biased towards this layer; if it is negative, it means that the position and level in the adjacent layer are more accurate, and the fusion information should be biased towards the adjacent layer; if it is zero, it means that the position and level accuracy in the two layers are consistent, and the reference of the fusion information to the two layers should also be consistent.

[0118] In this example, extract C i 'Class space regression quality information to obtain class space regression quality information D i Specifically include:

[0119] C′ with b×3×w×h i As input, it passes through 3×3×3 convolution kernel, 3×3×1 convolution kernel, and 1×1×1 convolution kernel in sequence to get the output D of b×1×w×h. i The convolutional layers are initialized with a normal distribution with a mean of 0 and a standard deviation of 0.01. To ensure initial convergence, the back propagation at the input needs to be cut off in the first 100 iterations.

[0120] In one example, step S24 also includes a feature extraction step of regression fusion pyramid:

[0121] R i , D i Input into the regression fusion pyramid and perform neighboring regression similarity calculation and neighboring information fusion calculation; further, the similarity calculation and neighboring information fusion calculation include:

[0122] Based on the regression space neighboring similarity calculation, the regression similarity index of the current layer and the adjacent layer at the same target detection position is obtained. Regression similarity index based on sigmoid function and class space regression quality information D i Calculate and weight to get the weight used to represent the degree of tendency between hierarchical information Then, the proportion of fusion information between the current layer and the adjacent layer is controlled when calculating the fusion of neighboring information in the regression space; the regression similarity index is calculated based on the Gaussian kernel function. Calculate and obtain the fuse selection used to represent the degree of difference between the hierarchical information Then, the ratio of fused information to original information is controlled during the fusion calculation of neighboring information in the regression space.

[0123] Specifically, the feature extraction process of the regression fusion pyramid in step S24 is as follows:

[0124] R i As the input of the regression fusion pyramid, the final result R is obtained i ' b×4×w×h , where R i '=R i ,i=3,7. When i=4,5,6, it is necessary to perform the fusion calculation of the neighboring information. When i=5, it is necessary to perform two different calculations and then fuse them again.

[0125] Specifically, when i=4,5, a bottom-up fusion method is adopted, and the similarity between multiple levels is calculated as follows:

[0126]

[0127] Di =D i -f(D i-1 )

[0128] Among them, f(·) is the convolution for downsampling.

[0129] When i=5,6, a top-down fusion method is used, and the similarity between multiple levels is calculated as follows:

[0130]

[0131] D i =D i -g(D i+1 )

[0132] Among them, g(·) is the upsampling by deconvolution.

[0133] The similarity results and class space regression similarity results are z-score standardized respectively, and D of b×1×w×h is * i Expanded to b×4×w×h, the specific standardized calculation formula is:

[0134]

[0135]

[0136] The standardized similarity and class space regression similarity results are calculated and weighted by the sigmoid function to obtain differentiable weights. The specific calculation formula is:

[0137]

[0138] Among them, λ is the weighting factor, which is 0.8 in this example; the differentiable weight

[0139] Compute the normalized class space regression similarity to obtain differentiable fuse selection When the class space regression similarity difference is large, it indicates that in the class space, the current layer and the neighboring layer do not both consider it to be a positive sample, so the strength of fusion needs to be appropriately weakened; the differentiable fuse selection The calculation formula is:

[0140]

[0141] The fusion calculation when i=4,5 is as follows:

[0142]

[0143] in, Represents the regression spatial feature information provided by the current layer, Represents the regression feature information provided by the adjacent upper layer, where and Used to control the fusion strength, that is, the ratio of fusion information to original information in the regression space.

[0144] The fusion calculation when i=5,6 is as follows:

[0145]

[0146] in, Indicates information provided by the adjacent lower layer.

[0147] Finally, R i =R' i , where the two R′5 obtained by the above calculation are calculated by information fusion as follows:

[0148]

[0149] R5=w R5 R u '5+(1-w R5 )R' d5

[0150] in, and R′ u5 They represent the fusion information obtained by top-down calculation when i=5 and 6 respectively. and R′5; and R' d5 They represent the fusion information obtained by bottom-up calculation when i=4 and 5 respectively. and R′5.

[0151] In one example, defect detection includes:

[0152] S31: Input the video stream data of the test piece to be inspected into the multi-scale space joint model; specifically, convert multiple images into tensor form, then b tensors are recorded as In∈P b×n×w×h , input to the multi-scale spatial joint model.

[0153] S32: Multi-scale spatial joint model extracts feature fusion information C of the video stream data of the test piece to be inspected i ', feature fusion information R i '; Specifically, each category feature extraction layer of the classification fusion pyramid extracts the category features of the input image and performs neighboring feature fusion processing to obtain feature fusion information C i '; Each regression feature extraction layer of the regression fusion pyramid extracts the regression features of the input image and performs neighboring feature fusion processing to obtain feature fusion information R i';

[0154] S33: Traversing C i 'For each point in the feature map, compare the data value of the point with the defect judgment threshold. If it is greater than, it is judged as a defect. i 'The 4D data of the corresponding position is obtained by locating the feature map, and the defect target frame of the point is drawn based on it, thereby realizing defect location. More specifically, the target frame is centered on the corresponding point and is divided into two parts by R′ i The 4D data is used as the distance from the point to the upper, lower, left, and right boxes, and NMS (non-maximum suppression) operation is performed on all candidate boxes to obtain the final result, and the target box is drawn.

[0155] In one example, the method further comprises an image preprocessing step:

[0156] Convert a single-channel image to an RGB channel image; specifically, convert P 1×w×h Expanded to P 3×w×h , where the data of each channel is consistent with the data of a single channel.

[0157] Calculate the pixel mean and standard deviation of the image, standardize the image, make the image satisfy the normal distribution, ensure the diversity of the image, and thus ensure the reliability and accuracy of the model training.

[0158] Combining the above examples, we can obtain the preferred visual detection method of this application, such as Figure 6 As shown, the specific steps include:

[0159] S1': Collect image sequence; specifically, use industrial light source to illuminate the test piece, and collect video stream data through industrial camera, where each frame of image sequence is recorded as P n×w×h , h and w represent the height and width of the image, and n represents the number of channels of the image;

[0160] S2': image preprocessing step;

[0161] S3': Build a multi-scale spatial joint model;

[0162] S4': training a multi-scale spatial joint model;

[0163] S5': Defect detection of video stream data of the test piece to be inspected is realized based on the multi-scale spatial joint model.

[0164] This example presents a visual inspection method based on multi-scale space union. It extracts the multi-scale space category and regression features of the image through a multi-scale space union model, and constructs fusion pyramids for the extracted category and regression features respectively. It also uses the information of the class space to guide the fusion of the regression space, and predicts the confidence of the category to which each pixel belongs based on the trained weights, thereby realizing defect judgment, and finally realizing automatic positioning of the defect position based on the regression distance.

[0165] In order to better illustrate the inventive concept of the present application, the multi-scale spatial joint model of the present application and the existing neural network models, including the RetinaNet model, the CenterNet model, the FCOS model, and the ATSS model, are now applied to the defect detection of the wrist flange. The specific test parameters are as follows:

[0166] 1. Test equipment

[0167] Industrial camera MV-SUF1200GM-T: sensor 1.1″CMOS, pixel size 3.45*3.45μm, global shutter, black and white camera.

[0168] Industrial lens MV-LD-16-10M-J: aperture F1.6~F16, target size 1″, 10 million pixels, 16mm lens focal length.

[0169] Bar light source CST-2BS14718-B for video stream data acquisition fill light: blue light.

[0170] Ring light source CST-RS12080-B for video stream data acquisition fill light: blue light.

[0171] 2. Test process

[0172] The industrial camera and light source are installed on the wrist flange of the robot arm through a self-made fixture. The terminal path of the robot arm is controlled so that the industrial camera can scan and record the entire pipeline under the constraint of being perpendicular to the surface of the pipeline weld. The collected image data is labeled and classified, and the divided training set is sent to the deep learning network training, and the trained network model is used on the test set for network performance testing.

[0173] 3. Parameter settings

[0174] Focal length: 0.1mm, aperture: F1.6, resolution: 1500x1200, video frame rate: 24fps, exposure time: 5ms, exposure gain: 20, strip light PWM: 160, ring light PWM: 135, scan time: 20s.

[0175] 4. Dataset Description

[0176] The data set for this experiment is 2021.1.18, which contains 59 pipelines, 43 of which are used as training sets and 16 as test sets. There are three types of defects, such as Figure 7 As shown, a is a burst shape defect, b is a scratch shape crack, and c is an indentation shape defect.

[0177] 5. Comparison of test results

[0178] Table 1 Performance comparison between this application and existing models

[0179]

[0180] Based on Table 1, it can be seen that the proposed defect discrimination probability has high accuracy; further, the actual detection results of the proposed model and the existing model are as follows: Figure 7-11 As shown, Figure 7 This is a schematic diagram of the defect detection results of the RetinaNet model; Figure 8 This is a schematic diagram of the defect detection results of the CenterNet model; Fig. 9 This is a schematic diagram of the defect detection results of the FCOS model; Fig.10 This is a schematic diagram of the defect detection results of the ATSS model; Fig.11 This is a schematic diagram of the defect detection results of the multi-scale spatial joint model of the present application. According to the above detection result diagram, it can be seen that the model of the present application can detect defects that other models cannot detect during the defect detection process, and for targets that can be detected by the above models, the confidence of the target box of this model is higher than that of other models, and it has good defect detection capabilities.

[0181] This embodiment provides a storage medium, which has the same inventive concept as a visual detection method for a multi-scale space joint model formed by any one of the above examples or a combination of multiple examples, and stores computer instructions thereon, which, when executed, execute the steps of the visual detection method for a multi-scale space joint model formed by any one of the above examples or a combination of multiple examples.

[0182] Based on this understanding, the technical solution of this embodiment, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0183] This embodiment also provides a terminal, which has the same inventive concept as the visual detection method of a multi-scale space joint model formed by any one or more of the above examples, including a memory and a processor, the memory stores computer instructions that can be run on the processor, and the processor executes the steps of the visual detection method of a multi-scale space joint model formed by any one or more of the above examples when running the computer instructions. The processor can be a single-core or multi-core central processing unit or a specific integrated circuit, or one or more integrated circuits configured to implement the present invention.

[0184] Each functional unit in the embodiment provided by the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0185] The above specific implementation methods are detailed descriptions of the present invention. It cannot be determined that the specific implementation methods of the present invention are limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions and substitutions can be made without departing from the concept of the present invention, which should be regarded as belonging to the protection scope of the present invention.

Claims

1. A visual detection method of a multi-scale spatial joint model, characterized in that: The method is implemented based on a multi-scale space joint model, wherein the multi-scale space joint model includes an FCOS basic network for extracting multi-scale category feature information and multi-scale regression feature information of an input image, wherein the multi-scale category feature information output end of the FCOS basic network is connected to a classification fusion pyramid, and the multi-scale regression feature information output end of the FCOS basic network is connected to a regression fusion pyramid; The classification fusion pyramid is connected to the regression fusion pyramid via the class space regression similarity module; the class space regression similarity module is used to extract the output feature fusion information C of the classification fusion pyramid. i 'Class space regression quality information D i , and regress the class space quality information D i Input regression fusion pyramid; classification fusion pyramid is based on multi-scale category feature information C i Further extract feature fusion information C i ', based on feature fusion information C i 'Defect category judgment; regression fusion pyramid based on multi-scale regression feature information R i , Class space regression quality information D i Further extract feature fusion information R i ', based on feature fusion information R i 'Locate defects; The visual detection method comprises the following steps: S1: Building a multi-scale spatial joint model; S2: training a multi-scale spatial joint model; S21: Extract training sample features based on the basic network to obtain multi-scale category feature information C i and multi-scale regression feature information R i ; S22: C i Input into the classification fusion pyramid to perform feature fusion calculation to obtain feature fusion information C i ', and based on C i Train the classification fusion pyramid; S23: Extract C i 'Class space regression quality information D i ; S24: Based on R i , D i Train the regression fusion pyramid; S25: Repeat steps S21-S24 until the multi-scale spatial joint model converges to obtain a trained multi-scale spatial joint model; S3: Defect detection of video stream data of the test piece to be inspected based on the multi-scale spatial joint model; The fusion pyramid includes: C i Input into the classification fusion pyramid to perform neighbor-level category similarity calculation and neighbor-level information fusion calculation, and extract class space regression quality information D based on the fusion information i ; and / or, R i , D i Input into the regression fusion pyramid and perform neighbor-level regression similarity calculation and neighbor-level information fusion calculation.

2. The visual detection method of a multi-scale spatial joint model according to claim 1, characterized in that: The classification fusion pyramid includes a third category feature extraction layer, a fourth category feature extraction layer, a fifth category feature extraction layer, a sixth category feature extraction layer, and a seventh category feature extraction layer. The third category feature extraction layer, the fourth category feature extraction layer, and the fifth category feature extraction layer perform feature fusion from bottom to top or from top to bottom, and the fifth category feature extraction layer, the sixth category feature extraction layer, and the seventh category feature extraction layer perform feature fusion from top to bottom or from bottom to top.

3. The visual detection method of a multi-scale spatial joint model according to claim 1, characterized in that: The regression fusion pyramid includes a third regression feature extraction layer, a fourth regression feature extraction layer, a fifth regression feature extraction layer, a sixth regression feature extraction layer, and a seventh regression feature extraction layer. The third regression feature extraction layer, the fourth regression feature extraction layer, and the fifth regression feature extraction layer perform feature fusion from bottom to top or from top to bottom, and the fifth regression feature extraction layer, the sixth regression feature extraction layer, and the seventh regression feature extraction layer perform feature fusion from top to bottom or from bottom to top.

4. The visual detection method of a multi-scale spatial joint model according to claim 1, characterized in that: The class space regression similarity module includes several convolutional layers connected in sequence.

5. The visual detection method of a multi-scale spatial joint model according to claim 1, characterized in that: The construction of the multi-scale spatial joint model includes: Adopt FCOS as the basic network; Based on the Head output results of the FCOS basic network, a classification fusion pyramid and a regression fusion pyramid are built respectively; Connect the classification fusion pyramid to the regression fusion pyramid through the class space regression similarity module.

6. The visual detection method of a multi-scale spatial joint model according to claim 1, characterized in that: The similarity calculation and neighbor level information fusion calculation include: Based on the class space neighboring level similarity calculation, the class space similarity index of the current level and the adjacent level at the same target detection position is obtained. Similarity index of class space based on sigmoid function Calculate and obtain the weight used to represent the degree of tendency between hierarchical information Then, when calculating the fusion of neighboring information in the class space, the proportion of the fusion information of the current layer and the adjacent layer is controlled; based on the Gaussian kernel function, the similarity index of the class space is Calculate and obtain the fuse selection used to represent the degree of difference between the hierarchical information Then, the ratio of fused information to original information is controlled during the fusion calculation of neighboring information in class space; and the class space regression quality information D is extracted based on the fused information. i , and / or, Based on the regression space neighboring level similarity calculation, the regression similarity index of the current level and the adjacent level at the same target detection position is obtained. Regression similarity index based on sigmoid function and class space regression quality information D i Calculate and weight to get the weight used to represent the degree of tendency between hierarchical information Then, the proportion of fusion information between the current layer and the adjacent layer is controlled when calculating the fusion of neighboring information in the regression space; the regression similarity index is calculated based on the Gaussian kernel function. Calculate and obtain the fuse selection used to represent the degree of difference between the hierarchical information Then, the ratio of fused information to original information is controlled during the fusion calculation of neighboring information in the regression space.

7. The visual detection method of a multi-scale spatial joint model according to claim 1, characterized in that: The defect detection includes: Inputting the video stream data of the test piece to be inspected into the multi-scale spatial joint model; The multi-scale spatial joint model extracts the feature fusion information of the video stream data of the test piece to be inspected. i ', feature fusion information R i '; Judgment C i 'Is it greater than C i 'Set the threshold value for the corresponding level. If it is greater than, it is judged as a defect and the location is related to C i 'Corresponding level R i ' and draw the target selection box based on the corresponding points to achieve defect positioning.

8. The visual detection method of a multi-scale spatial joint model according to claim 1, characterized in that: The method also includes a pre-processing step: Convert a single channel image to an RGB channel image; and / or, Calculate the pixel mean and standard deviation of the image and standardize the image so that it satisfies the normal distribution.

Images