Defect detection method and system based on multi-modal vision

The multimodal vision-based defect detection method addresses the challenge of accurately locating defects by integrating RGB and depth image features through cross-modal fusion and decoding, achieving enhanced defect representation and boundary refinement for precise defect positioning.

CN119090822APending Publication Date: 2024-12-06ZHENGZHOU UNIV

Patent Information

Application Number
CN202411113145.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-14
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

Existing defect detection technologies struggle to accurately locate surface defects and their boundaries due to the difficulty in effectively integrating and distinguishing between RGB and depth images, especially in complex backgrounds with unclear contrasts.

Method used

A defect detection method and system utilizing multimodal vision that performs feature extraction on RGB and depth images, followed by cross-modal feature fusion to generate an edge map containing defect edge features, and decodes these features to obtain a defect prediction map, employing techniques like SwinTransformer, deep mining feature fusion, context-aware edge detection, and hierarchical feature aggregation to enhance defect representation and boundary refinement.

Benefits of technology

The method effectively integrates RGB and depth information for deeper feature fusion, enhancing defect feature representation and achieving more accurate defect area positioning by combining coarse-grained and fine-grained edge information.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119090822A_ABST
    Figure CN119090822A_ABST
Patent Text Reader

Abstract

The invention discloses a defect detection method and system based on multi-modal vision, and the method comprises the following steps: carrying out the feature extraction of an RGB image and a depth image, and obtaining a multi-stage RGB coding feature and a multi-stage depth coding feature; performing cross-modal feature fusion on the multi-stage RGB coding features and the multi-stage depth coding features to obtain multi-stage RGB-D coding features, and generating an edge graph containing defect edge features; and decoding according to the multi-stage RGB-D coding features and the defect edge features to obtain a defect prediction map. According to the defect detection method disclosed by the invention, the depth information which can be semantically matched with the RGB features can be mined and utilized, and the RGB and the depth information are effectively integrated, so that deeper feature fusion is realized, and the representation of the defect features is enhanced; meanwhile, edge information of coarse granularity and fine granularity is combined for refining defect boundaries, so that more accurate defect area positioning is realized.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer vision and deep learning technology, and in particular to a defect detection method and system based on multimodal vision. Background Art

[0002] In the field of modern manufacturing and product maintenance, surface defect detection is crucial to ensure the stability of product quality and the safety of the system. With the rapid development of deep learning technology, the research and practice of intelligent defect detection methods have made significant progress, greatly improving the reliability and efficiency of quality control in industrial production. However, the current mainstream intelligent defect detection technology mainly relies on single RGB image information. When the color and texture of the object surface are similar to the background, the defects may be difficult to be effectively distinguished from the RGB image.

[0003] In response to this challenge, the development of depth imaging technology has brought new opportunities for defect detection. Depth images (Depth Map) provide rich geometric shape information, which helps the algorithm to more accurately identify defects based on the three-dimensional features of the object and adapt to changes in various scenes. However, the difference in quality and semantics between depth images and RGB images means that the two cannot be simply treated equally and the traditional feature fusion strategy is adopted. This method cannot give full play to the complementary advantages of the two in the field of surface defect detection. In addition, in defect scenes with complex backgrounds and unclear contrast between foreground and background, previous detection algorithms also find it difficult to accurately locate the boundary area of ​​the defect. Summary of the invention

[0004] The main problem to be solved by the present invention is to provide a defect detection method and system based on multimodal vision, so as to solve the problem that it is difficult to accurately locate the surface defects and boundary areas of defects of an object in the prior art.

[0005] In order to solve the above technical problems, a technical solution adopted by the present invention is to provide a defect detection method based on multimodal vision, the method comprising the following steps:

[0006] Feature extraction is performed on the RGB image and the depth image respectively to obtain multi-level RGB coding features and multi-level depth coding features; cross-modal feature fusion is performed on the multi-level RGB coding features and the multi-level depth coding features to obtain multi-level RGB-D coding features, and an edge map containing defect edge features is generated at the same time; decoding is performed according to the multi-level RGB-D coding features and the defect edge features to obtain a defect prediction map.

[0007] In some embodiments, before extracting features from the RGB image and the depth image to obtain multi-level RGB coding features and multi-level depth coding features, the step of:

[0008] The RGB image and the depth image are preprocessed, and the RGB image and the depth image are reshaped so that the sizes of the RGB image and the depth image are the same.

[0009] In some embodiments, the step of extracting features from the RGB image and the depth image to obtain multi-level RGB coding features and multi-level depth coding features comprises the following steps:

[0010] Extract five layers of features from the RGB image and the depth image respectively to obtain the i-th layer of RGB coding features and the i-th layer deep encoding feature i=1,2,3,4,5; when i=1, the i-th layer RGB encoding feature and the i-th layer deep encoding feature The size of is 1 / 4 of the size of the RGB image and the depth image; when i=2,3,4, the i-th layer RGB encoding feature and the i-th layer deep encoding feature The size of is the RGB encoding feature of the (i-1) layer and 1 / 2 of the size of the (i-1)th layer depth encoding feature; when i=5, the i-th layer RGB encoding feature and the i-th layer deep encoding feature The size of the feature vector is the same as the RGB encoding feature vector of the (i-1)th layer. and the (i-1)th layer deep encoding features The same size.

[0011] In some embodiments, the step of generating an edge map containing defective edge features comprises the steps of:

[0012] The multi-level RGB-D coding features are cross-modally fused with the multi-level RGB coding features and the multi-level depth coding features to generate an edge map containing defect edge features.

[0013] In some embodiments, decoding according to the multi-level RGB-D coding features and the defect edge features to obtain a defect prediction map includes the steps of:

[0014] Aggregate the encoding features of different levels and modes to emphasize the defect area; use the defect edge features to perceive and sharpen the defect boundary; decode to obtain multi-level decoding features, and use the activation function to obtain the defect prediction map.

[0015] In some embodiments, when decoding is performed according to the defect edge features to obtain a defect prediction map, a hybrid loss function is used:

[0016] L total =L BCE +LIOU +L Dice ,

[0017] Among them, including the BCE loss function L BCE , IOU loss function L IOU and Dice loss function L Dice ;

[0018] The BCE loss function is:

[0019]

[0020] The IOU loss function is:

[0021]

[0022] The Dice loss function is:

[0023] L Dice = l Dice (P edge ,G edge ),

[0024] Among them, l BCE represents BCE loss, l IOU represents the IOU loss, represents the RGB-D predicted image of the i-th layer, P depth represents the depth prediction image, P rgb represents the RGB predicted image, G represents the true value image, l Dice represents the Dice loss, P edge The edge graph G represents the edge features containing defects. edge Represents the edge truth image.

[0025] The present invention also provides a defect detection system based on multimodal vision, the system comprising deep information mining and edge perception network, the deep information mining and edge perception network comprising:

[0026] A feature extraction module is used to extract features from RGB images and depth images respectively to obtain multi-level RGB coding features and multi-level depth coding features; a feature fusion module is used to perform cross-modal feature fusion on the multi-level RGB coding features and the multi-level depth coding features to obtain multi-level RGB-D coding features, and simultaneously generate an edge map containing defect edge features; a decoding module is used to decode according to the multi-level RGB-D coding features and the defect edge features to obtain a defect prediction map.

[0027] In some embodiments, the system further includes a preprocessing module, which is used to preprocess the RGB image and the depth image, and reshape the RGB image and the depth image so that the RGB image and the depth image have the same size.

[0028] In some embodiments, the feature fusion module includes:

[0029] A deep mining feature fusion module is used to perform cross-modal feature fusion on the multi-level RGB coding features and the multi-level depth coding features to obtain multi-level RGB-D coding features; a context-aware edge detection module is used to perform cross-modal feature fusion on the multi-level RGB-D coding features with the multi-level RGB coding features and the multi-level depth coding features respectively to generate an edge map containing defect edge features.

[0030] In some embodiments, the decoding module includes:

[0031] A hierarchical feature aggregation module is used to aggregate features of different levels and modes to emphasize defect areas; an edge-guided feature enhancement module is used to utilize the defect edge features to perceive and sharpen defect boundaries; a prediction module is used to decode to obtain multi-level decoding features and use activation functions to obtain defect prediction maps.

[0032] The beneficial effects of the present invention are as follows: the present invention discloses a defect detection method and system based on multimodal vision, the method comprising the following steps: extracting features from RGB images and depth images respectively to obtain multi-level RGB coding features and multi-level depth coding features; cross-modal feature fusion of multi-level RGB coding features and multi-level depth coding features to obtain multi-level RGB-D coding features, and simultaneously generating an edge map containing defect edge features; decoding according to the multi-level RGB-D coding features and defect edge features to obtain a defect prediction map. The defect detection method of the present invention can mine and utilize depth information that can semantically match RGB features, effectively integrate RGB and depth information to achieve deeper feature fusion and enhance the representation of defect features; at the same time, it combines coarse-grained and fine-grained edge information for defect boundary refinement to achieve more accurate defect area positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a flow chart of an embodiment of a defect detection method based on multimodal vision of the present invention;

[0034] Figure 2 It is a principle diagram of an embodiment of a defect detection method based on multimodal vision of the present invention;

[0035] Figure 3 It is a specific flow chart of step S2 in an embodiment of a defect detection method based on multimodal vision of the present invention;

[0036] Figure 4 It is a data flow diagram of step S21 in an embodiment of a defect detection method based on multimodal vision of the present invention;

[0037] Figure 5 It is a data flow diagram of the SSA part of step S21 in an embodiment of a defect detection method based on multimodal vision of the present invention;

[0038] Figure 6 It is a data flow diagram of step S22 in an embodiment of a defect detection method based on multimodal vision of the present invention;

[0039] Figure 7 It is a specific flow chart of step S3 in an embodiment of a defect detection method based on multimodal vision of the present invention;

[0040] Figure 8 It is a data flow diagram of step S31 in an embodiment of a defect detection method based on multimodal vision of the present invention;

[0041] Fig. 9 It is a data flow diagram of step S32 in an embodiment of a defect detection method based on multimodal vision of the present invention;

[0042] Fig.10 It is a schematic diagram of comparison between a defect prediction image and a true value in an embodiment of a defect detection method based on multimodal vision of the present invention;

[0043] Fig.11 It is a structural block diagram of an embodiment of a defect detection system based on multimodal vision of the present invention;

[0044] Fig.12 It is a specific structural diagram of a feature fusion module in an embodiment of a defect detection system based on multimodal vision of the present invention;

[0045] Fig.13 It is a specific structural diagram of a decoding module in an embodiment of a defect detection system based on multimodal vision of the present invention. DETAILED DESCRIPTION

[0046] In order to facilitate the understanding of the present invention, the present invention is described in more detail below in conjunction with the accompanying drawings and specific embodiments. Preferred embodiments of the present invention are provided in the accompanying drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described in this specification. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present invention more thorough and comprehensive.

[0047] It should be noted that, unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used in this specification includes any and all combinations of one or more related listed items.

[0048] like Figure 1 As shown, a defect detection method based on multimodal vision of the present invention is shown, comprising the following steps:

[0049] Step S1: extract features from the RGB image and the depth image respectively to obtain multi-level RGB coding features and multi-level depth coding features.

[0050] It should be noted that before step S1, the RGB image and the depth image need to be preprocessed. After the data enhancement operation is performed simultaneously, the size (height and width) of the RGB image and the depth image are reshaped to 224×224, where the RGB image input channel is 3 and the depth image input channel is 1.

[0051] Specifically, combined Figure 2 As shown, step S1 includes the following steps:

[0052] Extract five layers of features from the RGB image and the depth image respectively to obtain the i-th layer RGB encoding feature and the i-th layer deep encoding feature i=1,2,3,4,5; when i=1, the i-th layer RGB encoding feature and the i-th layer deep encoding feature The size of is 1 / 4 of the size of the RGB image and the depth image; when i=2,3,4, the i-th layer RGB encoding feature and the i-th layer deep encoding feature The size of is the RGB encoding feature of the (i-1) layer and 1 / 2 of the size of the (i-1)th layer depth encoding feature; when i=5, the i-th layer RGB encoding feature and the i-th layer deep encoding feature The size of the feature vector is the same as the RGB encoding feature vector of the (i-1)th layer. and the (i-1)th layer deep encoding features The same size.

[0053] In this embodiment, two identical but independent SwinTransformer-S are used as encoders for RGB features and depth features, respectively, to extract multi-scale defect features from different modalities. Each encoder contains five layers, the first layer is downsampled by convolution, and the size of the first layer encoding feature is 1 / 4 of the size of the RGB image or the depth image, and the 2nd to 4th layers are downsampled by maximum pooling, and the size of the 2nd to 4th layers encoding features is 1 / 2 of the size of the previous layer (1st to 3rd layers). The 5th layer is not downsampled, and the size of the 5th layer encoding feature is the same as the size of the 4th layer encoding feature, that is, the size of the 1st layer RGB encoding feature and the 1st layer depth encoding feature is 56×56, the size of the 2nd layer RGB encoding feature and the 2nd layer depth encoding feature is 28×28, the size of the 3rd layer RGB encoding feature and the 3rd layer depth encoding feature is 14×14, the size of the 4th layer RGB encoding feature and the 4th layer depth encoding feature is 7×7, and the size of the 5th layer RGB encoding feature and the 5th layer depth encoding feature is 7×7.

[0054] In this process, the feature size is gradually compressed and the number of channels is gradually increased, so that the semantic information is gradually enhanced from the low layer to the high layer. To ensure the smooth transmission of detail information, the output of each layer is directly connected to the next layer. Therefore, the detail features of the front end of the network can be gradually transmitted to the deep layer of the network to compensate for the lack of detail information in the features extracted by the deep network. In this way, the features output in the encoding stage have multi-level and multi-scale information, providing more expressive defect edge features for the subsequent decoding process.

[0055] Among them, SwinTransformer is a deep learning model. The "Swin" in its name refers to "Shifted window". The characteristics of this model are that it adopts a multi-level structure and uses the shifted window method, focusing on its Hierarchical (multi-level) and Shifted Windows (shifted window) characteristics. Compared with the traditional visual Transformer, SwinTransformer can improve the feature representation ability while maintaining computational efficiency by shifting windows.

[0056] Step S2: Perform cross-modal feature fusion on the multi-level RGB encoding features and the multi-level depth encoding features to obtain multi-level RGB-D encoding features, and generate an edge map containing defect edge features.

[0057] Specifically, if Figure 3 As shown, step S2 includes the following sub-steps:

[0058] Step S21: Perform cross-modal feature fusion on the multi-level RGB coding features and the multi-level depth coding features to obtain multi-level RGB-D coding features.

[0059] It should be noted that there are semantic differences between RGB information and depth information, and the two do not completely match in terms of feature expression. Therefore, it is difficult to effectively complement the information of the two by using simple addition, multiplication or splicing to fuse features. In addition, in defect scenes, the information carried by depth images and RGB images is also different in terms of richness and quality. If the two modalities are blindly treated as equals and fused, the representation ability of the RGB modality may even be further reduced.

[0060] like Figure 4 As shown, the present application designs a deep-mining feature fusion module (DMFF) to mine valuable information from the deep modality to supplement the RGB encoding features and suppress the expression of redundant deep encoding features to achieve effective cross-modal feature fusion.

[0061] Among them, the i-th layer RGB encoding feature and the i-th layer deep encoding feature Perform cross-modal feature fusion to obtain the i-th layer RGB-D encoding features

[0062] Combination Figure 4 , first encode the RGB features of the i-th layer and the i-th layer deep encoding feature The concatenation is then performed, and then a wide receptive field is captured through continuous convolution and the channel is compressed to 1, and then the corresponding weight map is obtained after Sigmoid function activation processing. The weight map and the i-th layer deep encoding feature Perform element multiplication to obtain the i-th layer correction feature Thus, the i-th layer of deep encoding features is calibrated at the pixel level This process can be expressed as:

[0063]

[0064] Among them, σ represents the Sigmoid activation function, ⊙ represents element multiplication, and Cat represents the concatenation of features on the channel (corresponding to ), Conv 3×3 Represents a 3×3 convolution, where Conv 3×3 It includes two-dimensional convolution, BN layer and Relu activation function.

[0065] Furthermore, in defect scenes, given that RGB images have higher imaging quality and semantic richness than depth images, valuable depth information can often provide additional spatial information that has a high semantic similarity to the RGB encoding features themselves. Based on this point of view, the i-th layer RGB encoding feature and the i-th layer deep encoding feature After weighting, they are reshaped into Then perform matrix multiplication to obtain an association matrix R i ∈R C×C , this process can be expressed as:

[0066]

[0067] in, Represents matrix multiplication.

[0068] Furthermore, the correlation matrix R i ∈R C×C After Argmax processing, we get a C×1 correlation sequence Argmax(R i ), each element in the sequence represents the channel number of each channel of the RGB encoding feature that is most closely associated with the depth encoding feature. According to the correlation matrix R i The i-th layer correction feature The corresponding channels are extracted one by one and recombined in order to reconstruct the i-th layer reconstructed features. Then use the i-th layer to reconstruct the features With the i-th layer RGB encoding feature Add the elements to get the fusion features of the i-th layer This process can be expressed as:

[0069]

[0070] in, Denotes element-wise addition, and DFR represents the operation of sequentially extracting the deep feature channels most correlated with each RGB channel according to the correlation sequence and recombining them to obtain the reconstructed deep features.

[0071] Furthermore, if Figure 5 As shown, the i-th layer fusion features are fused through SSA (Synergistic Spatial Attention, collaborative spatial attention mechanism Further refinement, SSA can explore the commonalities between different modalities in space and complement each other. In SSA, the i-th layer RGB encoding features of different modal inputs The i-th layer rectified features And the i-th layer fusion feature After channel-wise pooling (CWP) and 7×7 convolution, each of them obtains an intermediate feature. And multiply them element by element to emphasize the common information of different modes in space, then add the three branches element by element to complement each other, and finally activate them with Sigmoid function to get the i-th layer spatial attention map This process can be expressed as:

[0072]

[0073] where mod∈{rd,wd,rgb}, Pooling represents channel pooling operation.

[0074] In this embodiment, the i-th layer spatial attention map is obtained through SSA After that, use the i-th layer spatial attention map For the i-th layer fusion feature Weighted, introducing the i-th layer fusion feature And input the i-th layer RGB encoding feature The residual connection of the i-th layer connection feature is obtained When i=1, the first layer connection features Perform 3×3 convolution to obtain the first layer RGB-D encoding features of the first layer DMFF output When i=2,3,4,5, the i-th layer connection feature The (i-1)th layer RGB-D encoding feature output by the previous layer DMFF after downsampling After concatenation, the 3×3 convolution is used to refine the i-th layer RGB-D encoding features output by the DMFF module of this layer. This process can be expressed as:

[0075]

[0076] Here, Down means downsampling.

[0077] In this process, after the encoding layer features of the two modalities are introduced into DMFF, RGB features are dominant and deep features are auxiliary. The attention mechanism is used to guide feature fusion to generate RGB-D stream features.

[0078] Step S22: Perform cross-modal feature fusion on the multi-level RGB-D coding features, the multi-level RGB coding features, and the multi-level depth coding features to generate an edge map containing defect edge features.

[0079] It should be noted that when the quality of the depth image is not ideal, its low-level features cannot retain the fine details of the edges, but the defective area can still be vaguely located in the deep network. That is, the depth image has poor accuracy in low-level texture details, but the defective area can be roughly located in the high-level low-resolution feature map with strong semantics.

[0080] like Figure 6 As shown, the present application designs a context-aware edge detection module (CAED), which utilizes a cross-attention mechanism to capture texture details between low-level RGB-D encoded features and RGB encoded features, as well as semantic information between high-level RGB-D encoded features and depth encoded features.

[0081] In this embodiment, the first layer RGB-D encoding features are captured and the first layer RGB encoding features Texture details between and 5th layer RGB-D encoding features and the 5th layer deep encoding features The semantic information between them is used to generate an edge graph P containing defect edge features. edge .

[0082] Combination Figure 2 and Figure 6 , for the first layer RGB-D encoding feature and the first layer RGB encoding features The number of channels is reduced by 1×1 convolution respectively; then the first layer RGB-D encoding features are Perform 1×1 convolution to adjust the number of channels and reshape to K l ∈R C×(H×W) and V l ∈R (H×W)×C , and at the same time the first layer RGB encoding features Reinventing Q l ∈R (H×W)×C Then, for K l and Q l Perform matrix multiplication to capture the global relationship between the two modal features, and then use the Softmax function to obtain the low-level edge attention map A. l ∈R (H×W)×(H×W) , the low-level edge attention map A l ∈R (H×W)×(H×W) With V l Perform matrix multiplication, map back to the original feature space through convolution, and introduce the first layer RGB-D encoding features Perform residual connection to obtain low-level edge features f l ∈R C×H×W .

[0083] Similarly, the same method is used to process the 5th layer RGB-D encoding features and the 5th layer deep encoding features Get high-level edge features f h ∈R C×H×W .

[0084] The above process can be expressed by the formula:

[0085]

[0086]

[0087] Furthermore, the low-level edge features f l and high-level edge features f h Combined with the texture and semantic information related to the defect edge, the texture and semantic information related to the defect edge are modeled. First, the low-level edge feature f l and high-level edge features f h It is refined through 3×3 convolution, then concatenated in the number of channels, and then processed through 3×3 convolution to obtain the defect edge feature f edge ∈R 256×H×W , which is rich in contextual information and highly edge-sensitive under the close supervision of the edge truth map. Next, the defect edge feature f edge After further 3×3 convolution, 1×1 convolution and Sigmoid function, the edge map P containing explicit defect boundary information is obtained. edge ∈R 1×H×W The above process can be expressed by the formula:

[0088]

[0089] In this process, through CAED, low-level and high-level features of different modalities are taken as input to balance the defect texture details and semantic information, and context-sensitive defect edge features are generated when learning to obtain edge maps.

[0090] Step S3: Decode the multi-level RGB-D encoding features and defect edge features to obtain a defect prediction map.

[0091] Specifically, if Figure 7 As shown, step S3 includes the following sub-steps:

[0092] Step S31: Aggregate the coding features of different levels and modalities to emphasize the defective areas.

[0093] It should be noted that in order to retain the defect details, this application is designed as follows Figure 8The Hierarchical Feature Aggregation (HFA) module shown aggregates features at different levels and adopts a progressive decoding strategy to transition the sampling process.

[0094] Combination Figure 8 First, the (i+1)th layer RGB-D encoding feature Upsampling, the i-th layer RGB-D encoding features Downsample to the same resolution, concatenate and perform 1×1 convolution to promote information fusion. The resulting fused features are then fed into the Receptive Field Block (RFB) to capture multiple receptive fields and compress channels. The aggregated encoded features from the RFB are then combined with the (i+1)th layer RGB-D decoded features that have been upsampled to the same resolution. Then, it is further upsampled to match the resolution of the low-level encoded features to obtain This process can be expressed as:

[0095]

[0096] Here, Up means upsampling.

[0097] Further, With the i-th layer RGB decoding feature and deep decoding features The SSA module is then used to enhance the representation of defective regions. Perform residual connection with the features output by the SSA module, and then connect it with the RGB-D encoding features of the i-th layer Splice to obtain the local RGB-D decoding features of the i-th layer This process can be expressed as:

[0098]

[0099] Here, SSA indicates that the corresponding operation of the SSA module in step S21 is performed according to the characteristics of the corresponding input SSA module.

[0100] In this embodiment, HFA prevents the loss of defect detail information during the sampling process by aggregating features of different levels and modalities, and at the same time, emphasizes the common significant defect areas in the decoding features of different modalities.

[0101] Step S32: Using defect edge features, sense and sharpen defect boundaries.

[0102] It should be noted that although spatial weighting is a commonly used edge prior method, this method is difficult to perceive the semantic difference between foreground and background and is affected by inaccurate edge maps. Since the boundary is essentially the division between foreground and background, it is equally important to enhance the network's understanding of contextual information. Therefore, this application designs the following Fig. 9 The edge-guided feature enhancement module (EGFE) shown here combines edge-sensitive contextual features and edge maps, and utilizes coarse-grained and fine-grained information to jointly refine the boundary area of ​​the defect.

[0103] Combination Fig. 9 First, the defect edge feature f edge The resolution is adjusted to match the local RGB-D decoded features of the i-th layer The resolution is consistent with that of the local RGB-D decoding feature of the i-th layer is the local RGB-D decoded feature from the previous HFA module. Subsequently, the number of channels is adjusted and the local RGB-D decoded feature of the i-th layer is matched by a multilayer perceptron (MLP). Then perform element-wise multiplication, and we get It can better distinguish the foreground and background and perceive the defect boundary in a coarse-grained manner. This process can be expressed as:

[0104]

[0105] Among them, MLP represents the operation inside the multi-layer perceptron.

[0106] Furthermore, the edge graph P edge Downsample so that the edge map P edge The resolution and The resolution of is consistent with , and element-wise multiplication is performed with it; then, Perform residual connection and refine the features through 3×3 convolution to obtain This is used to sharpen the defect boundary in a fine-grained manner. This process can be expressed as:

[0107]

[0108] Secondly, the channel attention mechanism (Efficient Channel Attention, ECA) is used to further focus on important channel features. Specifically, the global average pooling operation (Global Average Pooling, GAP) is used to Reshape to R C×1×1, use the one-dimensional convolution Conv1d with dynamic convolution kernel size to process it and activate it with Sigmoid function to get the channel weight, and then use the channel weight to Perform element-wise multiplication and weighted sum to get This process can be expressed as:

[0109]

[0110] Among them, Conv1D k represents a one-dimensional convolution with a kernel size of k, and k = |log2(C) / 2+1| odd , |…| odd Indicates taking the nearest odd value, C is The size of the number of channels.

[0111] In this embodiment, EGFE integrates information from local and global features, that is, context-edge-sensitive defect edge features and edge maps are integrated into the RGB-D decoding stream through EGFE to enhance the semantic distinction between foreground and background, thereby improving the perception of defect boundaries while sharpening boundary areas.

[0112] Step S33: Decode to obtain multi-level decoding features, and use an activation function to obtain a defect prediction map.

[0113] Among them, multi-layer RFB is used to decode the features of different modalities to obtain defect prediction maps of different modalities, and the multi-layer defect prediction maps are supervised by hybrid loss. The defect prediction maps of different modalities include depth prediction images, RGB prediction images, and RGB-D prediction images, and each prediction image shows the location of the defect in the corresponding object.

[0114] Combination Figure 2 , use RGB decoder, depth decoder and RGB-D decoder to decode the three modes. Each decoder includes 5 decoding layers, each decoding layer is composed of RFB, and the i-th decoding layer of different modes outputs the i-th RGB decoding feature The i-th deep decoding feature and the i-th RGB-D decoded feature Secondly, for the RGB decoder and the depth decoder, after the last decoding layer of each decoder outputs the corresponding decoding features, it is activated by the Sigmoid function and outputs the corresponding RGB prediction image and depth prediction image. For the RGB-D decoder, after the i-th decoding layer outputs the corresponding decoding features, it is activated by the Sigmoid function and outputs the corresponding i-th layer RGB-D prediction image.

[0115] like Fig.10As shown, the first and second columns of images are the original RGB images and depth images, respectively; the third to fifth columns are the depth prediction images, RGB prediction images, and RGB-D prediction images obtained by decoding, respectively; and the sixth column is the true value image, wherein the RGB-D prediction image is the prediction image output by the last decoding layer, and the true value image is the image that accurately displays the defect area. By comparison, it can be seen that the defect area displayed by the RGB-D prediction image is more consistent with the defect area in the true value image. Therefore, when the defect detection method based on multimodal vision of the present invention is used to detect the defects of an object, the position of the defect can be accurately located by obtaining the RGB-D prediction image.

[0116] In this embodiment, the decoder enhances the perception and integration capabilities of information of different scales while retaining the original features through the RFB module, and in order to keep the feature map size aligned, the features are sampled through bilinear interpolation during the decoding process.

[0117] Furthermore, in step S33, a hybrid loss function is used:

[0118] L total =L BCE +L IOU +L Dice ,

[0119] Among them, including the BCE loss function L BCE , IOU loss function L IOU and Dice loss function L Dice ;

[0120] The BCE loss function is:

[0121]

[0122] The IOU loss function is:

[0123]

[0124] The Dice loss function is:

[0125] L Dice = l Dice (P edge ,G edge ),

[0126] Among them, l BCE represents BCE loss, l IOU represents the IOU loss, represents the RGB-D predicted image of the i-th layer, P depth represents the depth prediction image, P rgb represents the RGB predicted image, G represents the true value image, lDice represents the Dice loss, P edge The edge graph G represents the edge features of the defect. edge Represents the edge truth image.

[0127] In this embodiment, on the one hand, the BCE loss and IoU loss are combined to supervise the defect prediction graph of the network. Defect prediction consists of three parts: P rgb and P depth (obtained from the final decoded layers of the RGB and depth modalities, respectively), and i = 1, 2, 3, 4, 5 (obtained from multiple RGB-D decoding layers). This approach can adopt a deep supervision strategy in the RGB-D modality, which can minimize information loss and promote the propagation of RGB-D modality gradients. On the other hand, Dice loss is used to supervise the predicted edge map to provide excellent noise resistance and adaptability to class imbalance.

[0128] Based on the same inventive concept, the present invention also provides a defect detection system based on multimodal vision, combined with Figure 2 and Fig.11 As shown, the system includes a depth-mining network with edge-sensative context (DMENet), which includes:

[0129] The feature extraction module 100 is used to extract features from the RGB image and the depth image respectively to obtain multi-level RGB coding features and multi-level depth coding features.

[0130] The feature fusion module 200 is used to perform cross-modal feature fusion on multi-level RGB coding features and multi-level depth coding features to obtain multi-level RGB-D coding features and generate an edge map containing defect edge features.

[0131] The decoding module 300 is used to decode according to the multi-level RGB-D coding features and the defect edge features to obtain a defect prediction map.

[0132] Furthermore, the system also includes a preprocessing module, which is used to preprocess the RGB image and the depth image, reshape the RGB image and the depth image, and make the RGB image and the depth image have the same size.

[0133] Furthermore, if Fig.12 As shown, the feature fusion module 200 includes:

[0134] The deep mining feature fusion module 201 is used to perform cross-modal feature fusion on multi-level RGB coding features and multi-level depth coding features to obtain multi-level RGB-D coding features.

[0135] The context-aware edge detection module 202 is used to perform cross-modal feature fusion on the multi-level RGB-D coding features with the multi-level RGB coding features and the multi-level depth coding features, respectively, to generate an edge map containing defect edge features.

[0136] Furthermore, if Fig.13 As shown, the decoding module 300 includes:

[0137] The hierarchical feature aggregation module 301 is used to aggregate features of different levels and modes to emphasize defect areas.

[0138] The edge-guided feature enhancement module 302 is used to perceive and sharpen defect boundaries using defect edge features.

[0139] The prediction module 303 is used for decoding to obtain multi-level decoding features and using an activation function to obtain a defect prediction map.

[0140] In the present application, other technical features in the above-mentioned defect detection system based on multimodal vision are the same as the features disclosed in the above-mentioned method embodiment, and will not be repeated here.

[0141] It can be seen that the present invention discloses a defect detection method and system based on multimodal vision, and the method includes the following steps: extracting features from RGB images and depth images respectively to obtain multi-level RGB coding features and multi-level depth coding features; cross-modal feature fusion of multi-level RGB coding features and multi-level depth coding features to obtain multi-level RGB-D coding features, and simultaneously generate an edge map containing defect edge features; decoding according to the multi-level RGB-D coding features and defect edge features to obtain a defect prediction map. The defect detection method of the present invention can mine and utilize depth information that can semantically match RGB features, effectively integrate RGB and depth information to achieve deeper feature fusion and enhance the representation of defect features; at the same time, it combines coarse-grained and fine-grained edge information for defect boundary refinement to achieve more accurate defect area positioning.

[0142] The above are only embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Any equivalent structural transformations made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are included in the protection scope of the present invention.

Claims

1. A defect detection method based on multimodal vision, characterized in that: The method comprises the following steps: Perform feature extraction on RGB images and depth images respectively to obtain multi-level RGB coding features and multi-level depth coding features; Performing cross-modal feature fusion on the multi-level RGB coding features and the multi-level depth coding features to obtain multi-level RGB-D coding features, and generating an edge map containing defect edge features; Decoding is performed according to the multi-level RGB-D coding features and the defect edge features to obtain a defect prediction map.

2. The defect detection method according to claim 1, characterized in that: Before extracting features from the RGB image and the depth image to obtain multi-level RGB coding features and multi-level depth coding features, the method further includes the following steps: The RGB image and the depth image are preprocessed, and the RGB image and the depth image are reshaped so that the sizes of the RGB image and the depth image are the same.

3. The defect detection method according to claim 2, characterized in that: The method of extracting features from the RGB image and the depth image respectively to obtain multi-level RGB coding features and multi-level depth coding features comprises the following steps: Extract five layers of features from the RGB image and the depth image respectively to obtain the i-th layer of RGB coding features and the i-th layer deep encoding feature Among them, when i=1, the i-th layer RGB encoding feature and the i-th layer deep encoding feature The size of is 1 / 4 of the size of the RGB image and the depth image; when i=2,3,4, the i-th layer RGB encoding feature and the i-th layer deep encoding feature The size of is the RGB encoding feature of the (i-1) layer and 1 / 2 of the size of the (i-1)th layer depth encoding feature; when i=5, the i-th layer RGB encoding feature and the i-th layer deep encoding feature The size of the feature vector is the same as the RGB encoding feature vector of the (i-1)th layer. and the (i-1)th layer deep encoding features The same size.

4. The defect detection method according to claim 1, characterized in that: The step of generating an edge map containing defect edge features comprises the following steps: The multi-level RGB-D coding features are cross-modally fused with the multi-level RGB coding features and the multi-level depth coding features to generate an edge map containing defect edge features.

5. The defect detection method according to claim 1, characterized in that: The method of decoding the multi-level RGB-D coding features and the defect edge features to obtain a defect prediction map comprises the following steps: Aggregate the coded features of different levels and modalities to emphasize defective areas; Using the defect edge features, the defect boundary is sensed and sharpened; The multi-level decoding features are obtained by decoding, and the activation function is used to obtain the defect prediction map.

6. The defect detection method according to claim 1, characterized in that: When decoding according to the defect edge features to obtain the defect prediction map, a hybrid loss function is used: L total =L BCE +L IOU +L Dice , Among them, including the BCE loss function L BCE , IOU loss function L IOU and Dice loss function L Dice ; The BCE loss function is: The IOU loss function is: The Dice loss function is: L Dice =l Dice (P edge ,G edge ), Among them, l BCE represents BCE loss, l IOU represents the IOU loss, represents the RGB-D predicted image of the i-th layer, P depth represents the depth prediction image, P rgb represents the RGB predicted image, G represents the true value image, l Dice represents the Dice loss, P edge The edge graph G represents the edge features containing defects. edge Represents the edge truth image.

7. A defect detection system based on multimodal vision, characterized in that: The system includes a deep information mining and edge perception network, and the deep information mining and edge perception network includes: A feature extraction module is used to extract features from RGB images and depth images respectively to obtain multi-level RGB coding features and multi-level depth coding features; A feature fusion module is used to perform cross-modal feature fusion on the multi-level RGB coding features and the multi-level depth coding features to obtain multi-level RGB-D coding features and generate an edge map containing defect edge features; A decoding module is used to decode according to the multi-level RGB-D coding features and the defect edge features to obtain a defect prediction map.

8. The defect detection system according to claim 7, characterized in that: The system also includes a preprocessing module, which is used to preprocess the RGB image and the depth image, and reshape the RGB image and the depth image so that the sizes of the RGB image and the depth image are the same.

9. The defect detection system according to claim 7, characterized in that: The feature fusion module includes: A deep mining feature fusion module is used to perform cross-modal feature fusion on the multi-level RGB coding features and the multi-level depth coding features to obtain multi-level RGB-D coding features; The context-aware edge detection module is used to perform cross-modal feature fusion on the multi-level RGB-D coding features with the multi-level RGB coding features and the multi-level depth coding features, respectively, to generate an edge map containing defect edge features.

10. The defect detection system according to claim 7, characterized in that: The decoding module comprises: Hierarchical feature aggregation module, used to aggregate features of different levels and modalities to emphasize defect areas; An edge-guided feature enhancement module, used to perceive and sharpen defect boundaries using the defect edge features; The prediction module is used to decode to obtain multi-level decoding features and use the activation function to obtain the defect prediction map.

Citation Information

Patent Citations

  • RGB-D image-based CLANet steel rail surface defect detection system and method

    CN114170174A

  • Salient target detection method and device

    CN116310394A

  • RGB-d multi-mode fusion personnel detection method based on asymmetric double-stream network

    WO2021088300A1

Cited By

  • Defect detection method and system for intelligent robot with body based on multi-modal vision

    CN121921321A