A spaceflight material crack image detection method based on CA-MLL two-stage reasoning

Abstract

The application discloses a kind of based on CA-MLL two segment reasoning aerospace material crack image detection method, comprising:1, using multi-scale adaptive feature extraction network to the image is handled, and the visual feature map of different scales is extracted;2, based on the visual feature map, generate multiple candidate defect regions with visual features;3, the crack data text prompt word is converted into text feature vector, the visual feature of candidate defect region is aligned with text feature vector across modal characteristics, and the semantic similarity of each candidate defect region and text prompt word is calculated;4, the defect region visual feature corresponding to the highest semantic similarity is input into multi-modal decision model, and the final crack category of candidate defect region is determined by multi-modal decision model, and crack classification result is output.The application improves the precision and accuracy of aerospace material crack detection, overcomes the characteristics that labeling is time-consuming and laborious in traditional defect detection, and model generalization ability is poor.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of intelligent manufacturing and industrial quality inspection, and particularly relates to a space material crack image detection method based on CA-MLL two-stage reasoning, which is a space material crack level detection method and is suitable for space material crack defect detection. BACKGROUND

[0002] With the development of neural network technology, the detection of industrial material defects is usually discriminated by using a neural network model. The traditional method is to train a neural network model by using known defect features to form a discrimination model, and then to extract features of industrial material defects, input the extracted features into the model, and realize the discrimination of defects. However, a large amount of training data is required in the training process. It is almost impossible to obtain a large amount of training data of different defects for space materials. Defect samples are difficult to obtain (cracks are difficult to produce in experiments), which leads to difficulty in training. The accuracy and precision of detection will be obviously affected by training with a small amount of defect data, especially for crack level discrimination. Therefore, how to use a neural network model to accurately judge the crack defects of space materials is a research topic at present. SUMMARY

[0003] In view of the above problems, the application provides a space material crack image detection method based on CA-MLL two-stage reasoning, which is a space material crack level detection method and is suitable for space material crack defect detection.

[0004] To achieve the above purpose, the technical scheme adopted by the application is as follows:

[0005] A space material crack level detection method based on an adaptive multi-scale architecture, comprising constructing crack data text prompt words, obtaining an image of a space material to be detected, and the method comprising:

[0006] Step 1, using a multi-scale adaptive feature extraction network to process the image to extract visual feature maps of different scales;

[0007] Step 2, based on the visual feature maps, generating a plurality of candidate defect regions with visual features;

[0008] Step 3, converting the crack data text prompt words into a text feature vector, and performing cross-modal feature alignment between the visual features of the candidate defect regions and the text feature vector, and calculating the semantic similarity between each candidate defect region and the text prompt words;

[0009] Step 4, inputting the defect region visual features corresponding to the highest semantic similarity into a multi-modal decision model, and the multi-modal decision model determines the final crack category of the candidate defect region and outputs a crack classification result.

[0010] The crack construction data text prompt word is constructed according to known cracks, including crack shape features and corresponding image information.

[0011] The multi-modal decision model is a multi-modal decision model for crack category classification formed by cross-modal feature alignment training of known crack data text and crack image data.

[0012] The scheme further comprises: the multi-scale adaptive feature extraction network of step 1 is a multi-scale adaptive feature extraction network based on an adaptive multi-scale Swin-Transformer architecture, and the different scale visual feature maps are extracted by inputting the image data to be detected into the multi-scale adaptive feature extraction network.

[0013] The scheme further comprises: the formula for calculating the semantic similarity is:

[0014] ;

[0015] Wherein: A is the visual feature of the candidate defect region; B is the text prompt word.

[0016] The scheme further comprises: the cross-modal feature alignment training of the multi-modal decision model of step 3 is to map the crack image data and the crack data text vector in the same dimension to form the feature alignment of the two different modalities, and calculate the loss function in the same space, so that similar semantics are close in the vector feature space, and dissimilar semantics are far apart in the vector feature space. Minimize the loss to obtain the optimal performance of the multi-modal decision model, and the loss calculation formula is:

[0017] ;

[0018] Wherein:

[0019] is the embedding of the image, is the embedding of the text, is the one-hot label of the connector sub-class, and C represents the number of categories, is a temperature parameter, represents the loss between the image and the text, and cos() is the cosine similarity.

[0020] The scheme further comprises: the crack data text prompt word is converted into a text feature vector by using a BERT model.

[0021] The scheme further comprises: the crack data text prompt word is constructed through a three-layer guiding architecture, and the three-layer guiding architecture comprises a material cognition layer, a feature discrimination cognition layer and a labeling execution layer; the material cognition layer is a professional classification of aerospace materials; the feature discrimination cognition layer is a discrimination rule of crack and non-crack features; the professional classification and the discrimination rule form a shape feature of the crack; and the labeling execution layer is image information labeling of the crack defect.

[0022] The scheme further comprises: the professional classification is a crack classification attribute proposed by an expert according to professional knowledge, and the crack classification attribute comprises fatigue cracks, stress corrosion cracks, thermal cracks, quenching cracks and hydrogen-induced cracks.

[0023] The discrimination rule of the crack and the non-crack features comprises scratches, machining textures, pollutants / deposits, reflections / optical artifacts and material organization features.

[0024] The image information labeling of the crack defect comprises boundary box labeling and attribute labeling.

[0025] The specification of the boundary box labeling execution comprises a minimum circumscribed rectangle closely fitted to the crack, a main body and branches containing the crack, a 1-2 pixel margin left on the edge of the boundary box and a coordinate format of the upper left corner and the lower right corner.

[0026] The specification of the attribute labeling execution comprises a crack category, a crack length, a crack width and a crack direction.

[0027] The scheme further comprises: the crack category comprises fatigue cracks, stress corrosion cracks, thermal cracks, quenching cracks, hydrogen-induced cracks and cracks of unknown types.

[0028] The method has the beneficial effects that the precision and accuracy of the aerospace material crack detection are improved, the work of collecting a large number of training samples for training in a traditional neural network judgment module is saved, and the characteristics of time-consuming and laborious labeling and poor model generalization ability in traditional defect detection are effectively overcome.

[0029] The application will be further described in detail in combination with the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0030] Figure 1 The figure is a flowchart of the method. DETAILED DESCRIPTION

[0031] The application discloses a CA-MLL two-stage reasoning-based aerospace material crack image detection method, which is a kind of aerospace material crack level detection method, comprising constructing a crack data text prompt word, obtaining an image of a detected aerospace material, and the method comprises:

[0032] Step 1, using a multi-scale adaptive feature extraction network to process the image, and extracting visual feature maps of different scales;

[0033] Step 2, based on the visual feature map, a plurality of candidate defect regions with visual features are generated;

[0034] Step 3, the crack data text prompt word is converted into a text feature vector, and the visual features of the candidate defect regions are aligned with the text feature vector in a cross-modal feature, and the semantic similarity of each candidate defect region and the text prompt word is calculated, and the calculation formula of the semantic similarity is:

[0035] ;

[0036] Wherein: A is the visual feature of the candidate defect region; B is the text prompt word;

[0037] The crack data text prompt word is converted into a text feature vector by using a BERT model;

[0038] Step 4, input the defect region visual feature corresponding to the highest semantic similarity into a multi-modal decision model, and the multi-modal decision model determines the final crack category of the candidate defect region and outputs the crack classification result;

[0039] The crack data text prompt word is constructed according to known cracks, including crack shape features and corresponding image information;

[0040] The multi-modal decision model is a multi-modal decision model for crack category classification trained by cross-modal feature alignment of known crack data text and crack image data.

[0041] Wherein: the multi-scale adaptive feature extraction network of step 1 is a multi-scale adaptive feature extraction network based on adaptive multi-scale Swin-Transformer architecture, and the processing of the image and the extraction of visual feature maps of different scales are obtained by inputting the image data to be detected into the multi-scale adaptive feature extraction network.

[0042] The adaptive multi-scale Swin-Transformer architecture is a neural network model, which is a widely used feature map acquisition processing model that can generalize to new image distributions and tasks in a zero-shot learning environment. Its structure includes: a Swin-Transformer backbone network, an adaptive multi-scale feature fusion module, and a context information enhancement module. First, input the HxW image into the feature extraction network, and the Swin Transformer outputs four-stage feature maps with a length and width of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original, and a channel number of C, 2C, 4C, and 8C, respectively. This network limits the calculation range of attention to non-overlapping local windows, saving computing resources while weakening the global perception ability of ViT. The context information enhancement module is combined with the Swin-Transformer, which uses dense connected dilated convolution to provide a larger receptive field on the feature map output in the last stage. The dilation rates are 2, 4, 8, and 12. When different scale feature maps are fused, after receiving the image data to be detected, first use sub-pixel convolution upsampling to obtain high-quality large-scale images, then combine with the hybrid attention mechanism to realize adaptive feature fusion. The most important module in Swin Transformer is the attention module based on the moving window, which contains a shifted windows multi-head self attention (SW-MSA) and a window multi-head self attention (W-MSA) based on the moving window. The other normalization layer and two layers of MLP remain the same as the original, and the GELU activation function is used.

[0043] The adaptive multi-scale feature fusion module mainly includes two parts: the SENet channel attention module and the CSAM spatial attention module. After the high-level feature map is upsampled by sub-pixel convolution, the channel attention mechanism is used to adjust the channel weight. The channel corresponding to the foreground response should obtain a larger weight, and correspondingly, the channel corresponding to the background interference information response should obtain a smaller weight. The channel attention part uses SE-Net, Fsq(·) as the "compression" step, uses global pooling to compress each channel into a constant, Fex(·, W) as the "excitation" step, uses a fully connected layer and an activation function to learn the weight of each channel, and finally uses F scale(·,·) applies weights to the original feature map to obtain the high-level feature map after the channel information is reweighted. The CSAM spatial attention module is different from the common spatial attention in that it can capture the mutual relationship in the space of two feature maps. Specifically, first, the high-level feature map and the low-level feature map are connected, and after 1x1, 3x3 two convolutions and softmax, the spatial attention weight matrix with a channel number of 1 belonging to the high level and the low level is obtained. After multiplying the weight matrix with the corresponding level feature map, the finally fused feature map is obtained.

[0044] wherein: the cross-modal feature alignment training of the multi-modal decision model is to map the crack image data and the vector of crack data text in the same dimension to form the feature alignment of two different modalities, calculate the loss function in the same space, so that similar semantics are close in the vector feature space, and dissimilar semantics are far apart in the vector feature space, minimize the loss to obtain the optimal performance of the multi-modal decision model, and the loss calculation formula is:

[0045] ;

[0046] wherein:

[0047] is the embedding of the image, is the embedding of the text, is the one-hot label of the connector subclass, and C represents the number of categories, is a temperature parameter, represents the loss between the image and the text, and cos() is the cosine similarity.

[0048] wherein: the feature alignment of two different modalities is to project the visual features of the candidate region and the text description (such as “irregular cracks on the metal surface”) to a shared latent space by using a multi-modal large model. The visual-text alignment is optimized by a dynamic weighted cross-modal contrastive loss function, wherein vi is a visual feature vector, ti is a text embedding, and λ is a task weight coefficient;

[0049] The visual feature vector (vi, output by the feature branch of the CA-MLL detector) of the crack candidate region is extracted, and the preset crack text description (such as “irregular cracks on the metal surface” and “horizontal cracks on the concrete surface”) is generated into a text embedding vector (ti) by a text encoder of a multi-modal large model; vi and ti are projected to the same dimension of the shared latent space to realize the dimension alignment of the visual features and the semantic information.

[0050] In the embodiment, the crack data text prompt word is implemented through a three-layer guiding architecture, and the three-layer guiding architecture is a material cognition layer, a feature discrimination cognition layer and a labeling execution layer.

[0051] The professional classification is a crack classification attribute proposed by an expert according to professional knowledge, and the crack classification attribute includes fatigue cracks, stress corrosion cracks, thermal cracks, quenching cracks and hydrogen-induced cracks.

[0052] The content contained is:

[0053] The fatigue crack is gradually expanded under the action of cyclic load and presents fatigue striations.

[0054] The stress corrosion crack is formed under the joint action of stress and corrosion environment, and the crack is thin and deep.

[0055] The thermal crack is generated in a thermal cycle or a welding process and is mostly present in a weld seam area.

[0056] The quenching crack is formed in a heat treatment process and is mostly linear.

[0057] The hydrogen-induced crack is caused by hydrogen embrittlement and is in a net shape or a dendritic shape.

[0058] The discrimination rule of the crack and the non-crack feature includes scratches, machining textures, pollutants / deposits, reflections / optical artifacts and material organization features.

[0059] The content contained is:

[0060] The scratch includes:

[0061] The scratch is a surface shallow damage without obvious depth.

[0062] The scratch has a blunt edge and appears as abrasion.

[0063] The scratch is along a single direction without branches.

[0064] The scratch has a smooth bottom and visible metal luster.

[0065] The key to discrimination is that the scratch has no depth shadow and the edge is not sharp.

[0066] The machining texture includes:

[0067] The machining texture is regularly arranged lines (parallel or concentric).

[0068] The machining texture has uniform depth and width.

[0069] The machining texture covers a large area.

[0070] Good texture continuity, no mutation;

[0071] Discrimination key: strong regularity, non-random distribution;

[0072] The pollutants / deposits:

[0073] Abnormal color but irregular shape;

[0074] Erased or washed away;

[0075] Fuzzy boundary, no sharp edge;

[0076] Discrimination key: attached to the surface, not the material body defect;

[0077] The light reflection / optical artifact:

[0078] Characteristics disappear or move when the observation angle changes;

[0079] Present highlights or strong reflections;

[0080] No fixed geometric shape;

[0081] Discrimination key: non-physical entity defect;

[0082] The material organization features:

[0083] Grain boundaries, phase transformation organization;

[0084] Uniform distribution, no obvious orientation;

[0085] Related to the microstructure of the material;

[0086] Discrimination key: material inherent characteristics, non-damage.

[0087] The image information annotation of the crack defect includes boundary box annotation and attribute annotation;

[0088] The specification of the boundary box annotation includes: the smallest circumscribed rectangle that closely fits the crack, the main body and obvious branches containing the crack, 1-2 pixel margin left on the edge of the boundary box, and the coordinate format is the upper left corner and the lower right corner;

[0089] The specification of the attribute annotation includes: crack category, crack length, crack width, and crack direction.

[0090] The crack category includes: fatigue crack, stress corrosion crack, thermal crack, quenching crack, hydrogen-induced crack, and crack of unknown type.

[0091] The above space material crack image detection method embodiment provides a multi-scale adaptive feature extraction network and a multi-modal decision model as two-stage reasoning of the overall crack detection model. The core goal is to realize efficient detection of the "open crack detection set" (i.e. a non-fixed data set containing crack data of different materials and different environments). The first stage is the "general positioning stage", which is responsible for accurately positioning all potential crack candidate regions from the input image, and does not rely on prior knowledge of specific crack categories. The second stage is the "semantic feature alignment stage", which is responsible for associating the visual features of the candidate regions with the crack-related text descriptions, and realizing accurate crack category determination and attribute (such as severity) evaluation. This architecture breaks through the limitations of traditional single-stage models in weak generalization ability in open scenarios, and can adapt to various crack detection tasks. In the construction of crack data text prompt words and the training of the multi-modal decision model, without the need for large-scale labeled samples, the multi-modal features related to the crack (such as the visual morphological features of the crack and the corresponding text description features) can be learned. Through the learning of a small amount of known labeled samples, the crack regions and attributes (such as crack features associated with materials) in the data to be labeled can be automatically identified, and preliminary labeling results (such as crack bounding boxes and category labels) can be generated. The core advantage is to greatly reduce the dependence on the number of labeled samples, and to improve the accuracy of labeling by combining multi-modal information (such as constraining the semantic consistency of visual labeling through text description), significantly improving the labeling efficiency, the precision and accuracy of space material crack detection, saving the work of collecting a large number of training samples required for training in the traditional neural network judgment module, and effectively overcoming the characteristics of time-consuming and laborious labeling and poor model generalization in traditional defect detection.

Claims

1. A spaceflight material crack image detection method based on CA-MLL two-stage reasoning, comprising constructing a crack data text prompt word and obtaining an image of a spaceflight material to be detected, characterized in that, The two reasoning stages: the first stage is a "general positioning stage", which is responsible for positioning all potential crack candidate regions from the input image, and does not rely on prior knowledge of specific crack categories; the second stage is a "semantic feature alignment stage", which is responsible for associating the visual features of the candidate regions with the crack-related text descriptions, and realizing the determination and attribute evaluation of the crack categories; the method comprises: Step 1, using a multi-scale adaptive feature extraction network to process the image, and extracting visual feature maps of different scales; Step 2, based on the visual feature map, a plurality of candidate defect regions with visual features are generated; Step 3, the crack data text prompt words are converted into text feature vectors, and the visual features of the candidate defect regions are aligned with the text feature vectors in a cross-modal manner, and the semantic similarity of each candidate defect region and the text prompt words is calculated; Step 4, input the defect region visual feature corresponding to the highest semantic similarity into a multi-modal decision model, and the multi-modal decision model determines the final crack category of the candidate defect region and outputs the crack classification result; The crack data text prompt words are constructed according to known cracks, including crack shape features and corresponding image information; The construction of the crack data text prompt words is realized through a three-layer guiding architecture, and the three-layer guiding architecture includes a material cognition layer, a feature discrimination cognition layer and a labeling execution layer, wherein: the material cognition layer is a professional classification of aerospace materials; the feature discrimination cognition layer is a discrimination rule for crack and non-crack features; the professional classification and the discrimination rule form the shape feature of the crack, and the labeling execution layer is image information labeling of the crack defect; The professional classification is a crack classification attribute proposed by experts according to professional knowledge, including: fatigue crack, stress corrosion crack, thermal crack, quenching crack and hydrogen-induced crack; The discrimination rule for crack and non-crack features includes: scratch, machining texture, contaminant / deposit, reflection / optical artifact and material organization feature; The image information labeling of the crack defect includes boundary box labeling and attribute labeling; The specifications of the boundary box labeling execution include: the smallest circumscribed rectangle closely fitted to the crack, the main body and branch containing the crack, the edge of the boundary box leaving a 1-2 pixel margin, and the coordinate format being the upper left corner and the lower right corner; The specifications of the attribute labeling execution include: crack category, crack length, crack width and crack direction; The multi-modal decision model is a multi-modal decision model for crack category classification trained by cross-modal feature alignment of known crack data text and crack image data; The cross-modal feature alignment training of the multi-modal decision model is to map the crack image data and the vector of the crack data text in the same dimension to form the alignment of the features of the two different modalities, calculate the loss function in the same space, so that similar semantics are close in the vector feature space, and dissimilar semantics are far apart in the vector feature space, minimize the loss to obtain the optimal performance of the multi-modal decision model, and the loss calculation formula is: ； wherein: is the embedding for the image, is the embedding for the text, is the one-hot label for the connector sub-class, C represents the number of classes, is the temperature parameter, represents the loss between image and text, cos() is the cosine similarity.

2. The detection method according to claim 1, characterized in that, The multi-scale adaptive feature extraction network in step 1 is a multi-scale adaptive feature extraction network based on an adaptive multi-scale Swin-Transformer architecture, and the processing of the image and the extraction of visual feature maps of different scales are obtained by inputting the image data to be detected into the multi-scale adaptive feature extraction network for feature extraction.

3. The method of claim 1, wherein The formula for calculating the semantic similarity is: ； Wherein: A is the visual feature of the candidate defect region; B is the text prompt word.

4. The method of claim 1, wherein The conversion of the crack data text prompt word into a text feature vector is by using a BERT model.

5. The method of claim 1, wherein The crack category includes fatigue cracks, stress corrosion cracks, thermal cracks, quenching cracks, hydrogen-induced cracks and cracks of unknown types.

Images