Steel surface damage identification method based on binocular machine vision

Abstract

The invention relates to a steel surface damage identification method based on binocular machine vision, and the method comprises the steps: constructing a binocular collection system, employing a visible light camera and an infrared camera to obtain a visible light image and an infrared image at the same time, heating a collection point through a thermal excitation source, and carrying out the recognition of the steel surface damage according to a temperature difference generated by the specific heat difference between a steel surface defect and an intact part. The method comprises the following steps: introducing an infrared camera to shoot the temperature distribution condition, combining with a visual image collected by a traditional visible light camera, then carrying out pre-processing such as alignment cutting on the two images, finally establishing a damage identification model by using a deep learning method, and fusing the characteristics of a corresponding damage part between the visible light image and the infrared image. Compared with a traditional single visible light image flaw detection method, additional temperature information is introduced, the advantages of two shooting types of a visible light camera and an infrared camera are combined, and the adaptability and robustness to environmental changes are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of steel structure damage identification, and in particular to a steel surface damage identification method using binocular machine vision. Background Art

[0002] Surface damage to steel structures has a significant impact on the overall performance and safe operation of cranes. Surface damage to steel structures, such as rust, can lead to uniform thinning of the steel structure and the creation of large rust pits in certain areas, which can easily cause stress concentration and may also lead to early damage to the local structure.

[0003] In recent years, the apparent damage method based on visible light machine vision of traditional cameras has been widely used for apparent damage detection of steel structures. Using image processing methods, it can accurately identify minor damage on the surface of steel structures, such as cracks, rust and deformation. Combined with technologies such as drones, it can realize the automatic collection of apparent damage images, avoiding direct contact between workers and the inspected structures, effectively avoiding errors and damage caused by human factors, and reducing the tediousness and labor intensity of manual operations.

[0004] However, visible light machine vision inspection methods also have the following disadvantages:

[0005] 1) Machine vision systems rely heavily on image quality, which can be affected by environmental factors such as lighting, temperature, and humidity. For example, insufficient or uneven lighting can result in blurred images or unclear features, thus affecting the accuracy of damage detection.

[0006] 2) Steel surfaces are usually coated. Different coatings have different reflection and scattering properties, resulting in different performance during detection. Therefore, conventional image processing technology may have difficulty in effectively distinguishing damage from images.

[0007] 3) For some complex damage forms, such as deep damage and coexistence of multiple types of damage, traditional visible light machine vision detection methods may be difficult to accurately identify and classify.

[0008] Based on this, it is necessary to study a steel surface damage recognition method based on binocular machine vision. Summary of the Invention

[0009] In view of this, the purpose of the present invention is to provide a steel surface damage identification method based on binocular machine vision, which can effectively solve the problems of unstable image quality and difficulty in accurately identifying various complex damages caused by visible light machine vision methods based on traditional cameras under the influence of complex environmental conditions and material properties.

[0010] To achieve the above object, the technical solution adopted by the present invention is:

[0011] A method for identifying steel surface damage using binocular machine vision comprises the following steps:

[0012] S100: Build binocular acquisition system and image processing system;

[0013] The acquisition system includes a mobile platform, a visible light camera, an infrared camera, and a thermal excitation source. The path and acquisition points of the mobile platform are set, and the visible light camera, infrared camera, and thermal excitation source are mounted on the mobile platform.

[0014] The image processing system includes a feature matching module and a damage recognition model;

[0015] S200: image acquisition;

[0016] The mobile platform moves to the collection point, where the thermal excitation source heats the collection point. The visible light camera and infrared camera capture images of the heated steel surface. Each capture simultaneously acquires visible light and infrared images.

[0017] S300: image preprocessing;

[0018] The feature matching module extracts key feature points from the corresponding visible light image and infrared image, matches the key feature points in the two images, and identifies and crops the overlapping areas in the two images;

[0019] S400: Establish a damage identification model and identify damage categories;

[0020] The damage identification model includes input layer, feature identification layer, feature fusion layer and classification layer;

[0021] Input the key feature points of the overlapping areas in the visible light image and the infrared image from the input layer;

[0022] The feature recognition layer extracts the texture and edge feature vectors of the damaged area on the steel surface from the key feature points of the visible light image.

[0023] The feature recognition layer extracts the feature vector of the damaged area on the steel surface from the key feature points of the visible light image by analyzing the texture and edge of the steel surface.

[0024] The feature recognition layer analyzes the temperature distribution on the steel surface and extracts the feature vector of the damaged area on the steel surface from the key feature points of the infrared image. The classification layer calculates the probability of each damage type based on the comprehensive feature vector and outputs the damage type with the highest probability.

[0025] S500: Verify the recognition result;

[0026] The output results are manually verified against the actual situation, and the verification data is recorded and uploaded to the image processing system to optimize the damage identification model.

[0027] Furthermore, the acquisition system also includes a standard heat source, which is set in the shooting range of the infrared camera, and the temperature of the standard heat source is recorded in the image processing system.

[0028] Furthermore, the method further includes step S150: configuring an infrared camera;

[0029] Set the emissivity of the steel surface to be measured in the infrared camera;

[0030] Adjust the distance and tilt angle between the visible light camera and the infrared camera to ensure that the overlap rate of the shooting fields of the visible light camera and the infrared camera exceeds 90%.

[0031] Furthermore, in step S200, the thermal excitation source continuously heats the acquisition point, and the visible light camera and the infrared camera capture images of the steel surface at different heating times, thereby obtaining multiple sets of images of the same acquisition point at different temperatures;

[0032] In step S400, multiple sets of images at the same acquisition point at different temperatures are input into the damage recognition model together, and multiple sets of comprehensive feature vectors are obtained through the feature fusion layer;

[0033] Multiple sets of feature vectors are fused into a final feature vector, so that the classification layer can determine the damage type based on the final feature vector.

[0034] Furthermore, the damage identification model is trained by the following steps:

[0035] Repeat steps S100-S200 to photograph steel materials with different types of damage under different lighting conditions and temperature conditions;

[0036] Steps S300 - S500 are performed on each set of images until the error of the output result is reduced to within the allowable range.

[0037] Furthermore, in step S300, a pixel value probability distribution map describing the pixel value distribution of each cropped image is generated;

[0038] In step S400, the pixel value probability distribution map is input into the input layer and used to fuse the feature vectors of the damaged areas of the two images.

[0039] Furthermore, the feature fusion layer obtains a comprehensive feature vector by fusing the following steps:

[0040] S401: feature transformation;

[0041] Perform a linear transformation on each eigenvector to ensure that they are in the same feature space;

[0042] Feature transformation formula:

[0043] F′ v =W v F v +b v ;

[0044] F′ ir =W ir F ir +b ir ;

[0045] Among them, F v is the feature vector extracted from the visible light image;

[0046] F ir is the feature vector extracted from the infrared image;

[0047] W v and W ir is the transformation matrix; b v and b ir is the bias vector;

[0048] S402: Setting up self-attention mechanism;

[0049] The self-attention mechanism is used to calculate the attention weight between two feature vectors to enhance the expression of important features:

[0050] Self-attention mechanism formula:

[0051] Where: Q=K=V=[F v ',F ir '], Q is the query matrix; K is the key matrix; V is the value matrix;

[0052] d k is the characteristic dimension;

[0053] The softmax function is used to calculate the attention weight;

[0054] d k is the characteristic dimension;

[0055] S403: Feature weighting;

[0056] The feature vectors are weighted and combined using the weights obtained through the self-attention mechanism;

[0057] Weighted combination formula: F att =Attention(F′ v ,F′ ir );

[0058] Among them F att is the weighted eigenvector;

[0059] S404: Fusion features;

[0060] The weighted feature vector is fused with the original transformed feature vector to form a comprehensive feature vector. The fused comprehensive feature vector is linearly transformed to further compress and integrate feature information.

[0061] Fusion feature formula:

[0062] F fused =concat(F att ,F′ v ,F′ ir );

[0063] F fused =W f F fused +b f ;

[0064] Among them, F fused is the comprehensive eigenvector, W f is the weight matrix of the fusion layer, b f is the bias vector.

[0065] The beneficial effects of the above technical solution are:

[0066] (1) The present invention constructs a binocular acquisition system, uses a visible light camera and an infrared camera to simultaneously acquire visible light images and infrared images, heats the acquisition point through a thermal excitation source, and introduces an infrared camera to capture the temperature distribution of the steel surface defects and the intact parts based on the temperature difference. The intuitive images acquired by the traditional visible light camera are combined, and the two images are pre-processed by alignment and cropping. Finally, a deep learning method is used to establish a damage recognition model, which integrates the features of the corresponding damage parts between the visible light image and the infrared image, thereby outputting an accurate apparent damage type and achieving the recognition of steel surface damage. Compared with the traditional single visible light image flaw detection method, this method introduces additional temperature information and does not rely solely on the visible light image. For example, it combines the advantages of visible light cameras and infrared cameras, increases adaptability and robustness to environmental changes, and can effectively solve the problems of unstable image quality caused by complex environmental conditions and material properties and difficulty in accurately identifying various complex damages. It can identify defect types more accurately and efficiently, and reduce the false detection rate caused by a single camera type, especially for some complex damage forms such as deep damage, coexistence of multiple types of damage, and subtle and potential damage. It can effectively monitor and evaluate the health status and potential problems of steel structures, provide more comprehensive and accurate steel structure service performance evaluation services, and provide important basis for formulating reasonable maintenance and overhaul plans, thereby reducing the maintenance cost and risk of steel structures.

[0067] (2) The present invention utilizes a feature recognition layer and a feature fusion layer to deeply fuse the features of the visible light image and the infrared image to generate a comprehensive feature vector. The classification layer calculates the probability of the damage type based on the comprehensive feature vector, thereby improving the accuracy and reliability of the classification. Unlike the simple superposition of two camera mechanisms, the present invention significantly improves the robustness and accuracy of damage recognition, thereby forming a more comprehensive and efficient steel surface damage recognition system. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 It is a functional diagram of the acquisition system and image processing system;

[0069] Figure 2 This is the flow chart for image feature extraction and fusion;

[0070] Figure 3 Another embodiment of image feature extraction and fusion;

[0071] Figure 4 This is another implementation method for extracting and fusing image features. DETAILED DESCRIPTION

[0072] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0073] Example 1. This example aims to provide a method for identifying steel surface damage using binocular machine vision, which is mainly used for surface inspection of crane steel structures. It addresses the problems of unstable image quality and difficulty in accurately identifying diverse and complex damage caused by visible light machine vision methods based on traditional cameras under the influence of complex environmental conditions and material properties.

[0074] A method for identifying steel surface damage using binocular machine vision comprises the following steps:

[0075] S100: Build binocular acquisition system and image processing system;

[0076] like Figure 1The acquisition system includes a mobile platform, a visible light camera, an infrared camera, and a thermal excitation source. The mobile platform can be a drone or a contact robot. The mobile platform is equipped with a visible light camera, an infrared camera, and a thermal excitation source. The visible light camera is a traditional camera, mainly used to capture intuitive color images. The infrared camera obtains temperature distribution images by capturing infrared radiation emitted by steel. The thermal excitation source is a controllable laser heater, hot air heater, etc., which is mainly used to heat the acquisition point. The surface temperature difference formed by the difference in specific heat capacity between the damaged area and the intact area is displayed on the infrared image through the infrared camera, thereby identifying the damaged area. The acquisition point of the mobile platform needs to be set manually in advance. According to the surface shape and physical distance parameters of the steel structure, the optimal viewpoint is designed, and the detection trajectory is accurately pre-planned. The acquisition system will go to shoot. How to control the mobile platform to go to the acquisition point and how to control peripherals such as cameras to perform image capture are existing technologies and are widely used in this field. They will not be repeated here.

[0077] The carrier of the image processing system is a computer, which is mainly used to receive and process the image data input by the acquisition system. It includes a feature matching module and a damage recognition model. The feature matching module is mainly used to pre-process visible light images and damage recognition images, while the damage recognition model uses deep learning to determine the damage type.

[0078] S200: image acquisition;

[0079] The mobile platform moves to the collection point, the thermal excitation source heats the collection point, and the visible light camera and infrared camera capture images of the heated steel surface. Each shot simultaneously acquires visible light and infrared images.

[0080] S300: image preprocessing;

[0081] The captured images are processed for noise, using different noise reduction methods for different types of noise. Contrast enhancement and smoothing filtering are then performed to improve image quality and reduce the impact of interference on subsequent damage identification. Furthermore, the infrared images are processed using temperature calibration and pseudo-color mapping to more intuitively observe temperature distribution and changes. These image preprocessing methods are all existing technologies and will not be detailed here.

[0082] The feature matching module uses a feature detection algorithm to extract key feature points from the corresponding visible light image and infrared image. It then uses a feature matching algorithm to match the key feature points in the two images and spatially align the corresponding visible light image and infrared image, thereby identifying and cropping the overlapping areas in the two images, ensuring that in the subsequent analysis process, the two types of images are processed in exactly the same field of view.

[0083] S400: Establish a damage identification model and identify damage categories;

[0084] The damage recognition model is based on a convolutional neural network model, which includes an input layer, a feature recognition layer, a feature fusion layer, and a classification layer. The key feature points of the overlapping areas of the visible light image and the infrared image obtained in the preprocessing are input into the input layer.

[0085] like Figure 2 The feature recognition layer uses edge detection, texture analysis and other methods to identify the damaged area based on the key feature points of the visible light image and determine the damaged area; at the same time, the feature recognition layer analyzes the temperature distribution on the surface of the steel, calculates the temperature gradient and the temperature difference between each pixel and its surrounding pixels, identifies the area with drastic temperature changes, and thus identifies the damaged area; calculates the spatial relationship between the respective damaged areas of the two, and combines the damage information of the two images to determine the final damaged area, so as to accurately locate the identified damage and obtain the spatial position and size information of the damaged area.

[0086] The feature recognition layer extracts feature vectors of texture and edge located in the final damage area from key feature points of the visible light image, and extracts thermal feature vectors located in the final damage area from key feature points of the infrared image.

[0087] The feature fusion layer finds the correspondence between the features of the damaged areas of the two images, fuses the feature vectors of the damaged areas of the two images and obtains a comprehensive feature vector.

[0088] The classification layer calculates the probability of each damage type based on the comprehensive feature vector and outputs the damage type with the highest probability.

[0089] The feature fusion layer obtains the comprehensive feature vector by fusing the following steps:

[0090] S401: feature transformation;

[0091] Perform a linear transformation on each eigenvector to ensure that they are in the same feature space;

[0092] Feature transformation formula:

[0093] F′ v =W v F v +b v ;

[0094] F′ ir =W ir F ir +b ir ;

[0095] Among them, F v is the feature vector extracted from the visible light image; F iris the feature vector extracted from the infrared image; W v and W ir is the transformation matrix; b v and b ir is the bias vector.

[0096] The transformation matrix and bias vector are parameters determined through model training. During the training process, the model adjusts these parameters based on the training data to minimize the loss function (such as classification error).

[0097] S402: Set up a self-attention mechanism; the purpose of setting up a self-attention mechanism is to dynamically adjust the weights of features by calculating the correlation between them, so that the model can better capture the relationship between features.

[0098] The self-attention mechanism formula is used to calculate the attention weight between two feature vectors to enhance the expression of important features:

[0099] Self-attention mechanism formula:

[0100] Where: Q=K=V=[F v ',F ir '] ， That is, Q, K and V are all composed of F v ' and F ir ', Q is the query matrix, which is used to query the vector set of other features; K is the key matrix, which is used to calculate the vector set similarity with the query feature; V is the value matrix, which is the vector set containing the actual eigenvalues; d k is the feature dimension, used for scaling; the softmax function is used to calculate the attention weight; d k is the characteristic dimension.

[0101] S403: Feature weighting;

[0102] The feature vectors are weighted and combined using the weights obtained through the self-attention mechanism;

[0103] Weighted combination formula: F att =Attention(F′ v ,F′ ir ); where F att is the weighted eigenvector.

[0104] The specific steps for using the attention mechanism for feature weighting and the detailed explanation of the formula are as follows:

[0105] S4031: Calculate similarity score;

[0106] Calculate the similarity score between feature vectors, that is, the dot product between the query feature Q and the key feature K, and then divide it by To zoom: Here QK T Represents the dot product between the query feature and the key feature, resulting in a similarity matrix, divided by This is to prevent the dot product value from being too large, thereby affecting the calculation result of the softmax function.

[0107] S4032: Calculate attention weight;

[0108] Apply the softmax function to the similarity score matrix to obtain the attention weights: weights = softmax(scores); the softmax function converts the similarity scores into a probability distribution so that the sum of all attention weights is 1.

[0109] S4033: weighted summation;

[0110] Use the attention weight to perform weighted summation on the value features to obtain the weighted feature representation: F att =weights·V.

[0111] S404: Fusion features;

[0112] The weighted feature vector is fused with the original transformed feature vector to form a comprehensive feature vector. The fused comprehensive feature vector is further compressed and integrated through linear transformation. The fusion feature formula is:

[0113] F fused =concat(F att ,F′ v ,F′ ir );

[0114] F fused =W f F fused +b f ;

[0115] Among them, F fused is the comprehensive eigenvector, W f is the transformation matrix, b f is the bias vector. In machine learning and deep learning, the concat function refers to the concatenation operation of feature vectors. Through the concatenation operation, feature vectors from different sources or different modalities can be combined into a larger comprehensive feature vector.

[0116] S500: Verify the recognition result;

[0117] Manually verify the output results with the actual situation, record and upload the verification data to the image processing system, optimize the damage recognition model, and maintain continuous learning of the damage recognition model.

[0118] The damage identification model is trained through the following steps:

[0119] Collect steel structures with known damage types and repeat steps S100-S200 to photograph steel with different types of damage under different lighting and temperature conditions to obtain comprehensive image information to cope with the complex inspection environment of crane steel structures;

[0120] Execute steps S300-S500 for each set of images, manually verify the classification results of the neural network with the actual situation, record each result, calculate its error probability, and adjust the model according to the verification results until the output result error is reduced to within the allowable range.

[0121] Example 2: This example is basically the same as Example 1, except that the configuration parameters of the infrared camera are optimized. This example further illustrates the structure.

[0122] In this embodiment, the step S150 is further included: configuring an infrared camera;

[0123] The emissivity of the steel surface to be measured is set in the infrared camera. Emissivity is a measure of the material surface's ability to emit infrared radiation. The emissivity determines how much energy the surface radiates at a given temperature. Different materials and surfaces (such as coatings, rust, etc.) have different emissivities, and the emissivity of steel surfaces is usually low. Setting the emissivity can prevent the actual temperature from being underestimated. The emissivity of normal steel structure surfaces needs to be detected in advance and configured in the infrared camera.

[0124] Adjust the distance and tilt angle between the visible light camera and the infrared camera to ensure that the overlap rate of the shooting fields of the visible light camera and the infrared camera exceeds 90%, so as to facilitate the determination of the overlapping area, improve the accuracy of image alignment and cropping, reduce errors during image fusion, and ensure the accuracy of feature matching and damage identification.

[0125] In this embodiment, the acquisition system also includes a standard heat source, which is a black body with a known temperature. The standard heat source is set in the shooting range of the infrared camera. The temperature of the standard heat source is recorded in the image processing system, and the infrared image is temperature calibrated.

[0126] Example 3: This example is basically the same as Example 1, except that this example further illustrates the structure.

[0127] In this embodiment, in step S300, the pixel values ​​in the image are counted, the frequency of occurrence of each pixel value is calculated, the frequency of each pixel value is divided by the total number of pixels, and the probability of each pixel value is obtained. The calculated pixel value probability distribution is represented in the form of a histogram and a vector, thereby obtaining a pixel value probability distribution graph describing the pixel value distribution. A pixel value probability distribution graph is also generated for each cropped image.

[0128] In step S400, if Figure 3 , the pixel value probability distribution map is input into the input layer and participates in the fusion of the feature vectors of the damaged areas of the two images. The pixel value probability distribution map enhances the accuracy of the damage recognition model by providing the overall brightness and contrast distribution information of the image.

[0129] Example 4: This example is basically the same as Example 1, except that images at different temperatures are collected during the heating process of the excitation source. This example further illustrates the structure.

[0130] In this embodiment, in step S200, the thermal excitation source continuously heats the collection point, and the visible light camera and the infrared camera capture images of the steel surface at different heating times, thereby obtaining multiple sets of images of the same collection point at different temperatures.

[0131] In step S400, Figure 4 ,Multiple sets of images at the same acquisition point at different temperatures are input into the damage recognition model together, and multiple sets of comprehensive feature vectors are obtained through the feature fusion layer;

[0132] Multiple groups of feature vectors are fused into a final feature vector using a weighted average algorithm, so that the classification layer can judge the damage type according to the final feature vector. This can more comprehensively utilize the temperature feature information and improve the recognition performance of the model.

Claims

1. A method for identifying steel surface damage using binocular machine vision, characterized by: The following steps are involved: S100: Build binocular acquisition system and image processing system; The acquisition system includes a mobile platform, a visible light camera, an infrared camera, and a thermal excitation source. The path and acquisition points of the mobile platform are set, and the visible light camera, infrared camera, and thermal excitation source are mounted on the mobile platform. The image processing system includes a feature matching module and a damage recognition model; S200: image acquisition; The mobile platform moves to the collection point, where the thermal excitation source heats the collection point. The visible light camera and infrared camera capture images of the heated steel surface. Each capture simultaneously acquires visible light and infrared images. S300: image preprocessing; The feature matching module extracts key feature points from the corresponding visible light image and infrared image, matches the key feature points in the two images, and identifies and crops the overlapping areas in the two images; S400: Establish a damage identification model and identify damage categories; The damage identification model includes input layer, feature identification layer, feature fusion layer and classification layer; Input the key feature points of the overlapping areas in the visible light image and the infrared image from the input layer; The feature recognition layer extracts the feature vector of the damaged area on the steel surface from the key feature points of the visible light image by analyzing the texture and edge of the steel surface. The feature recognition layer extracts the feature vector of the damaged area on the steel surface from the key feature points of the infrared image by analyzing the temperature distribution on the steel surface; The feature fusion layer finds the correspondence between the features of the damaged areas of the two images, fuses the feature vectors of the damaged areas of the two images and obtains a comprehensive feature vector; The classification layer calculates the probability of each damage type based on the comprehensive feature vector and outputs the damage type with the highest probability; S500: Verify the recognition result; The output results are manually verified against the actual situation, and the verification data is recorded and uploaded to the image processing system to optimize the damage identification model.

2. The method for identifying steel surface damage using binocular machine vision according to claim 1, characterized in that: The acquisition system further includes a standard heat source, which is arranged in the shooting range of the infrared camera, and the temperature of the standard heat source is recorded in the image processing system.

3. The method for identifying steel surface damage using binocular machine vision according to claim 1, characterized in that: The step S150 is also included: configuring an infrared camera; Set the emissivity of the steel surface to be measured in the infrared camera; Adjust the distance and tilt angle between the visible light camera and the infrared camera to ensure that the overlap rate of the shooting fields of the visible light camera and the infrared camera exceeds 90%.

4. The method for identifying steel surface damage using binocular machine vision according to claim 1, characterized in that: In step S200, the thermal excitation source continuously heats the collection point, and the visible light camera and the infrared camera capture images of the steel surface at different heating times, thereby obtaining multiple sets of images of the same collection point at different temperatures; In step S400, multiple sets of images at the same acquisition point at different temperatures are input into the damage recognition model together, and multiple sets of comprehensive feature vectors are obtained through the feature fusion layer; Multiple sets of feature vectors are fused into a final feature vector, so that the classification layer can determine the damage type based on the final feature vector.

5. The method for identifying steel surface damage using binocular machine vision according to claim 1, characterized in that: The damage identification model is trained by the following steps: Repeat steps S100-S200 to photograph steel materials with different types of damage under different lighting conditions and temperature conditions; Steps S300 - S500 are performed on each set of images until the error of the output result is reduced to within the allowable range.

6. The method for identifying steel surface damage using binocular machine vision according to claim 1, characterized in that: In the step S300, a pixel value probability distribution map describing the pixel value distribution of each cropped image is also generated; In step S400, the pixel value probability distribution map is input into the input layer and used to fuse the feature vectors of the damaged areas of the two images.

7. The method for identifying steel surface damage using binocular machine vision according to claim 1, characterized in that: The feature fusion layer obtains the comprehensive feature vector through the following steps: S401: feature transformation; Perform a linear transformation on each eigenvector to ensure that they are in the same feature space; Feature transformation formula: F′ v =W v F v +b v ; F′ ir =W ir F ir +b ir ; Among them, F v is the feature vector extracted from the visible light image; F ir is the feature vector extracted from the infrared image; W v and W ir is the transformation matrix; b v and b ir is the bias vector; S402: Setting up self-attention mechanism; The self-attention mechanism is used to calculate the attention weight between two feature vectors to enhance the expression of important features: Self-attention mechanism formula: Where: Q=K=V=[F v ',F ir '], Q is the query matrix; K is the key matrix; V is the value matrix; d k is the characteristic dimension; The softmax function is used to calculate the attention weight; d k is the characteristic dimension; S403: Feature weighting; The feature vectors are weighted and combined using the weights obtained through the self-attention mechanism; Weighted combination formula: F att =Attention(F′ v ,F′ ir ); Among them F att is the weighted eigenvector; S404: Fusion features; The weighted feature vector is fused with the original transformed feature vector to form a comprehensive feature vector. The fused comprehensive feature vector is linearly transformed to further compress and integrate feature information. Fusion feature formula: F fused =concat(F att ,F′ v ,F′ ir ); F fused =W f F fused +b f ; Among them, F fused is the comprehensive eigenvector, W f is the weight matrix of the fusion layer, b f is the bias vector.

Images