Target detection method in dark light environment

Abstract

The invention provides a target detection method in a dark light environment. The target detection method comprises the following steps: step 1), training decomposition model illumination map and reflection map decomposition; step 2), introducing an attention mechanism, and fusing the depth information provided by the millimeter wave radar with the reflection component to obtain object features; the method comprises the steps of (1) obtaining image features, (2) obtaining object features, (3) introducing an attention fusion mechanism and fusing the image features and the object features, and (4) calculating and predicting the position and category of a target through a YOLO frame detection head. The method not only provides a new view angle and a technical path for target detection in a dark light environment, but also shows the huge potential of interdisciplinary technology fusion in solving complex problems.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision target detection, and in particular relates to a target detection method in a dark light environment. Background Art

[0002] Object detection technology is an extremely important research direction in the field of computer vision. It has a wide range of application scenarios and huge application potential. It has played a huge role in daily production and life, such as autonomous driving, text processing, face recognition and other fields. With the advancement of scientific research, many object detection algorithms have gradually been developed. The YOLO series of algorithms is one of the most widely used object detection algorithms. When it was first released, it attracted widespread attention for its fast detection capabilities. After multiple versions of iterations, the YOLO algorithm not only retains the advantage of fast detection speed, but also makes up for the shortcoming of low detection accuracy.

[0003] However, most of the existing target detection algorithms are only studied in scenes assuming normal lighting. The performance of this type of target detection algorithm in dark scenes such as night, tunnels and bad weather is usually greatly reduced. These scenes require more accurate target detection, but the current research on target detection in dark scenes is very limited. Compared with scenes with normal brightness, images in dark scenes have many characteristics that have a negative impact on the detection effect, such as low brightness, high noise and large color deviation. This leads to a great difference between dark scenes and normal lighting scenes, that is, algorithms for normal lighting scenes are often not effectively applied to dark scenes. However, common target detection tasks in dark scenes have very broad application prospects, such as intelligent driving, monitoring and drones in dark environments. However, the current research on dark images mainly focuses on designing efficient dark enhancement algorithms, that is, using artificial intelligence algorithms to visually improve the brightness of the image so that the objects in the image have clearer contours and details. Although the above strategy can reduce the difference in brightness between dark light images and normal light images, the existing dark light enhancement algorithm will inevitably bring about problems such as noise amplification and color deviation amplification, resulting in the subsequent feature extraction network being unable to extract high-quality features, ultimately leading to unsatisfactory detection results. In addition, the structure of mutually independent enhancement and detection models in series is a non-end-to-end model structure, which will increase the model calculation amount and reduce the detection speed. Summary of the invention

[0004] The purpose of this invention is to overcome the shortcomings of traditional target detection methods in low-light environments by improving existing target detection algorithms. The improved method can effectively make up for the limitations of pure images in low-light environments, effectively extract the feature information of objects in the scene, and maintain the efficiency of the end-to-end model, thereby significantly improving the target detection performance in low-light environments while maintaining the efficiency of the algorithm. This method can be widely used in the fields of autonomous driving, security monitoring, drones, etc.

[0005] To achieve the above object, the present invention proposes a target detection method in a dark environment, comprising the following steps:

[0006] Step 1), train the decomposition model to decompose the illumination map and the reflection map; introduce the decomposition model, analyze the input image, and decompose the illumination map and the reflection map into the illumination map component and the reflection map component;

[0007] Step 2) Introduce the attention mechanism to fuse the 2D point cloud data provided by the millimeter-wave radar with the reflection map component to obtain the object features;

[0008] Step 3), introduce the attention fusion mechanism, fuse the image features obtained by splicing the input image and the illumination map with the object features, and use the YOLO backbone network to process the image to extract the final image features;

[0009] Step 4) The YOLO framework detection head is used to calculate the position and category of the predicted target. In the final target detection stage, the detection head uses the final image features to accurately calculate the position and category of the predicted target.

[0010] The present invention has the following beneficial technical effects:

[0011] The present invention gives full play to the advantage of the Retinex theory that the reflectance map component of the image can reflect the characteristics of the object itself. By using the depth information provided by the millimeter-wave radar to enhance the reflectance map component of the image, the reflectance map features in the image can be extracted more effectively and accurately. This feature is referred to as "object feature" in the present invention. Finally, by using a pyramid feature generation mechanism and a self-attention feature fusion mechanism, these object features are effectively integrated into the image feature extraction process of the YOLO target detection framework in multiple scale dimensions, thereby enhancing the image features in the original YOLO algorithm and improving the algorithm's detection accuracy of targets in low-light environments. This method not only provides a new perspective and technical path for target detection in low-light environments, but also demonstrates the great potential of interdisciplinary technology integration in solving complex problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 Schematic diagram of the process of the target detection method in the present invention;

[0013] Figure 2 It is a framework diagram of the target detection algorithm in the present invention;

[0014] Figure 3 Decompose the model graph into illumination graph and reflection graph;

[0015] Figure 4 This is the structure diagram of the object feature extraction module;

[0016] Figure 5 It is a schematic diagram of the CSP structure;

[0017] Figure 6 Schematic diagram of the process of fusing body features with image features;

[0018] Figure 7 Schematic diagram of the self-attention fusion mechanism structure;

[0019] Figure 8 This is the architecture diagram of the dark light target detection system. DETAILED DESCRIPTION

[0020] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of this application.

[0021] Reference Figure 1 , Figure 2 , as an embodiment of the present invention, provides a method for detecting a target in a dark environment, comprising the following:

[0022] Step 1) First, we need to build a target detection dataset with millimeter wave data. This dataset not only contains image data, but also combines millimeter wave radar data, providing a rich multimodal data resource. The images and millimeter wave radar data in the dataset are taken synchronously and appear in pairs.

[0023] The dataset contains about 6,270 frames of data, each of which includes an image and the corresponding millimeter-wave radar point cloud data. All data were collected in low-light environments, covering common indoor and outdoor scenes, thus ensuring the diversity and practicality of the dataset. In order to support accurate target detection algorithm evaluation, the bounding box information of each human target in the dataset is labeled in detail.

[0024] The millimeter wave dataset is carefully divided into training set and validation set in a 4:1 ratio. 5016 frames of data are allocated as training set for model learning and optimization, and the remaining 1254 frames of data are used as validation set to evaluate the generalization ability and actual performance of the model.

[0025] Step 2) Train the decomposition model to decompose the illumination map and the reflection map. Figure 3 As shown in the figure, a decomposition model of illumination map and reflectance map based on convolutional neural network is proposed. The decomposition model is composed of 5 layers of 3×3 convolutions with ReLU activation function, and a multi-head self-attention module MSA is introduced after the third convolution layer, so that the decomposition model can utilize the global information of the image. The attention module realizes attention encoding by evenly dividing the input vector into 64 blocks. Finally, a decoupling head module is introduced after the fifth convolution layer to obtain the final illumination map and reflectance map decomposition result. The decoupling head module is specifically composed of two parallel 3×3 convolutions with ReLU activation function and a Sigmoid function. The structure of this module is shown in Figure 3 shown.

[0026] In order to train the decomposition model, it is necessary to create constraints for the illumination map and the reflection map. When training the decomposition model, a bright light image needs to be input at the same time. With a dark light picture , and the illumination components of the bright light image obtained by the decomposition model With the reflection component , and get the illumination component of the dark light picture at the same time With the reflection component Then the bright light image can be simultaneously or and Reconstruction. The reconstruction process can be expressed as:

[0027] ;

[0028] There are also corresponding:

[0029] ;

[0030] in, Representing the binary operation of the reconstruction operation, the following reconstruction loss function is introduced:

[0031] ;

[0032] At the same time, since the reflection component R is independent of illumination, the reflection component R of a paired bright light image and dark light image should be as consistent as possible. Based on this conclusion, the following reflection consistency loss function is introduced:

[0033] ;

[0034] The final decomposition model loss function can be expressed as:

[0035] ;

[0036] in, is a loss function, which controls the proportion of reconstruction loss and reflection consistency loss in the overall loss. In this embodiment, The value of is 0.4, and the image reconstruction operation Defined as element-wise multiplication.

[0037] Step 3) introduces the attention mechanism, and fuses the point cloud data provided by the millimeter-wave radar with the reflection component to obtain the object features; in order to improve the detection accuracy and robustness of the target detection algorithm in a dark light environment, the target detection algorithm framework proposed in the present invention fuses the data provided by the millimeter-wave radar. With its high resolution and penetration, the millimeter-wave radar can provide rich distance and speed information for target detection. However, the raw data output by the millimeter-wave radar is usually a three-dimensional point cloud, and directly processing this data is complex and computationally intensive. In order to effectively utilize this information and simplify the processing flow, the present invention adopts a method of mapping 3D point cloud data to 2D data. The specific data mapping process includes two parts, namely coordinate mapping and distance mapping. Among them, coordinate mapping refers to mapping the horizontal and vertical coordinates in the three-dimensional point cloud data output by the radar to the horizontal and vertical coordinates of the two-dimensional image. This step converts complex three-dimensional spatial information into an easy-to-process two-dimensional image format, greatly simplifying the subsequent processing flow. Distance mapping refers to converting the distance information in the point cloud data into the grayscale value of the image, thereby realizing the preservation of the target distance information.

[0038] After coordinate mapping and distance mapping, a 2D image reflecting the millimeter wave radar point cloud data can be preliminarily obtained. In order to further improve the quality of the image and reduce the influence of noise, the present invention also uses a Gaussian filter to perform image smoothing.

[0039] Through the above mapping method, the present invention can effectively process millimeter wave radar data. This processing method reduces the complexity of data processing while maintaining the important features provided by millimeter wave radar data, such as distance and position information, providing a new data source for image-based target detection technology. By fusing millimeter wave radar data, the target detection method can improve the accuracy and robustness of detection in complex environments, especially in low light or bad weather conditions.

[0040] After successfully obtaining the 2D mapping results of the millimeter-wave radar data and decomposing the illumination map and reflection map components obtained by the input image, the present invention innovatively introduces the point cloud data obtained by the millimeter-wave device. This step is extremely critical because it enables the millimeter-wave detection information to be deeply integrated with the reflection map, thereby generating comprehensive object features rich in essential attributes such as object distance and contour. This fusion process includes three calculation stages, namely, the reflection map and millimeter-wave feature extraction stage, the feature fusion stage, and the pyramid feature generation stage.

[0041] Specifically include: millimeter wave point cloud data mapping. The detection distance limit of the millimeter wave radar can be expressed as L, then the detection distance limits in the left and right directions, vertical directions, and front and back directions on the millimeter wave radar detection plane are expressed as , and Correspondingly, the point cloud coordinates in the three directions can be expressed as , and ,at the same time, satisfy , satisfy , satisfy Then, the resolution of the mapped target 2D image is defined as , then the process of left-right and vertical coordinate mapping can be expressed as:

[0042] ;

[0043] ;

[0044] The process of distance mapping is expressed as:

[0045] ;

[0046] in, Function represents the rounding function. is a constant, representing the maximum value of the image grayscale.

[0047] The millimeter wave point cloud data is deeply integrated with the reflection map to generate comprehensive object features rich in essential attributes such as object distance and contour. This fusion process includes three calculation stages, namely, the reflection map and millimeter wave feature extraction stage, feature fusion stage, and pyramid feature generation stage.

[0048] The first is the feature extraction stage of the reflection map and millimeter-wave 2D radar data. The present invention designs two parallel feature extraction networks, which are responsible for mining the rich feature information contained in the reflection map and millimeter-wave 2D radar data respectively. The goal of this stage is to extract feature information that can reflect the essential properties of the object from two different types of data sources, image reflection map and millimeter-wave 2D radar data.

[0049] Next is the feature fusion stage. The present invention adopts an advanced multi-head attention mechanism (MSA), which can effectively integrate the reflection map features and the millimeter wave 2D radar data features. In this way, not only the unique information of each feature is retained, but also the complementarity and enhancement between features are achieved, resulting in richer and more delicate object features.

[0050] Finally, a pyramid feature extraction structure is used to realize the generation of object features of different scales. This design enables the model of the present invention to flexibly adapt to the detection of objects of different sizes and distances, greatly improving the applicability and detection performance of the model.

[0051] The structural design of the entire feature extraction and fusion module is as follows Figure 4 Through this module, the present invention successfully achieves the goal of extracting and fusing key object features from the reflection map component of the input image and the millimeter-wave 2D radar data, providing strong feature support for subsequent target detection and ensuring the performance of the model in a dark environment.

[0052] First, the point cloud data obtained by the millimeter wave is mapped into a vector in 2D space by plane mapping, and the original distance information is retained by the element value. In this process, a certain threshold filtering method is also used to remove noise. After the millimeter wave 2D radar data is obtained, it is sent to the feature extraction model in parallel with the reflection map. The reflection map and the millimeter wave 2D radar data use the same feature extraction model, and in order to extract features more effectively, the present invention introduces a CSP structure in the feature extraction model. The schematic diagram of the CSP structure is shown in FIG. Figure 5 As shown in the figure, the structure divides the input feature map into two parts, called the foreground part and the background part. The foreground part usually contains more important features, while the background part contains relatively less important features. The two parts of the feature map are processed by different network stages. Afterwards, the two parts of the feature map will be fused together. This cross design enables the model to have better feature extraction capabilities. The feature extraction model contains a 6-layer CSP structure.

[0053] The reflection map features obtained by the feature extraction model are expressed as , the millimeter wave characteristics are expressed as , introduce a multi-head attention (MSA) to merge them. First, we need to and According to The structure of the output vector group is obtained , and then set up 3 learnable weight matrices , so that:

[0054] ;

[0055] Among them, (Q, K, V) is a set of query vectors calculated, and then the attention result is obtained by multiplying Q and K. The attention result is the fused feature :

[0056] ;

[0057] in, is the square root of the first dimension of the weight matrix. , and Expand to multiple, that is, form a multi-head attention mechanism, and then concatenate the output results of each attention head with a learnable weight matrix Multiply and fuse the result to the input vector Output vectors of the same shape Finally, the output vector Reorganize the input into and , and then add the two together to get the final object feature .

[0058] After obtaining the object feature T, it is passed as follows Figure 4 The pyramid structure shown in the figure is mapped into features of different sizes in order to be integrated with image features at different stages. The structure contains several convolutional layers and upsampling operations. First, the feature map convolution operation is performed from top to bottom to integrate stronger position information into the lower feature map, and then the feature map is upsampled from bottom to top. At the same time, the feature map of the previous stage is spliced ​​and integrated to integrate stronger semantic information into the high-level feature map, and finally three features of different sizes from large to small are obtained. .

[0059] Step 4) Introduce the self-attention fusion mechanism to fuse object features with image features.

[0060] The present invention adopts the target detection enrichment based on the YOLO architecture as the basic target detection algorithm to achieve fast and accurate recognition of targets in images. The core of the YOLO architecture lies in its powerful backbone network, which is responsible for deep image feature extraction of the input image. The structure of this backbone network includes four key calculation stages, which can effectively capture image features from low-level to high-level to support complex target detection tasks. For the sake of simplicity, the present invention marks the output results of these four calculation stages as ,in Represents the feature output of the i-th calculation stage.

[0061] In addition, in the object feature extraction process, object features at three scales were successfully extracted, and these features are represented as These object features are consistent in shape with the image features extracted by the YOLO backbone network except for the number of channels. This design makes the feature fusion process more direct and effective.

[0062] In order to achieve effective feature fusion, this paper introduces a fusion function , which is responsible for fusing object features with image features. With this function, the feature fusion result of each stage can be concisely expressed by the following formula:

[0063] ;

[0064] in, Represents the calculation result of the i-th stage. After the introduction of feature fusion, the input of the i-th calculation stage of the original YOLO backbone network is changed from the original Change to At the same time, the previously obtained illumination map is spliced ​​into the input image to obtain the image feature P to enhance the illumination semantics of the input image. First, the image feature P is input into the YOLO backbone network and a series of convolution operations are performed to obtain , and then compare it with the largest size object feature After the self-attention fusion function, we get ,and After convolution calculation, we get And in the next step with the object features of the second size Perform self-attention fusion to obtain Calculated Object features with minimum size Fusion to obtain fusion features , after calculation, we finally get the feature . The calculation steps of the four features are as follows Figure 6 As shown in Figure 1. In this way, not only the original feature information of the image is retained, but also additional object features provided by sensors such as millimeter-wave radar are introduced, which greatly enriches the model's ability to perceive the target and improves the accuracy and robustness of target detection. This feature fusion strategy is particularly important in low-light conditions because it can significantly improve the model's detection performance for targets in low-light environments. The structural diagram of the fusion module is shown in Figure 1. Figure 7 shown.

[0065] Since object features focus more on the characteristics of the object itself, and image features focus more on the high-level semantics of the image, there will be structural differences between object features and image features. In order to more effectively fuse these two features, the present invention combines the designs in CA and EMA, and designs a self-attention fusion mechanism, which can fully consider the structural differences of the two features while fusing the features. It inputs the input vector into three parallel calculation paths, namely the longitudinal global path, the transverse global path and the local path, which are used to capture the longitudinal, transverse and local feature distribution information of the feature vector, and then use this information to achieve the effect of self-attention fusion. On this basis, the present invention also introduces a cross-spatial attention mechanism, so that the results of local attention and the results of global attention are cross-calculated, and the attention features of different scales are further fused, so that the spatial semantic features are better evenly distributed in the feature vector.

[0066] The cross-spatial attention module accepts two given input feature vectors and then extracts the attention weights of the grouped feature maps through three parallel routes. The first two parallel paths are on the global attention branch, and the third path is on the local attention branch. In the global attention branch, a 1D global average pooling operation is used to encode the channels along the vertical and horizontal spatial directions of the features respectively. In the local attention branch, only one 3×3 kernel convolution is stacked to capture the local inter-channel attention encoding. In this way, the attention not only encodes the information between channels to adjust the importance of different channels, but also saves the precise spatial structure information into the channels.

[0067] This mechanism produces two attention results, one of which is the output of the global attention branch, and the other is the output of the local attention branch. In order to cross-fuse the results of the two branches, it is necessary to use 2D global average pooling and a 1×1 convolution to encode the global spatial information in the output of each branch. This global spatial information will be used to cross-fuse the calculation results of the two branches. At the same time, for efficient calculation, the natural nonlinear function Softmax of the 2D Gaussian mapping is used to fit the linear transformation at the output of the two branches. The fused attention result is obtained by cross-multiplying the global spatial information encoding result obtained above with the attention results of the two branches. Finally, the output attention fusion result of this module is the result of adding the two generated spatial attention weight values.

[0068] Step 5) The YOLO framework detection head is used to accurately calculate the location and category of the predicted target; the YOLOv5 detection head, which adopts a multi-scale prediction method, is used to enable the model to detect objects on the input image at different scales to adapt to objects of different sizes. Three detection heads of different scales are used for prediction, and the feature information at each level is effectively integrated through the combination of the feature pyramid network (FPN) and the path aggregation network (PAN). This structure enhances the model's detection performance for objects of various sizes, especially small and large objects.

[0069] Step 6) Train the algorithm model. The optimizer uses stochastic gradient descent (SGD). This is a commonly used optimization algorithm used to minimize the loss function of the model during training and improve the performance of the model by iteratively updating the weights of the model. The learning rate scheduler uses the cosine annealing scheduling strategy. This method gradually reduces the learning rate according to the cosine function, which helps to smoothly adjust the learning rate in the later stages of training, avoid oscillations during training, and thus find a better local minimum. The initial learning rate is set to 0.0032. This is the step size of each weight update at the beginning of training. The setting of the initial learning rate has an important impact on the training speed and final performance of the model. The final learning rate is set to 0.12. This is the learning rate that the learning rate scheduler expects to achieve at the end of training. Combined with the initial learning rate, it forms a path for the learning rate to change over time. The momentum is set to 0.843. Momentum helps accelerate SGD optimization in related directions and suppress oscillations. It adjusts the update direction by weighted accumulation of past gradients, which can speed up training and improve efficiency. The weight decay is set to 0.00036. Weight decay is a regularization technique used to prevent overfitting of the model by slightly reducing the weight values ​​to make the model more robust. The warm-up period is set to 2.0. This is the initial stage at the beginning of training, and the learning rate will gradually increase from a smaller value to the initial set learning rate, which helps the stability of the model in the early stages of training. The warm-up momentum is set to 0.5. During the warm-up period, the momentum value will gradually adjust from this lower starting value to the initial set momentum value to help the model adapt to the learning process more smoothly in the early stages of training. The warm-up bias learning rate is set to 0.05. This is the learning rate used for the bias parameter during the warm-up period, which is usually set to a lower value than the weight learning rate to avoid over-adjusting the bias parameter at the beginning of training. The model is trained on the training set for 60 rounds. This process not only improves the model's detection ability in low-light environments, but also enhances the model's ability to process millimeter-wave radar data.

[0070] Step 7) Deploy the trained target detection model to the terminal device to detect and identify targets in a dark environment. The architecture diagram of the dark light target detection system is shown in the figure below. Figure 8As shown in the figure, the prototype system of low-light target detection adopts the classic client / server architecture design to achieve efficient data processing and user interaction. In this highly specialized system, the server undertakes the core task of model reasoning, while the client focuses on data collection, storage and display.

Claims

1. A method for detecting a target in a dark environment, characterized in that: The method comprises the following steps: Step 1), train the decomposition model to decompose the illumination map and the reflection map; introduce the decomposition model, analyze the input image, and decompose the illumination map and the reflection map into the illumination map component and the reflection map component; Step 2) Introduce the attention mechanism to fuse the 2D point cloud data provided by the millimeter-wave radar with the reflection map component to obtain the object features; Step 3), introduce the attention fusion mechanism, fuse the image features obtained by splicing the input image and the illumination map with the object features, and use the YOLO backbone network to process the image to extract the final image features; Step 4) The YOLO framework detection head is used to calculate the position and category of the predicted target. In the final target detection stage, the detection head uses the final image features to accurately calculate the position and category of the predicted target.

2. The method according to claim 1, characterized in that The decomposition model in step 1 is a light map reflectance map decomposition model based on a convolutional neural network, specifically: The decomposition model is composed of 5 layers of 3×3 convolutions with ReLU activation functions, and a multi-head self-attention module MSA is introduced after the third layer of convolution, so that the decomposition model can utilize the global information of the image. The attention module MSA realizes attention encoding by evenly dividing the input vector into 64 blocks; finally, a decoupling head module is introduced after the fifth layer of convolution to obtain the final illumination map reflectance map decomposition result. The decoupling head module is specifically composed of two parallel 3×3 convolutions with ReLU activation functions and a Sigmoid function.

3. The method according to claim 2, characterized in that In step 1, the decomposition model is trained; at the same time, a bright light image is input With a dark light picture , and the illumination components of the bright light image obtained by the decomposition model With the reflection component , and get the illumination component of the dark light picture at the same time With the reflection component , then the bright light image is simultaneously passed through or and Reconstruction, the reconstruction process is expressed as: ； There are also corresponding: ； in, Representing the binary operation of the reconstruction operation, the following reconstruction loss function is introduced: ； The following reflection consistency loss function is introduced: ； The final decomposition model loss function is expressed as: ； in, are loss functions, which respectively control the proportion of reconstruction loss and reflection consistency loss in the overall loss.

4. The method according to claim 1, characterized in that: Step 2 includes three calculation stages, namely, the reflection map and millimeter wave feature extraction stage, the feature fusion stage, and the pyramid feature generation stage. In the reflection map and millimeter wave feature extraction stage, the 3D point cloud data output by the millimeter wave radar is mapped to 2D data and the millimeter wave feature extraction is performed using a feature extraction model; and another parallel feature extraction model performs reflection map feature extraction on the reflection map component. In the feature fusion stage, a multi-head attention mechanism is introduced to fuse the emission map feature extraction results and the millimeter wave feature extraction results obtained in the previous stage. In the pyramid feature generation stage, the feature results obtained in the previous stage are mapped through a pyramid structure to generate a multi-scale feature representation to obtain the object features. .

5. The method according to claim 4, characterized in that Mapping 3D point cloud data to 2D data includes two parts, namely coordinate mapping and distance mapping. Coordinate mapping refers to mapping the horizontal and vertical coordinates in the 3D point cloud data output by the radar to the horizontal and vertical coordinates of the two-dimensional image; distance mapping refers to converting the distance information in the point cloud data into the grayscale value of the image. It includes: The detection distance limit of the millimeter-wave radar is expressed as L, and the detection distance limits in the left and right directions, vertical directions, and front and back directions on the millimeter-wave radar detection plane are expressed as , and , correspondingly, the point cloud coordinates in the three directions are expressed as , and ,at the same time, satisfy , satisfy , satisfy Then, the resolution of the mapped target 2D image is defined as , the process of left-right and vertical coordinate mapping is expressed as: ； ； The process of distance mapping is expressed as: ； in, Function represents the rounding function. is a constant, representing the maximum value of the image grayscale.

6. The method according to claim 4, characterized in that The reflection map features obtained by the feature extraction model are expressed as , the millimeter wave characteristics are expressed as , introduce a multi-head attention MSA to merge them; first and Perform structural division respectively to obtain the output vector group , and then set the weight matrix , so that: ； Among them, (Q, K, V) is a set of query vectors calculated, and then the attention result is obtained by multiplying Q and K. The attention result is the fused feature : ； in, is the square root of the first dimension of the weight matrix. , and Expand to multiple, that is, form a multi-head attention mechanism, and then concatenate the output results of each attention head with a learnable weight matrix Multiply and fuse the result to the input vector Output vectors of the same shape Finally, the output vector Reorganize the input into and , and then add the two together to get the final object feature .

7. The method according to claim 4, characterized in that The self-attention fusion mechanism in step 3 is as follows: the input vector is input into three parallel calculation paths, namely the vertical global path, the horizontal global path and the local path, which are used to capture the vertical, horizontal and local feature distribution information of the feature vector respectively, and then use this information to realize self-attention fusion; The cross-spatial attention module is further introduced. The cross-spatial attention module accepts two given input feature vectors and then extracts the attention weights of the grouped feature maps through three parallel paths. The first two parallel paths are on the global attention branch, and the third path is on the local attention branch. In the global attention branch, a 1D global average pooling operation is used to encode the channel along the vertical and horizontal spatial directions of the feature respectively. In the local attention branch, only one 3×3 kernel convolution is stacked to capture the local inter-channel attention code; this module produces two attention results, one of which is the output of the global attention branch, and the other is the output of the local attention branch; 2D global average pooling and a 1×1 convolution are used to encode the global spatial information in the output of each branch, and the global spatial information will be used to cross-fuse the calculation results of the two branches. At the same time, the natural nonlinear function Softmax of the 2D Gaussian mapping is used at the output of the two branches to fit the linear transformation. Finally, the output attention fusion result of this module is the result of adding the two generated spatial attention weight values.

Images