A target positioning method and system based on multi-modal fusion
By combining multimodal information from radar and visual sensors, the resolution and anti-occlusion issues of target detection in complex scenarios are solved, achieving more efficient and accurate target localization.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SUN YAT SEN UNIVERSITY SHENZHEN
- Filing Date
- 2026-02-04
- Publication Date
- 2026-05-29
AI Technical Summary
Existing target detection methods based on communication and computer vision suffer from low resolution and insufficient resistance to occlusion in complex scenes, which affects the accuracy and reliability of target detection.
By fusing multimodal information from radar and vision sensors, the base station detection information provided by radar guides the vision module to perform fine detection of local areas and performs multimodal fusion correction, thereby improving the three-dimensional positioning accuracy of the target.
It significantly reduces the target confusion and false detection problems that are prone to occur with single sensors, and improves the robustness and reliability of target detection, especially maintaining high detection efficiency in low-resolution and complex environments.
Smart Images
Figure CN122110086A_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of target localization technology, and proposes a target localization method and system based on multimodal fusion. Background Technology
[0002] Traditional wireless communication technologies are primarily used for data transmission. However, in the 5G-A era, increased computing power and optimized algorithms in the communications field enable communication technologies to be used for target detection, much like traditional sensing technologies. By sending reference signals and measuring wireless channel characteristics, target location information can be obtained, and more accurate target detection, localization, and behavior analysis can be achieved in complex environments. This, in turn, promotes the development of "communication-sensing-computing integration."
[0003] However, existing communication-based and computer vision-based detection methods each have their limitations: radar sensors excel at detecting target distance and velocity, but perform poorly in resolution and target recognition; visual sensors provide rich target appearance information, but lack reliability in depth perception and under complex lighting conditions. Many current studies consider how to efficiently fuse communication and visual perception results at the outcome level. However, numerous outcome-oriented fusion methods have not fundamentally solved the low resolution of base station detection and the poor resistance to occlusion, truncation, and ambient light variations in visual detection, thus affecting the application of communication-vision fusion target detection in scenarios requiring high resolution and experiencing significant environmental disturbances. Summary of the Invention
[0004] To overcome the shortcomings of the prior art in terms of insufficient positioning capability in complex scenarios, this invention proposes a target positioning method and system based on multimodal fusion.
[0005] To achieve the above-mentioned technical effects, the technical solution of the present invention is as follows: A target localization method based on multimodal fusion includes the following steps: The visual image of the area where the target to be detected is located is obtained based on the visual terminal, and the visual image is input into the visual detection model to obtain the two-dimensional detection box of the target to be detected. The radar channel state information of the area where the target to be detected is located is obtained based on the radar base station. If the number of radar base stations is one, the azimuth information of the target to be detected in the radar base station coordinate system is obtained based on pseudo-spectral peak search. Otherwise, the radar channel state information is input into the angle prediction model to obtain the azimuth information of the target to be detected in the radar base station coordinate system. The azimuth information in the radar base station coordinate system is projected onto the visual terminal coordinate system to obtain a spatial constraint region; the two-dimensional detection box is filtered based on the spatial constraint region to obtain a filtered two-dimensional detection box; for the filtered two-dimensional detection box, the three-dimensional position of the target to be detected in the visual terminal coordinate system is generated based on the expected size of the target to be detected.
[0006] As a preferred embodiment, the target localization method, after obtaining the three-dimensional position of the target to be detected in the visual terminal coordinate system, further performs multimodal fusion correction. The steps of the multimodal fusion correction include: The filtered two-dimensional detection box is projected as a visual ray in three-dimensional space, and the three-dimensional position of the intersection point of the azimuth information in the radar base station coordinate system and the visual ray is calculated based on the geometric constraints of the base station. Calculate the residual between the three-dimensional position of the intersection point and the three-dimensional position of the target to be detected in the visual terminal coordinate system. If the residual is greater than a preset threshold, the positioning result is removed; otherwise, the three-dimensional position of the target to be detected in the visual terminal coordinate system is adjusted on the visual ray to minimize the residual.
[0007] As a preferred embodiment, the step of obtaining the azimuth information of the target to be detected in the radar base station coordinate system based on pseudo-spectral peak search includes: A covariance matrix is constructed based on radar channel state information and eigenvalue decomposition is performed to obtain eigenvalues and eigenvectors. A pseudo-spectrum is then calculated based on the eigenvalues and eigenvectors. The elevation and azimuth angles of the target to be detected are obtained by searching within the pseudo-spectrum. The calculation formula for the pseudo-spectrum is as follows:
[0008] As a preferred embodiment, the step of inputting radar channel state information into the angle prediction model to obtain the azimuth information of the target to be detected in the radar base station coordinate system includes: The radar channel state information is converted into a time-domain channel impulse response, and a range-Doppler spectrum is generated based on a two-dimensional FFT. Static clutter is then filtered and removed based on a local maximum threshold. The calculation formula for the range-Doppler spectrum is as follows:
[0009] The range-Doppler spectrum is input into the angle prediction model to obtain the angle information of the target under the radar base station coordinate system; Based on the angle information and the geometric relationship between the two base stations, the three-dimensional coordinates of the target to be detected in the radar base station coordinate system are determined.
[0010] As a preferred embodiment, the angle prediction model comprises a CNN network, a Transformer encoder, and a multilayer perceptron connected in sequence. The CNN network extracts local features from the range-Doppler spectrum and adds a learnable vector at the beginning of the local features. Global feature modeling is performed based on the Transformer encoder. The multilayer perceptron regresses the pitch and azimuth angles of the target to be detected, and inverse normalization maps the pitch and azimuth angles to the actual angle range. The expression is as follows:
[0011]
[0012] As a preferred embodiment, the angle prediction model uses the mean absolute error as the loss function during training, and its expression is as follows: .
[0013] As a preferred embodiment, the target to be detected is a vehicle, and the visual detection model is trained by retaining only the vehicle label.
[0014] This invention also proposes a target localization system based on multimodal fusion, which, when applied to the aforementioned target localization method based on multimodal fusion, includes: The visual detection module is used to obtain a visual image of the area where the target to be detected is located, and to process the visual image using a visual detection model to obtain a two-dimensional detection box of the target to be detected. The radar signal processing module is used to acquire radar channel state information of the area where the target to be detected is located. If the number of radar base stations is one, the azimuth information of the target to be detected in the radar base station coordinate system is obtained based on pseudo-spectral peak search; otherwise, the radar channel state information is input into the angle prediction model to obtain the azimuth information of the target to be detected in the radar base station coordinate system. The cross-modal filtering module is used to project the azimuth information in the radar base station coordinate system to the visual terminal coordinate system to obtain a spatial constraint region; filter the two-dimensional detection box based on the spatial constraint region to obtain a filtered two-dimensional detection box; and generate the three-dimensional position of the target in the visual terminal coordinate system based on the expected size of the target to be detected for the filtered two-dimensional detection box.
[0015] The present invention also proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement a target localization method based on multimodal fusion as described in the present invention.
[0016] The present invention also proposes a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements a target localization method based on multimodal fusion as described in the present invention.
[0017] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention integrates the base station detection scheme provided by radar with the detection scheme of visual sensors. Radar compensates for the shortcomings of vision in depth estimation and illumination dependence, while vision compensates for the deficiencies of radar in angular resolution and target category recognition. The synergy between the two significantly reduces the target confusion and false detection problems that are prone to occur with single sensors, thereby improving the robustness and reliability of overall perception. Attached Figure Description
[0018] Figure 1 This is a flowchart illustrating the implementation of a target localization method based on multimodal fusion proposed in Example 1. Figure 2 This is a schematic diagram of the angle prediction model in Example 1; Figure 3 This is a schematic diagram of the algorithm flow within the angle prediction model of Example 1; Figure 4 This is a structural diagram of a target localization system based on multimodal fusion proposed in Example 2; Figure 5 This is a schematic diagram of the target detection and localization scenario in Example 3; Figure 6 This is a slice of the target detection simulation scene in Example 3; Figure 7 This is a comparison chart of the F1 scores of different detection methods in Example 3. Detailed Implementation
[0019] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0020] Example 1 This embodiment proposes a target localization method based on multimodal fusion, and the implementation flowchart is as follows. Figure 1 As shown.
[0021] This embodiment proposes a target localization method based on multimodal fusion, which includes the following steps: S1. Obtain a visual image of the area where the target to be detected is located based on the visual terminal, and input the visual image into the visual detection model to obtain a two-dimensional detection box of the target to be detected; S2. Based on the radar base station, obtain the radar channel state information of the area where the target to be detected is located. If the number of radar base stations is one, then the azimuth information of the target to be detected in the radar base station coordinate system is obtained based on the pseudo-spectral peak search; otherwise, the radar channel state information is input into the angle prediction model to obtain the azimuth information of the target to be detected in the radar base station coordinate system. S3. Project the azimuth information in the radar base station coordinate system to the visual terminal coordinate system to obtain a spatial constraint region; filter the two-dimensional detection box based on the spatial constraint region to obtain a filtered two-dimensional detection box; generate the three-dimensional position of the target in the visual terminal coordinate system based on the expected size of the target to be detected for the filtered two-dimensional detection box.
[0022] In this embodiment, the existing method of directly fusing detection results is transformed into one that uses radar base stations to provide the target's azimuth information, guiding the vision module to perform fine detection of local areas in the image. This achieves more efficient and accurate target detection and localization, and ensures that superior detection performance can be maintained even in low-resolution images and scenarios with strong environmental interference.
[0023] In an optional embodiment, after obtaining the three-dimensional position of the target to be detected in the visual terminal coordinate system, the target localization method further performs multimodal fusion correction, the steps of which include: The filtered two-dimensional detection box is projected as a visual ray in three-dimensional space, and the three-dimensional position of the intersection point of the azimuth information in the radar base station coordinate system and the visual ray is calculated based on the geometric constraints of the base station. Calculate the residual between the three-dimensional position of the intersection point and the three-dimensional position of the target to be detected in the visual terminal coordinate system. If the residual is greater than a preset threshold, the positioning result is removed; otherwise, the three-dimensional position of the target to be detected in the visual terminal coordinate system is adjusted on the visual ray to minimize the residual.
[0024] As an example, in single-base station mode, the intersection point is obtained by finding the ellipsoid with the base station as the center and the radius being the distance in the azimuth information; in dual-base station mode, the intersection point is obtained by finding the ellipsoid with the base station as the focus and the sum of the paths to the two focuses being the distance in the azimuth information, and the intersection point is solved by numerical methods; and a geometric consistency check is performed to eliminate NLOS false targets.
[0025] In this embodiment, the geometric residual is the difference between the distance from the target point to the base station and the equivalent ranging in the case of a single base station; otherwise, it is the difference between the sum of the distances from the target point to multiple base stations and the equivalent propagation path length.
[0026] In an optional embodiment, the step of obtaining the azimuth information of the target to be detected in the radar base station coordinate system based on pseudo-spectral peak search includes: A covariance matrix is constructed based on radar channel state information and eigenvalue decomposition is performed to obtain eigenvalues and eigenvectors. A pseudo-spectrum is then calculated based on the eigenvalues and eigenvectors. The elevation and azimuth angles of the target to be detected are obtained by searching within the pseudo-spectrum. The calculation formula for the pseudo-spectrum is as follows:
[0027] In this embodiment, the statistical properties of the covariance matrix are used to suppress noise and multipath interference. The pseudo-spectral peak search method can simultaneously estimate the angle of arrival of multiple targets, thereby achieving efficient multi-target resolution and parameter estimation.
[0028] In an optional embodiment, the step of inputting radar channel state information into the angle prediction model to obtain the azimuth information of the target to be detected in the radar base station coordinate system includes: The radar channel state information is converted into a time-domain channel impulse response, and a range-Doppler spectrum is generated based on a two-dimensional FFT. Static clutter is then filtered and removed based on a local maximum threshold. The calculation formula for the range-Doppler spectrum is as follows:
[0029] The range-Doppler spectrum is input into the angle prediction model to obtain the angle information of the target under the radar base station coordinate system; Based on the angle information and the geometric relationship between the two base stations, the three-dimensional coordinates of the target to be detected in the radar base station coordinate system are determined.
[0030] Furthermore, let the rotation matrices around the x, y, and z axes be Rx, Ry, and Rz, respectively. The total rotation matrix R used for 3D reconstruction is:
[0031] By obtaining the translation matrix t of the visual terminal relative to the radar base station, synthesizing the extrinsic parameter matrix [R|t] of the visual terminal, and combining it with the intrinsic parameter matrix K and the scaling factor s, the coordinates (X, Y, Z) of the target to be detected in the real world can be obtained:
[0032] The orientation information provided by the base station is then back-projected onto the image plane point (u0, v0) to generate a spatially constrained region (ROI=[u0-w / 2, v0-h / 2, u0+w / 2, v0+h / 2]). Based on the spatially constrained region, the two-dimensional candidate bounding boxes detected by YOLO are filtered, and only the bounding boxes with the highest confidence whose center point falls within the spatially constrained region are retained.
[0033] In this embodiment, by converting the original radar channel state information (CSI) into range-Doppler spectrum (RD-spectrum), the separation of multiple targets in the range and velocity dimensions can be achieved. Secondly, the angle prediction model CNN-VIT can accurately map the elevation and azimuth information of the target in the radar base station coordinate system from the spectral features.
[0034] In an optional embodiment, the angle prediction model includes a CNN network, a Transformer encoder, and a multilayer perceptron connected in sequence. The CNN network extracts local features from the range-Doppler spectrum and adds a learnable vector at the beginning of the local features. Global feature modeling is performed based on the Transformer encoder. The multilayer perceptron regresses the pitch and azimuth angles of the target to be detected, and inverse normalization maps the pitch and azimuth angles to the actual angle range. The expression is as follows:
[0035]
[0036] like Figure 2 The figure shown is a schematic diagram of the angle prediction model.
[0037] like Figure 3 The diagram shown is a schematic of the algorithm flow within the angle prediction model.
[0038] In this embodiment, a CNN network is introduced to perform convolutional encoding on local sub-regions, achieving spatial downsampling and channel upscaling. The original CSI data dimension is 40*256*4*1638. After radar signal processing, the number of channels becomes the number of CSI candidate targets * 4 * the two-dimensional arrangement of the antenna array (N*4*8*16). This reduces the number of tokens while preserving the microstructural features related to the target angle, improving the efficiency of subsequent global modeling. Simultaneously, by introducing learnable variables, the Transformer encoder can learn the long-range dependencies between different range units and different Doppler units at a global scale. This enhances the ability to discriminate the true angle of the target in multipath reflection, low signal-to-noise ratio, and occlusion environments, significantly improving the continuity and stability of angle estimation and reducing errors caused by spurious peaks and angle jumps.
[0039] In an optional embodiment, the angle prediction model uses the mean absolute error as the loss function during training, and its expression is as follows: .
[0040] In this embodiment, the mean absolute error is less sensitive to outliers and isolated points, thereby guiding the model to learn more generalizable and robust angle mapping relationships.
[0041] In one optional embodiment, the target to be detected is a vehicle, and the visual detection model is trained by retaining only the vehicle label.
[0042] As an example, the visual detection model retains only three categories: "car," "bus," and "truck," denoted as a unified category set to reduce category conflict and improve recall. Several online augmentations are applied to the training set to significantly mitigate overfitting. The visual detection model is YOLOv11s, with weights W_coco initialized and the first few layers of the network frozen to reduce training parameters and shorten training time. The AdamW optimizer is used, and a co-annealing learning strategy is implemented to balance fast convergence, fine-tuning, and overfitting prevention, resulting in a highly robust final model.
[0043] Example 2 This embodiment proposes a target localization system based on multimodal fusion, applying a target localization method based on multimodal fusion proposed in Embodiment 1. For example... Figure 4 The diagram shown is an architecture diagram of a target localization system based on multimodal fusion in this embodiment.
[0044] This embodiment proposes a target localization system based on multimodal fusion, including: The visual detection module is used to obtain a visual image of the area where the target to be detected is located, and to process the visual image using a visual detection model to obtain a two-dimensional detection box of the target to be detected. The radar signal processing module is used to acquire radar channel state information of the area where the target to be detected is located. If the number of radar base stations is one, the azimuth information of the target to be detected in the radar base station coordinate system is obtained based on pseudo-spectral peak search; otherwise, the radar channel state information is input into the angle prediction model to obtain the azimuth information of the target to be detected in the radar base station coordinate system. The cross-modal filtering module is used to project the azimuth information in the radar base station coordinate system to the visual terminal coordinate system to obtain a spatial constraint region; filter the two-dimensional detection box based on the spatial constraint region to obtain a filtered two-dimensional detection box; and generate the three-dimensional position of the target in the visual terminal coordinate system based on the expected size of the target to be detected for the filtered two-dimensional detection box.
[0045] It is understood that the system in this embodiment corresponds to the method in Embodiment 1 above, and the options in Embodiment 1 above are also applicable to this embodiment, so they will not be described again here.
[0046] Example 3 This embodiment applies the target localization method based on multimodal fusion proposed in Embodiment 1.
[0047] like Figure 5 The image shows a schematic diagram of a target detection and localization scenario. Figure 5 The given scenario involves an urban street environment, including roads, vehicles, and buildings. Conventional single-modal detection techniques face several potential detection challenges. Dense urban buildings can easily cause radar signal obstruction or multipath interference. Furthermore, visual terminals may be affected by changes in lighting, weather, or viewing angle limitations, leading to unstable target recognition. This application utilizes a radar base station and a visual terminal to jointly detect and locate targets such as vehicles on the road. Figure 6 The image shown is a slice of a simulated scene for object detection.
[0048] Figure 6 The algorithm is divided into four scenes: top left, top right, bottom left, and bottom right, representing scenarios A, B, C, and D, respectively. Scenes A and C contain small to medium-sized buildings, which may partially obscure vehicles at intersections but will not completely block them. These scenes correspond to open scenes such as school intersections and regular road intersections in reality. Scenes B and D, compared to A and C, add tall buildings, which cover a certain area of the road within the field of view and completely obscure all vehicles within that area, making them impossible to detect directly using visual methods. These scenes correspond to high-density school districts and densely built-up urban areas, respectively. Scene D also includes parking areas where cars are not counted in the road vehicle detection and need to be removed, posing new requirements for the algorithm. All scenes include buildings, roads similar to those in real-world scenarios, and randomly generated traffic flow. Visual terminals can capture targets within line of sight that are not obstructed or partially truncated. Buildings may obstruct or completely block some vehicles to varying degrees, and visual terminals cannot directly capture the visual information of these vehicles. Radar base stations can transmit radar signals to measure scene information within their coverage area. The measurement information implicitly includes traffic flow information. The radar information is further processed into three-dimensional target information and then transmitted to the visual terminal. The visual terminal combines the three-dimensional information provided by the base station to assist visual detection and finally provide all measurement results.
[0049] The four scenarios A, B, C, and D were detected and the results were recorded using the method described in this application, YOLO standalone detection, and CSI standalone detection.
[0050] like Figure 7 The figure shown is a comparison chart of F1 scores for different detection methods.
[0051] from Figure 7It can be seen that the YOLO standalone detection method performs best in simple scenarios (such as scenario B) with less occlusion, small changes in light, and high traffic density within line of sight, indicating that it can effectively detect targets in a single mode. However, its F1 score gradually decreases as scene complexity increases and in scenarios with large changes in light intensity. The CSI standalone detection method can ignore the influence of line of sight to detect targets within the base station coverage area, but it tends to detect targets with faster movement speeds. In scenarios A and B, where vehicle speeds are higher and more targets are occluded, the detection effect is significantly better than the YOLO standalone detection method. In particular, the best performance is achieved in scenario B, where buildings are dense and reflections are more frequent. As the reflection paths in the scene decrease, the detection effect decreases in scenario C. In scenario D, which is a parking lot area with relatively slower vehicle speeds, the CSI standalone detection method performs the worst, indicating that its accuracy in target orientation measurement is relatively low but its recall rate is higher when used alone. In comparison, the method of this application has the highest F1 score in all scenarios, demonstrating its advantages in both simple and complex scenarios. In scenarios C and D, where the vehicle speed is slow and there are many targets within the line of sight of the occluded area, the fusion method combines the effects of two single-modal detection methods, and its F1 score is also higher than the other two methods.
[0052] Example 4 This embodiment proposes a computer device, including a memory and a processor. The memory stores computer-readable instructions, which, when executed by the processor, cause the processor to perform the steps of the target localization method based on multimodal fusion proposed in Embodiment 1.
[0053] Example 5 This embodiment proposes a storage medium storing computer-readable instructions, wherein when the computer-readable instructions are executed by a processor, they implement the steps of the target localization method based on multimodal fusion proposed in Embodiment 1.
[0054] By way of example, the storage medium includes, but is not limited to, USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks or optical disks, and other media capable of storing program code.
[0055] By way of example, the instructions, programs, code sets, or instruction sets may be implemented using conventional programming languages.
[0056] By way of example, the processor includes, but is not limited to, smartphones, personal computers, servers, network devices, etc., for performing all or part of the steps of the pad via identification and positioning method described in Example 1.
[0057] The terminology used in the accompanying drawings is for illustrative purposes only and should not be construed as limiting the scope of this patent. Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.
Claims
1. A target localization method based on multimodal fusion, characterized in that, Includes the following steps: The visual image of the area where the target to be detected is located is obtained based on the visual terminal, and the visual image is input into the visual detection model to obtain the two-dimensional detection box of the target to be detected. The radar channel state information of the area where the target to be detected is located is obtained based on the radar base station. If the number of radar base stations is one, the azimuth information of the target to be detected in the radar base station coordinate system is obtained based on pseudo-spectral peak search. Otherwise, the radar channel state information is input into the angle prediction model to obtain the azimuth information of the target to be detected in the radar base station coordinate system. The azimuth information in the radar base station coordinate system is projected onto the visual terminal coordinate system to obtain the spatially constrained region. The two-dimensional detection boxes are filtered based on the spatial constraint region to obtain filtered two-dimensional detection boxes; for the filtered two-dimensional detection boxes, the three-dimensional position of the target to be detected in the visual terminal coordinate system is generated based on the expected size of the target to be detected.
2. The target localization method based on multimodal fusion according to claim 1, characterized in that, After obtaining the three-dimensional position of the target to be detected in the visual terminal coordinate system, the target localization method further performs multimodal fusion correction. The steps of the multimodal fusion correction include: The filtered two-dimensional detection box is projected as a visual ray in three-dimensional space, and the three-dimensional position of the intersection point of the azimuth information in the radar base station coordinate system and the visual ray is calculated based on the geometric constraints of the base station. Calculate the residual between the three-dimensional position of the intersection point and the three-dimensional position of the target to be detected in the visual terminal coordinate system. If the residual is greater than a preset threshold, the positioning result is removed; otherwise, the three-dimensional position of the target to be detected in the visual terminal coordinate system is adjusted on the visual ray to minimize the residual.
3. The target localization method based on multimodal fusion according to claim 1, characterized in that, The step of obtaining the azimuth information of the target under the radar base station coordinate system based on pseudo-spectral peak search includes: A covariance matrix is constructed based on radar channel state information and eigenvalue decomposition is performed to obtain eigenvalues and eigenvectors. A pseudo-spectrum is then calculated based on the eigenvalues and eigenvectors. The elevation and azimuth angles of the target to be detected are obtained by searching within the pseudo-spectrum. The calculation formula for the pseudo-spectrum is as follows: in, For the angle of the space to be measured, The steering vector is the phase difference generated when the radar signal reaches each element of the antenna array. It is the noise subspace matrix composed of eigenvectors.
4. The target localization method based on multimodal fusion according to claim 1, characterized in that, The step of inputting radar channel state information into the angle prediction model to obtain the azimuth information of the target to be detected in the radar base station coordinate system includes: The radar channel state information is converted into a time-domain channel impulse response, and a range-Doppler spectrum is generated based on a two-dimensional FFT. Static clutter is then filtered and removed based on a local maximum threshold. The calculation formula for the range-Doppler spectrum is as follows: in, For the symbolic dimension of the Fourier transform, The inverse Fourier transform of the subcarrier dimension. This refers to radar channel status information; The range-Doppler spectrum is input into the angle prediction model to obtain the angle information of the target under the radar base station coordinate system; Based on the angle information and the geometric relationship between the two base stations, the three-dimensional coordinates of the target to be detected in the radar base station coordinate system are determined.
5. The target localization method based on multimodal fusion according to claim 4, characterized in that, The angle prediction model comprises a CNN network, a Transformer encoder, and a multilayer perceptron connected in sequence. The CNN network extracts local features from the range-Doppler spectrum and adds a learnable vector to the beginning of the local features. Global feature modeling is performed based on the Transformer encoder. The multilayer perceptron regresses the pitch and azimuth angles of the target to be detected, and inverse normalization is used to convert the pitch angle... The expression for mapping the azimuth angle (AOA) to the actual angle range is as follows: in, and The range of maximum and minimum pitch angles set for the model. and The range of maximum and minimum azimuth angles set for the model.
6. The target localization method based on multimodal fusion according to claim 5, characterized in that, The angle prediction model uses the mean absolute error as the loss function during training, and its expression is as follows: in, For the output of the prediction model, The label represents the true value from the actual perspective.
7. The target localization method based on multimodal fusion according to any one of claims 1 to 6, characterized in that, The target to be detected is a vehicle, and the visual detection model is trained by retaining only the vehicle label.
8. A target localization system based on multimodal fusion, employing the target localization method based on multimodal fusion as described in any one of claims 1 to 7, characterized in that, include: The visual detection module is used to obtain a visual image of the area where the target to be detected is located, and to process the visual image using a visual detection model to obtain a two-dimensional detection box of the target to be detected. The radar signal processing module is used to acquire radar channel state information of the area where the target to be detected is located. If the number of radar base stations is one, the azimuth information of the target to be detected in the radar base station coordinate system is obtained based on pseudo-spectral peak search; otherwise, the radar channel state information is input into the angle prediction model to obtain the azimuth information of the target to be detected in the radar base station coordinate system. The cross-modal filtering module is used to project the azimuth information in the radar base station coordinate system to the visual terminal coordinate system to obtain the spatially constrained region; The two-dimensional detection boxes are filtered based on the spatial constraint region to obtain filtered two-dimensional detection boxes; for the filtered two-dimensional detection boxes, the three-dimensional position of the target to be detected in the visual terminal coordinate system is generated based on the expected size of the target to be detected.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the target localization method based on multimodal fusion as described in any one of claims 1 to 7.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the target localization method based on multimodal fusion as described in any one of claims 1 to 7.