Spatial positioning method based on multi-modal visual fusion

Abstract

The invention relates to the technical field of spatial positioning, and discloses a spatial positioning method based on multi-modal visual fusion. According to the method, mutual calibration among a laser radar, a short-focus camera and a long-focus camera is realized through a sensor joint calibration algorithm, a spliced image acquired by a bifocal lens of a # imgabs0 # angular color camera is processed, image data in an electric power operation scene shot by a joint calibrated camera system is read, the image is preprocessed, and the image data is acquired. Target detection is carried out on the preprocessed image through a deep learning method, identification and positioning of a target object are realized, a laser radar emits a laser beam to detect the target to obtain data, point cloud data are processed in combination with a visual program, and point cloud mapping, fitting reconstruction and position calculation are carried out on image information obtained by a # imgabs1 # angular color camera, so that the target object is identified and positioned. By means of the method, the precision of power operation complex scene space positioning is improved, and the position detection and recognition error is controlled to be 10 mm or below.

Description

Technical Field

[0001] The present invention relates to the field of spatial positioning technology, and in particular to a multi-modal vision fusion spatial positioning method. Background Art

[0002] The objects in power operation scenes are complex and of various shapes. Identifying and spatially locating objects such as electric poles, houses, and trees in power operation scenes can obtain the spatial position of the objects, and provide assistance for subsequent equipment obstacle avoidance, defect detection, and intelligent management in power scenes. At present, the spatial positioning methods of objects in power operation scenes are mainly divided into two categories: three-dimensional point cloud recognition and positioning and two-dimensional image recognition and positioning. The former is to obtain three-dimensional point cloud data of the scene through three-dimensional laser scanning technology, thereby constructing a three-dimensional model of the scene containing information of multiple objects. The latter is to combine environmental images obtained by multiple sensors such as infrared sensors and visual sensors, and use image processing algorithms for recognition and positioning.

[0003] The current multi-sensor fusion positioning method is difficult to apply to more complex actual scenarios and cannot meet the image recognition and spatial positioning requirements in power operation scenarios, resulting in low positioning accuracy and large errors in position detection and recognition when performing spatial positioning in complex power operation scenarios. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention provides a spatial positioning method for multimodal visual fusion, which has the function of realizing mutual calibration between laser radar, short-focus camera and long-focus camera through sensor joint calibration algorithm. The short-focus lens of the angular color camera collects a large range of stitched images for processing, thereby realizing the scanning, recording and recognition of the global scene. The telephoto lens of the angular color camera collects and processes the field of view information with high definition at a long distance, so as to realize high-precision recognition of distant target objects, process the above-mentioned two-dimensional image information, read the two-dimensional information obtained by the joint calibrated camera system to take the picture data in the power distribution operation scene, pre-process the image, and use the deep learning method to perform target detection on the pre-processed image to realize the recognition and positioning of the target object, use the laser radar to emit a laser beam to detect the target, collect the reflected beam to form a point cloud and obtain data, combine the visual program to process the point cloud data, and The image information obtained by the angular color camera is used for point cloud mapping, fitting reconstruction and position calculation. The above method improves the accuracy of spatial positioning in complex scenes of power operations and controls the position detection and recognition error within 10 The following advantages solve the above problems.

[0005] To achieve the above object, the present invention provides the following technical solution: a spatial positioning method of multimodal visual fusion, comprising the following steps: S1. Through the sensor joint calibration algorithm, the laser radar, short-focus camera and long-focus camera are calibrated with each other, and the target is constructed. Labels are attached to the four corners of the target. Short-focus camera, The target image information collected by the telephoto camera and the laser radar is classified according to the timestamp, the feature points of the above-mentioned captured image and the calibration image are calculated, the feature points are matched, and the internal parameters of the camera are calculated. , the coordinates of the four tags on the target in the image coordinate system, the distance between the camera and the four tags, and the rotation and translation matrix between the image coordinate system and the camera coordinate system. The point with the smallest reprojection error after rotation and translation is selected as the external parameter result. At the same time, according to the target information collected by the lidar, the center point coordinates of the four tags are obtained by the fitting circle method. According to the coordinates of the four tags on the target in the world coordinate system, the external parameters of the radar are calculated. ,pass The decomposition method is used to obtain the rotation and translation matrix from the camera to the radar, completing the joint calibration algorithm of the camera and the radar; S2, yes The short-focus lens of the angular color camera collects a large range of stitched images for processing, thereby realizing the scanning, recording and recognition of the global scene; S3, yes The telephoto lens of the angular color camera collects high-definition field of view information at a long distance for processing, thereby achieving high-precision recognition of distant target objects; S4, processing the two-dimensional image information obtained by the short-focus lens and the long-focus lens in S2 and S3 respectively, reading the two-dimensional information obtained by the image data of the power distribution operation scene taken by the jointly calibrated camera system, pre-processing the image, specifically including denoising, image enhancement, light intensity adjustment and other operations, performing target detection on the pre-processed image through a deep learning method, and realizing recognition and positioning of the target object; S5, using laser radar to emit laser beams to detect targets, and collect reflected beams to form point clouds and obtain data; S6, process point cloud data through visual programs and combine The image information obtained by the angular color camera is used for point cloud mapping, fitting reconstruction and position calculation.

[0006] Preferably, the S1 calculates the camera's internal parameters based on the target image information collected by the telephoto camera and the laser radar. , the calculation formula is as follows:

[0007] In the formula, represents the internal parameters of the camera, and Indicates that the camera is Axis and The focal length on axis, and Represents the coordinates of the center of the image.

[0008] Preferably, S1 calculates the coordinates of the four tags on the target in the image coordinate system according to the target image information collected by the telephoto camera and the laser radar, and the calculation formula is as follows:

[0009] In the formula, Represents the label in the image coordinate system The coordinates of Represents the horizontal pixel coordinate, Represents the vertical pixel coordinate, represents the internal parameters of the camera, Represents the label in the world coordinate system The three-dimensional coordinates of , the added 1 is used to indicate homogeneous coordinates for matrix multiplication.

[0010] Preferably, S1 calculates the distance from the camera to the four tags according to the target image information collected by the telephoto camera and the laser radar, and the calculation formula is as follows:

[0011] In the formula, Indicates the camera is at The distance between tags, the world coordinates of the tags are expressed as , , ,in , indicating four labels, , , Represents the 3D coordinates in the camera coordinate system.

[0012] Preferably, S1 calculates the rotation and translation matrix between the image coordinate system and the camera coordinate system according to the target image information collected by the telephoto camera and the laser radar, and the calculation formula is as follows: Combining the rotation matrix and the translation vector, we can construct a 4*4 external parameter matrix, which is as follows:

[0013] The rotation matrix is ​​calculated by the least squares method. The commonly used rotation matrix is ​​expressed as follows:

[0014] Translation Vector Represents the position of the origin of the camera coordinate system in the world coordinate system, and its calculation is obtained by the following formula:

[0015] Once the rotation matrix and displacement vector are obtained, the point in unit homogeneous coordinate form is mapped from the image coordinate system to the camera coordinate system in the following way:

[0016] In summary, the rotation and translation matrix between the image coordinate system and the camera coordinate system is as follows:

[0017] This matrix will be used to transform points in the image coordinate system to points in the camera coordinate system.

[0018] Preferably, the S1 calculates the external parameters of the radar according to the coordinates of the four tags on the target in the world coordinate system. , the calculation formula is as follows:

[0019] In the formula, represents the external parameters of the radar, represents the internal parameters of the camera, Represents the rotation and translation matrix between the image coordinate system and the camera coordinate system, Represents the label in the world coordinate system The three-dimensional coordinates of Represents the translation vector.

[0020] Preferably, the S1 is The decomposition method is used to obtain the rotation and translation matrix from the camera to the radar. The calculation formula is as follows:

[0021] In the formula, Represents the rotation and translation matrix from camera to radar, Indicates that Decompose the calculated rotation matrix, Represents the translation vector calculated using the rotation matrix and the center of mass.

[0022] Preferably, the step S4 performs denoising on the obtained two-dimensional image information, and the denoising formula is as follows:

[0023] In the formula, represents the pixel value of the two-dimensional image after denoising, Represents the original two-dimensional image pixel value, represents the neighborhood area, Indicate point and The Euclidean distance between represents the Gaussian spatial function, represents the intensity Gaussian weight function.

[0024] Preferably, the S6 processes the point cloud data through a visual program and performs a radius filtering operation on the point cloud. The radius filtering formula is as follows:

[0025] In the formula, Ly(i) represents the neighborhood, r represents the preset radius, (x j ,y j ,z j ) represents a point in the point cloud dataset.

[0027] Preferably, the point cloud image mapping sub-process description: The point cloud image mapping algorithm is the main algorithm for mapping point clouds to images to confirm the spatial position of objects. The specific description is as follows: (1) Selecting a target object in a two-dimensional image based on the image recognition result; (2) Input the original point cloud data, camera intrinsic parameter matrix, and camera-radar calibration matrix into the algorithm; (3) Map all radar point clouds to two-dimensional images based on the calibration data; (4) Calculate the ratio of the number of point cloud points mapped in the target object area in (1) to the number of pixel points in the target object mask. If the ratio exceeds a threshold, the object in the point cloud area is considered to be the target object identified in the image. (5) After two-dimensional image comparison, the confirmed mapped target object point cloud is reversely inferred to the original point cloud data based on the calibration data, and finally the position and pose of the target object is obtained in the original point cloud.

[0028] Compared with the prior art, the present invention provides a spatial positioning method of multimodal visual fusion, which has the following beneficial effects: The present invention realizes mutual calibration between laser radar, short-focus camera and long-focus camera through sensor joint calibration algorithm. The short-focus lens of the angular color camera collects a large range of stitched images for processing, thereby realizing the scanning, recording and recognition of the global scene. The telephoto lens of the angular color camera collects and processes the field of view information with high definition at a long distance, so as to realize high-precision recognition of distant target objects, process the above-mentioned two-dimensional image information, read the two-dimensional information obtained by the joint calibrated camera system to take the picture data in the power distribution operation scene, pre-process the image, and use the deep learning method to perform target detection on the pre-processed image to realize the recognition and positioning of the target object, use the laser radar to emit a laser beam to detect the target, collect the reflected beam to form a point cloud and obtain data, combine the visual program to process the point cloud data, and The image information obtained by the angular color camera is used for point cloud mapping, fitting reconstruction and position calculation. The above method improves the accuracy of spatial positioning in complex scenes of power operations and controls the position detection and recognition error within 10 the following. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a schematic diagram of the steps of the method of the present invention; Figure 2 Processing logic diagram for point cloud mapping. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] The current multi-sensor fusion positioning method is difficult to apply to more complex actual scenes and cannot meet the image recognition and spatial positioning requirements in power operation scenes, resulting in low positioning accuracy and large position detection and recognition errors. Therefore, a multi-modal visual fusion spatial positioning method is proposed. Figure 1 , the method comprises the following steps: S1. Through the sensor joint calibration algorithm, the laser radar, short-focus camera and long-focus camera are calibrated with each other, and the target is constructed. Labels are attached to the four corners of the target. Short-focus camera, The target image information collected by the telephoto camera and the laser radar is classified according to the timestamp, the feature points of the above-mentioned captured image and the calibration image are calculated, the feature points are matched, and the internal parameters of the camera are calculated. , the coordinates of the four tags on the target in the image coordinate system, the distance between the camera and the four tags, and the rotation and translation matrix between the image coordinate system and the camera coordinate system. The point with the smallest reprojection error after rotation and translation is selected as the external parameter result. At the same time, according to the target information collected by the lidar, the center point coordinates of the four tags are obtained by the fitting circle method. According to the coordinates of the four tags on the target in the world coordinate system, the external parameters of the radar are calculated. ,pass The solution obtains the rotation and translation matrix from camera to radar, completing the joint calibration algorithm of camera and radar; Among them, the camera's internal parameters The calculation formula is as follows:

[0032] Camera intrinsic parameters are important parameters that describe the internal optical properties of the camera. They are crucial to the performance and accuracy of computer vision and image processing applications. Intrinsic parameters can accurately describe the focal length, principal point position, and distortion characteristics of the camera, thereby improving the accuracy of 3D reconstruction and object measurement. Correct intrinsic parameters help to accurately convert image coordinates into real world coordinates. In the formula, represents the internal parameters of the camera, and Indicates that the camera is Axis and The focal length on axis, and Indicates the coordinates of the image center. In a multi-camera system, the intrinsic parameters of each camera need to be calibrated and synchronized to ensure that images from different perspectives can be correctly merged and parsed, thereby achieving higher-dimensional depth information acquisition; The coordinate calculation formula of the four labels on the target in the image coordinate system is as follows:

[0033] The four labels on the target can be used as reference points to help calibrate at specific locations in the image. These reference points can be used for subsequent measurements and analysis to ensure the accuracy and consistency of the process. In the formula, Represents the label in the image coordinate system The coordinates of Represents the horizontal pixel coordinate, Represents the vertical pixel coordinate, represents the internal parameters of the camera, Represents the label in the world coordinate system The added 1 is used to represent the homogeneous coordinates for matrix multiplication. By calculating the coordinates of the target label in the image coordinate system, the camera can be calibrated to obtain the internal and external parameters of the camera, which is crucial for the geometric transformation of the image, 3D reconstruction and other optical system analysis. The distance calculation formula from the camera to the four tags is as follows:

[0034] In object detection and recognition tasks, knowing the distance between the camera and the tag can enhance the accuracy of the recognition algorithm, especially at different distances, the appearance of the object may be significantly different. In the formula, Indicates the camera is at The distance between tags, the world coordinates of the tags are expressed as , , ,in , indicating four labels, , , Represents the 3D coordinates of the camera coordinate system, which can provide key data for vision-based real-time systems by calculating the distance between the camera and the tag; The calculation formula of the rotation and translation matrix between the image coordinate system and the camera coordinate system is as follows: Combining the rotation matrix and the translation vector, we can construct a 4*4 external parameter matrix, which is as follows:

[0035] The rotation matrix is ​​calculated by the least squares method. The commonly used rotation matrix is ​​expressed as follows:

[0036] Translation Vector Represents the position of the origin of the camera coordinate system in the world coordinate system, and its calculation is obtained by the following formula:

[0037] Once the rotation matrix and displacement vector are obtained, the point in unit homogeneous coordinate form is mapped from the image coordinate system to the camera coordinate system in the following way:

[0038] In summary, the rotation and translation matrix between the image coordinate system and the camera coordinate system is as follows:

[0039] This matrix will be used to convert points in the image coordinate system to points in the camera coordinate system. By calculating the rotation and translation matrices, different coordinate systems can be unified into a standard coordinate system, which is very important for processing data from different sensors and performing multi-view fusion tasks. At the same time, the calculation of the rotation and translation matrix is ​​an important part of camera calibration. Through calibration, the camera's shooting parameters can be accurately understood, thereby improving the accuracy of subsequent image processing; The calculation formula of the radar's external parameters is as follows:

[0040] After knowing the external parameters of the radar, it is easier to analyze the dynamic information such as the object's motion trajectory, speed and direction, which is extremely important for applications such as motion detection and trajectory prediction. In the formula, represents the external parameters of the radar, represents the internal parameters of the camera, Represents the rotation and translation matrix between the image coordinate system and the camera coordinate system, Represents the label in the world coordinate system The three-dimensional coordinates of Represents the translation vector, converting radar data into a common world coordinate system, which helps to achieve 3D reconstruction and understanding of the scene. Through accurate external parameters, different objects in the scene can be effectively identified and classified; The calculation formula of the rotation and translation matrix from camera to radar is as follows:

[0041] The rotation and translation matrix calculation from camera to radar is the basis for realizing sensor fusion and improving environmental perception. It can significantly improve the safety, efficiency and accuracy of the positioning system, making the system smarter and more responsive. By making full use of the advantages of different sensors, it can cope with more complex real-time environments and various application scenarios. In the formula, Represents the rotation and translation matrix from camera to radar, Indicates that Decompose the calculated rotation matrix, represents the translation vector calculated using the rotation matrix and the center of mass; S2, yes The short-focus lens of the angular color camera collects a large range of stitched images for processing, thereby realizing the scanning, recording and recognition of the global scene; The color camera acquisition technology can identify and measure the target object with high precision. The cameras collect image data separately, and the short-focus lens can obtain a wide field of view for scanning, recording and identifying the global scene; S3, yes The telephoto lens of the angular color camera collects high-definition field of view information at a long distance for processing, thereby achieving high-precision recognition of distant target objects; The telephoto lens has a long-distance, high-definition field of view and is used for high-precision recognition of distant target objects. S4, processing the two-dimensional image information obtained by the short-focus lens and the long-focus lens in S2 and S3 respectively, reading the two-dimensional information obtained by the image data of the power distribution operation scene taken by the jointly calibrated camera system, pre-processing the image, specifically including denoising, image enhancement, light intensity adjustment and other operations, performing target detection on the pre-processed image through a deep learning method, and realizing recognition and positioning of the target object; The obtained two-dimensional image information is denoised, and the denoising formula is as follows:

[0042] The denoised image can make the feature extraction process, such as edge detection and corner detection, more accurate and improve the effect of computer vision algorithms. In the formula, represents the pixel value of the two-dimensional image after denoising, Represents the original two-dimensional image pixel value, represents the neighborhood area, Indicate point and The Euclidean distance between represents the Gaussian spatial function, The intensity Gaussian weight function is shown. Denoising a two-dimensional image can not only improve the visual quality, but also enhance the accuracy and efficiency of image processing and analysis. In computer vision tasks, denoising is an important step to obtain high-quality results. S5, using laser radar to emit laser beams to detect targets, and collect reflected beams to form point clouds and obtain data; LiDAR acquisition technology uses non-contact laser ranging technology, which detects targets by emitting laser beams and collects reflected beams to form point clouds and obtain data; S6, process point cloud data through visual programs and combine The image information obtained by the angular color camera is used for point cloud mapping, fitting reconstruction and position calculation; The point cloud data is processed by the visual program, and the point cloud is subjected to radius filtering. The radius filtering formula is as follows:

[0043] Radius filtering can remove noise in low-density areas while retaining the key features and geometric shapes of high-density areas, helping to preserve the true outline of the object. In the formula, represents the neighborhood, Indicates the preset radius. Represents the points in the point cloud data set. Radius filtering can reduce the number of points in the point cloud, simplify data processing, and reduce the complexity of subsequent calculations, which is especially important when processing large-scale point clouds. Point cloud image mapping sub-process description: The point cloud image mapping algorithm is the main algorithm for mapping point clouds to images to confirm the spatial position of objects. The specific description is as follows: (1) Selecting a target object in a two-dimensional image based on the image recognition result; (2) Input the original point cloud data, camera intrinsic parameter matrix, and camera-radar calibration matrix into the algorithm; (3) Map all radar point clouds to two-dimensional images based on the calibration data; (4) Calculate the ratio of the number of point cloud points mapped in the target object area in (1) to the number of pixel points in the target object mask. If the ratio exceeds a threshold, the object in the point cloud area is considered to be the target object identified in the image. After two-dimensional image comparison, the confirmed mapped target object point cloud is reversely inferred to the original point cloud data according to the calibration data, and finally the position and pose of the target object is obtained in the original point cloud.

[0044] By using this method, the accuracy of spatial positioning in complex scenes of power operations is improved, and the detection and recognition error of the position is controlled within 10 the following.

[0045] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A spatial positioning method for multimodal visual fusion, characterized by: The following steps are involved: S1. Through the sensor joint calibration algorithm, the laser radar, short-focus camera and long-focus camera are calibrated with each other, and the target is constructed. Labels are attached to the four corners of the target. Short-focus camera, The target image information collected by the telephoto camera and the laser radar is classified according to the timestamp, the feature points of the above-mentioned captured image and the calibration image are calculated, the feature points are matched, and the internal parameters of the camera are calculated. , the coordinates of the four tags on the target in the image coordinate system, the distance between the camera and the four tags, and the rotation and translation matrix between the image coordinate system and the camera coordinate system. The point with the smallest reprojection error after rotation and translation is selected as the external parameter result. At the same time, according to the target information collected by the lidar, the center point coordinates of the four tags are obtained by the fitting circle method. According to the coordinates of the four tags on the target in the world coordinate system, the external parameters of the radar are calculated. ,pass The decomposition method is used to obtain the rotation and translation matrix from the camera to the radar, completing the joint calibration algorithm of the camera and the radar; S2, yes The short-focus lens of the angular color camera collects a large range of stitched images for processing, thereby realizing the scanning, recording and recognition of the global scene; S3, yes The telephoto lens of the angular color camera collects high-definition field of view information at a long distance for processing, thereby achieving high-precision recognition of distant target objects; S4, processing the two-dimensional image information obtained by the short-focus lens and the long-focus lens in S2 and S3 respectively, reading the two-dimensional information obtained by the image data of the power distribution operation scene taken by the jointly calibrated camera system, pre-processing the image, specifically including denoising, image enhancement, light intensity adjustment and other operations, performing target detection on the pre-processed image through a deep learning method, and realizing recognition and positioning of the target object; S5, using laser radar to emit laser beams to detect targets, and collect reflected beams to form point clouds and obtain data; S6, process point cloud data through visual programs and combine The image information obtained by the angular color camera is used for point cloud mapping, fitting reconstruction and position calculation.

2. The spatial positioning method of multimodal visual fusion according to claim 1, characterized in that: S1 calculates the camera's internal parameters based on the target image information collected by the telephoto camera and the laser radar , the calculation formula is as follows: In the formula, represents the internal parameters of the camera, and Indicates that the camera is Axis and The focal length on axis, and Represents the coordinates of the center of the image.

3. The spatial positioning method of multimodal visual fusion according to claim 2, characterized in that: The S1 calculates the coordinates of the four tags on the target in the image coordinate system according to the target image information collected by the telephoto camera and the laser radar. The calculation formula is as follows: In the formula, Represents the label in the image coordinate system The coordinates of Represents the horizontal pixel coordinate, Represents the vertical pixel coordinate, represents the internal parameters of the camera, Represents the label in the world coordinate system The three-dimensional coordinates of , the added 1 is used to indicate homogeneous coordinates for matrix multiplication.

4. The spatial positioning method of multimodal visual fusion according to claim 3, characterized in that: S1 calculates the distance from the camera to the four tags based on the target image information collected by the telephoto camera and the laser radar. The calculation formula is as follows: In the formula, Indicates the camera is at The distance between tags, the world coordinates of the tags are expressed as , , ,in , indicating four labels, , Represents the 3D coordinates in the camera coordinate system.

5. The spatial positioning method of multimodal visual fusion according to claim 4, characterized in that: The S1 calculates the rotation and translation matrix between the image coordinate system and the camera coordinate system according to the target image information collected by the telephoto camera and the laser radar. The calculation formula is as follows: Combining the rotation matrix and the translation vector, we can construct a 4*4 external parameter matrix, which is as follows: The rotation matrix is ​​calculated by the least squares method. The commonly used rotation matrix is ​​expressed as follows: Translation Vector Represents the position of the origin of the camera coordinate system in the world coordinate system, and its calculation is obtained by the following formula: Once the rotation matrix and displacement vector are obtained, the point in unit homogeneous coordinate form is mapped from the image coordinate system to the camera coordinate system in the following way: In summary, the rotation and translation matrix between the image coordinate system and the camera coordinate system is as follows: This matrix will be used to transform points in the image coordinate system to points in the camera coordinate system.

6. The spatial positioning method of multimodal visual fusion according to claim 5, characterized in that: S1 calculates the external parameters of the radar based on the coordinates of the four tags on the target in the world coordinate system. , the calculation formula is as follows: In the formula, represents the external parameters of the radar, represents the internal parameters of the camera, Represents the rotation and translation matrix between the image coordinate system and the camera coordinate system, Represents the label in the world coordinate system The three-dimensional coordinates of Represents the translation vector.

7. The spatial positioning method of multimodal visual fusion according to claim 6, characterized in that: The S1 is passed The decomposition method is used to obtain the rotation and translation matrix from the camera to the radar. The calculation formula is as follows: In the formula, Represents the rotation and translation matrix from camera to radar, Indicates that Decompose the calculated rotation matrix, Represents the translation vector calculated using the rotation matrix and the center of mass.

8. The spatial positioning method of multimodal visual fusion according to claim 1, characterized in that: The S4 performs denoising on the obtained two-dimensional image information, and the denoising formula is as follows: In the formula, represents the pixel value of the two-dimensional image after denoising, Represents the original two-dimensional image pixel value, represents the neighborhood area, Indicate point and The Euclidean distance between represents the Gaussian spatial function, represents the intensity Gaussian weight function.

9. The spatial positioning method of multimodal visual fusion according to claim 1, characterized in that: The S6 processes the point cloud data through a visual program and performs a radius filtering operation on the point cloud. The radius filtering formula is as follows: In the formula, represents the neighborhood, Indicates the preset radius. Represents a point in a point cloud dataset.

10. The spatial positioning method of multimodal visual fusion according to claim 1, characterized in that: The S6 point cloud image mapping sub-process description: The point cloud image mapping algorithm is the main algorithm for mapping point clouds to images to confirm the spatial position of objects. The specific description is as follows: (1) Selecting a target object in a two-dimensional image based on the image recognition result; (2) Input the original point cloud data, camera intrinsic parameter matrix, and camera-radar calibration matrix into the algorithm; (3) Map all radar point clouds to two-dimensional images based on the calibration data; (4) Calculate the ratio of the number of point cloud points mapped in the target object area in (1) to the number of target object mask pixels. If the ratio exceeds a threshold, the object in the point cloud area is considered to be the target object identified in the image. After two-dimensional image comparison, the confirmed mapped target object point cloud is reversely inferred to the original point cloud data according to the calibration data, and finally the position and pose of the target object is obtained in the original point cloud.

Images