Homogeneous multi-modal feature fusion and interaction three-dimensional object detection system and method
By fusing the data of camera and lidar in the three-dimensional object detection system, feature mapping, enhancement and fusion are solved, and the problem of insufficient accuracy and robustness of three-dimensional object detection in the prior art is achieved, and more efficient three-dimensional objects are achieved Test.
Patent Information
- Application Number
- CN202411321814.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-23
- Publication Date
- 2025-02-18
AI Technical Summary
The existing three-dimensional object detection technology relies on lidar, and its accuracy and robustness are insufficient, and feature information is difficult to be identified and utilized by detection algorithms, so the detection efficiency is low.
Using a three-dimensional object detection system that integrates and interacts homogeneous multimodal features, two-dimensional image data and three-dimensional point cloud data are collected through cameras and lidar, feature mapping, enhancement and fusion, and the feature fusion module is used to map point cloud features and images. The pixel features are fused to generate the final three-dimensional object data information.
It improves the accuracy and robustness of three-dimensional object detection, makes feature information easier to be recognized and utilized by detection algorithms, and improves detection efficiency.
Smart Images
Figure CN119478916A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of three-dimensional object detection, and in particular to a three-dimensional object detection system and method for homogeneous multi-modal feature fusion and interaction. Background Art
[0002] 3D object detection technology determines the shape, size and position of an object by collecting 3D coordinate information in space. It uses advanced measuring instruments and technologies, such as laser scanners, depth cameras, binocular stereo vision systems, etc., to perform non-contact 3D data collection on objects. Subsequently, the collected 3D data is processed through data processing and analysis software to generate a 3D model of the object, and further perform shape analysis, size measurement, position positioning and other operations. However, the existing 3D object detection technology still has shortcomings. First, the existing 3D object detection technology mainly relies on laser radar to provide 3D data of objects, which is not enough to ensure the accuracy and robustness of 3D object detection. Second, in the existing 3D object detection technology, the processed feature information is difficult to be recognized and utilized by the detection algorithm. Third, for scenarios where 3D object detection technology is frequently used, there are also high requirements for the detection efficiency of 3D objects. Therefore, it is necessary to design a 3D object detection system and method for homogeneous multimodal feature fusion and interaction. Summary of the invention
[0003] The purpose of the present invention is to provide a three-dimensional object detection system and method for homogeneous multi-modal feature fusion and interaction to solve the problems raised in the above background technology.
[0004] To achieve the above-mentioned purpose, the present invention provides the following technical solutions: a three-dimensional object detection system with homogeneous multimodal feature fusion and interaction, comprising a camera adjustment board, an image acquisition module is arranged on the camera adjustment board, the image acquisition module is controlled to be connected to an image processing module, the image processing module is controlled to be connected to an image output module, the image processing module and the image output module are arranged on the camera adjustment board, the image output module is controlled to be connected to an image receiving module, the image receiving module is arranged on a receiving adjustment board, a point cloud receiving module and a point cloud docking module are arranged on one side of the receiving adjustment board, the point cloud receiving module is controlled to be connected to the point cloud docking module, an image docking module is arranged on the other side of the receiving adjustment board, the point cloud receiving module is controlled to be connected to the point cloud output module, the point cloud output module is controlled to be connected to the point cloud processing module, the point cloud processing module is controlled to be connected to the point cloud acquisition module, and the point cloud acquisition module, the point cloud processing module and the point cloud output module are all arranged on the radar adjustment board.
[0005] As a further technical solution of the present invention, the image docking module controls the connection with an internal parameter acquisition module, the image docking module controls the connection with an external parameter acquisition module, the external parameter acquisition module controls the connection with a parameter extraction module, the parameter extraction module controls the connection with an internal parameter acquisition module, the parameter extraction module controls the connection with a depth acquisition module, the depth acquisition module controls the connection with a point cloud docking module, the point cloud docking module controls the connection with a point cloud analysis module, and the point cloud analysis module controls the connection with a grid construction module.
[0006] As a further technical solution of the present invention, the parameter extraction module is controlled and connected to a pixel conversion module, the pixel conversion module is controlled and connected to a three-dimensional corresponding module, the three-dimensional corresponding module is controlled and connected to a three-dimensional image module, the three-dimensional image module is arranged on a mapping adjustment board, an internal parameter acquisition module, an external parameter acquisition module, a depth acquisition module and a parameter extraction module are arranged on one side of the mapping adjustment board, and the other side of the mapping adjustment board is controlled and connected to a pixel conversion module, a three-dimensional corresponding module and a three-dimensional image module.
[0007] As a further technical solution of the present invention, the three-dimensional image module is controlled and connected to an image extraction module, the image extraction module is controlled and connected to an image allocation module, the image allocation module is controlled and connected to an image voxel module, the image voxel module is controlled and connected to a grid construction module, the grid construction module is arranged on a voxel adjustment board, a point cloud analysis module is arranged on one side of the voxel adjustment board, and an image extraction module, an image allocation module and an image voxel module are arranged on the other side of the voxel adjustment board.
[0008] As a further technical solution of the present invention, the image voxel module is controlled to be connected with a contrast module, the contrast module is controlled to be connected with an image sharpening module, the image sharpening module is controlled to be connected with an image denoising module, the image denoising module is controlled to be connected with an image fusion module, the image fusion module is arranged on an enhancement adjustment board, the contrast module, the image sharpening module and the image denoising module are arranged on one side of the enhancement adjustment board, the point cloud denoising module is controlled to be connected with the point cloud docking module, the point cloud denoising module is controlled to be connected with a downsampling module, the downsampling module is controlled to be connected with a point cloud extraction module, the point cloud extraction module is controlled to be connected with a point cloud fusion module, the point cloud fusion module, the point cloud extraction module and the downsampling module are arranged on the enhancement adjustment board.
[0009] As a further technical solution of the present invention, the image fusion module is controlled to be connected to the feature fusion module, the feature fusion module is controlled to be connected to the point cloud fusion module, the feature fusion module is controlled to be connected to the detection output module, the feature fusion module is controlled to be connected to the feature storage module, the feature storage module is controlled to be connected to the feature recognition module, the feature recognition module is controlled to be connected to the feature classification module, the feature classification module is controlled to be connected to the feature learning module, the feature learning module is arranged on the learning adjustment board, and the feature storage module, the feature recognition module and the feature classification module are arranged on the learning adjustment board.
[0010] A three-dimensional object detection method for homogeneous multi-modal feature fusion and interaction includes the following steps: step one, acquisition mapping; step two, enhanced fusion; step three, recognition learning; In the above step 1, the three-dimensional point cloud data and the two-dimensional image data of the object are extracted respectively, and the two-dimensional image features are mapped in the three-dimensional space; In the above step 2, the point cloud features and the image voxel features are enhanced respectively, and fusion detection is performed; In the above step three, the features are stored and recognition learning is performed.
[0011] As a further technical solution of the present invention, in the step one, the three-dimensional point cloud information of the object is collected by the point cloud acquisition module, and then the point cloud information is preliminarily processed by the point cloud processing module and then transmitted to the point cloud output module, the point cloud output module transmits the point cloud data to the point cloud receiving module, and the point cloud receiving module transmits the point cloud data to the point cloud docking module, the image of the object is collected by the image acquisition module, and the collected image is transmitted to the image processing module, the image processing module preliminarily processes the image, and calibrates the internal and external parameters of the camera during image acquisition, and then transmits it to the image output module, the image output module transmits the image to the image receiving module, the image information is transmitted to the image docking module through the image receiving module, the data information in the image docking module is extracted by the internal parameter acquisition module and the external parameter acquisition module, the data information in the point cloud docking module is extracted by the depth acquisition module, and the internal parameter acquisition module and the external parameter acquisition module are calibrated by the parameter extraction module. The data information extracted from the depth acquisition module continues to be collected and integrated, and is transmitted to the pixel conversion module. The pixel conversion module converts the two-dimensional data of the image into pixel points, and transmits them to the three-dimensional corresponding module. The three-dimensional corresponding module corresponds the converted pixel points to the position in the three-dimensional space, and the three-dimensional image module integrates the image features corresponding to the three-dimensional space. The point cloud docking module transmits the object point cloud features to the point cloud analysis module. The point cloud analysis module analyzes the point cloud features of the object to obtain the distribution and density of the point cloud, and transmits the obtained data to the grid construction module. The grid construction module constructs a voxel grid according to the input data, and the image extraction module extracts the three-dimensional image features in the three-dimensional image module, and transmits the extracted three-dimensional image features to the image allocation module. The three-dimensional image features are allocated to the constructed voxel grid according to the position information through the image allocation module and the image voxel module, so as to obtain homogeneous image voxel features.
[0012] As a further technical solution of the present invention, in the step 2, the image voxel features are transmitted to the contrast module through the image voxel module, the contrast features in the image voxel features are enhanced through the contrast module, and then transmitted to the image sharpening module, the image voxel features are sharpened through the image sharpening module, and then transmitted to the image denoising module, the noise points in the image voxel features are removed through the image denoising module, and the processed image voxel features are transmitted to the image fusion module; at the same time, the object point cloud features are transmitted to the point cloud denoising module through the point cloud docking module, the noise points in the point cloud features are removed through the point cloud denoising module, and then transmitted to the downsampling module, the number of data points in the point cloud is reduced through the downsampling module, the key features are retained, and then transmitted to the point cloud extraction module, the geometric features and statistical features in the point cloud features are extracted through the point cloud extraction module, the processed point cloud features are transmitted to the point cloud fusion module, the point cloud features in the point cloud fusion module and the image pixel features in the image fusion module are transmitted to the feature fusion module, the two are fused through the feature fusion module to obtain the final three-dimensional object data information, and it is transmitted to the detection output module for output.
[0013] As a further technical solution of the present invention, in the step three, the fused three-dimensional object feature data is transmitted to the feature storage module through the feature fusion module, the features in the feature storage module are identified through the feature recognition module, and then the identified features are classified through the feature classification module, and the features of different classifications are learned through the feature learning module for subsequent rapid processing of similar features.
[0014] Compared with the prior art, the beneficial effects of the present invention are as follows: the homogeneous multimodal feature fusion and interactive three-dimensional object detection system and method transmits both the point cloud features in the point cloud fusion module and the image pixel features in the image fusion module to the feature fusion module, fuses the two through the feature fusion module, obtains the final three-dimensional object data information, and transmits it to the detection output module for output, and improves the accuracy and robustness of three-dimensional object detection by fusing data from lidar and camera and utilizing their complementary advantages; enhances the contrast features in the image voxel features through the contrast module, and then transmits them to the image sharpening module, sharpens the image voxel features through the image sharpening module, and then transmits them to the image denoising module, removes the noise in the image voxel features through the image denoising module, and transmits the processed image voxel features to the image fusion module; at the same time, through the point cloud docking module The block transmits the object point cloud features to the point cloud denoising module, removes the noise in the point cloud features through the point cloud denoising module, and then transmits it to the downsampling module, reduces the number of data points in the point cloud through the downsampling module, retains the key features, and then transmits it to the point cloud extraction module, extracts the geometric features and statistical features in the point cloud features through the point cloud extraction module, and transmits the processed point cloud features to the point cloud fusion module, which enhances the image voxel features and point cloud features to make them easier to be recognized and used by the detection algorithm; the fused three-dimensional object feature data is transmitted to the feature storage module through the feature fusion module, the features in the feature storage module are recognized through the feature recognition module, and then the recognized features are classified through the feature classification module, and the features of different classifications are learned through the feature learning module for subsequent rapid processing of similar features, which can speed up the detection efficiency of three-dimensional objects after a period of use. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] Figure 1 It is a schematic diagram of the overall structure of the present invention; Figure 2 This is a module flow chart on the camera adjustment board of the present invention; Figure 3 It is a module flow chart on the radar adjustment board of the present invention; Figure 4 It is a module flow chart on the receiving and regulating board in the present invention; Figure 5 It is a flowchart of the modules on the mapping adjustment board in the present invention; Figure 6 It is a module flow chart on the voxel adjustment board in the present invention; Figure 7 It is a flow chart of modules on the enhanced adjustment board in the present invention; Figure 8 It is a partial flow chart of the present invention; Fig. 9A flow chart of the modules on the learning and regulating board in the present invention; Fig.10 is a flow chart of the method of the present invention; In the figure: 1. Camera adjustment board; 2. Receiving adjustment board; 3. Mapping adjustment board; 4. Voxel adjustment board; 5. Enhancement adjustment board; 6. Feature fusion module; 7. Detection output module; 8. Learning adjustment board; 9. Radar adjustment board; 11. Image acquisition module; 12. Image processing module; 13. Image output module; 21. Image receiving module; 22. Point cloud receiving module; 23. Point cloud docking module; 24. Image docking module; 31. Internal reference acquisition module; 32. External reference acquisition module; 33. Depth acquisition module; 34. Parameter extraction module; 35. Pixel conversion module; 36. Three-dimensional correspondence module; 37. Three-dimensional image module; 41. Point cloud analysis module; 42. Mesh construction module; 43. Image extraction module; 44. Image allocation module; 45. Image voxel module; 51. Contrast module; 52. Image sharpening module; 53. Image denoising module; 54. Image fusion module; 55. Point cloud denoising module; 56. Downsampling module; 57. Point cloud extraction module; 58. Point cloud fusion module; 81. Feature storage module; 82. Feature recognition module; 83. Feature classification module; 84. Feature learning module; 91. Point cloud acquisition module; 92. Point cloud processing module; 93. Point cloud output module. DETAILED DESCRIPTION
[0016] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0017] Please see attached Figure 1 -Attached Fig. 9An embodiment of the present invention provides a three-dimensional object detection system with homogeneous multimodal feature fusion and interaction, including a camera adjustment board 1, an image acquisition module 11 is arranged on the camera adjustment board 1, the image acquisition module 11 is controlled to be connected to an image processing module 12, the image processing module 12 is controlled to be connected to an image output module 13, the image processing module 12 and the image output module 13 are arranged on the camera adjustment board 1, the image output module 13 is controlled to be connected to an image receiving module 21, the image receiving module 21 is arranged on a receiving adjustment board 2, a point cloud receiving module 22 and a point cloud docking module 23 are arranged on one side of the receiving adjustment board 2, the point cloud receiving module 22 is controlled to be connected to the point cloud docking module 23, and an image docking module 23 is arranged on the other side of the receiving adjustment board 2. The point cloud receiving module 22 is controlled to be connected to the point cloud output module 93, the point cloud output module 93 is controlled to be connected to the point cloud processing module 92, the point cloud processing module 92 is controlled to be connected to the point cloud acquisition module 91, the point cloud acquisition module 91, the point cloud processing module 92 and the point cloud output module 93 are all arranged on the radar adjustment board 9, the image docking module 24 is controlled to be connected to the internal reference acquisition module 31, the image docking module 24 is controlled to be connected to the external reference acquisition module 32, the external reference acquisition module 32 is controlled to be connected to the parameter extraction module 34, the parameter extraction module 34 is controlled to be connected to the internal reference acquisition module 31, the parameter extraction module 34 is controlled to be connected to the depth acquisition module 33, the depth acquisition module 33 is controlled to be connected to the point cloud docking module 23, and the point cloud acquisition module 91 is controlled to be connected to the point cloud acquisition module 91. The cloud docking module 23 controls the connection with the point cloud analysis module 41, the point cloud analysis module 41 controls the connection with the grid construction module 42, the parameter extraction module 34 controls the connection with the pixel conversion module 35, the pixel conversion module 35 controls the connection with the three-dimensional correspondence module 36, the three-dimensional correspondence module 36 controls the connection with the three-dimensional image module 37, the three-dimensional image module 37 is arranged on the mapping adjustment board 3, one side of the mapping adjustment board 3 is arranged with the internal reference acquisition module 31, the external reference acquisition module 32, the depth acquisition module 33 and the parameter extraction module 34, the other side of the mapping adjustment board 3 controls the connection with the pixel conversion module 35, the three-dimensional correspondence module 36 and the three-dimensional image module 37, the three-dimensional image module 37 controls the connection with the image extraction module 43, the image extraction module 43 is controlled to be connected with an image allocation module 44, the image allocation module 44 is controlled to be connected with an image voxel module 45, the image voxel module 45 is controlled to be connected with a grid construction module 42, the grid construction module 42 is arranged on the voxel adjustment board 4, a point cloud analysis module 41 is arranged on one side of the voxel adjustment board 4, the image extraction module 43, the image allocation module 44 and the image voxel module 45 are arranged on the other side of the voxel adjustment board 4, the image voxel module 45 is controlled to be connected with a contrast module 51, the contrast module 51 is controlled to be connected with an image sharpening module 52, the image sharpening module 52 is controlled to be connected with an image denoising module 53, the image denoising module 53 is controlled to be connected with an image fusion module 54, the image fusion module 54 is arranged on the enhancement adjustment board 5,A contrast module 51, an image sharpening module 52 and an image denoising module 53 are arranged on one side of the enhancement adjustment board 5, and a point cloud denoising module 55 is arranged on the other side of the enhancement adjustment board 5. The point cloud denoising module 55 is controlled to be connected to the point cloud docking module 23, and the point cloud denoising module 55 is controlled to be connected to a downsampling module 56, and the downsampling module 56 is controlled to be connected to a point cloud extraction module 57, and the point cloud extraction module 57 is controlled to be connected to a point cloud fusion module 58. The point cloud fusion module 58, the point cloud extraction module 57 and the downsampling module 56 are arranged on the enhancement adjustment board 5, and the image fusion module 54 is controlled to be connected to the point cloud docking module 23. The feature fusion module 6 is connected to the feature fusion module 6, which controls the connection with the point cloud fusion module 58, the feature fusion module 6 controls the connection with the detection output module 7, the feature fusion module 6 controls the connection with the feature storage module 81, the feature storage module 81 controls the connection with the feature recognition module 82, the feature recognition module 82 controls the connection with the feature classification module 83, the feature classification module 83 controls the connection with the feature learning module 84, the feature learning module 84 is arranged on the learning adjustment board 8, and the learning adjustment board 8 is provided with the feature storage module 81, the feature recognition module 82 and the feature classification module 83.
[0018] Please see attached Fig.10 , an embodiment provided by the present invention: a three-dimensional object detection method with homogeneous multi-modal feature fusion and interaction, comprising the following steps: step one, acquisition mapping; step two, enhanced fusion; step three, recognition learning; In the above step 1, the three-dimensional point cloud data and the two-dimensional image data of the object are extracted respectively, and the two-dimensional image features are mapped in the three-dimensional space; the three-dimensional point cloud information of the object is collected by the point cloud collection module 91, and then the point cloud information is preliminarily processed by the point cloud processing module 92 and transmitted to the point cloud output module 93, and the point cloud output module 93 transmits the point cloud data to the point cloud receiving module 22, and the point cloud receiving module 22 transmits the point cloud data to the point cloud docking module 23, and the image of the object is collected by the image collection module 11, and the collected image is transmitted to the point cloud output module 93. The image is input to the image processing module 12, which performs preliminary processing on the image and calibrates the internal and external parameters of the camera during image acquisition, and then transmits it to the image output module 13, which transmits the image to the image receiving module 21, and the image information is transmitted to the image docking module 24 through the image receiving module 21. The data information in the image docking module 24 is extracted through the internal parameter acquisition module 31 and the external parameter acquisition module 32, and the data information in the point cloud docking module 23 is extracted through the depth acquisition module 33. 34 continues to collect and integrate the data information extracted from the internal reference acquisition module 31, the external reference acquisition module 32 and the depth acquisition module 33, and transmits them to the pixel conversion module 35. The pixel conversion module 35 converts the two-dimensional data of the image into pixel points, and transmits them to the three-dimensional correspondence module 36. The three-dimensional correspondence module 36 corresponds the converted pixel points to the position in the three-dimensional space, and the three-dimensional image module 37 integrates the image features corresponding to the three-dimensional space, and transmits the object point cloud features to the point cloud analysis module 41 through the point cloud docking module 23. The point cloud analysis module 41 analyzes the point cloud features of the object to obtain the distribution and density of the point cloud, and transmits the obtained data to the grid construction module 42. The grid construction module 42 constructs a voxel grid according to the input data, and at the same time, the image extraction module 43 extracts the three-dimensional image features in the three-dimensional image module 37, and transmits the extracted three-dimensional image features to the image allocation module 44. The three-dimensional image features are allocated to the constructed voxel grid according to the position information through the image allocation module 44 and the image voxel module 45, so as to obtain homogeneous image voxel features; In the above step 2, the point cloud features and the image voxel features are enhanced respectively, and fusion detection is performed; the image voxel features are transmitted to the contrast module 51 through the image voxel module 45, the contrast features in the image voxel features are enhanced through the contrast module 51, and then transmitted to the image sharpening module 52, the image voxel features are sharpened through the image sharpening module 52, and then transmitted to the image denoising module 53, the noise points in the image voxel features are removed through the image denoising module 53, and the processed image voxel features are transmitted to the image fusion module 54; at the same time, the object point cloud features are transmitted to the point cloud denoising module 55 through the point cloud docking module 23 , the noise points in the point cloud features are removed by the point cloud denoising module 55, and then transmitted to the downsampling module 56, the number of data points in the point cloud is reduced by the downsampling module 56, the key features are retained, and then transmitted to the point cloud extraction module 57, the geometric features and statistical features in the point cloud features are extracted by the point cloud extraction module 57, the processed point cloud features are transmitted to the point cloud fusion module 58, the point cloud features in the point cloud fusion module 58 and the image pixel features in the image fusion module 54 are transmitted to the feature fusion module 6, the two are fused by the feature fusion module 6, the final three-dimensional object data information is obtained, and it is transmitted to the detection output module 7 for output; In the above step three, the features are stored and recognized and learned; the fused three-dimensional object feature data is transmitted to the feature storage module 81 through the feature fusion module 6, the features in the feature storage module 81 are recognized by the feature recognition module 82, and then the recognized features are classified by the feature classification module 83, and the features of different classifications are learned by the feature learning module 84 for the subsequent rapid processing of similar features.
[0019] Based on the above, the advantage of the present invention is that when the present invention is in use, the point cloud features in the point cloud fusion module 58 and the image pixel features in the image fusion module 54 are transmitted to the feature fusion module 6, and the two are fused through the feature fusion module 6 to obtain the final three-dimensional object data information, and it is transmitted to the detection output module 7 for output, by fusing the data from the lidar and the camera and utilizing their complementary advantages, the accuracy and robustness of three-dimensional object detection are improved; the contrast features in the image voxel features are enhanced through the contrast module 51, and then transmitted to the image sharpening module 52, and the image voxel features are sharpened through the image sharpening module 52, and then transmitted to the image denoising module 53, and the noise points in the image voxel features are removed through the image denoising module 53, and the processed image voxel features are transmitted to the image fusion module 54; at the same time, the object point cloud features are transmitted to the point cloud through the point cloud docking module 23 In the denoising module 55, the noise points in the point cloud features are removed by the point cloud denoising module 55, and then transmitted to the downsampling module 56. The number of data points in the point cloud is reduced by the downsampling module 56, and the key features are retained. Then, the point cloud is transmitted to the point cloud extraction module 57. The geometric features and statistical features in the point cloud features are extracted by the point cloud extraction module 57, and the processed point cloud features are transmitted to the point cloud fusion module 58. By enhancing the image voxel features and point cloud features, it is easier to be recognized and used by the detection algorithm; the fused three-dimensional object feature data is transmitted to the feature storage module 81 through the feature fusion module 6, and the features in the feature storage module 81 are recognized by the feature recognition module 82, and then the recognized features are classified by the feature classification module 83, and the features of different classifications are learned by the feature learning module 84 for subsequent rapid processing of similar features, which can speed up the detection efficiency of three-dimensional objects after a period of use.
[0020] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.
Claims
1. A three-dimensional object detection system with homogeneous multimodal feature fusion and interaction, comprising a camera adjustment board (1), characterized in that: The camera adjustment board (1) is provided with an image acquisition module (11), the image acquisition module (11) is controlled to be connected to an image processing module (12), the image processing module (12) is controlled to be connected to an image output module (13), the image processing module (12) and the image output module (13) are arranged on the camera adjustment board (1), the image output module (13) is controlled to be connected to an image receiving module (21), the image receiving module (21) is arranged on a receiving adjustment board (2), and a point cloud receiving module (22) and a point cloud receiving module (21) are arranged on one side of the receiving adjustment board (2). A docking module (23), the point cloud receiving module (22) is controlled to be connected to the point cloud docking module (23), an image docking module (24) is arranged on the other side of the receiving adjustment board (2), the point cloud receiving module (22) is controlled to be connected to the point cloud output module (93), the point cloud output module (93) is controlled to be connected to the point cloud processing module (92), the point cloud processing module (92) is controlled to be connected to the point cloud acquisition module (91), and the point cloud acquisition module (91), the point cloud processing module (92) and the point cloud output module (93) are all arranged on the radar adjustment board (9).
2. The homogeneous multimodal feature fusion and interactive three-dimensional object detection system according to claim 1, characterized in that: The image docking module (24) is controlled to be connected to an internal reference acquisition module (31), the image docking module (24) is controlled to be connected to an external reference acquisition module (32), the external reference acquisition module (32) is controlled to be connected to a parameter extraction module (34), the parameter extraction module (34) is controlled to be connected to an internal reference acquisition module (31), the parameter extraction module (34) is controlled to be connected to a depth acquisition module (33), the depth acquisition module (33) is controlled to be connected to a point cloud docking module (23), the point cloud docking module (23) is controlled to be connected to a point cloud analysis module (41), and the point cloud analysis module (41) is controlled to be connected to a grid construction module (42).
3. The homogeneous multi-modal feature fusion and interactive three-dimensional object detection system according to claim 2, characterized in that: The parameter extraction module (34) is controlled to be connected to a pixel conversion module (35), the pixel conversion module (35) is controlled to be connected to a three-dimensional correspondence module (36), the three-dimensional correspondence module (36) is controlled to be connected to a three-dimensional image module (37), the three-dimensional image module (37) is arranged on a mapping adjustment board (3), an internal reference acquisition module (31), an external reference acquisition module (32), a depth acquisition module (33) and a parameter extraction module (34) are arranged on one side of the mapping adjustment board (3), and the pixel conversion module (35), the three-dimensional correspondence module (36) and the three-dimensional image module (37) are controlled to be connected to the other side of the mapping adjustment board (3).
4. The homogeneous multi-modal feature fusion and interactive three-dimensional object detection system according to claim 3, characterized in that: The three-dimensional image module (37) is controlled to be connected to an image extraction module (43), the image extraction module (43) is controlled to be connected to an image allocation module (44), the image allocation module (44) is controlled to be connected to an image voxel module (45), the image voxel module (45) is controlled to be connected to a grid construction module (42), the grid construction module (42) is arranged on a voxel adjustment board (4), a point cloud analysis module (41) is arranged on one side of the voxel adjustment board (4), and the image extraction module (43), the image allocation module (44) and the image voxel module (45) are arranged on the other side of the voxel adjustment board (4).
5. The homogeneous multi-modal feature fusion and interactive three-dimensional object detection system according to claim 4, characterized in that: The image voxel module (45) is controlled to be connected to a contrast module (51), the contrast module (51) is controlled to be connected to an image sharpening module (52), the image sharpening module (52) is controlled to be connected to an image denoising module (53), the image denoising module (53) is controlled to be connected to an image fusion module (54), the image fusion module (54) is arranged on an enhancement adjustment board (5), and the contrast module (51), the image sharpening module (52) and the image denoising module (53) are arranged on one side of the enhancement adjustment board (5). A point cloud denoising module (55) is arranged on the other side of the enhancement adjustment board (5); the point cloud denoising module (55) is control-connected to the point cloud docking module (23); the point cloud denoising module (55) is control-connected to a downsampling module (56); the downsampling module (56) is control-connected to a point cloud extraction module (57); the point cloud extraction module (57) is control-connected to a point cloud fusion module (58); the point cloud fusion module (58), the point cloud extraction module (57) and the downsampling module (56) are arranged on the enhancement adjustment board (5).
6. The homogeneous multi-modal feature fusion and interactive three-dimensional object detection system according to claim 5, characterized in that: The image fusion module (54) is controlled to be connected to the feature fusion module (6), the feature fusion module (6) is controlled to be connected to the point cloud fusion module (58), the feature fusion module (6) is controlled to be connected to the detection output module (7), the feature fusion module (6) is controlled to be connected to the feature storage module (81), the feature storage module (81) is controlled to be connected to the feature recognition module (82), the feature recognition module (82) is controlled to be connected to the feature classification module (83), the feature classification module (83) is controlled to be connected to the feature learning module (84), the feature learning module (84) is arranged on the learning adjustment board (8), and the learning adjustment board (8) is arranged with the feature storage module (81), the feature recognition module (82) and the feature classification module (83).
7. A three-dimensional object detection method based on homogeneous multimodal feature fusion and interaction, comprising the following steps: Step 1: acquisition and mapping; Step 2: enhanced fusion; Step 3: recognition and learning; the characteristics are: In the above step 1, the three-dimensional point cloud data and the two-dimensional image data of the object are extracted respectively, and the two-dimensional image features are mapped in the three-dimensional space; In the above step 2, the point cloud features and the image voxel features are enhanced respectively, and fusion detection is performed; In the above step three, the features are stored and recognition learning is performed.
8. The three-dimensional object detection method of homogeneous multimodal feature fusion and interaction according to claim 7, characterized in that: In the step 1, the three-dimensional point cloud information of the object is collected by the point cloud collection module (91), and then the point cloud information is preliminarily processed by the point cloud processing module (92) and then transmitted to the point cloud output module (93), the point cloud output module (93) transmits the point cloud data to the point cloud receiving module (22), and the point cloud receiving module (22) transmits the point cloud data to the point cloud docking module (23), the image of the object is collected by the image collection module (11), and the collected image is transmitted to the image processing module (12), and the image processing module (12) performs preliminarily processing on the image , and calibrate the internal and external parameters of the camera during image acquisition, and then transmit them to the image output module (13), the image output module (13) transmits the image to the image receiving module (21), the image information is transmitted to the image docking module (24) through the image receiving module (21), the data information in the image docking module (24) is extracted through the internal parameter acquisition module (31) and the external parameter acquisition module (32), the data information in the point cloud docking module (23) is extracted through the depth acquisition module (33), the internal parameter acquisition module (31) and the external parameter acquisition module (32) are transmitted to the image receiving module (21), and the image information is transmitted to the image docking module (24) through the parameter extraction module (34). The data information extracted from the acquisition module (32) and the depth acquisition module (33) is further collected and integrated, and transmitted to the pixel conversion module (35). The pixel conversion module (35) converts the two-dimensional data of the image into pixel points, and transmits them to the three-dimensional correspondence module (36). The three-dimensional correspondence module (36) corresponds the converted pixel points to the positions in the three-dimensional space, and integrates the image features corresponding to the three-dimensional space through the three-dimensional image module (37). The point cloud features of the object are transmitted to the point cloud analysis module (41) through the point cloud docking module (23). 1) Analyze the point cloud features of the object to obtain the distribution and density of the point cloud, and transmit the obtained data to the grid construction module (42). The grid construction module (42) constructs a voxel grid according to the input data. At the same time, the image extraction module (43) extracts the three-dimensional image features in the three-dimensional image module (37), and transmits the extracted three-dimensional image features to the image allocation module (44). The three-dimensional image features are allocated to the constructed voxel grid according to the position information through the image allocation module (44) and the image voxel module (45), so as to obtain homogeneous image voxel features.
9. The three-dimensional object detection method of homogeneous multimodal feature fusion and interaction according to claim 7, characterized in that: In the second step, the image voxel features are transmitted to the contrast module (51) through the image voxel module (45), the contrast features in the image voxel features are enhanced through the contrast module (51), and then transmitted to the image sharpening module (52), the image voxel features are sharpened through the image sharpening module (52), and then transmitted to the image denoising module (53), the noise points in the image voxel features are removed through the image denoising module (53), and the processed image voxel features are transmitted to the image fusion module (54); at the same time, the object point cloud features are transmitted to the point cloud denoising module (55) through the point cloud docking module (23), and the point cloud denoising module (55) removes the point cloud noise. The noise points in the cloud features are then transmitted to the downsampling module (56), which reduces the number of data points in the point cloud and retains the key features. The noise points are then transmitted to the point cloud extraction module (57), which extracts the geometric features and statistical features in the point cloud features. The processed point cloud features are transmitted to the point cloud fusion module (58), and the point cloud features in the point cloud fusion module (58) and the image pixel features in the image fusion module (54) are transmitted to the feature fusion module (6). The two are fused through the feature fusion module (6) to obtain the final three-dimensional object data information, which is then transmitted to the detection output module (7) for output.
10. The three-dimensional object detection method of homogeneous multi-modal feature fusion and interaction according to claim 7, characterized in that: In the step three, the fused three-dimensional object feature data is transmitted to the feature storage module (81) through the feature fusion module (6), the features in the feature storage module (81) are identified through the feature recognition module (82), and then the identified features are classified through the feature classification module (83), and the features of different classifications are learned through the feature learning module (84) for the subsequent rapid processing of similar features.