Multi-modal three-dimensional target detection method and system
By collecting camera multi-view RGB images in real time, acquiring lidar point clouds, and performing feature fusion processing, the problem of poor modal information fusion in the existing technology is solved, and the precise detection and detection efficiency of small targets is improved.
Patent Information
- Application Number
- CN202411416275.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-11
- Publication Date
- 2025-02-28
AI Technical Summary
The prior art is difficult to effectively integrate the modal information generated by cameras and lidars, resulting in poor detection effects on small targets and reducing detection efficiency.
The preset camera collects multi-view RGB images in real time, acquires the lidar point cloud, and inputs it into the corresponding feature encoder to generate the original image features and point cloud features. Then, the target image features and point cloud features are fused through the feature fusion module to generate BEV spatial features, and finally encode and output object information.
Accurate detection of various targets is achieved, detection efficiency is improved, and modal information of cameras and lidar can be effectively integrated.
Smart Images
Figure CN119540938A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of radar technology, and in particular to a multi-modal three-dimensional target detection method and system. Background Art
[0002] With the advancement of science and technology and the rapid development of productivity, radar technology has also developed rapidly. Among them, most of the existing radars can realize the function of three-dimensional target detection to obtain the position, category and geometric information of dynamic and static targets in the environment, so as to enable autonomous driving vehicles to accurately identify the driving environment.
[0003] Among them, lidar and camera, as the two core sensors in the autonomous driving perception system, each uses different methods to capture the surrounding environment information. Specifically, the existing lidar will emit a light beam of a specific wavelength and the corresponding receiver will reflect the signal to generate corresponding point cloud data, so as to locate the corresponding target object.
[0004] Furthermore, when existing cameras or lidars face smaller targets, the corresponding modal information generated by the two cannot be effectively fused together, resulting in poor detection of small targets, thereby correspondingly reducing detection efficiency. Summary of the invention
[0005] Based on this, the purpose of the present invention is to provide a multimodal three-dimensional target detection method and system to solve the problem that the prior art cannot effectively fuse the modal information generated by the camera and the lidar, resulting in poor detection effect on small targets.
[0006] The first aspect of the embodiment of the present invention proposes: A multimodal three-dimensional target detection method, wherein the method comprises: The multi-view RGB images of the target object are collected in real time by a preset camera, and the multi-view RGB images are input into a preset image feature encoder to output corresponding original image features; Acquire a laser radar point cloud corresponding to the laser radar in real time, and input the laser radar point cloud into a preset laser radar feature encoder to output corresponding point cloud features; The original image features are converted to generate corresponding target image features, and the target image features and the point cloud features are simultaneously input into a preset feature fusion module to output corresponding BEV space features; The BEV spatial features are encoded to output object information corresponding to the target object in real time.
[0007] The beneficial effect of the present invention is that by collecting multi-view RGB pictures by the camera in real time, the required original image features can be obtained accordingly. Similarly, by acquiring the lidar point cloud in real time, the required point cloud features can be obtained accordingly. Based on this, the preset feature fusion module can effectively fuse the two at present to output the required BEV spatial features, and finally perform corresponding encoding to output the required object information, thereby effectively and accurately detecting various targets and improving the detection efficiency.
[0008] Furthermore, the step of converting the original image features to generate corresponding target image features includes: When the original image features are acquired in real time, the camera angle of view corresponding to the preset camera is detected in real time; When the camera viewing angle is acquired in real time, a preset BEV space is correspondingly called out in a preset program, and a BEV viewing angle corresponding to the preset BEV space is correspondingly detected; The conversion process of the original image features is completed according to the camera viewing angle and the BEV viewing angle to generate the target image features accordingly.
[0009] Furthermore, the step of completing the conversion processing of the original image features according to the camera perspective and the BEV perspective to generate the target image features accordingly includes: When the camera perspective and the BEV perspective are respectively acquired, emitting corresponding camera light through the camera perspective to the BEV perspective; Detecting in real time a number of characteristic pixels corresponding to the interior of the camera light, and correspondingly detecting a number of discrete points corresponding to the BEV viewing angle; The target image features are correspondingly generated according to the plurality of feature pixels and the plurality of discrete points.
[0010] Furthermore, the step of correspondingly generating the target image features according to the plurality of characteristic pixels and the plurality of discrete points includes: When a plurality of the characteristic pixels are acquired respectively, a target characteristic value corresponding to each of the characteristic pixels is detected in real time; When a plurality of the discrete points are respectively acquired, a discrete interval corresponding to each of the discrete points is detected in real time, and a target threshold corresponding to the discrete interval is detected in real time; Each of the target feature values is mapped to a target threshold corresponding thereto, so as to generate the target image feature accordingly.
[0011] Furthermore, the step of simultaneously inputting the target image features and the point cloud features into a preset feature fusion module to output corresponding BEV spatial features includes: When the target image features and the point cloud features are respectively acquired, the corresponding feature selector is called out in real time in the preset feature fusion module; The feature selector outputs the target feature ratio that matches the target image feature and the point cloud feature in real time, and completes the fusion processing of the target image feature and the point cloud feature based on the target feature ratio to generate the BEV spatial feature accordingly.
[0012] Furthermore, the step of completing the fusion processing of the target image feature and the point cloud feature based on the target feature ratio to generate the BEV space feature accordingly includes: When the target feature ratio is obtained in real time, the corresponding target convolution layer is called out in the preset feature fusion module; Detecting original convolution parameters contained in the target convolution layer in real time, and adaptively adjusting the original convolution parameters according to the target feature ratio to generate corresponding target convolution parameters; The target image features and the point cloud features are fused based on the target convolution parameters to generate the BEV spatial features accordingly.
[0013] Furthermore, the step of fusing the target image features and the point cloud features based on the target convolution parameters to generate the BEV spatial features accordingly includes: When the target convolution parameters are obtained in real time, the adapted initial neural network is called out in real time; The target convolution parameters and the initial neural network are fused to generate a corresponding target neural network, and the target image features and the point cloud features are fused through the target neural network to generate the BEV spatial features accordingly.
[0014] The second aspect of the embodiment of the present invention proposes: A multimodal three-dimensional target detection system, wherein the system comprises: An acquisition module is used to acquire multi-view RGB images of a target object in real time through a preset camera, and input the multi-view RGB images into a preset image feature encoder to output corresponding original image features; An acquisition module is used to acquire a laser radar point cloud corresponding to the laser radar in real time, and input the laser radar point cloud into a preset laser radar feature encoder to output corresponding point cloud features; A processing module, used for converting the original image features to generate corresponding target image features, and simultaneously inputting the target image features and the point cloud features into a preset feature fusion module to output corresponding BEV space features; The output module is used to encode the BEV spatial features to output object information corresponding to the target object in real time.
[0015] Furthermore, the processing module is specifically used for: When the original image features are acquired in real time, the camera angle of view corresponding to the preset camera is detected in real time; When the camera viewing angle is acquired in real time, a preset BEV space is correspondingly called out in a preset program, and a BEV viewing angle corresponding to the preset BEV space is correspondingly detected; The conversion process of the original image features is completed according to the camera viewing angle and the BEV viewing angle to generate the target image features accordingly.
[0016] Furthermore, the processing module is specifically used for: When the camera perspective and the BEV perspective are respectively acquired, emitting corresponding camera light through the camera perspective to the BEV perspective; Detecting in real time a number of characteristic pixels corresponding to the interior of the camera light, and correspondingly detecting a number of discrete points corresponding to the BEV viewing angle; The target image features are correspondingly generated according to the plurality of feature pixels and the plurality of discrete points.
[0017] Furthermore, the processing module is specifically used for: When a plurality of the characteristic pixels are acquired respectively, a target characteristic value corresponding to each of the characteristic pixels is detected in real time; When a plurality of the discrete points are respectively acquired, a discrete interval corresponding to each of the discrete points is detected in real time, and a target threshold corresponding to the discrete interval is detected in real time; Each of the target feature values is mapped to a target threshold corresponding thereto, so as to generate the target image feature accordingly.
[0018] Furthermore, the processing module is specifically used for: When the target image features and the point cloud features are respectively acquired, the corresponding feature selector is called out in real time in the preset feature fusion module; The feature selector outputs the target feature ratio that matches the target image feature and the point cloud feature in real time, and completes the fusion processing of the target image feature and the point cloud feature based on the target feature ratio to generate the BEV spatial feature accordingly.
[0019] Furthermore, the processing module is specifically used for: When the target feature ratio is obtained in real time, the corresponding target convolution layer is called out in the preset feature fusion module; Detecting original convolution parameters contained in the target convolution layer in real time, and adaptively adjusting the original convolution parameters according to the target feature ratio to generate corresponding target convolution parameters; The target image features and the point cloud features are fused based on the target convolution parameters to generate the BEV spatial features accordingly.
[0020] Furthermore, the processing module is specifically used for: When the target convolution parameters are obtained in real time, the adapted initial neural network is called out in real time; The target convolution parameters and the initial neural network are fused to generate a corresponding target neural network, and the target image features and the point cloud features are fused through the target neural network to generate the BEV spatial features accordingly.
[0021] The third aspect of the embodiment of the present invention proposes: A computer comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the multimodal three-dimensional target detection method as described above is implemented.
[0022] The fourth aspect of the embodiments of the present invention proposes: A readable storage medium stores a computer program thereon, wherein when the program is executed by a processor, the multimodal three-dimensional target detection method as described above is implemented.
[0023] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] Figure 1 A flowchart of a multimodal three-dimensional target detection method provided by the first embodiment of the present invention; Figure 2 A structural block diagram of a multimodal three-dimensional object detection system provided for the third embodiment of the present invention.
[0025] The following specific implementation manner will further illustrate the present invention in conjunction with the above-mentioned drawings. DETAILED DESCRIPTION
[0026] In order to facilitate the understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. Several embodiments of the present invention are given in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive.
[0027] It should be noted that when an element is referred to as being "fixed to" another element, it may be directly on the other element or there may be a central element. When an element is considered to be "connected to" another element, it may be directly connected to the other element or there may be a central element at the same time. The terms "vertical", "horizontal", "left", "right" and similar expressions used herein are for illustrative purposes only.
[0028] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.
[0029] See also Figure 1 , shown is a multimodal three-dimensional target detection method provided by the first embodiment of the present invention. The multimodal three-dimensional target detection method provided by this embodiment can effectively and accurately detect various types of targets, thereby improving the detection efficiency.
[0030] Specifically, this embodiment provides: A multimodal three-dimensional target detection method specifically comprises the following steps: Step S10, collecting multi-view RGB images of the target object in real time through a preset camera, and inputting the multi-view RGB images into a preset image feature encoder to output corresponding original image features; Step S20, acquiring a laser radar point cloud corresponding to the laser radar in real time, and inputting the laser radar point cloud into a preset laser radar feature encoder to output corresponding point cloud features; Step S30, converting the original image features to generate corresponding target image features, and simultaneously inputting the target image features and the point cloud features into a preset feature fusion module to output corresponding BEV space features; Step S40: encoding the BEV spatial features to output object information corresponding to the target object in real time.
[0031] Specifically, in this embodiment, it should be noted that in order to objectively and accurately complete the detection of various objects and obtain the required object information, the detection accuracy of the camera and radar can be improved accordingly. Based on this, it is necessary to perform corresponding fusion processing on the information currently obtained by the two, and determine the required target object to complete the corresponding detection. Specifically, the present invention will first randomly collect a multi-view RGB (red, green, and blue color channel) image of an object through a pre-installed camera. At the same time, the current multi-view RGB image is immediately input into the preset image feature encoder, and the required original image features can be further output.
[0032] Furthermore, the laser radar needs to be processed. Preferably, in order to accurately obtain the information collected by the current laser radar, preferably, the present invention will first obtain the laser radar point cloud corresponding to the current laser radar. At the same time, the current laser radar point cloud will be further input into the preset laser radar feature encoder, and the required point cloud features can be output accordingly. Preferably, the point cloud features provided by the present invention are specifically BEV (bird's eye view) features. Furthermore, in order to accurately obtain the required object information, further fusion processing is required. Specifically, the present invention will first perform corresponding conversion processing on the above-mentioned original image features, and can convert them into the required target image features. Based on this, the current target image features and the current point cloud features can be simultaneously input into the preset feature fusion module for corresponding feature fusion processing, and can be further fused into the required BEV space features. On this basis, finally, only the current BEV space features need to be encoded accordingly, and the object information corresponding to the current object can be finally output. It should be pointed out that the object can be a large object or various small objects, and the object information provided by the present invention specifically includes target category, depth information, 2D bounding box, and 3D bounding box information. Through the above method, the recognition of various types of targets can be completed objectively and accurately, and the detection efficiency is correspondingly improved.
[0033] Second embodiment Furthermore, the step of converting the original image features to generate corresponding target image features includes: When the original image features are acquired in real time, the camera angle of view corresponding to the preset camera is detected in real time; When the camera viewing angle is acquired in real time, a preset BEV space is correspondingly called out in a preset program, and a BEV viewing angle corresponding to the preset BEV space is correspondingly detected; The conversion process of the original image features is completed according to the camera viewing angle and the BEV viewing angle to generate the target image features accordingly.
[0034] Furthermore, the step of completing the conversion processing of the original image features according to the camera perspective and the BEV perspective to generate the target image features accordingly includes: When the camera perspective and the BEV perspective are respectively acquired, emitting corresponding camera light through the camera perspective to the BEV perspective; Detecting in real time a number of characteristic pixels corresponding to the interior of the camera light, and correspondingly detecting a number of discrete points corresponding to the BEV viewing angle; The target image features are correspondingly generated according to the plurality of feature pixels and the plurality of discrete points.
[0035] Furthermore, the step of correspondingly generating the target image features according to the plurality of characteristic pixels and the plurality of discrete points includes: When a plurality of the characteristic pixels are acquired respectively, a target characteristic value corresponding to each of the characteristic pixels is detected in real time; When a plurality of the discrete points are respectively acquired, a discrete interval corresponding to each of the discrete points is detected in real time, and a target threshold corresponding to the discrete interval is detected in real time; Each of the target feature values is mapped to a target threshold corresponding thereto, so as to generate the target image feature accordingly.
[0036] Furthermore, the step of simultaneously inputting the target image features and the point cloud features into a preset feature fusion module to output corresponding BEV spatial features includes: When the target image features and the point cloud features are respectively acquired, the corresponding feature selector is called out in real time in the preset feature fusion module; The feature selector outputs the target feature ratio that matches the target image feature and the point cloud feature in real time, and completes the fusion processing of the target image feature and the point cloud feature based on the target feature ratio to generate the BEV spatial feature accordingly.
[0037] Furthermore, the step of completing the fusion processing of the target image feature and the point cloud feature based on the target feature ratio to generate the BEV space feature accordingly includes: When the target feature ratio is obtained in real time, the corresponding target convolution layer is called out in the preset feature fusion module; Detecting original convolution parameters contained in the target convolution layer in real time, and adaptively adjusting the original convolution parameters according to the target feature ratio to generate corresponding target convolution parameters; The target image features and the point cloud features are fused based on the target convolution parameters to generate the BEV spatial features accordingly.
[0038] Furthermore, the step of fusing the target image features and the point cloud features based on the target convolution parameters to generate the BEV spatial features accordingly includes: When the target convolution parameters are obtained in real time, the adapted initial neural network is called out in real time; The target convolution parameters and the initial neural network are fused to generate a corresponding target neural network, and the target image features and the point cloud features are fused through the target neural network to generate the BEV spatial features accordingly.
[0039] In addition, in this embodiment, it should be noted that after the required original image features are obtained in real time through the above steps, in order to objectively and accurately complete the conversion process of the current original image features, preferably, the present invention will first call out the BEV space for operation, and synchronously detect the corresponding BEV perspective, and further, it is necessary to detect the camera perspective corresponding to the above preset camera. Based on this, in the actual conversion process, the present invention will first emit the corresponding camera light to the current BEV perspective through the current camera perspective, and at the same time, in the current camera light, it detects several feature pixels contained in it in real time, and at the same time, it detects several discrete points corresponding to the current BEV perspective, based on this, the present invention will further detect the target feature value corresponding to each current feature pixel in real time, similarly, in order to facilitate subsequent conversion, the present invention will also detect the discrete intervals corresponding to each current discrete point, and can synchronously detect the target threshold corresponding to each current discrete interval, wherein it should be pointed out that each target feature value is a specific value, based on this, each current target feature value can be further mapped to the corresponding target threshold according to its size, so that the required target image features can be converted accordingly.
[0040] Furthermore, after further acquiring the required target image features and point cloud features through the above steps, in order to objectively and accurately complete the corresponding fusion processing, preferably, the present invention will first call out the pre-set feature fusion module, based on which, further call out the feature selector contained in the current feature fusion module, and at the same time, output the required target feature ratio in real time through the feature selector. Furthermore, in order to facilitate subsequent fusion, the present invention will further input the current target feature ratio into the target convolution layer of the above feature fusion module, based on which, in the current target convolution layer, the current target convolution is detected in real time. The original convolution parameters corresponding to the layer can be directly replaced with the current target feature ratio on this basis, and the required target convolution parameters can be generated. Furthermore, since the target convolution parameters are only a series of values, the fusion processing cannot be completed. Based on this, the present invention also needs to further call out the initial neural network adapted thereto, and immediately fuse the current target convolution parameters and the current initial neural network, so as to further generate the required target neural network, and finally complete the corresponding fusion processing through the target neural network to generate the required BEV spatial features for subsequent processing.
[0041] See also Figure 2 , the third embodiment of the present invention provides: A multimodal three-dimensional target detection system, wherein the system comprises: An acquisition module is used to acquire multi-view RGB images of a target object in real time through a preset camera, and input the multi-view RGB images into a preset image feature encoder to output corresponding original image features; An acquisition module is used to acquire a laser radar point cloud corresponding to the laser radar in real time, and input the laser radar point cloud into a preset laser radar feature encoder to output corresponding point cloud features; A processing module, used for converting the original image features to generate corresponding target image features, and simultaneously inputting the target image features and the point cloud features into a preset feature fusion module to output corresponding BEV space features; The output module is used to encode the BEV spatial features to output object information corresponding to the target object in real time.
[0042] Furthermore, the processing module is specifically used for: When the original image features are acquired in real time, the camera angle of view corresponding to the preset camera is detected in real time; When the camera viewing angle is acquired in real time, a preset BEV space is correspondingly called out in a preset program, and a BEV viewing angle corresponding to the preset BEV space is correspondingly detected; The conversion process of the original image features is completed according to the camera viewing angle and the BEV viewing angle to generate the target image features accordingly.
[0043] Furthermore, the processing module is specifically used for: When the camera perspective and the BEV perspective are respectively acquired, emitting corresponding camera light through the camera perspective to the BEV perspective; Detecting in real time a number of characteristic pixels corresponding to the interior of the camera light, and correspondingly detecting a number of discrete points corresponding to the BEV viewing angle; The target image features are correspondingly generated according to the plurality of feature pixels and the plurality of discrete points.
[0044] Furthermore, the processing module is specifically used for: When a plurality of the characteristic pixels are acquired respectively, a target characteristic value corresponding to each of the characteristic pixels is detected in real time; When a plurality of the discrete points are respectively acquired, a discrete interval corresponding to each of the discrete points is detected in real time, and a target threshold corresponding to the discrete interval is detected in real time; Each of the target feature values is mapped to a target threshold corresponding thereto, so as to generate the target image feature accordingly.
[0045] Furthermore, the processing module is specifically used for: When the target image features and the point cloud features are respectively acquired, the corresponding feature selector is called out in real time in the preset feature fusion module; The feature selector outputs the target feature ratio that matches the target image feature and the point cloud feature in real time, and completes the fusion processing of the target image feature and the point cloud feature based on the target feature ratio to generate the BEV spatial feature accordingly.
[0046] Furthermore, the processing module is specifically used for: When the target feature ratio is obtained in real time, the corresponding target convolution layer is called out in the preset feature fusion module; Detecting original convolution parameters contained in the target convolution layer in real time, and adaptively adjusting the original convolution parameters according to the target feature ratio to generate corresponding target convolution parameters; The target image features and the point cloud features are fused based on the target convolution parameters to generate the BEV spatial features accordingly.
[0047] Furthermore, the processing module is specifically used for: When the target convolution parameters are obtained in real time, the adapted initial neural network is called out in real time; The target convolution parameters and the initial neural network are fused to generate a corresponding target neural network, and the target image features and the point cloud features are fused through the target neural network to generate the BEV spatial features accordingly.
[0048] A fourth embodiment of the present invention provides a computer, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the multimodal three-dimensional target detection method as described above when executing the computer program.
[0049] A fifth embodiment of the present invention provides a readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the multimodal three-dimensional target detection method as described above is implemented.
[0050] In summary, the multimodal three-dimensional target detection method and system provided by the above embodiments of the present invention can effectively and accurately detect various types of targets, thereby improving detection efficiency.
[0051] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.
[0052] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.
[0053] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.
[0054] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0055] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.
[0056] The above-described embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the present invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the attached claims.
Claims
1. A multimodal three-dimensional target detection method, characterized in that: The method comprises: The multi-view RGB images of the target object are collected in real time by a preset camera, and the multi-view RGB images are input into a preset image feature encoder to output corresponding original image features; Acquire a laser radar point cloud corresponding to the laser radar in real time, and input the laser radar point cloud into a preset laser radar feature encoder to output corresponding point cloud features; The original image features are converted to generate corresponding target image features, and the target image features and the point cloud features are simultaneously input into a preset feature fusion module to output corresponding BEV space features; The BEV spatial features are encoded to output object information corresponding to the target object in real time.
2. The multimodal three-dimensional target detection method according to claim 1, characterized in that: The step of converting the original image features to generate corresponding target image features comprises: When the original image features are acquired in real time, the camera angle of view corresponding to the preset camera is detected in real time; When the camera viewing angle is acquired in real time, a preset BEV space is correspondingly called out in a preset program, and a BEV viewing angle corresponding to the preset BEV space is correspondingly detected; The conversion process of the original image features is completed according to the camera viewing angle and the BEV viewing angle to generate the target image features accordingly.
3. The multimodal three-dimensional target detection method according to claim 2, characterized in that: The step of completing the conversion processing of the original image features according to the camera perspective and the BEV perspective to generate the target image features accordingly includes: When the camera perspective and the BEV perspective are respectively acquired, emitting corresponding camera light to the BEV perspective through the camera perspective; Detecting in real time a number of characteristic pixels contained in the camera light, and detecting a number of discrete points contained in the BEV viewing angle; The target image features are correspondingly generated according to the plurality of feature pixels and the plurality of discrete points.
4. The multimodal three-dimensional target detection method according to claim 3, characterized in that: The step of correspondingly generating the target image features according to the plurality of feature pixels and the plurality of discrete points comprises: When a plurality of the characteristic pixels are acquired respectively, a target characteristic value corresponding to each of the characteristic pixels is detected in real time; When a plurality of the discrete points are respectively acquired, a discrete interval corresponding to each of the discrete points is detected in real time, and a target threshold corresponding to the discrete interval is detected in real time; Each of the target feature values is mapped to a target threshold corresponding thereto, so as to generate the target image feature accordingly.
5. The multimodal three-dimensional target detection method according to claim 1, characterized in that: The step of simultaneously inputting the target image features and the point cloud features into a preset feature fusion module to output corresponding BEV spatial features includes: When the target image features and the point cloud features are respectively acquired, the corresponding feature selector is called out in real time in the preset feature fusion module; The feature selector outputs the target feature ratio that matches the target image feature and the point cloud feature in real time, and completes the fusion processing of the target image feature and the point cloud feature based on the target feature ratio to generate the BEV spatial feature accordingly.
6. The multimodal three-dimensional target detection method according to claim 5, characterized in that: The step of completing the fusion processing of the target image feature and the point cloud feature based on the target feature ratio to generate the BEV space feature accordingly includes: When the target feature ratio is obtained in real time, the corresponding target convolution layer is called out in the preset feature fusion module; Detecting original convolution parameters contained in the target convolution layer in real time, and adaptively adjusting the original convolution parameters according to the target feature ratio to generate corresponding target convolution parameters; The target image features and the point cloud features are fused based on the target convolution parameters to generate the BEV spatial features accordingly.
7. The multimodal three-dimensional target detection method according to claim 6, characterized in that: The step of fusing the target image features and the point cloud features based on the target convolution parameters to generate the BEV spatial features accordingly includes: When the target convolution parameters are obtained in real time, the adapted initial neural network is called out in real time; The target convolution parameters and the initial neural network are fused to generate a corresponding target neural network, and the target image features and the point cloud features are fused through the target neural network to generate the BEV spatial features accordingly.
8. A multimodal three-dimensional target detection system, characterized in that: The system comprises: An acquisition module is used to acquire multi-view RGB images of a target object in real time through a preset camera, and input the multi-view RGB images into a preset image feature encoder to output corresponding original image features; An acquisition module is used to acquire a laser radar point cloud corresponding to the laser radar in real time, and input the laser radar point cloud into a preset laser radar feature encoder to output corresponding point cloud features; A processing module, used for converting the original image features to generate corresponding target image features, and simultaneously inputting the target image features and the point cloud features into a preset feature fusion module to output corresponding BEV space features; The output module is used to encode the BEV spatial features to output object information corresponding to the target object in real time.
9. A computer comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the multimodal three-dimensional target detection method as described in any one of claims 1 to 7 is implemented.
10. A readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the multimodal three-dimensional target detection method as described in any one of claims 1 to 7 is implemented.