Target detection method and device, computer device and storage medium

Abstract

The application relates to a target detection method and device, computer equipment and a storage medium, belongs to the technical field of intelligent driving, and comprises the following steps: acquiring initial multi-modal perception data, preprocessing the initial multi-modal perception data, and obtaining target multi-modal perception data; selecting a target encoder network to extract feature data in the target multi-modal perception data; determining target features corresponding to the feature data based on a pre-constructed bird's eye view space; and performing target detection on the target features to obtain a target detection result. The application can realize high-precision perception in a multi-source sensor fusion scheme, and at the same time, necessary single-mode perception schemes are considered when sensor signals are missing, so that the strong adaptability of automatic driving to special working conditions is achieved, and the safety of automatic driving is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent driving, in particular to a target detection method and device, computer equipment and a storage medium. BACKGROUND

[0002] Several problems of the multi-modal BEV (Bird's Eye View) scheme are as follows: 1. The image provides 2D features of the detected target, while the point cloud provides 3D features. Mapping the point cloud features to the image space (3D→2D) will lose the depth information of the detected target, cause geometric distortion, and reduce the accuracy of the geometric task (such as 3D target detection). Mapping the image features to the point cloud space (2D→3D) will cause semantic density difference because only a small number of pixel points will match the 3D features. The 3D information of the camera needs to rely on the point cloud space coordinates for query, so the camera must be bound to the Lidar (Laser Radar) to work. When the Lidar cannot work due to coverage or failure, the whole perception system will be paralyzed, and the robustness is poor. 2. Due to the high cost of the Lidar, and in view of the life defect of the ring-type mechanical Lidar, in the feasible mass production scheme, a single (1x) front view (-60°~+60°) semi-solid Lidar + multiple (6x) ring camera blind filling sensor deployment scheme is usually used. Therefore, the coordination problem of the Lidar and the camera during driving needs to be considered. Therefore, how to improve the robustness of the perception system in the case of radar and camera collaborative perception to face multi-modal tasks, and thus to strengthen the driving safety, is a problem to be solved. SUMMARY

[0003] Therefore, it is necessary to provide a target detection method, device, computer equipment and storage medium which can improve the robustness of the perception system in the case of radar and camera collaborative perception to face multi-modal tasks, and thus to strengthen the driving safety.

[0004] In a first aspect, a target detection method is provided, which comprises:

[0005] Obtaining initial multi-modal perception data, and pre-processing the initial multi-modal perception data to obtain target multi-modal perception data;

[0006] Selecting a target encoder network to extract feature data in the target multi-modal perception data;

[0007] Based on a pre-constructed bird's eye view space, determining a target feature corresponding to the feature data, performing target detection on the target feature, and obtaining a target detection result.

[0008] Optionally, the pre-processing method of the multi-modal perception data comprises:

[0009] acquiring a timestamp of multi-modal perception data, the multi-modal perception data comprising: image data acquired based on a camera and / or point cloud data acquired based on a radar, the timestamp of the multi-modal perception data comprising: a first time value corresponding to same-frame image data generated based on a plurality of image data and / or a second time value corresponding to same-frame point cloud data generated based on a plurality of point cloud data;

[0010] in response to detecting that an absolute value of a difference between the first time value and the second time value is less than or equal to a first preset threshold, defining the target multi-modal perception data as one frame of multi-modal data generated based on the same-frame image data and the same-frame point cloud data, and assigning a new timestamp to the one frame of multi-modal data;

[0011] in response to detecting that the absolute value of the difference between the first time value and the second time value is greater than the first preset threshold and the timestamp of the same-frame image data has been updated, defining the target multi-modal perception data as image multi-modal data, and assigning a new timestamp to the image multi-modal data;

[0012] in response to detecting that the absolute value of the difference between the first time value and the second time value is greater than the first preset threshold and the timestamp of the same-frame point cloud data has been updated, defining the target multi-modal perception data as point cloud multi-modal data, and assigning a new timestamp to the point cloud multi-modal data;

[0013] encapsulating the one frame of multi-modal data or the image multi-modal data or the point cloud multi-modal data as a tensor of a target dimension.

[0014] Optionally, the method for acquiring the timestamp of the multi-modal perception data comprises:

[0015] acquiring a first error value between timestamps corresponding to a plurality of image data and / or a second error value between timestamps corresponding to a plurality of point cloud data;

[0016] in response to detecting that all of the first error values are less than a second preset threshold, merging the plurality of image data into same-frame image data, adjusting the timestamp of the same-frame image data, and defining the adjusted timestamp as the first time value;

[0017] in response to detecting that there is an error value greater than or equal to the second preset threshold in the first error values, defining the timestamp of a previous same-frame image data as the first time value;

[0018] in response to detecting that all of the second error values are less than the second preset threshold, merging the plurality of point cloud data into same-frame point cloud data, adjusting the timestamp of the same-frame point cloud data, and defining the adjusted timestamp as the second time value;

[0019] In response to detecting that there is an error value greater than or equal to a third preset threshold in the second error value, define a timestamp of a previous same-frame point cloud data as a second time value.

[0020] Optionally, the target encoder network is selected to extract feature data in the target multi-modal perception data, comprising:

[0021] In response to detecting that the target multi-modal perception data is a frame of multi-modal data, respectively perform feature extraction on corresponding tensors in the frame of multi-modal data based on the point cloud encoder network and the image encoder network, to obtain first feature data;

[0022] In response to detecting that the target multi-modal perception data is image multi-modal data, perform feature extraction on a tensor corresponding to the image multi-modal data based on the image encoder network, to obtain second feature data;

[0023] In response to detecting that the target multi-modal perception data is point cloud multi-modal data, perform feature extraction on a tensor corresponding to the point cloud multi-modal data based on the point cloud encoder network, to obtain third feature data.

[0024] Optionally, based on a pre-constructed bird's eye view space, determining a target feature corresponding to the feature data comprises:

[0025] Defining a target range covered by multi-source sensor perception as the bird's eye view space, and dividing the bird's eye view space into a target number of bird's eye view voxel units;

[0026] In response to detecting that the feature data is first feature data, respectively acquiring three-dimensional image features and three-dimensional point cloud features corresponding to the first feature data;

[0027] In response to detecting that the feature data is second feature data, performing multi-scale feature fusion on the second feature data based on a target neural network, and acquiring pixel depth corresponding to the second feature data, combining image features after multi-scale feature fusion and the pixel depth to obtain three-dimensional image features;

[0028] In response to detecting that the feature data is third feature data, performing multi-scale feature fusion on the third feature data based on a target neural network to obtain three-dimensional point cloud features.

[0029] Optionally, in response to detecting the three-dimensional image features and / or the three-dimensional point cloud features, the method further comprises:

[0030] Respectively mapping the three-dimensional image features and / or the three-dimensional point cloud features into the bird's eye view voxel units to obtain target bird's eye view features;

[0031] Based on the time sequence queue, a historical aerial view feature is acquired, and feature fusion is performed on the target aerial view feature based on the historical aerial view feature, to obtain a fusion feature containing time sequence information, and meanwhile, the target aerial view feature is added to the time sequence queue, and a historical aerial view feature in the time sequence queue is deleted correspondingly;

[0032] In response to detecting that the fusion feature does not contain a three-dimensional point cloud feature or a three-dimensional image feature, the fusion feature is defined as the target feature;

[0033] In response to detecting that the fusion feature contains a three-dimensional point cloud feature and a three-dimensional image feature, the fusion feature is input into an aerial view network to extract an aerial view feature, and multi-scale feature fusion is performed on the aerial view feature to obtain the target feature.

[0034] Optionally, the method for obtaining the target detection result comprises:

[0035] In response to detecting that the target feature is a feature corresponding to the first feature data, the target feature is input into a fusion stream detection head to perform target detection, to obtain a target detection result;

[0036] In response to detecting that the target feature is a feature corresponding to the second feature data, the target feature is input into a camera stream detection head to perform target detection, to obtain a target detection result;

[0037] In response to detecting that the target feature is a feature corresponding to the third feature data, the target feature is input into a radar stream detection head to perform target detection, to obtain a target detection result.

[0038] In a second aspect, a target detection device is provided, and the device comprises:

[0039] A preprocessing module is configured to acquire initial multi-modal perception data, and perform preprocessing on the initial multi-modal perception data, to obtain target multi-modal perception data;

[0040] A feature data extraction module is configured to select a target encoder network to extract feature data in the target multi-modal perception data;

[0041] A target detection module is configured to determine a target feature corresponding to the feature data based on a pre-constructed aerial view space, and perform target detection on the target feature, to obtain a target detection result.

[0042] In a third aspect, a computer device is provided, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the following steps when executing the computer program:

[0043] Obtaining initial multi-modal perception data, and preprocessing the initial multi-modal perception data to obtain target multi-modal perception data;

[0044] Selecting a target encoder network to extract feature data in the target multi-modal perception data;

[0045] Based on the pre-constructed bird's eye view space, determining the target feature corresponding to the feature data, and performing target detection on the target feature to obtain a target detection result.

[0046] In a fourth aspect, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the following steps are implemented:

[0047] Obtaining initial multi-modal perception data, and preprocessing the initial multi-modal perception data to obtain target multi-modal perception data;

[0048] Selecting a target encoder network to extract feature data in the target multi-modal perception data;

[0049] Based on the pre-constructed bird's eye view space, determining the target feature corresponding to the feature data, and performing target detection on the target feature to obtain a target detection result.

[0050] In a fifth aspect, a computer program product is provided, which includes a computer program. When the computer program is executed by a processor, the following steps are implemented:

[0051] Obtaining initial multi-modal perception data, and preprocessing the initial multi-modal perception data to obtain target multi-modal perception data;

[0052] Selecting a target encoder network to extract feature data in the target multi-modal perception data;

[0053] Based on the pre-constructed bird's eye view space, determining the target feature corresponding to the feature data, and performing target detection on the target feature to obtain a target detection result.

[0054] The above target detection method, device, computer equipment and storage medium, the method includes: obtaining initial multi-modal perception data, and preprocessing the initial multi-modal perception data to obtain target multi-modal perception data; selecting a target encoder network to extract feature data in the target multi-modal perception data; based on the pre-constructed bird's eye view space, determining the target feature corresponding to the feature data, and performing target detection on the target feature to obtain a target detection result, the application can realize high-precision perception in a multi-source sensor fusion scheme, while taking into account the necessary single-mode perception scheme when the sensor signal is missing, to complete the strong adaptability of automatic driving to special working conditions and improve the safety of automatic driving. BRIEF DESCRIPTION OF DRAWINGS

[0055] Figure 1 A flowchart of a target detection method in an embodiment;

[0056] Figure 2 A flowchart of a target detection method in an embodiment;

[0057] Figure 3 A flowchart of a target detection method in an embodiment;

[0058] Figure 4 A flowchart of a target detection method in an embodiment;

[0059] Figure 5 A flowchart of a target detection method in an embodiment;

[0060] Figure 6 A structural block diagram of a target detection device in an embodiment;

[0061] Figure 7 An internal structure diagram of a computer device in an embodiment. DETAILED DESCRIPTION

[0062] In order to make the objectives, technical solutions and advantages of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0063] It should be understood that, in the description of the present application, unless the context clearly requires otherwise, the terms "comprise", "comprise", and the like in the entire specification should be interpreted as inclusive rather than exclusive or exhaustive meaning; that is, the meaning of "including but not limited to".

[0064] It should also be understood that the terms "first", "second" and the like are only for descriptive purposes and should not be interpreted as indicating or implying relative importance. In addition, in the description of the present application, unless otherwise stated, the meaning of "multiple" is two or more.

[0065] It should be noted that the terms "S1", "S2" and the like are only used for the purpose of describing the steps and do not particularly indicate the order or sequence, nor are they used to limit the present application, which is merely for the convenience of describing the method of the present application, and cannot be understood as indicating the sequence of the steps. In addition, the technical solutions of various embodiments can be combined with each other, but it must be based on the realization of a person skilled in the art, when the combination of technical solutions appears contradictory or unachievable, it should be considered that the combination of technical solutions does not exist, nor is it within the protection scope required by the present application.

[0066] Multi-modal is essential for accurate and reliable autonomous driving systems, where cameras can provide rich 2D texture, color information for target detection objects, which are irreplaceable in semantic recognition, but it is difficult to restore the depth features of the detected objects under 2D information; while Lidar can provide the position and depth features of the detected objects in 3D space, which has important value in target recognition and action acquisition, but the point cloud is limited by sparsity and cannot capture the surface features of the object. The two types of sensors complement each other and are indispensable in L3+ car-level autonomous driving. Multi-modal fusion is the most mainstream autonomous driving solution at present, which aligns the image features extracted from camera pictures and the point cloud features extracted from Liar point clouds in the "bird's eye view" (BEV) space, and performs subsequent multi-task processing such as segmentation or detection in a unified space, which usually has higher precision advantage compared to single sensor tasks; according to the background art, there are 2 problems in the prior art, for problem 2, the coordination problem of Lidar and camera during driving, including: joint detection scheme in the full coverage range of semi-solid Lidar and camera in the main viewing angle (-60°~+60°); pure vision scheme in the camera-only coverage range in the blind angle (+60°~+300°, semi-solid Lidar blind area); pure point cloud scheme of semi-solid Lidar alone due to camera failure (physical falling off, exposure problem, etc.); and pure vision scheme of camera alone due to semi-solid Lidar failure (physical failure, interference, point cloud sparsity problem, etc.).

[0067] To solve the above technical problems, the application provides a target detection method, device and equipment and a storage medium, constructs a high-robustness unified automatic driving perception framework for multi-modal tasks, can realize high-precision perception in a multi-source sensor fusion scheme, at the same time, gives consideration to necessary single-modal perception schemes when sensor signal is missing, to complete strong adaptability of automatic driving to special working conditions, improve the safety of automatic driving, the framework structure is simple, has strong expansibility, such as encoder composition, multi-modal data composition, etc. are not specially required, is convenient to deploy at the vehicle end, hardly has extra occupation of limited computing resources, and can be competent for pure visual perception, pure radar perception, multi-modal perception and constraint perception and other tasks, can effectively reduce the deployment cost, and also facilitates secondary development based on it.

[0068] In one embodiment, as shown in Figure 1 a target detection method is provided, comprising the following steps:

[0069] S1: acquiring initial multi-modal perception data, and pre-processing the initial multi-modal perception data to obtain target multi-modal perception data.

[0070] It should be noted that the multi-modal perception data can be obtained by a multi-source sensor, and the time interval for obtaining the multi-modal perception data can be set according to actual requirements. The multi-source sensor can include a camera, a radar (which can be a laser radar-Lidar), etc. The number of cameras and radars can be multiple. The multi-modal perception data can include image data obtained based on the camera and / or point cloud data obtained based on the radar, etc. Data preprocessing refers to fuzzy alignment of multi-modal data according to a time stamp, and then encapsulating the aligned data as a tensor of a specified dimension for the subsequent feature encoding part.

[0071] In some specific embodiments, the pre-processing method of the multi-modal perception data includes:

[0072] acquiring a time stamp of multi-modal perception data, the multi-modal perception data including image data obtained based on a camera and / or point cloud data obtained based on a radar, the time stamp of the multi-modal perception data including a first time value corresponding to same-frame image data generated based on multiple image data and / or a second time value corresponding to same-frame point cloud data generated based on multiple point cloud data, wherein the time stamp refers to a mark or record of a specific time;

[0073] In response to detecting that the absolute value of the difference between the first time value and the second time value is less than or equal to a first preset threshold, defining the target multi-modal perception data as one frame of multi-modal data generated based on the same frame image data and the same frame point cloud data, and assigning a new timestamp to the one frame of multi-modal data, wherein the first preset threshold can be set according to actual needs, such as 30 ms, etc., that is, when the absolute value of the difference between the first time value and the second time value is less than or equal to 30 ms, it indicates that the detected data includes image data and point cloud data, which can generate one frame of multi-modal data.

[0074] In response to detecting that the absolute value of the difference between the first time value and the second time value is greater than the first preset threshold and the timestamp of the same frame image data has been updated, defining the target multi-modal perception data as image multi-modal data, and assigning a new timestamp to the image multi-modal data.

[0075] In response to detecting that the absolute value of the difference between the first time value and the second time value is greater than the first preset threshold and the timestamp of the same frame point cloud data has been updated, defining the target multi-modal perception data as point cloud multi-modal data, and assigning a new timestamp to the point cloud multi-modal data.

[0076] In some embodiments, the method for obtaining the timestamp of the multi-modal perception data includes:

[0077] Obtaining a first error value between the timestamps corresponding to the plurality of image data and / or a second error value between the timestamps corresponding to the plurality of point cloud data, wherein the first error value and the second error value can be defined as a set of error values, including one or more error values, for example, if there are two timestamps corresponding to the image data, the first error value only includes one error value, if there are three timestamps, the first error value includes three error values, and so on, which will not be repeated here.

[0078] In response to detecting that the first error value is less than a second preset threshold, merging the plurality of image data into same frame image data, adjusting the timestamp of the same frame image data, and defining the adjusted timestamp as the first time value, wherein the second preset threshold can be set according to actual needs, and the first time value can be determined according to the timestamps corresponding to the plurality of image data, such as adjusting other timestamps according to the value of any selected timestamp or the average value corresponding to the plurality of timestamps as the standard, and defining the adjusted timestamp as the first time value, etc.

[0079] In response to detecting that there is an error value greater than or equal to the second preset threshold in the first error value, the timestamp of the previous same-frame image data is defined as the first time value. That is, if one or more of the timestamps corresponding to the plurality of image data is always not updated, a case where the error value between the plurality of timestamps is large will occur. At this time, the timestamp of the same-frame image data defined in the previous time is used as the first time value for calculation.

[0080] In response to detecting that the second error values are all less than the second preset threshold, the plurality of point cloud data is merged into same-frame point cloud data, and the timestamp of the same-frame point cloud data is adjusted. The adjusted timestamp is defined as the second time value. The third preset threshold can be set according to actual needs. Similarly, the second time value can be determined according to the timestamps corresponding to the plurality of point cloud data. For example, the value of any selected one of the plurality of timestamps or the average value corresponding to the plurality of timestamps is used as a standard to adjust the other timestamps. The adjusted timestamp is defined as the second time value.

[0081] In response to detecting that there is an error value greater than or equal to the third preset threshold in the second error value, the timestamp of the previous same-frame point cloud data is defined as the second time value. That is, if one or more of the timestamps corresponding to the plurality of point cloud data is always not updated, a case where the error value between the plurality of timestamps is large will occur. At this time, the timestamp of the same-frame point cloud data defined in the previous time is used as the second time value for calculation.

[0082] Specifically, the image data and the point cloud data are respectively synchronously subscribed through the message_filters library. Because synchronous subscription is passively executed, the system may be unable to normally synchronize the data sources of the camera or the radar due to physical disconnection or network congestion and the like (for example, the system is scheduled to synchronize 6 cameras, but one of the cameras malfunctions, so the system can only monitor the data of 5 cameras, and continuously fails to meet the synchronization condition. The timestamp t1 of the camera stream is always not updated, but the radar works normally, the radar data is synchronized every 30 ms, and the timestamp t2 of the radar stream is continuously updated. Therefore, after a period of time, the timestamp t2 of the radar data is greater than t1, and exceeds 30 ms. The system monitors that the camera data is no longer updated, but the radar data is continuously updated. Subsequently, perception is performed based on the radar stream. Conversely, perception is performed based on the camera stream. Based on this, when the target multi-modal perception data is image multi-modal data, the absolute value of the difference between the first time value and the second time value is greater than the first preset threshold, which can also mean that the difference between the first time value and the second time value is greater than the first preset threshold. When the target multi-modal perception data is point cloud multi-modal data, the absolute value of the difference between the first time value and the second time value is greater than the first preset threshold, which can also mean that the difference between the second time value and the first time value is greater than the first preset threshold.

[0083] The one frame of multi-modal data or the image multi-modal data or the point cloud multi-modal data is packaged as a tensor of a target dimension, where the target dimension can be set according to actual needs, the tensor corresponding to the radar data can be processed based on a radar stream, the tensor corresponding to the camera data can be processed based on a camera stream, and the tensor containing the camera data and the radar data can be processed based on a fusion stream.

[0084] In the above implementation, by preprocessing the obtained initial multi-modal perception data to obtain target multi-modal perception data, a target encoder network required to be adopted can be determined to quickly extract feature data, and the speed of target detection can be improved.

[0085] S2: selecting a target encoder network to extract feature data in the target multi-modal perception data.

[0086] It should be noted that the target encoder network can include a point cloud encoder network and an image encoder network, the point cloud encoder network is used to process data corresponding to the radar stream, the image encoder network is used to process data corresponding to the camera stream, and the feature data refers to perception features corresponding to the target multi-modal perception data.

[0087] In some specific implementations, based on the target multi-modal perception data, the target encoder network is selected to extract feature data in the target multi-modal perception data, including:

[0088] In response to detecting that the target multi-modal perception data is one frame of multi-modal data, feature extraction is performed on corresponding tensors in the one frame of multi-modal data based on the point cloud encoder network and the image encoder network respectively to obtain first feature data, where a feature encoding part uses a special encoder to extract features according to the preprocessed multi-modal data, specifically, the point cloud encoder network can select PointPillars, CenterPoint or VoxelNet as a 3D backbone network, and the image encoder network can select ResNet-50 or Dual-Swin-Tiny (Swin-Transfromer) as a 2D backbone network, that is, the first feature data includes 3D features extracted based on the 3D backbone network and 2D features extracted based on the 2D backbone network;

[0089] In response to detecting that the target multi-modal perception data is image multi-modal data, feature extraction is performed on a tensor corresponding to the image multi-modal data based on the image encoder network to obtain second feature data, where the image encoder network is the same as above and will not be described again, and the second feature data extracted by the image encoder network is 2D feature;

[0090] In response to detecting that the target multi-modal perception data is point cloud multi-modal data, performing feature extraction on a tensor corresponding to the point cloud multi-modal data based on a point cloud encoder network to obtain third feature data, wherein the point cloud encoder network is the same as described above, and the second feature data extracted by the point cloud encoder network is a 3D feature.

[0091] In the above embodiments, according to different attributes of multi-modal perception data, a target encoder network is selected to extract corresponding 2D features and / or 3D features for subsequent determination of a detection head for target detection. This can achieve high-precision perception in a multi-source sensor fusion scheme while taking into account the necessary single-modal perception scheme when sensor signals are missing, thereby improving the strong adaptability of autonomous driving to special working conditions and improving the safety of autonomous driving.

[0092] S3: determining a target feature corresponding to the feature data based on a pre-constructed bird's eye view space, performing target detection on the target feature to obtain a target detection result.

[0093] It should be noted that the bird's eye view refers to BEV, and the bird's eye view space refers to the BEV space. The BEV (Bird's Eye View) space provides a unified space for fusion processing of data of different modalities at the feature level, which can effectively improve the detection accuracy and is more conducive to time sequence fusion to form a 4D space.

[0094] In some specific embodiments, determining a target feature corresponding to the feature data based on a pre-constructed bird's eye view space comprises:

[0095] The target range covered by the multi-source sensor perception is defined as the bird's eye view space, and the bird's eye view space is divided into a target number of bird's eye view voxel units. Specifically, in order to retain rich semantic information obtained from a camera and accurate spatial information obtained from a radar, it is necessary to convert features of different sensors into a shared bird's eye view representation space. The present application selects the effective range covered by the detector (i.e., the sensor) for perception as the bird's eye view space, and the space is uniformly divided into a limited number of bird's eye view voxel units in the radar coordinate in advance, wherein the effective range can be set according to actual requirements.

[0096] In response to detecting that the feature data is first feature data, three-dimensional image features and three-dimensional point cloud features corresponding to the first feature data are obtained, respectively, wherein the three-dimensional image features are 3D image features, and the three-dimensional point cloud features are 3D point cloud features. Specifically, the first feature data includes feature data corresponding to image data and point cloud data, so it is necessary to extract 3D image features and 3D point cloud features, respectively.

[0097] In response to detecting that the feature data is second feature data, performing multi-scale feature fusion on the second feature data based on a target neural network, and obtaining pixel depth corresponding to the second feature data, combining the image feature after multi-scale feature fusion and the pixel depth to obtain three-dimensional image feature, wherein the three-dimensional image feature is a 3D image feature;

[0098] In response to detecting that the feature data is third feature data, performing multi-scale feature fusion on the third feature data based on a target neural network to obtain three-dimensional point cloud feature, wherein the three-dimensional point cloud feature is a 3D point cloud feature.

[0099] The extraction method of the 3D image feature is: after the 2D feature is extracted, multi-scale feature fusion is performed using FPN (Feature Pyramid Network) to improve the detection capability, and at the same time, the image 2D pixel points and 3D bird's eye view voxel space are mapped according to the camera-radar internal and external parameters (i.e. sensor internal and external parameters) and the affine network through the Lift-Splat-Shoot (LSS) method, so as to obtain the depth information (i.e. pixel depth) lost by the camera due to optical imaging, and the 2D feature after multi-scale feature fusion and the pixel depth are combined to reconstruct the 3D image feature; the extraction method of the 3D point cloud feature is: after the 3D feature is extracted, multi-scale feature fusion is performed using FPN (Feature Pyramid Network) to obtain the 3D point cloud feature, which is completely independent based on radar stream and camera stream, when image data and point cloud data exist at the same time, multi-threading can be used to analyze and process the two, then in the bird's eye view space, the features from multiple modalities are unified, a full convolution network is used to process possible local misalignment problems, TransFusion-L or Dual-Swin-Tiny can be selected as the backbone for feature fusion to obtain a fusion stream, i.e. using TransFusion-L or Dual-Swin-Tiny to perform feature fusion on the obtained 3D image feature and 3D point cloud feature.

[0100] In some specific embodiments, in response to detecting the three-dimensional image feature and / or the three-dimensional point cloud feature, the method further comprises:

[0101] map the three-dimensional image features and / or the three-dimensional point cloud features into the bird's eye view voxel units respectively to obtain target bird's eye view features; wherein, because the point cloud feature encoding retains 3D position information, the three-dimensional point cloud features can be directly mapped into the corresponding bird's eye view voxel units, and the 3D image features can be mapped into the corresponding bird's eye view voxel units by aligning (BEV pooling) the camera stream features and the bird's eye view voxel units, after being mapped into the bird's eye view space, both the three-dimensional image features and the three-dimensional point cloud features can be defined as the target bird's eye view features, which can be used as the unified feature encoding of the current frame, i.e. the bird's eye view features of this encoding;

[0102] Based on the time sequence queue, the historical bird's eye view features are obtained, and based on the historical bird's eye view features, the target bird's eye view features are fused to obtain the fusion features containing time sequence information, at the same time, the target bird's eye view features are added to the time sequence queue and the historical bird's eye view features in the time sequence queue are deleted correspondingly, wherein the historical bird's eye view features refer to the unified feature encoding of historical n frames of data, each feature encoding contains corresponding time sequence information, wherein the value of n can be set according to actual needs, for example, n is 8, at this time, the time sequence queue includes the unified feature encoding of 8 frames of data, this step is specifically to splice the radar stream / camera stream / fusion stream features unified into the bird's eye view space which have been preprocessed with weights as the unified feature encoding of the current frame and the unified feature encoding of the historical n frames of data to obtain the fusion features; the number of deleted historical bird's eye view features is the same as the number of target bird's eye view features newly added to the time sequence queue, the time sequence information can be obtained by selecting CustomResnet as the backbone to perform BEV encoding, outputting 4D-BEV features (i.e. target bird's eye view features), and storing the unified feature encoding of the current frame (i.e. target bird's eye view features) in the time sequence feature storage part, i.e. establishing the time sequence queue, if n is 8, the unified feature encoding of the current frame in the bird's eye view encoding part is pressed into the back end of the time sequence queue, and the data of the historical 9th frame is popped out from the front end of the time sequence queue to update the time sequence features.

[0103] In some embodiments, after obtaining the fusion features, the method further comprises:

[0104] In response to detecting that there is no three-dimensional point cloud feature or three-dimensional image feature in the fusion features, the fusion features are defined as the target features, i.e. only camera stream related data or radar stream related data is detected, then the fusion features corresponding to the processing results of the camera stream related data or the radar stream related data are defined as the target features, which are used as input to the corresponding detection head for target detection;

[0105] In response to detecting that the fusion feature includes a three-dimensional point cloud feature and a three-dimensional image feature, the fusion feature is input into an aerial view network to extract an aerial view feature, and the aerial view feature is subjected to multi-scale feature fusion to obtain the target feature. That is, when the fusion feature is a fusion stream related data processing result, secondary feature extraction needs to be performed on the processing result, and a multi-scale feature fusion network (such as FPN_LSS) is used to perform multi-scale feature fusion on the extracted feature to obtain the target feature.

[0106] In some embodiments, the target feature is input into a corresponding detection head for target detection to obtain a target detection result, including:

[0107] In response to detecting that the target feature is a feature corresponding to the first feature data, the target feature is input into a fusion stream detection head for target detection to obtain a target detection result.

[0108] In response to detecting that the target feature is a feature corresponding to the second feature data, the target feature is input into a camera stream detection head for target detection to obtain a target detection result.

[0109] In response to detecting that the target feature is a feature corresponding to the third feature data, the target feature is input into a radar stream detection head for target detection to obtain a target detection result.

[0110] Specifically, the multi-task detection head part can be based on the bird's eye view feature to realize the detection or segmentation task by circumscribing different task detection heads according to specific task targets, wherein the detection head includes a camera stream detection head, a radar stream detection head and a fusion stream detection head, which correspond to a camera stream feature extraction network, a radar stream feature extraction network and a fusion stream network respectively, the network is composed of different networks for extracting corresponding features in the above steps, and the network and its corresponding detection head can be pre-trained through related data, specifically: under the premise of unified bird's eye view feature, the historical camera stream feature after bird's eye view encoding is connected to the corresponding task detection head for training to obtain the camera stream feature extraction network and the camera stream detection head; the historical radar stream feature after bird's eye view encoding is connected to the corresponding task detection head for training to obtain the radar stream feature extraction network and the radar stream detection head; the trained camera stream extraction network and the trained radar stream feature extraction network are gradient frozen, and then the fusion stream network and the fusion stream detection head are obtained by connecting the bird's eye view encoder and the corresponding task detection head for training; further, the multi-modal data needs to be dynamically switched according to actual needs in extreme cases, in the overlapping area of radar and camera coverage, the fusion stream network is called to infer the multi-modal data to obtain high-precision detection or segmentation effect; when only the camera works normally in the radar failure or blind area, the camera stream network is called to infer the image data; when only the radar perception is normal in the camera failure or poor perception quality, the radar stream network is called to infer the point cloud data, so that the perception system better adapts to multi-modal tasks and improves the robustness.

[0111] The overall structure flow can include: model initialization including loading of point cloud and image feature extraction Backbone and radar stream, camera stream, fusion stream task Head, bird's eye view space construction, etc.; multi-modal data preprocessing part includes acquiring various sensor perception information at regular intervals, fuzzy alignment according to timestamp and tensor packaging; the scheduling part includes switching the general encoder network according to the composition of the current frame multi-modal data: switching the radar stream for point cloud encoding when the current frame data only contains point cloud information; switching the camera stream for image encoding when the current frame data only contains image information; using point cloud encoding and image encoding in parallel inference when the current frame data contains point cloud and image information, and performing data fusion in the bird's eye view space; the encoder network receives multi-modal tensor for inference, and after feature extraction and fusion, outputs unified features and combines historical 8 frame data from the time sequence queue to input into the bird's eye view network for bird's eye view encoding; update the time sequence queue; the encoded bird's eye view feature is connected to the corresponding detection head or segmentation head for inference to complete the target detection of multi-modal tasks.

[0112] In the above embodiment, the way of adapting the multi-detection head under the unified representation of the bird's eye view realizes the robustness requirement in the automatic driving multi-source sensor fusion perception scheme, so that the automatic driving platform can flexibly carry various sensors, and has the advantages of multi-modal fusion and single-modal perception. A time sequence queue module is added to the multi-modal fusion perception, 4D perception is realized by using historical bird's eye view features, which helps to reduce the perception blind area and solves the occlusion problem to a certain extent, realizes the strong adaptability of automatic driving to special working conditions, and improves the safety of automatic driving.

[0113] In some embodiments, the method further comprises:

[0114] Obtaining target detection results in a plurality of preset time periods, and using a first identifier to mark the same target detection results, wherein the preset time period can be set according to actual needs, such as one day, one week, one month, etc., and the preferred value is one week;

[0115] In response to detecting that the number of occurrences of the first identifier in the corresponding time range of the target time node is greater than a first preset value, using a second identifier to mark the time range corresponding to the target time node, wherein the time range can be set according to actual needs, such as 120 ms, the target time node is any optional time point in a day, and the first preset value can be set according to actual needs, such as 5 times;

[0116] In response to detecting that the number of occurrences of the second identifier in a plurality of preset time periods is greater than a second preset value, obtaining the importance of the time range corresponding to the target time node according to a preset mapping table, wherein the second preset value can be set according to actual needs, such as 6 when the number of preset time periods is 10, the preset mapping table includes the mapping relationship between the time range corresponding to the time node and its importance, and the importance can be set by the driver or determined by expert evaluation, such as a higher risk coefficient of driving a vehicle at the time node, then the importance is higher;

[0117] In response to detecting that the importance is less than a third preset value, using a third identifier to mark the time range corresponding to the target time node, and if the current time node of the next time period is within the time range corresponding to the target time node, using the target detection result directly as the final result without re-computing.

[0118] In the above embodiment, the target detection result that can be directly used for the final result of a time node is determined by multiple angles, and the importance of the time node is considered, which further improves the target detection efficiency while ensuring driving safety.

[0119] To further illustrate the steps of the above method, the present application is exemplified based on a specific landing scenario adapted to the framework, as shown in Figure 5 The scheme takes a bus as the carrier, and its sensor part contains 1 Falcon Kinetic 150 line laser radar as the main laser, responsible for front view (-60°~+60°) perception, 1-4 Robin-W laser radars as blind filling lasers, 1 8 million front view camera, 4 500 million ring view cameras, and 1 500 million rear view camera, model Leopard-ar0233, and the specific installation position is as shown in Figure 5 .

[0120] The vehicle-mounted computing platform is equipped with ROS2 environment (open source platform) and DDS (Data Distribute Service) communication middleware to support multi-modal data subscription and publication, frame synchronization and preprocessing functions. The sensor collects data at a fixed frequency, and through the corresponding driver, it is converted into a ROS2 topic (message, a communication carrier for asynchronous communication between publishers and subscribers) and published to DDS. The vehicle-mounted computing platform is connected with the sensor through Ethernet / serial port, etc., and synchronously subscribes to multi-modal data and performs frame synchronization through the DDS communication middleware, as shown in Figure 2As shown, the multi-modal data of the scene is divided into point cloud data part obtained by laser radar and image data part obtained by camera; N1 (default 6) camera topics (sensor_msgs: :msg: :Image format) and N2 (default 2) radar topics (sensor_msgs: :msg: :PointCloud2 format) are respectively synchronously subscribed by using the message_filters library, and the timestamp ambiguity alignment is performed by using the TimeSynchronizer and ApproximateTime to calibrate, so as to obtain one frame of image data and one frame of point cloud data; the standard of the timestamp ambiguity alignment is that the timestamp error of the default several subscribed topics is within a certain range, and then the data is integrated into one frame of data; the alignment standard selected in this step is 30 ms; a timer is added in the program, and frame synchronization is performed once every 30 ms; the latest calibrated image data and point cloud data are executed when the synchronization program is executed; if the absolute value of the timestamp difference is less than or equal to 30 ms, the multi-modal data is merged into one frame of multi-modal data, which contains a point image part and a point cloud part; if the absolute value of the timestamp difference is greater than 30 ms and the image data timestamp is updated, the multi-modal data only contains the image part; if the absolute value of the timestamp difference is greater than 30 ms and the point cloud data timestamp is updated, the multi-modal data only contains the point cloud part; the timestamp is updated to the timestamp of the frame synchronization program execution; the function of this part is to distinguish the same frame of multi-modal data from the continuous data stream so as to facilitate subsequent model reasoning.

[0121] The multi-modal encoder structure and stream division are as follows Figure 3As shown, it can be divided into camera stream part, radar stream part and fusion stream part, as a specific model implementation, set the camera stream read-in image tensor, format (N, C, W, H), wherein N represents the number of camera sources (default 6), C represents the number of color channels (default RGB three channels), W and H are image pixels (default 256*704), the camera stream part selects ResNet50 as the 2D feature extraction backbone, uses FPN (Feature Pyramid Network) for multi-scale feature fusion to improve the detection ability, in addition, the camera intrinsic matrix and the image tensor are input into the LSSViewTransformer model to infer the pixel depth information, and the 2D image feature is combined to reconstruct the 3D image feature, and the 3D image feature is followed by CenterHead as a detection head that can independently perform target detection tasks; the radar stream reads in the point cloud tensor, format (X, Y, Z, I), wherein X, Y and Z are three-dimensional coordinates of point cloud data in the radar coordinate system, and I is the reflectivity, the radar stream part selects VoxelNet as the 3D feature extraction backbone, and also uses FPN for multi-scale feature fusion to improve the detection ability, and obtains 3D point cloud features, and then CenterHead is connected as a detection head that can also independently perform target detection tasks. The application sets X: [-54.0, 54.0, 0.6], Y: [-54.0, 54.0, 0.6], Z: [-5.0, 3.0, 0.2] in the laser radar coordinate system to build a BEV space, format "coordinates: [lower limit, upper limit, step]", unit: meter, thus the BEV space is divided into [180*180*40] grids, and this space is also the unified coding space of image / point cloud 3D features; then the TransFusion-L model is selected to perform image and / or point cloud multi-modal data fusion, and the heterogeneous features are weighted and averaged in each BEV unit; the feature channel number is set to 80, and the BEV data is added with time sequence information, and the historical 8 frames of data are combined, then the BEV data format is [(1+8)*80]*180*180 (Z direction information is not considered in the BEV weighting operation); the BEV coding part selects CustomResnet+FPN_LSS as the feature extraction backbone and performs multi-scale fusion, and then CenterHead is connected as a detection head to perform target detection tasks; as described before, during model training, it needs to be performed in the order of camera stream->radar stream->fusion stream, so that there is a matching CenterHead detection head that can directly perform target detection tasks under different data coding (3D image coding / 3D point cloud coding / BEV coding), which makes the framework of the application can execute tasks under image / point cloud / fusion multi-modal, so the robustness of the function under different conditions can be realized; specifically as Figure 4As shown, after acquiring the same frame multi-modal data, it is first split into image topic and point cloud topic; with the help of the cv_bridge: toCvCopy method provided by the visual library OpenCV, the image topic of type sensor_msgs: msg: Image can be converted into an unsigned char type pixel tensor, and the pcl: fromROSMsg method provided by the PCL_conversion library can be used to convert the point cloud topic of type sensor_msgs: msg: PointCloud2 into XYZI format, so as to further integrate into the data format required by the multi-modal encoder; when only the point cloud or image part is included in the specific frame multi-modal data, the radar stream / camera stream is called for inference, while when both are included, the fusion stream part is called for inference to improve accuracy and add timing information to solve the problem of occlusion, etc., in addition, the multi-modal composition mentioned in the present application can also add the corresponding encoding structure including lidar+camera+millimeter wave, etc. to realize the robustness requirement.

[0122] In the above target detection method, the method comprises: based on a multi-source sensor, acquiring initial multi-modal perception data, and pre-processing the initial multi-modal perception data to obtain target multi-modal perception data; based on the target multi-modal perception data, selecting a target encoder network, and based on the target encoder network, extracting feature data in the target multi-modal perception data; based on a pre-constructed BEV space, determining a target feature corresponding to the feature data, and inputting the target feature into a corresponding detection head for target detection to obtain a target detection result, the present application proposes a high-robustness unified automatic driving perception framework for multi-modal tasks, which realizes the robustness requirement in the automatic driving multi-source sensor fusion perception scheme in the mode of adapting multiple detection heads under the BEV unified representation, which makes the automatic driving platform can flexibly carry various sensors, and has the advantages of multi-modal fusion and single-modal perception; at the same time, a timing queue module is added for multi-modal fusion perception, which realizes 4D perception by using historical BEV features, which is helpful to reduce the perception blind area and can solve the problem of occlusion to a certain extent; a complete set of data closed-loop perception scheme is proposed, including multi-source message subscription, frame synchronization, model establishment and training strategy, etc., and the unified multi-modal task is completed by using stream collaboration, which can be used as a bottom template to realize the secondary development of high-level functions; while realizing high-precision perception in the multi-source sensor fusion scheme, the necessary single-modal perception scheme is also considered when the sensor signal is missing, so as to complete the strong adaptability of the automatic driving to special working conditions and improve the safety of the automatic driving.

[0123] It should be understood that, although Figures 1-4The steps in the flowchart are shown in sequence according to the arrows, but the steps are not necessarily executed in the order indicated by the arrows. Unless otherwise explicitly stated herein, the execution of the steps is not strictly limited in sequence, and the steps can be executed in other orders. Moreover, Figures 1-4 At least part of the steps in the flowchart can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of the sub-steps or stages is not necessarily sequential, but can be alternately executed with other steps or at least part of the sub-steps or stages of other steps.

[0124] In one embodiment, as shown in Figure 6 A target detection device is provided, comprising a preprocessing module, a feature data extraction module, and a target detection module, wherein:

[0125] The preprocessing module is configured to obtain initial multi-modal perception data, and preprocess the initial multi-modal perception data to obtain target multi-modal perception data.

[0126] The feature data extraction module is configured to select a target encoder network to extract feature data in the target multi-modal perception data.

[0127] The target detection module is configured to determine target features corresponding to the feature data based on a pre-constructed bird's eye view space, and perform target detection on the target features to obtain a target detection result.

[0128] As a more preferred embodiment, in the embodiment of the present application, the preprocessing module is specifically configured to:

[0129] Obtain a timestamp of multi-modal perception data, the multi-modal perception data including image data obtained based on a camera and / or point cloud data obtained based on a radar, the timestamp of the multi-modal perception data including a first time value corresponding to same-frame image data generated based on multiple image data and / or a second time value corresponding to same-frame point cloud data generated based on multiple point cloud data.

[0130] In response to detecting that the absolute value of the difference between the first time value and the second time value is less than or equal to a first preset threshold, defining the target multi-modal perception data as one frame of multi-modal data generated based on same-frame image data and same-frame point cloud data, and assigning a new timestamp to the one frame of multi-modal data.

[0131] in response to detecting that the absolute value of the difference between the first time value and the second time value is greater than a first preset threshold and the timestamp of the same frame image data has been updated, defining the target multi-modal perception data as image multi-modal data and assigning a new timestamp to the image multi-modal data;

[0132] in response to detecting that the absolute value of the difference between the first time value and the second time value is greater than a first preset threshold and the timestamp of the same frame point cloud data has been updated, defining the target multi-modal perception data as point cloud multi-modal data and assigning a new timestamp to the point cloud multi-modal data;

[0133] encapsulating the one frame of multi-modal data or the image multi-modal data or the point cloud multi-modal data as a tensor of a target dimension.

[0134] As a more preferred embodiment, in the embodiment of the present application, the preprocessing module is specifically further used for:

[0135] obtaining a first error value between corresponding timestamps of a plurality of image data and / or a second error value between corresponding timestamps of a plurality of point cloud data;

[0136] in response to detecting that all the first error values are less than a second preset threshold, merging the plurality of image data into same frame image data and adjusting the timestamp of the same frame image data, and defining the adjusted timestamp as a first time value;

[0137] in response to detecting that there is an error value greater than or equal to the second preset threshold in the first error values, defining the timestamp of a previous same frame image data as the first time value;

[0138] in response to detecting that all the second error values are less than a second preset threshold, merging the plurality of point cloud data into same frame point cloud data and adjusting the timestamp of the same frame point cloud data, and defining the adjusted timestamp as a second time value;

[0139] in response to detecting that there is an error value greater than or equal to the third preset threshold in the second error values, defining the timestamp of a previous same frame point cloud data as the second time value.

[0140] As a more preferred embodiment, in the embodiment of the present application, the feature data extraction module is specifically used for:

[0141] in response to detecting that the target multi-modal perception data is one frame of multi-modal data, respectively extracting features from corresponding tensors in the one frame of multi-modal data based on a point cloud encoder network and an image encoder network to obtain first feature data;

[0142] In response to detecting that the target multi-modal perception data is image multi-modal data, performing feature extraction on a tensor corresponding to the image multi-modal data based on an image encoder network to obtain second feature data;

[0143] In response to detecting that the target multi-modal perception data is point cloud multi-modal data, performing feature extraction on a tensor corresponding to the point cloud multi-modal data based on a point cloud encoder network to obtain third feature data.

[0144] As a more preferred embodiment, in the embodiment of the present application, the target detection module is specifically used for:

[0145] defining a target range covered by multi-source sensors as the bird's eye view space, and dividing the bird's eye view space into a target number of bird's eye view voxel units;

[0146] In response to detecting that the feature data is the first feature data, respectively acquiring three-dimensional image features and three-dimensional point cloud features corresponding to the first feature data;

[0147] In response to detecting that the feature data is the second feature data, performing multi-scale feature fusion on the second feature data based on a target neural network, and acquiring pixel depth corresponding to the second feature data, combining the image features after multi-scale feature fusion and the pixel depth to obtain three-dimensional image features;

[0148] In response to detecting that the feature data is the third feature data, performing multi-scale feature fusion on the third feature data based on a target neural network to obtain three-dimensional point cloud features.

[0149] As a more preferred embodiment, in the embodiment of the present application, the target detection module is specifically used for:

[0150] respectively mapping the three-dimensional image features and / or the three-dimensional point cloud features into the bird's eye view voxel units to obtain target bird's eye view features;

[0151] based on a time sequence queue, acquiring historical bird's eye view features, and based on the historical bird's eye view features, performing feature fusion on the target bird's eye view features to obtain fused features containing time sequence information, simultaneously adding the target bird's eye view features to the time sequence queue and correspondingly deleting the historical bird's eye view features in the time sequence queue;

[0152] In response to detecting that the fused features do not contain three-dimensional point cloud features or three-dimensional image features, defining the fused features as the target features;

[0153] In response to detecting that the fusion feature has the three-dimensional point cloud feature and the three-dimensional image feature, inputting the fusion feature into an aerial view network to extract an aerial view feature, and performing multi-scale feature fusion on the aerial view feature to obtain the target feature.

[0154] As a more preferred embodiment, the target detection module is further configured to:

[0155] In response to detecting that the target feature is the feature corresponding to the first feature data, inputting the target feature into a fusion stream detection head for target detection to obtain a target detection result.

[0156] In response to detecting that the target feature is the feature corresponding to the second feature data, inputting the target feature into a camera stream detection head for target detection to obtain a target detection result.

[0157] In response to detecting that the target feature is the feature corresponding to the third feature data, inputting the target feature into a radar stream detection head for target detection to obtain a target detection result.

[0158] The specific limitations of the target detection device can be referred to the limitations of the target detection method described above, which will not be repeated here. Each module in the above target detection device can be realized by software, hardware and their combination. Each module described above can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so that the processor calls and executes the operations corresponding to each module.

[0159] In one embodiment, a computer device is provided, which can be a terminal, and its internal structure diagram can be as shown in Figure 7 The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The network interface of the computer device is configured to communicate with external terminals through network connection. The computer program is executed by the processor to implement a target detection method. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the computer device, or an external keyboard, touchpad or mouse, etc.

[0160] Those skilled in the art can understand that Figure 7The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0161] In one embodiment, a computer device is provided, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the following steps when executing the computer program:

[0162] S1: obtaining initial multi-modal perception data, and pre-processing the initial multi-modal perception data to obtain target multi-modal perception data;

[0163] S2: selecting a target encoder network to extract feature data in the target multi-modal perception data;

[0164] S3: determining a target feature corresponding to the feature data based on a pre-constructed bird's eye view space, and performing target detection on the target feature to obtain a target detection result.

[0165] In one embodiment, the processor further implements the following steps when executing the computer program:

[0166] obtaining a timestamp of multi-modal perception data, the multi-modal perception data comprising image data obtained based on a camera and / or point cloud data obtained based on a radar, the timestamp of the multi-modal perception data comprising a first time value corresponding to same-frame image data generated based on a plurality of image data and / or a second time value corresponding to same-frame point cloud data generated based on a plurality of point cloud data;

[0167] in response to detecting that an absolute value of a difference between the first time value and the second time value is less than or equal to a first preset threshold, defining the target multi-modal perception data as one frame of multi-modal data generated based on same-frame image data and same-frame point cloud data, and assigning a new timestamp to the one frame of multi-modal data;

[0168] in response to detecting that the absolute value of the difference between the first time value and the second time value is greater than the first preset threshold and the timestamp of the same-frame image data has been updated, defining the target multi-modal perception data as image multi-modal data, and assigning a new timestamp to the image multi-modal data;

[0169] in response to detecting that the absolute value of the difference between the first time value and the second time value is greater than the first preset threshold and the timestamp of the same-frame point cloud data has been updated, defining the target multi-modal perception data as point cloud multi-modal data, and assigning a new timestamp to the point cloud multi-modal data;

[0170] encapsulate the one frame of multi-modal data or the image multi-modal data or the point cloud multi-modal data as a tensor of a target dimension.

[0171] In one embodiment, the processor, when executing the computer program, also implements the following steps:

[0172] obtain a first error value between timestamps corresponding to the plurality of image data and / or a second error value between timestamps corresponding to the plurality of point cloud data;

[0173] In response to detecting that all of the first error values are less than a second preset threshold, merge the plurality of image data into same-frame image data, and adjust the timestamps of the same-frame image data, with the adjusted timestamps defined as a first time value;

[0174] In response to detecting that there is an error value greater than or equal to the second preset threshold among the first error values, define the timestamp of the previous same-frame image data as the first time value;

[0175] In response to detecting that all of the second error values are less than a second preset threshold, merge the plurality of point cloud data into same-frame point cloud data, and adjust the timestamps of the same-frame point cloud data, with the adjusted timestamps defined as a second time value;

[0176] In response to detecting that there is an error value greater than or equal to a third preset threshold among the second error values, define the timestamp of the previous same-frame point cloud data as the second time value.

[0177] In one embodiment, the processor, when executing the computer program, also implements the following steps:

[0178] In response to detecting that the target multi-modal perception data is one frame of multi-modal data, respectively perform feature extraction on corresponding tensors in the one frame of multi-modal data based on a point cloud encoder network and an image encoder network, to obtain first feature data;

[0179] In response to detecting that the target multi-modal perception data is image multi-modal data, perform feature extraction on a tensor corresponding to the image multi-modal data based on an image encoder network, to obtain second feature data;

[0180] In response to detecting that the target multi-modal perception data is point cloud multi-modal data, perform feature extraction on a tensor corresponding to the point cloud multi-modal data based on a point cloud encoder network, to obtain third feature data.

[0181] In one embodiment, the processor, when executing the computer program, also implements the following steps:

[0182] A target range of a multi-source sensor perception coverage is defined as the bird's eye view space, and the bird's eye view space is divided into a target number of bird's eye view voxel units;

[0183] In response to detecting that the feature data is first feature data, three-dimensional image features and three-dimensional point cloud features corresponding to the first feature data are respectively acquired;

[0184] In response to detecting that the feature data is second feature data, multi-scale feature fusion is performed on the second feature data based on a target neural network, and pixel depth corresponding to the second feature data is acquired, three-dimensional image features after multi-scale feature fusion and the pixel depth are combined to obtain three-dimensional image features;

[0185] In response to detecting that the feature data is third feature data, multi-scale feature fusion is performed on the third feature data based on a target neural network to obtain three-dimensional point cloud features.

[0186] In one embodiment, the processor, when executing the computer program, also implements the following steps:

[0187] The three-dimensional image features and / or the three-dimensional point cloud features are respectively mapped into the bird's eye view voxel units to obtain target bird's eye view features;

[0188] Based on a time sequence queue, historical bird's eye view features are acquired, and feature fusion is performed on the target bird's eye view features based on the historical bird's eye view features to obtain fused features containing time sequence information, and the target bird's eye view features are added to the time sequence queue and corresponding historical bird's eye view features in the time sequence queue are deleted;

[0189] In response to detecting that there is no three-dimensional point cloud feature or three-dimensional image feature in the fused features, the fused features are defined as the target features;

[0190] In response to detecting that there are three-dimensional point cloud features and three-dimensional image features in the fused features, the fused features are input into a bird's eye view network to extract bird's eye view features, and multi-scale feature fusion is performed on the bird's eye view features to obtain the target features.

[0191] In one embodiment, the processor, when executing the computer program, also implements the following steps:

[0192] In response to detecting that the target feature is a feature corresponding to the first feature data, the target feature is input into a fusion stream detection head for target detection to obtain a target detection result;

[0193] In response to detecting that the target feature is a feature corresponding to the second feature data, the target feature is input into a camera stream detection head for target detection to obtain a target detection result;

[0194] In response to detecting that the target feature is a feature corresponding to third feature data, inputting the target feature into a radar stream detection head for target detection to obtain a target detection result.

[0195] In one embodiment, a computer readable storage medium is provided, and a computer program is stored on the computer readable storage medium. The computer program is executed by a processor to implement the following steps:

[0196] S1: obtaining initial multi-modal perception data, and preprocessing the initial multi-modal perception data to obtain target multi-modal perception data;

[0197] S2: selecting a target encoder network to extract feature data in the target multi-modal perception data;

[0198] S3: determining a target feature corresponding to the feature data based on a pre-constructed bird's eye view space, and performing target detection on the target feature to obtain a target detection result.

[0199] In one embodiment, the computer program is executed by the processor to further implement the following steps:

[0200] Obtaining a timestamp of multi-modal perception data, the multi-modal perception data including image data based on a camera and / or point cloud data based on a radar, the timestamp of the multi-modal perception data including a first time value corresponding to same-frame image data generated based on multiple image data and / or a second time value corresponding to same-frame point cloud data generated based on multiple point cloud data;

[0201] In response to detecting that an absolute value of a difference between the first time value and the second time value is less than or equal to a first preset threshold, defining the target multi-modal perception data as one frame of multi-modal data generated based on same-frame image data and same-frame point cloud data, and assigning a new timestamp to the one frame of multi-modal data;

[0202] In response to detecting that the absolute value of the difference between the first time value and the second time value is greater than the first preset threshold and the timestamp of the same-frame image data has been updated, defining the target multi-modal perception data as image multi-modal data, and assigning a new timestamp to the image multi-modal data;

[0203] In response to detecting that the absolute value of the difference between the first time value and the second time value is greater than the first preset threshold and the timestamp of the same-frame point cloud data has been updated, defining the target multi-modal perception data as point cloud multi-modal data, and assigning a new timestamp to the point cloud multi-modal data;

[0204] Encapsulating the one frame of multi-modal data or the image multi-modal data or the point cloud multi-modal data as a tensor of a target dimension.

[0205] In one embodiment, the computer program, when executed by the processor, further implements the following steps:

[0206] obtaining a first error value between timestamps corresponding to the plurality of image data and / or a second error value between timestamps corresponding to the plurality of point cloud data;

[0207] in response to detecting that all of the first error values are less than a second preset threshold, merging the plurality of image data into same-frame image data and adjusting the timestamps of the same-frame image data, the adjusted timestamps being defined as first time values;

[0208] in response to detecting that there is an error value greater than or equal to the second preset threshold among the first error values, defining the timestamp of the previous same-frame image data as the first time value;

[0209] in response to detecting that all of the second error values are less than a second preset threshold, merging the plurality of point cloud data into same-frame point cloud data and adjusting the timestamps of the same-frame point cloud data, the adjusted timestamps being defined as second time values;

[0210] in response to detecting that there is an error value greater than or equal to the third preset threshold among the second error values, defining the timestamp of the previous same-frame point cloud data as the second time value.

[0211] In one embodiment, the computer program, when executed by the processor, further implements the following steps:

[0212] in response to detecting that the target multi-modal perception data is one frame of multi-modal data, respectively extracting features from corresponding tensors in the one frame of multi-modal data based on the point cloud encoder network and the image encoder network to obtain first feature data;

[0213] in response to detecting that the target multi-modal perception data is image multi-modal data, extracting features from a tensor corresponding to the image multi-modal data based on the image encoder network to obtain second feature data;

[0214] in response to detecting that the target multi-modal perception data is point cloud multi-modal data, extracting features from a tensor corresponding to the point cloud multi-modal data based on the point cloud encoder network to obtain third feature data.

[0215] In one embodiment, the computer program, when executed by the processor, further implements the following steps:

[0216] defining a target range covered by multi-source sensors as the bird's eye view space, and dividing the bird's eye view space into a target number of bird's eye view voxel units;

[0217] in response to detecting that the feature data is first feature data, respectively acquiring three-dimensional image features and three-dimensional point cloud features corresponding to the first feature data;

[0218] in response to detecting that the feature data is second feature data, performing multi-scale feature fusion on the second feature data based on a target neural network, and acquiring pixel depth corresponding to the second feature data, combining the image features after multi-scale feature fusion and the pixel depth to obtain three-dimensional image features;

[0219] in response to detecting that the feature data is third feature data, performing multi-scale feature fusion on the third feature data based on a target neural network to obtain three-dimensional point cloud features.

[0220] In one embodiment, the computer program is further implemented when executed by the processor to perform the following steps:

[0221] respectively mapping the three-dimensional image features and / or the three-dimensional point cloud features into the aerial view voxel units to obtain target aerial view features;

[0222] based on a time sequence queue, acquiring historical aerial view features, and based on the historical aerial view features, performing feature fusion on the target aerial view features to obtain fusion features containing time sequence information, while adding the target aerial view features to the time sequence queue and correspondingly deleting the historical aerial view features in the time sequence queue;

[0223] in response to detecting that the fusion features do not contain three-dimensional point cloud features or three-dimensional image features, defining the fusion features as the target features;

[0224] in response to detecting that the fusion features contain three-dimensional point cloud features and three-dimensional image features, inputting the fusion features into an aerial view network to extract aerial view features, and performing multi-scale feature fusion on the aerial view features to obtain the target features.

[0225] In one embodiment, the computer program is further implemented when executed by the processor to perform the following steps:

[0226] in response to detecting that the target features are features corresponding to the first feature data, inputting the target features into a fusion stream detection head for target detection to obtain a target detection result;

[0227] in response to detecting that the target features are features corresponding to the second feature data, inputting the target features into a camera stream detection head for target detection to obtain a target detection result;

[0228] in response to detecting that the target features are features corresponding to the third feature data, inputting the target features into a radar stream detection head for target detection to obtain a target detection result.

[0229] In one embodiment, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the following steps:

[0230] S1: obtaining initial multi-modal perception data, and pre-processing the initial multi-modal perception data to obtain target multi-modal perception data;

[0231] S2: selecting a target encoder network to extract feature data in the target multi-modal perception data;

[0232] S3: determining a target feature corresponding to the feature data based on a pre-constructed bird's eye view space, and performing target detection on the target feature to obtain a target detection result.

[0233] In one embodiment, the computer program, when executed by the processor, further implements the following steps:

[0234] obtaining a timestamp of multi-modal perception data, the multi-modal perception data comprising image data obtained based on a camera and / or point cloud data obtained based on a radar, the timestamp of the multi-modal perception data comprising a first time value corresponding to same-frame image data generated based on multiple image data and / or a second time value corresponding to same-frame point cloud data generated based on multiple point cloud data;

[0235] in response to detecting that an absolute value of a difference between the first time value and the second time value is less than or equal to a first preset threshold, defining the target multi-modal perception data as one frame of multi-modal data generated based on same-frame image data and same-frame point cloud data, and assigning a new timestamp to the one frame of multi-modal data;

[0236] in response to detecting that the absolute value of the difference between the first time value and the second time value is greater than the first preset threshold and the timestamp of the same-frame image data has been updated, defining the target multi-modal perception data as image multi-modal data, and assigning a new timestamp to the image multi-modal data;

[0237] in response to detecting that the absolute value of the difference between the first time value and the second time value is greater than the first preset threshold and the timestamp of the same-frame point cloud data has been updated, defining the target multi-modal perception data as point cloud multi-modal data, and assigning a new timestamp to the point cloud multi-modal data;

[0238] encapsulating the one frame of multi-modal data or the image multi-modal data or the point cloud multi-modal data as a tensor of a target dimension.

[0239] In one embodiment, the computer program, when executed by the processor, further implements the following steps:

[0240] obtaining a first error value between timestamps corresponding to the plurality of image data and / or a second error value between timestamps corresponding to the plurality of point cloud data;

[0241] in response to detecting that all of the first error values are less than a second preset threshold, merging the plurality of image data into same-frame image data and adjusting timestamps of the same-frame image data, the adjusted timestamps being defined as first time values;

[0242] in response to detecting that there is an error value greater than or equal to the second preset threshold among the first error values, defining a timestamp of a previous same-frame image data as the first time value;

[0243] in response to detecting that all of the second error values are less than a third preset threshold, merging the plurality of point cloud data into same-frame point cloud data and adjusting timestamps of the same-frame point cloud data, the adjusted timestamps being defined as second time values;

[0244] in response to detecting that there is an error value greater than or equal to the third preset threshold among the second error values, defining a timestamp of a previous same-frame point cloud data as the second time value.

[0245] In one embodiment, the computer program, when executed by the processor, further implements the following steps:

[0246] in response to detecting that the target multi-modal perception data is one frame of multi-modal data, respectively extracting features of corresponding tensors in the one frame of multi-modal data based on the point cloud encoder network and the image encoder network to obtain first feature data;

[0247] in response to detecting that the target multi-modal perception data is image multi-modal data, extracting features of a tensor corresponding to the image multi-modal data based on the image encoder network to obtain second feature data;

[0248] in response to detecting that the target multi-modal perception data is point cloud multi-modal data, extracting features of a tensor corresponding to the point cloud multi-modal data based on the point cloud encoder network to obtain third feature data.

[0249] In one embodiment, the computer program, when executed by the processor, further implements the following steps:

[0250] defining a target range covered by the multi-source sensor perception as the bird's eye view space, and dividing the bird's eye view space into a target number of bird's eye view voxel units;

[0251] in response to detecting that the feature data is the first feature data, respectively obtaining three-dimensional image features and three-dimensional point cloud features corresponding to the first feature data;

[0252] In response to detecting that the feature data is second feature data, performing multi-scale feature fusion on the second feature data based on a target neural network, obtaining pixel depth corresponding to the second feature data, combining the image feature after multi-scale feature fusion and the pixel depth to obtain three-dimensional image feature;

[0253] In response to detecting that the feature data is third feature data, performing multi-scale feature fusion on the third feature data based on a target neural network to obtain three-dimensional point cloud feature.

[0254] In one embodiment, the computer program, when executed by the processor, further implements the following steps:

[0255] Map the three-dimensional image feature and / or the three-dimensional point cloud feature into the bird's eye view voxel unit respectively to obtain target bird's eye view feature;

[0256] Based on the time sequence queue, obtain historical bird's eye view feature, and based on the historical bird's eye view feature, perform feature fusion on the target bird's eye view feature to obtain fusion feature containing time sequence information, at the same time, add the target bird's eye view feature to the time sequence queue and correspondingly delete the historical bird's eye view feature in the time sequence queue;

[0257] In response to detecting that the fusion feature does not contain three-dimensional point cloud feature or three-dimensional image feature, define the fusion feature as the target feature;

[0258] In response to detecting that the fusion feature contains three-dimensional point cloud feature and three-dimensional image feature, input the fusion feature into the bird's eye view network to extract bird's eye view feature, and perform multi-scale feature fusion on the bird's eye view feature to obtain the target feature.

[0259] In one embodiment, the computer program, when executed by the processor, further implements the following steps:

[0260] In response to detecting that the target feature is the feature corresponding to the first feature data, input the target feature into the fusion stream detection head for target detection to obtain target detection result;

[0261] In response to detecting that the target feature is the feature corresponding to the second feature data, input the target feature into the camera stream detection head for target detection to obtain target detection result;

[0262] In response to detecting that the target feature is the feature corresponding to the third feature data, input the target feature into the radar stream detection head for target detection to obtain target detection result.

[0263] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, storage, databases, or other media in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0264] The technical features of the above embodiments can be combined in any way. In order to make the description simple, not all possible combinations of the technical features in the above embodiments are described, but as long as the combinations of the technical features do not exist, they should be considered as the scope of the present application.

[0265] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, some modifications and improvements can be made, which are all within the protection scope of the present application.

Claims

1. A target detection method, characterized in that, The method includes: Acquire initial multimodal sensing data and preprocess the initial multimodal sensing data to obtain target multimodal sensing data; A target encoder network is selected to extract feature data from the target multimodal sensing data; Based on the pre-constructed bird's-eye view space, the target features corresponding to the feature data are determined, and target detection is performed on the target features to obtain the target detection results; Based on the pre-constructed bird's-eye view space, the target features corresponding to the feature data are determined as follows: The target range perceived and covered by the multi-source sensor is defined as the bird's-eye view space, and the bird's-eye view space is divided into bird's-eye view voxel units of the number of targets; In response to detecting that the feature data is the first feature data, the three-dimensional image features and three-dimensional point cloud features corresponding to the first feature data are acquired respectively; In response to detecting that the feature data is the second feature data, multi-scale feature fusion is performed on the second feature data based on the target neural network, and the pixel depth corresponding to the second feature data is obtained. The image features after multi-scale feature fusion and the pixel depth are combined to obtain the three-dimensional image features. In response to the detection that the feature data is the third feature data, multi-scale feature fusion is performed on the third feature data based on the target neural network to obtain three-dimensional point cloud features; In response to detecting 3D image features and / or 3D point cloud features, the method further includes: The three-dimensional image features and / or three-dimensional point cloud features are respectively mapped into the bird's-eye view voxel unit to obtain the target bird's-eye view features; Based on the time-series queue, historical bird's-eye view features are obtained, and based on the historical bird's-eye view features, the target bird's-eye view features are fused to obtain fused features containing time-series information. At the same time, the target bird's-eye view features are added to the time-series queue, and the historical bird's-eye view features in the time-series queue are deleted accordingly. When it is detected that no 3D point cloud feature or 3D image feature is found in the fused feature, the fused feature is defined as the target feature; In response to the detection of the presence of 3D point cloud features and 3D image features in the fused features, the fused features are input into the bird's-eye view network to extract bird's-eye view features, and multi-scale feature fusion is performed on the bird's-eye view features to obtain the target features.

2. The target detection method according to claim 1, characterized in that, Preprocessing methods for multimodal sensing data include: The timestamps of the multimodal sensing data are obtained, including: image data acquired by a camera and / or point cloud data acquired by radar. The timestamps of the multimodal sensing data include: a first time value corresponding to the same frame image data generated based on multiple image data and / or a second time value corresponding to the same frame point cloud data generated based on multiple point cloud data. In response to detecting that the absolute value of the difference between the first time value and the second time value is less than or equal to a first preset threshold, the target multimodal perception data is defined as a frame of multimodal data generated based on the same frame image data and the same frame point cloud data, and a new timestamp is assigned to the frame of multimodal data. In response to detecting that the absolute value of the difference between the first time value and the second time value is greater than a first preset threshold and the timestamp of the same frame image data has been updated, the target multimodal perception data is defined as image multimodal data, and a new timestamp is assigned to the image multimodal data; In response to detecting that the absolute value of the difference between the first time value and the second time value is greater than a first preset threshold and the timestamp of the point cloud data in the same frame has been updated, the target multimodal perception data is defined as point cloud multimodal data, and a new timestamp is assigned to the point cloud multimodal data; The multimodal data of a frame, the multimodal data of an image, or the multimodal data of a point cloud are encapsulated into a tensor of the target dimension.

3. The target detection method according to claim 2, characterized in that, The method for obtaining the timestamps of the multimodal sensing data includes: Obtain a first error value between timestamps corresponding to multiple image data and / or a second error value between timestamps corresponding to multiple point cloud data; In response to the detection that all of the first error values ​​are less than the second preset threshold, the multiple image data are merged into the same frame image data, and the timestamp of the same frame image data is adjusted, and the adjusted timestamp is defined as the first time value. In response to detecting an error value greater than or equal to a second preset threshold among the first error values, the timestamp of the previous same frame image data is defined as the first time value; In response to the detection that the second error values ​​are all less than the second preset threshold, the multiple point cloud data are merged into point cloud data in the same frame, and the timestamp of the point cloud data in the same frame is adjusted, and the adjusted timestamp is defined as the second time value. In response to the detection of an error value greater than or equal to a third preset threshold in the second error value, the timestamp of the previous point cloud data in the same frame is defined as the second time value.

4. The target detection method according to claim 1, characterized in that, Selecting a target encoder network to extract feature data from the target multimodal sensing data includes: In response to detecting that the target multimodal sensing data is a frame of multimodal data, feature extraction is performed on the corresponding tensors in the frame of multimodal data based on the point cloud encoder network and the image encoder network to obtain the first feature data; In response to detecting that the target multimodal sensing data is image multimodal data, feature extraction is performed on the tensor corresponding to the image multimodal data based on the image encoder network to obtain second feature data; In response to the detection that the target multimodal sensing data is point cloud multimodal data, feature extraction is performed on the tensor corresponding to the point cloud multimodal data based on the point cloud encoder network to obtain third feature data.

5. The target detection method according to claim 1, characterized in that, The method for obtaining the target detection result includes: When a target feature is detected to be the feature corresponding to the first feature data, the target feature is input into the fusion stream detection head for target detection to obtain the target detection result; When a target feature is detected to be the feature corresponding to the second feature data, the target feature is input into the camera stream detection head for target detection to obtain the target detection result; When a target feature is detected to be the feature corresponding to the third feature data, the target feature is input into the radar flow detection head for target detection, and a target detection result is obtained.

6. A target detection apparatus for implementing the target detection method as described in claim 1, characterized in that, The device includes: The preprocessing module is used to acquire initial multimodal sensing data and preprocess the initial multimodal sensing data to obtain target multimodal sensing data. The feature data extraction module is used to select the target encoder network to extract feature data from the target multimodal sensing data; The target detection module is used to determine the target features corresponding to the feature data based on the pre-constructed bird's-eye view space, perform target detection on the target features, and obtain the target detection result.

7. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 5.

Images