Target detection method and device, computer equipment and storage medium

Abstract

The invention relates to a target detection method and device, computer equipment and a storage medium, and belongs to the technical field of intelligent driving, and the method comprises the steps: obtaining initial multi-modal sensing data, and carrying out the preprocessing of the initial multi-modal sensing data, and obtaining target multi-modal sensing data; selecting a target encoder network to extract feature data in the target multi-modal sensing data; and based on a pre-constructed aerial view space, determining a target feature corresponding to the feature data, and performing target detection on the target feature to obtain a target detection result. According to the method and the device, a necessary single-mode sensing scheme when sensor signals are lost can be considered while a multi-source sensor fusion scheme realizes high-precision sensing, so that the strong adaptive capacity of automatic driving to special working conditions is completed, and the safety of automatic driving is improved.

Description

Technical Field

[0001] The present application relates to the field of intelligent driving technology, and in particular to a target detection method, device, computer equipment and storage medium. Background Art

[0002] There are several problems with the implementation of the multimodal BEV (Bird's Eye View) solution: 1. Images provide 2D features of the detection target, while point clouds provide 3D features. Mapping point cloud features to image space (3D→2D) will lose the depth information of the detection target, causing geometric distortion and reducing the accuracy of geometric tasks (such as 3D target detection). Mapping image features to point cloud space (2D→3D) will cause semantic density differences because only a small number of pixels will match the 3D features. In addition, reconstructing the camera's 3D information requires querying the point cloud space coordinates, which means that the camera must be bound to Lidar (laser radar) to work. 1. When AR cannot cover a location or fails to work due to a malfunction, the entire perception system will be paralyzed and have poor robustness. 2. Due to the high cost of Lidar and the life span of surround-view mechanical Lidar, a feasible mass production solution usually adopts a sensor deployment solution of a single (1×) forward-looking (-60°~+60°) semi-solid Lidar + multiple (6×) surround-view cameras to fill in the blind spots. In this case, it is necessary to consider the coordination of Lidar and camera during driving. Therefore, how to improve the robustness of the perception system for multimodal tasks in the case of collaborative perception of radar and camera, thereby enhancing driving safety, is an urgent problem to be solved. Summary of the invention

[0003] Based on this, it is necessary to provide a target detection method, device, computer equipment and storage medium that can improve the robustness of the perception system for multimodal tasks when radar and camera perform collaborative perception, thereby enhancing driving safety, in response to the above technical problems.

[0004] In a first aspect, a target detection method is provided, the method comprising:

[0005] Acquiring initial multimodal sensing data, and preprocessing the initial multimodal sensing data to obtain target multimodal sensing data;

[0006] Selecting a target encoder network to extract feature data from the target multimodal perception data;

[0007] Based on the pre-constructed bird's-eye view space, the target features corresponding to the feature data are determined, and target detection is performed on the target features to obtain a target detection result.

[0008] Optionally, the preprocessing method of multimodal perception data includes:

[0009] Obtaining a timestamp of multimodal perception data, the multimodal perception data comprising: image data acquired based on a camera and / or point cloud data acquired based on a radar, the timestamp of the multimodal perception data comprising: a first time value corresponding to same-frame image data generated based on a plurality of image data and / or a second time value corresponding to same-frame point cloud data generated based on a plurality of point cloud data;

[0010] In response to detecting that the absolute value of the difference between the first time value and the second time value is less than or equal to a first preset threshold, defining the target multimodal perception data as a frame of multimodal data generated based on the same-frame image data and the same-frame point cloud data, and assigning a new timestamp to the frame of multimodal data;

[0011] In response to detecting that the absolute value of the difference between the first time value and the second time value is greater than a first preset threshold and the timestamp of the same frame image data has been updated, defining the target multimodal perception data as image multimodal data, and assigning a new timestamp to the image multimodal data;

[0012] In response to detecting that the absolute value of the difference between the first time value and the second time value is greater than a first preset threshold and the timestamp of the point cloud data in the same frame has been updated, defining the target multimodal perception data as point cloud multimodal data, and assigning a new timestamp to the point cloud multimodal data;

[0013] The one frame of multimodal data or the image multimodal data or the point cloud multimodal data is encapsulated as a tensor of a target dimension.

[0014] Optionally, the method for obtaining the timestamp of the multimodal sensing data includes:

[0015] Acquire a first error value between timestamps corresponding to a plurality of image data and / or a second error value between timestamps corresponding to a plurality of point cloud data;

[0016] In response to detecting that the first error values ​​are all smaller than a second preset threshold, merging the plurality of image data into same-frame image data, adjusting the timestamps of the same-frame image data, and defining the adjusted timestamps as a first time value;

[0017] In response to detecting that there is an error value greater than or equal to a second preset threshold value in the first error values, defining the timestamp of the previous image data of the same frame as a first time value;

[0018] In response to detecting that the second error values ​​are all smaller than a second preset threshold, merging the plurality of point cloud data into same-frame point cloud data, adjusting the timestamps of the same-frame point cloud data, and defining the adjusted timestamps as a second time value;

[0019] In response to detecting that the second error values ​​include an error value greater than or equal to a third preset threshold, a timestamp of previous point cloud data in the same frame is defined as a second time value.

[0020] Optionally, selecting a target encoder network to extract feature data from the target multimodal perception data includes:

[0021] In response to detecting that the target multimodal perception data is a frame of multimodal data, based on a point cloud encoder network and an image encoder network, feature extraction is performed on corresponding tensors in the frame of multimodal data to obtain first feature data;

[0022] In response to detecting that the target multimodal perception data is image multimodal data, based on an image encoder network, performing feature extraction on a tensor corresponding to the image multimodal data to obtain second feature data;

[0023] In response to detecting that the target multimodal perception data is point cloud multimodal data, based on a point cloud encoder network, feature extraction is performed on a tensor corresponding to the point cloud multimodal data to obtain third feature data.

[0024] Optionally, based on the pre-constructed bird's-eye view space, determining the target feature corresponding to the feature data includes:

[0025] The target range covered by the multi-source sensor perception is defined as the bird's-eye view space, and the bird's-eye view space is divided into bird's-eye view voxel units of the target number;

[0026] In response to detecting that the feature data is first feature data, respectively acquiring a three-dimensional image feature and a three-dimensional point cloud feature corresponding to the first feature data;

[0027] In response to detecting that the feature data is second feature data, performing multi-scale feature fusion on the second feature data based on a target neural network, obtaining a pixel depth corresponding to the second feature data, and combining an image feature after the multi-scale feature fusion with the pixel depth to obtain a three-dimensional image feature;

[0028] In response to detecting that the feature data is the third feature data, multi-scale feature fusion is performed on the third feature data based on the target neural network to obtain a three-dimensional point cloud feature.

[0029] Optionally, in response to detecting the three-dimensional image feature and / or the three-dimensional point cloud feature, the method further includes:

[0030] Mapping the three-dimensional image features and / or the three-dimensional point cloud features into the bird's-eye view voxel units respectively to obtain target bird's-eye view features;

[0031] Based on the time series queue, historical bird's-eye view features are obtained, and based on the historical bird's-eye view features, feature fusion is performed on the target bird's-eye view features to obtain fusion features containing time series information, and at the same time, the target bird's-eye view features are added to the time series queue, and the historical bird's-eye view features in the time series queue are correspondingly deleted;

[0032] In response to detecting that there is no three-dimensional point cloud feature or three-dimensional image feature in the fused feature, defining the fused feature as the target feature;

[0033] In response to detecting the presence of three-dimensional point cloud features and three-dimensional image features in the fused features, the fused features are input into a bird's-eye view network to extract the bird's-eye view features, and multi-scale feature fusion is performed on the bird's-eye view features to obtain the target features.

[0034] Optionally, the method for obtaining the target detection result includes:

[0035] In response to detecting that the target feature is a feature corresponding to the first feature data, inputting the target feature into the fusion stream detection head for target detection to obtain a target detection result;

[0036] In response to detecting that the target feature is a feature corresponding to the second feature data, inputting the target feature into a camera stream detection head for target detection to obtain a target detection result;

[0037] In response to detecting that the target feature is a feature corresponding to the third feature data, the target feature is input into the radar stream detection head for target detection to obtain a target detection result.

[0038] In a second aspect, a target detection device is provided, the device comprising:

[0039] A preprocessing module, used to obtain initial multimodal perception data and preprocess the initial multimodal perception data to obtain target multimodal perception data;

[0040] A feature data extraction module, used to select a target encoder network to extract feature data from the target multimodal perception data;

[0041] The target detection module is used to determine the target features corresponding to the feature data based on the pre-constructed bird's-eye view space, perform target detection on the target features, and obtain target detection results.

[0042] In a third aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the following steps are implemented:

[0043] Acquiring initial multimodal sensing data, and preprocessing the initial multimodal sensing data to obtain target multimodal sensing data;

[0044] Selecting a target encoder network to extract feature data from the target multimodal perception data;

[0045] Based on the pre-constructed bird's-eye view space, the target features corresponding to the feature data are determined, and target detection is performed on the target features to obtain a target detection result.

[0046] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0047] Acquiring initial multimodal sensing data, and preprocessing the initial multimodal sensing data to obtain target multimodal sensing data;

[0048] Selecting a target encoder network to extract feature data from the target multimodal perception data;

[0049] Based on the pre-constructed bird's-eye view space, the target features corresponding to the feature data are determined, and target detection is performed on the target features to obtain a target detection result.

[0050] In a fifth aspect, a computer program product is provided, the computer program product comprising a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0051] Acquiring initial multimodal sensing data, and preprocessing the initial multimodal sensing data to obtain target multimodal sensing data;

[0052] Selecting a target encoder network to extract feature data from the target multimodal perception data;

[0053] Based on the pre-constructed bird's-eye view space, the target features corresponding to the feature data are determined, and target detection is performed on the target features to obtain a target detection result.

[0054] The above-mentioned target detection method, device, computer equipment and storage medium, the method includes: obtaining initial multimodal perception data, and preprocessing the initial multimodal perception data to obtain target multimodal perception data; selecting a target encoder network to extract feature data in the target multimodal perception data; based on a pre-constructed bird's-eye view space, determining the target features corresponding to the feature data, performing target detection on the target features, and obtaining target detection results. The present application can achieve high-precision perception in a multi-source sensor fusion solution while taking into account the necessary single-modal perception solution when the sensor signal is missing, so as to complete the strong adaptability of autonomous driving to special working conditions and improve the safety of autonomous driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 Schematic diagram of a target detection method in one embodiment;

[0056] Figure 2 A schematic diagram of a synchronization subscription and frame synchronization process of a target detection method in an embodiment;

[0057] Figure 3 A schematic diagram of a multimodal encoder structure and stream division of a target detection method in one embodiment;

[0058] Figure 4 is another schematic diagram of a process of a target detection method in one embodiment;

[0059] Figure 5 A schematic diagram of multi-source sensor distribution of a target detection method in one embodiment;

[0060] Figure 6 is a structural block diagram of a target detection device in one embodiment;

[0061] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0063] It should be understood that in the description of the present application, unless the context clearly requires otherwise, words such as "include", "comprises", and the like throughout the specification should be interpreted as including rather than being exclusive or exhaustive; that is, as including but not limited to.

[0064] It should also be understood that the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of this application, unless otherwise specified, "plurality" means two or more.

[0065] It should be noted that the terms "S1", "S2", etc. are only used for the purpose of describing the steps, and do not specifically refer to the order or sequence, nor are they used to limit the present application. They are only for the convenience of describing the method of the present application, and cannot be understood as indicating the order of the steps. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in this field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection required by the present application.

[0066] Multimodality is crucial for accurate and reliable autonomous driving systems. Cameras can provide rich 2D texture and color information of target detection objects, which is irreplaceable in semantic recognition. However, it is difficult to restore the depth characteristics of the detected objects under 2D information. Lidar can provide the position and depth characteristics of the detected targets in 3D space, which is of great value in target recognition and motion acquisition. However, point clouds are limited by sparseness and cannot capture surface features of objects. The two types of sensors complement each other and are indispensable in L3+ automotive-grade autonomous driving. Multimodal fusion is currently the most mainstream autonomous driving solution. It aligns the image features extracted from camera images and the point cloud features extracted from Liar point clouds in the "bird's eye view" (BEV) space, and performs subsequent multi-task processing in the unified space, such as segmentation or detection. Compared with single-sensor tasks, it usually has a higher accuracy advantage. According to the background technology, there are two problems in the existing technology. For problem 2, the coordination problem of Lidar and camera during driving, including: a joint detection solution within the full coverage range of semi-solid Lidar and camera in the main viewing angle (-60°~+60°); a pure vision solution within the camera's single coverage range in the blind viewing angle (+60°~+300°, semi-solid Lidar blind area); a pure point cloud solution when the semi-solid Lidar works alone due to camera failure (physical detachment, exposure problem, etc.); and a pure vision solution when the camera works alone due to semi-solid Lidar failure (physical failure, interference, point cloud sparse problem, etc.).

[0067] In order to solve the above technical problems, the present application provides a target detection method, device, equipment and storage medium to construct a highly robust unified autonomous driving perception framework for multimodal tasks. While achieving high-precision perception in a multi-source sensor fusion scheme, it can also take into account the necessary single-modal perception scheme when the sensor signal is missing, so as to complete the strong adaptability of autonomous driving to special working conditions and improve the safety of autonomous driving. The framework has a simple structure and strong scalability. For example, there are no special requirements for encoder composition and multimodal data composition. It is easy to deploy on the vehicle side and takes up almost no additional limited computing resources. It is capable of tasks such as pure visual perception, pure radar perception, multimodal perception and constraint perception, which can effectively reduce deployment costs and facilitate secondary development based on it.

[0068] In one embodiment, Figure 1 As shown, a target detection method is provided, comprising the following steps:

[0069] S1: Acquire initial multimodal perception data, and preprocess the initial multimodal perception data to obtain target multimodal perception data.

[0070] It should be noted that multimodal perception data can be acquired through multi-source sensors, and the acquisition time interval can be set according to actual needs. The multi-source sensors may include cameras, radars (which may be laser radars - Lidar), etc., and the number of cameras and radars may be multiple. The multimodal perception data may include image data acquired based on the camera and / or point cloud data acquired based on the radar, etc.; data preprocessing refers to fuzzy alignment of the multimodal data according to the timestamp, and then encapsulating the aligned data into a tensor of a specified dimension for the subsequent feature encoding part.

[0071] In some specific implementations, the preprocessing method of multimodal perception data includes:

[0072] Acquiring a timestamp of multimodal perception data, the multimodal perception data comprising: image data acquired based on a camera and / or point cloud data acquired based on a radar, the timestamp of the multimodal perception data comprising: a first time value corresponding to same-frame image data generated based on a plurality of image data and / or a second time value corresponding to same-frame point cloud data generated based on a plurality of point cloud data, wherein a timestamp refers to a mark or record of a specific time;

[0073] In response to detecting that the absolute value of the difference between the first time value and the second time value is less than or equal to a first preset threshold, defining the target multimodal perception data as a frame of multimodal data generated based on the same-frame image data and the same-frame point cloud data, and assigning a new timestamp to the frame of multimodal data, wherein the first preset threshold can be set according to actual needs, such as 30ms, that is, when the absolute value of the difference between the first time value and the second time value is less than or equal to 30ms, it indicates that the detected data includes image data and point cloud data, which can generate a frame of multimodal data;

[0074] In response to detecting that the absolute value of the difference between the first time value and the second time value is greater than a first preset threshold and the timestamp of the same frame image data has been updated, defining the target multimodal perception data as image multimodal data, and assigning a new timestamp to the image multimodal data;

[0075] In response to detecting that the absolute value of the difference between the first time value and the second time value is greater than a first preset threshold and the timestamp of the point cloud data in the same frame has been updated, the target multimodal perception data is defined as point cloud multimodal data, and a new timestamp is assigned to the point cloud multimodal data.

[0076] In some specific implementations, the method for obtaining the timestamp of the multimodal sensing data includes:

[0077] Obtain a first error value between timestamps corresponding to a plurality of image data and / or a second error value between timestamps corresponding to a plurality of point cloud data, wherein the first error value and the second error value may be defined as an error value set, which includes one or more error values. For example, if there are two timestamps corresponding to the image data, the first error value includes only one error value; if there are three timestamps, the first error value includes three error values, and so on, which will not be described in detail.

[0078] In response to detecting that the first error values ​​are all less than a second preset threshold, the multiple image data are merged into same-frame image data, and the timestamps of the same-frame image data are adjusted, and the adjusted timestamps are defined as a first time value, wherein the second preset threshold can be set according to actual needs, and the first time value can be determined according to timestamps corresponding to the multiple image data, such as adjusting other timestamps based on the value of any one of the multiple timestamps or the average value corresponding to the multiple timestamps, and defining the adjusted timestamp as the first time value, etc.;

[0079] In response to detecting that there is an error value greater than or equal to a second preset threshold value in the first error values, the timestamp of the previous image data in the same frame is defined as the first time value, that is, if one or more timestamps corresponding to the multiple image data are never updated, there will be a situation where the error values ​​between the multiple timestamps are large. In this case, the timestamp of the image data in the same frame defined last time is used as the first time value used for calculation this time;

[0080] In response to detecting that the second error values ​​are all less than the second preset threshold, the multiple point cloud data are merged into the same-frame point cloud data, and the timestamps of the same-frame point cloud data are adjusted, and the adjusted timestamps are defined as the second time value, wherein the third preset threshold can be set according to actual needs. Similarly, the second time value can be determined according to the timestamps corresponding to the multiple point cloud data, such as adjusting the other timestamps based on the value of any one of the multiple timestamps or the average value corresponding to the multiple timestamps, and defining the adjusted timestamp as the second time value, etc.;

[0081] In response to detecting that there is an error value greater than or equal to a third preset threshold value in the second error value, the timestamp of the previous point cloud data in the same frame is defined as the second time value, that is, if one or more timestamps corresponding to the multiple point cloud data are never updated, there will be a situation where the error values ​​between the multiple timestamps are large. At this time, the timestamp of the point cloud data in the same frame defined last time is used as the second time value used for calculation this time;

[0082] Specifically, the above image data and point cloud data are subscribed to synchronously through the message_filters library. Because the synchronous subscription is executed passively, the system may not be able to synchronize the data source of the camera or radar due to physical loss or network congestion (for example, the system reserves 6 cameras for time synchronization, but one of the cameras fails, so the system can only monitor the data of 5 cameras, and the synchronization condition is continuously not met. The timestamp t1 of the camera stream will never be updated, but the radar works normally, the radar data is synchronized once every 30ms, and the timestamp t2 of the radar stream is continuously updated. Therefore, after a period of time, the radar data time When the time stamp t2>t1, and after more than 30ms, the system detects that the camera data is no longer updated, but the radar data is continuously updated, and perception will be performed based on the radar stream in the future, and vice versa. Based on this, when the target multimodal perception data is image multimodal data, the absolute value of the difference between the first time value and the second time value is greater than the first preset threshold, which may also mean that the difference between the first time value and the second time value is greater than the first preset threshold. When the target multimodal perception data is point cloud multimodal data, the absolute value of the difference between the first time value and the second time value is greater than the first preset threshold, which may also mean that the difference between the second time value and the first time value is greater than the first preset threshold.

[0083] The one frame of multimodal data or the image multimodal data or the point cloud multimodal data is encapsulated as a tensor of a target dimension, wherein the target dimension can be set according to actual needs, the tensor corresponding to the radar data can be processed based on the radar stream, the tensor corresponding to the camera data can be processed based on the camera stream, and the tensor containing both the camera data and the radar data can be processed based on the fusion stream.

[0084] In the above implementation, by preprocessing the acquired initial multimodal perception data, target multimodal perception data is obtained, which can be used to determine the target encoder network to be adopted to quickly extract feature data, thereby improving the speed of target detection.

[0085] S2: Select a target encoder network to extract feature data from the target multimodal perception data.

[0086] It should be noted that the target encoder network may include a point cloud encoder network and an image encoder network. The point cloud encoder network is used to process data corresponding to the radar stream, and the image encoder network is used to process data corresponding to the camera stream. The feature data refers to the perceptual features corresponding to the target multimodal perception data.

[0087] In some specific implementations, based on the target multimodal perception data, selecting a target encoder network to extract feature data from the target multimodal perception data includes:

[0088] In response to detecting that the target multimodal perception data is a frame of multimodal data, based on the point cloud encoder network and the image encoder network, feature extraction is performed on the corresponding tensor in the frame of multimodal data to obtain first feature data, wherein the feature encoding part uses a special encoder to extract features according to the preprocessed multimodal data. Specifically, the point cloud encoder network can select PointPillars, CenterPoint or VoxelNet as the 3D backbone network, and the image encoder network can select ResNet-50 or Dual-Swin-Tiny (Swin-Transfromer) as the 2D backbone network, that is, the first feature data includes 3D features extracted based on the 3D backbone network and 2D features extracted based on the 2D backbone network;

[0089] In response to detecting that the target multimodal perception data is image multimodal data, based on an image encoder network, feature extraction is performed on a tensor corresponding to the image multimodal data to obtain second feature data, wherein the image encoder network is the same as above and will not be described in detail herein, and the second feature data extracted is a 2D feature;

[0090] In response to detecting that the target multimodal perception data is point cloud multimodal data, based on the point cloud encoder network, feature extraction is performed on the tensor corresponding to the point cloud multimodal data to obtain third feature data, wherein the point cloud encoder network is the same as above and will not be repeated here, and the second feature data extracted is a 3D feature.

[0091] In the above-mentioned embodiment, according to the different attributes of the multimodal perception data, a target encoder network is selected to extract the corresponding 2D features and / or 3D features for subsequent determination of the detection head for target detection. While achieving high-precision perception in a multi-source sensor fusion scheme, it is possible to take into account the necessary single-modal perception scheme when the sensor signal is missing, so as to achieve the strong adaptability of autonomous driving to special working conditions and improve the safety of autonomous driving.

[0092] S3: Based on the pre-constructed bird's-eye view space, determine the target features corresponding to the feature data, perform target detection on the target features, and obtain a target detection result.

[0093] It should be noted that the bird's-eye view refers to BEV, and the bird's-eye view space refers to the BEV space. The BEV (Bird's Eye View) space provides a unified space to fuse data of different modalities at the feature level, which can effectively improve the detection accuracy and make it easier to perform time series fusion to form a 4D space.

[0094] In some specific implementations, determining the target feature corresponding to the feature data based on the pre-constructed bird's-eye view space includes:

[0095] The target range covered by multi-source sensor perception is defined as the bird's-eye view space, and the bird's-eye view space is divided into bird's-eye view voxel units of the target number. Specifically, in order to retain the rich semantic information obtained from the camera and the accurate spatial information obtained from the radar, the features of different sensors need to be converted into a shared bird's-eye view representation space. The present application selects the effective range covered by the detector (i.e., sensor) perception as the bird's-eye view space, and evenly divides the space into a finite number of bird's-eye view voxel units in advance in the radar coordinates, wherein the effective range can be set according to actual needs;

[0096] In response to detecting that the feature data is the first feature data, respectively obtaining a three-dimensional image feature and a three-dimensional point cloud feature corresponding to the first feature data, wherein the three-dimensional image feature is a 3D image feature, and the three-dimensional point cloud feature is a 3D point cloud feature. Specifically, the first feature data includes feature data corresponding to the image data and the point cloud data, so it is necessary to extract the 3D image feature and the 3D point cloud feature respectively;

[0097] In response to detecting that the feature data is second feature data, performing multi-scale feature fusion on the second feature data based on a target neural network, obtaining a pixel depth corresponding to the second feature data, and combining the image feature after the multi-scale feature fusion and the pixel depth to obtain a three-dimensional image feature, wherein the three-dimensional image feature is a 3D image feature;

[0098] In response to detecting that the feature data is the third feature data, multi-scale feature fusion is performed on the third feature data based on the target neural network to obtain a three-dimensional point cloud feature, wherein the three-dimensional point cloud feature is a 3D point cloud feature.

[0099] Among them, the method for extracting 3D image features is as follows: after extracting 2D features, FPN (Feature Pyramid Network) is used to perform multi-scale feature fusion to improve the detection capability. At the same time, the image 2D pixel points and the 3D bird's-eye view voxel space are mapped according to the camera-radar internal and external parameters (i.e., sensor internal and external parameters) and the affine network through the Lift-Splat-Shoot (LSS) method, so as to obtain the depth information (i.e., pixel depth) lost by the camera optical imaging, and the 2D features and pixel depth after multi-scale feature fusion are combined to reconstruct the 3D image features; the method for extracting 3D point cloud features is as follows: after extracting 3D features, FPN (Feature Pyramid Network) is used to perform multi-scale feature fusion to improve the detection capability. Network) for multi-scale feature fusion to obtain 3D point cloud features. Based on the complete independence of radar stream and camera stream, when it is detected that both image data and point cloud data exist at the same time, they can be analyzed and processed simultaneously through multi-threading, and then in the bird's-eye view space, the features from multi-modality are unified, and the full convolutional network is used to deal with possible local misalignment problems. TransFusion-L or Dual-Swin-Tiny can be selected as the backbone for feature fusion to obtain a fused stream, that is, TransFusion-L or Dual-Swin-Tiny is used to fuse the obtained 3D image features and 3D point cloud features.

[0100] In some specific embodiments, in response to detecting the 3D image feature and / or the 3D point cloud feature, the method further includes:

[0101] The three-dimensional image features and / or the three-dimensional point cloud features are respectively mapped into the bird's-eye view voxel units to obtain the target bird's-eye view features; wherein, because the point cloud feature encoding retains the 3D position information, the three-dimensional point cloud features can be directly mapped into the corresponding bird's-eye view voxel units, and the camera stream features and the bird's-eye view voxel units can be processed by pooling and aligning (BEVpooling) to map the 3D image features into the corresponding bird's-eye view voxel units. After being mapped into the bird's-eye view space, the three-dimensional image features and the three-dimensional point cloud features can both be defined as the target bird's-eye view features, which can be used as the unified feature encoding of the current frame, that is, the bird's-eye view features encoded this time;

[0102] Based on the time series queue, the historical bird's-eye view features are obtained, and based on the historical bird's-eye view features, the target bird's-eye view features are feature fused to obtain fused features containing time series information. At the same time, the target bird's-eye view features are added to the time series queue, and the historical bird's-eye view features in the time series queue are deleted accordingly. The historical bird's-eye view features refer to the unified feature coding of the historical n frames of data, and each feature coding contains the corresponding time series information. The n value can be set according to actual needs. For example, if n is 8, then the time series queue includes the unified feature coding of 8 frames of data. This step specifically uses the radar stream / camera stream / fusion stream features that have been weighted preprocessed in the bird's-eye view space as the unified feature coding of the current frame. The unified feature coding of one feature code and the historical n-frame data are concatenated to obtain the fused feature; the number of deleted historical bird's-eye view features is the same as the number of target bird's-eye view features newly added to the timing queue. The timing information can be obtained and BEV encoded by selecting CustomResnet as the backbone, and the 4D-BEV feature (i.e., the target bird's-eye view feature) is output, and the unified feature coding of the current frame (i.e., the target bird's-eye view feature) is stored in the timing feature storage part, that is, a timing queue is established. If n is 8, the unified feature coding of the current frame in the bird's-eye view coding part is pushed into the back end of the timing queue, and the data of the historical 9th ​​frame is popped out from the front end of the timing queue to update the timing feature.

[0103] In some specific embodiments, after obtaining the fusion feature, the method further includes:

[0104] In response to detecting that there is no three-dimensional point cloud feature or three-dimensional image feature in the fused feature, defining the fused feature as the target feature, that is, only camera stream related data or radar stream related data is detected, and then defining the fused feature corresponding to the processing result of the camera stream related data or radar stream related data as the target feature for input into the corresponding detection head for target detection;

[0105] In response to detecting the presence of three-dimensional point cloud features and three-dimensional image features in the fused features, the fused features are input into the bird's-eye view network to extract the bird's-eye view features, and the bird's-eye view features are subjected to multi-scale feature fusion to obtain the target features. That is, when the fused features are the results of fusion stream related data processing, it is necessary to perform secondary feature extraction on the processing results, and use a multi-scale feature fusion network (such as FPN_LSS) to perform multi-scale feature fusion on the extracted features to obtain the target features.

[0106] In some specific implementations, the target feature is input into a corresponding detection head for target detection, and obtaining a target detection result includes:

[0107] In response to detecting that the target feature is a feature corresponding to the first feature data, inputting the target feature into the fusion stream detection head for target detection to obtain a target detection result;

[0108] In response to detecting that the target feature is a feature corresponding to the second feature data, inputting the target feature into a camera stream detection head for target detection to obtain a target detection result;

[0109] In response to detecting that the target feature is a feature corresponding to the third feature data, the target feature is input into the radar stream detection head for target detection to obtain a target detection result.

[0110] Specifically, the multi-task detection head part can connect different task detection heads based on the bird's-eye view features to achieve detection or segmentation tasks according to specific task objectives, wherein the detection head includes a camera stream detection head, a radar stream detection head and a fusion stream detection head, which correspond to the camera stream feature extraction network, the radar stream feature extraction network and the fusion stream network respectively. The network is composed of different networks used to extract corresponding features in the above steps. The network and its corresponding detection head can be pre-trained with relevant data, specifically: under the premise of unifying the bird's-eye view features, the historical camera stream features encoded by the bird's-eye view are connected to the corresponding task detection head for training to obtain the camera stream feature extraction network and the camera stream detection head; the historical radar stream features encoded by the bird's-eye view are connected to the corresponding task detection head for training to obtain Radar stream feature extraction network and radar stream detection head; freeze the gradients of the trained camera stream extraction network and the trained radar stream feature extraction network, fuse them, and then connect them to the bird's-eye view encoder and the corresponding task detection head for training to obtain the fused stream network and the fused stream detection head; further, in extreme cases, multimodal data needs to dynamically switch networks according to actual needs. In areas where radar and camera coverage overlap, the fused stream network is called to infer the multimodal data to obtain high-precision detection or segmentation effects; in areas where the radar fails or is in a blind spot, when only the camera works normally, the camera stream network is called to infer the image data; when the camera fails or the perception quality is poor, only the radar perception is normal, and the radar stream network is called to infer the point cloud data. In this way, the perception system can better adapt to multimodal tasks and improve robustness.

[0111] Its overall structural process may include: model initialization including point cloud and image feature extraction Backbone and loading of radar stream, camera stream, fusion stream task Head, bird's-eye view space construction, etc.; the multimodal data preprocessing part includes the regular acquisition of various sensor perception information, and fuzzy alignment and tensor encapsulation according to timestamps; the scheduling part includes switching the general encoder network according to the composition of the current frame multimodal data: when the current frame data only contains point cloud information, switch the radar stream for point cloud encoding; when the current frame data only contains image information, switch the camera stream for image encoding; when the current frame data contains both point cloud and image information, use point cloud encoding and image encoding for parallel reasoning, and perform data fusion in the bird's-eye view space; the encoder network receives the multimodal tensor for reasoning, completes feature extraction and fusion, outputs unified features and combines the historical 8 frames of data from the time series queue, and inputs them into the bird's-eye view network for bird's-eye view encoding; updates the time series queue; the encoded bird's-eye view features are connected to the corresponding detection head or segmentation head for reasoning to complete the target detection of the multimodal task.

[0112] In the above implementation, the method of adapting multiple detection heads under the unified representation of the bird's-eye view realizes the robustness requirements in the multi-source sensor fusion perception scheme of autonomous driving, so that the autonomous driving platform can flexibly carry various sensors, compatible with the advantages of multimodal fusion and single-modal perception, and adds a timing queue module to multimodal fusion perception. 4D perception is achieved by using historical bird's-eye view features, which is helpful in reducing perception blind spots and can solve the occlusion problem to a certain extent, thereby realizing the strong adaptability of autonomous driving to special working conditions and improving the safety of autonomous driving.

[0113] In some embodiments, the method further comprises:

[0114] Obtain target detection results within a plurality of preset time periods, and mark the same target detection results using the first identifier, wherein the preset time period can be set according to actual needs, such as one day, one week, and one month, etc., and the preferred value is one week;

[0115] In response to detecting that the number of occurrences of the first identifier within the time range corresponding to the target time node is greater than the first preset value, the time range corresponding to the target time node is marked with the second identifier, wherein the time range can be set according to actual needs, such as 120ms, etc., the target time node is an optional time point in a day, and the first preset value can be set according to actual needs, such as 5 times, etc.;

[0116] In response to detecting that the number of occurrences of the second identifier in a plurality of preset time periods is greater than a second preset value, the importance of the time range corresponding to the target time node is obtained according to a preset mapping table, wherein the second preset value can be set according to actual needs, such as when the number of preset time periods is 10, the second preset value can be 6, and the preset mapping table includes a mapping relationship between the time ranges corresponding to the plurality of time nodes and their importance, and the importance can be set autonomously by the driver or can be determined by expert evaluation, and if the risk factor of driving the vehicle at the time node is high, the importance is high;

[0117] In response to detecting that the importance is less than a third preset value, the time range corresponding to the target time node is marked using a third identifier. If the current time node of the next time period is within the time range corresponding to the target time node, the target detection result is directly used as the final result without recalculation.

[0118] In the above implementation, the target detection result that can be directly used for the final result of a time node is determined from multiple angles, and the importance of the time node is taken into consideration, thereby further improving the target detection efficiency while ensuring driving safety.

[0119] To further illustrate the steps of the above method, this application provides an example based on a specific implementation scenario adapted to the framework, such as Figure 5 As shown in the figure, it shows the corresponding multi-modal sensor composition and layout. This solution uses a bus as the carrier. Its sensor part includes 1 Falcon Kinetic 150-line laser radar as the main laser, responsible for forward vision (-60°~+60°) perception, 1 to 4 Robin-W laser radars as blind spot lasers, 1 8 million front-view camera, 4 5 million surround-view cameras, and 1 5 million rear-view camera. The model is Leopard-ar0233. The specific installation locations are as follows: Figure 5 shown.

[0120] The vehicle computing platform is equipped with ROS2 environment (open source platform) and DDS (Data Distribute Service) communication middleware to support the subscription, publishing, frame synchronization and preprocessing functions of multimodal data. The sensor collects data at a fixed frequency and converts it into ROS2 topic (message, a communication carrier for asynchronous communication between publishers and subscribers) through the corresponding driver and publishes it to DDS. The vehicle computing platform is connected to the sensor through Ethernet / serial port, etc., and synchronously subscribes to multimodal data and performs frame synchronization from the DDS communication middleware. Figure 2As shown in the figure, the multimodal data of the scene is divided into the point cloud data part obtained by the lidar and the image data part obtained by the camera; the message_filters library is used to synchronously subscribe to N1 (default is 6) camera topics (sensor_msgs::msg::Image format) and N2 (default is 2) radar topics (sensor_msgs::msg::PointCloud2 format) respectively, and the timestamp fuzzy alignment is performed by TimeSynchronizer and ApproximateTime to calibrate and obtain a frame of image data and a frame of point cloud data. The standard of timestamp fuzzy alignment is that the timestamp errors of several subscribed topics are within a certain range by default, and they are integrated into one frame. data, the alignment standard selected in this step is 30ms, a timer is added to the program, and frame synchronization is performed every 30ms. When the synchronization program is executed, the latest calibrated image data and point cloud data are used. If the absolute value of the timestamp difference between the two is less than or equal to 30ms, they are merged into a frame of multimodal data, which contains a point image part and a point cloud part; if the absolute value of the timestamp difference between the two is greater than 30ms and the image data timestamp is updated, the updated multimodal data only contains the image part; if the absolute value of the timestamp difference between the two is greater than 30ms and the point cloud data timestamp is updated, the updated multimodal data only contains the point cloud part, and the synchronized timestamp is updated to the timestamp of the frame synchronization program execution. The function of this part is to distinguish the same frame multimodal data from the continuously throughput data stream for subsequent model reasoning.

[0121] Multimodal encoder structure and stream division Figure 3As shown in the figure, it can be divided into camera stream part, radar stream part and fusion stream part. As a specific model implementation method, the camera stream is set to read the image tensor in the format of (N, C, W, H), where N represents the number of camera sources (the default is 6), C represents the number of color channels (the default is RGB three channels), W and H are image pixels (the default is 256*704), and the camera stream part uses ResNet50 as the 2D feature extraction backbone, using FPN (Feature Pyramid In addition, the camera intrinsic parameter matrix and image tensor are used to input the LSSViewTransformer model to infer the pixel depth information, and the 3D image features are combined with the 2D image features to reconstruct the 3D image features. The 3D image features are connected to the CenterHead as the detection head to independently perform the target detection task; the radar stream reads the point cloud tensor in the format of (X, Y, Z, I), where X, Y, and Z are the three-dimensional coordinates of the point cloud data in the radar coordinate system, and I is the reflection intensity. The radar stream uses VoxelNet as the 3D feature extraction backbone, and also uses FPN for multi-scale feature fusion to improve the detection capability, obtain 3D point cloud features, and then connect CenterHead as the detection head to independently perform the target detection task. This application sets the laser radar coordinate system X: [-54.0, 54.0, 0.6], Y: [-54.0, 54.0, 0.6], Z: [-5.0, 3.0, 0.2] to construct the BEV space, the format is "coordinate: [lower limit, upper limit, step length]", the unit is meter, so the BEV space is divided into [180*180*40] grids, this space is also the image / point cloud 3D feature unified encoding space; then select the TransFusion-L model to perform image and / or point cloud multimodal data fusion, and perform weighted averaging of heterogeneous features in each BEV unit; set the number of feature channels to 80, and choose to add timing information to the BEV data, merge the historical 8 frames of data, then the BEV data format is [(1+8)*8 0]*180*180 (Z direction information is not considered in BEV weighting operation); In the BEV encoding part, CustomResnet+FPN_LSS is selected as the feature extraction backbone and multi-scale fusion is performed, followed by CenterHead as the detection head for target detection tasks; As mentioned above, when training the model, it is necessary to follow the order of camera stream → radar stream → fusion stream, so that there is a matching CenterHead detection head under different data encodings (3D image encoding / 3D point cloud encoding / BEV encoding) to directly perform target detection tasks, which enables the framework of this application to perform tasks under image / point cloud / fusion multi-modality, so that the robustness of functions under different conditions can be achieved; Specifically, Figure 4As shown, after obtaining the multimodal data of the same frame, it is first split into image topic and point cloud topic; with the help of the cv_bridge::toCvCopy method provided by the visual library OpenCV, the image topic of type sensor_msgs::msg::Image can be converted into a pixel tensor of type unsigned char, and the point cloud topic of type sensor_msgs::msg::PointCloud2 can be converted into XYZI format using the pcl::fromROSMsg method provided by the PCL_conversion library, so as to further integrate it into the data format required by the multimodal encoder; when only the point cloud or image part is contained in the multimodal data of a specific frame, the radar stream / camera stream is called for reasoning, and when both are contained, the fusion stream is called for partial reasoning to improve accuracy and add timing information to solve occlusion problems, etc. In addition, the multimodal composition mentioned in this application can also add corresponding coding structures including lidar + camera + millimeter wave to achieve robustness requirements.

[0122] In the above-mentioned target detection method, the method includes: based on multi-source sensors, obtaining initial multimodal perception data, and preprocessing the initial multimodal perception data to obtain target multimodal perception data; based on the target multimodal perception data, selecting a target encoder network, and based on the target encoder network, extracting feature data in the target multimodal perception data; based on the pre-constructed BEV space, determining the target features corresponding to the feature data, and inputting the target features into the corresponding detection head for target detection to obtain the target detection result. This application proposes a highly robust unified autonomous driving perception framework for multimodal tasks, and realizes the robustness of the autonomous driving multi-source sensor fusion perception scheme by adapting multiple detection heads under the unified representation of BEV. The solution enables the autonomous driving platform to be flexibly equipped with various sensors, and is compatible with the advantages of multimodal fusion and single-modal perception. At the same time, a time series queue module is added to multimodal fusion perception, and 4D perception is achieved by using historical BEV features, which is helpful in reducing perception blind spots and can solve the occlusion problem to a certain extent. A complete set of data closed-loop perception solutions is proposed, including multi-source message subscription, frame synchronization, model building and training strategies, etc., and the unification of multimodal tasks is completed by stream collaboration, which can be used as an underlying template to realize secondary development of advanced functions. While achieving high-precision perception in the multi-source sensor fusion solution, it can also take into account the necessary single-modal perception solution when the sensor signal is missing, so as to complete the strong adaptability of autonomous driving to special working conditions and improve the safety of autonomous driving.

[0123] It should be understood that although Figure 1-Figure 4The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1-Figure 4 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0124] In one embodiment, Figure 6 As shown, a target detection device is provided, comprising: a preprocessing module, a feature data extraction module and a target detection module, wherein:

[0125] A preprocessing module, used to obtain initial multimodal perception data and preprocess the initial multimodal perception data to obtain target multimodal perception data;

[0126] A feature data extraction module, used to select a target encoder network to extract feature data from the target multimodal perception data;

[0127] The target detection module is used to determine the target features corresponding to the feature data based on the pre-constructed bird's-eye view space, perform target detection on the target features, and obtain target detection results.

[0128] As a preferred implementation, in the embodiment of the present invention, the preprocessing module is specifically used for:

[0129] Obtaining a timestamp of multimodal perception data, the multimodal perception data comprising: image data acquired based on a camera and / or point cloud data acquired based on a radar, the timestamp of the multimodal perception data comprising: a first time value corresponding to same-frame image data generated based on a plurality of image data and / or a second time value corresponding to same-frame point cloud data generated based on a plurality of point cloud data;

[0130] In response to detecting that the absolute value of the difference between the first time value and the second time value is less than or equal to a first preset threshold, defining the target multimodal perception data as a frame of multimodal data generated based on the same-frame image data and the same-frame point cloud data, and assigning a new timestamp to the frame of multimodal data;

[0131] In response to detecting that the absolute value of the difference between the first time value and the second time value is greater than a first preset threshold and the timestamp of the same frame image data has been updated, defining the target multimodal perception data as image multimodal data, and assigning a new timestamp to the image multimodal data;

[0132] In response to detecting that the absolute value of the difference between the first time value and the second time value is greater than a first preset threshold and the timestamp of the point cloud data in the same frame has been updated, defining the target multimodal perception data as point cloud multimodal data, and assigning a new timestamp to the point cloud multimodal data;

[0133] The one frame of multimodal data or the image multimodal data or the point cloud multimodal data is encapsulated as a tensor of a target dimension.

[0134] As a preferred implementation, in the embodiment of the present invention, the preprocessing module is further used for:

[0135] Acquire a first error value between timestamps corresponding to a plurality of image data and / or a second error value between timestamps corresponding to a plurality of point cloud data;

[0136] In response to detecting that the first error values ​​are all smaller than a second preset threshold, merging the plurality of image data into same-frame image data, adjusting the timestamps of the same-frame image data, and defining the adjusted timestamps as a first time value;

[0137] In response to detecting that there is an error value greater than or equal to a second preset threshold value in the first error values, defining the timestamp of the previous image data of the same frame as a first time value;

[0138] In response to detecting that the second error values ​​are all smaller than a second preset threshold, merging the plurality of point cloud data into same-frame point cloud data, adjusting the timestamps of the same-frame point cloud data, and defining the adjusted timestamps as a second time value;

[0139] In response to detecting that the second error values ​​include an error value greater than or equal to a third preset threshold, a timestamp of previous point cloud data in the same frame is defined as a second time value.

[0140] As a preferred implementation, in the embodiment of the present invention, the feature data extraction module is specifically used for:

[0141] In response to detecting that the target multimodal perception data is a frame of multimodal data, based on a point cloud encoder network and an image encoder network, feature extraction is performed on corresponding tensors in the frame of multimodal data to obtain first feature data;

[0142] In response to detecting that the target multimodal perception data is image multimodal data, based on an image encoder network, performing feature extraction on a tensor corresponding to the image multimodal data to obtain second feature data;

[0143] In response to detecting that the target multimodal perception data is point cloud multimodal data, based on a point cloud encoder network, feature extraction is performed on a tensor corresponding to the point cloud multimodal data to obtain third feature data.

[0144] As a preferred implementation, in the embodiment of the present invention, the target detection module is specifically used for:

[0145] The target range covered by the multi-source sensor perception is defined as the bird's-eye view space, and the bird's-eye view space is divided into bird's-eye view voxel units of the target number;

[0146] In response to detecting that the feature data is first feature data, respectively acquiring a three-dimensional image feature and a three-dimensional point cloud feature corresponding to the first feature data;

[0147] In response to detecting that the feature data is second feature data, performing multi-scale feature fusion on the second feature data based on a target neural network, obtaining a pixel depth corresponding to the second feature data, and combining an image feature after the multi-scale feature fusion with the pixel depth to obtain a three-dimensional image feature;

[0148] In response to detecting that the feature data is the third feature data, multi-scale feature fusion is performed on the third feature data based on the target neural network to obtain a three-dimensional point cloud feature.

[0149] As a preferred implementation, in the embodiment of the present invention, the target detection module is further used for:

[0150] Mapping the three-dimensional image features and / or the three-dimensional point cloud features into the bird's-eye view voxel units respectively to obtain target bird's-eye view features;

[0151] Based on the time series queue, historical bird's-eye view features are obtained, and based on the historical bird's-eye view features, feature fusion is performed on the target bird's-eye view features to obtain fusion features containing time series information, and at the same time, the target bird's-eye view features are added to the time series queue, and the historical bird's-eye view features in the time series queue are correspondingly deleted;

[0152] In response to detecting that there is no three-dimensional point cloud feature or three-dimensional image feature in the fused feature, defining the fused feature as the target feature;

[0153] In response to detecting the presence of three-dimensional point cloud features and three-dimensional image features in the fused features, the fused features are input into a bird's-eye view network to extract the bird's-eye view features, and multi-scale feature fusion is performed on the bird's-eye view features to obtain the target features.

[0154] As a preferred implementation, in the embodiment of the present invention, the target detection module is further used for:

[0155] In response to detecting that the target feature is a feature corresponding to the first feature data, inputting the target feature into the fusion stream detection head for target detection to obtain a target detection result;

[0156] In response to detecting that the target feature is a feature corresponding to the second feature data, inputting the target feature into a camera stream detection head for target detection to obtain a target detection result;

[0157] In response to detecting that the target feature is a feature corresponding to the third feature data, the target feature is input into the radar stream detection head for target detection to obtain a target detection result.

[0158] For the specific definition of the target detection device, please refer to the definition of the target detection method above, which will not be repeated here. Each module in the above target detection device can be implemented in whole or in part by software, hardware and a combination thereof. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0159] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 7 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a target detection method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0160] Those skilled in the art will understand that Figure 7The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0161] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0162] S1: Acquire initial multimodal perception data, and preprocess the initial multimodal perception data to obtain target multimodal perception data;

[0163] S2: Selecting a target encoder network to extract feature data from the target multimodal perception data;

[0164] S3: Based on the pre-constructed bird's-eye view space, determine the target features corresponding to the feature data, perform target detection on the target features, and obtain a target detection result.

[0165] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0166] Obtaining a timestamp of multimodal perception data, the multimodal perception data comprising: image data acquired based on a camera and / or point cloud data acquired based on a radar, the timestamp of the multimodal perception data comprising: a first time value corresponding to same-frame image data generated based on a plurality of image data and / or a second time value corresponding to same-frame point cloud data generated based on a plurality of point cloud data;

[0167] In response to detecting that the absolute value of the difference between the first time value and the second time value is less than or equal to a first preset threshold, defining the target multimodal perception data as a frame of multimodal data generated based on the same-frame image data and the same-frame point cloud data, and assigning a new timestamp to the frame of multimodal data;

[0168] In response to detecting that the absolute value of the difference between the first time value and the second time value is greater than a first preset threshold and the timestamp of the same frame image data has been updated, defining the target multimodal perception data as image multimodal data, and assigning a new timestamp to the image multimodal data;

[0169] In response to detecting that the absolute value of the difference between the first time value and the second time value is greater than a first preset threshold and the timestamp of the point cloud data in the same frame has been updated, defining the target multimodal perception data as point cloud multimodal data, and assigning a new timestamp to the point cloud multimodal data;

[0170] The one frame of multimodal data or the image multimodal data or the point cloud multimodal data is encapsulated as a tensor of a target dimension.

[0171] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0172] Acquire a first error value between timestamps corresponding to a plurality of image data and / or a second error value between timestamps corresponding to a plurality of point cloud data;

[0173] In response to detecting that the first error values ​​are all smaller than a second preset threshold, merging the plurality of image data into same-frame image data, adjusting the timestamps of the same-frame image data, and defining the adjusted timestamps as a first time value;

[0174] In response to detecting that there is an error value greater than or equal to a second preset threshold value in the first error values, defining the timestamp of the previous image data of the same frame as a first time value;

[0175] In response to detecting that the second error values ​​are all smaller than a second preset threshold, merging the plurality of point cloud data into same-frame point cloud data, adjusting the timestamps of the same-frame point cloud data, and defining the adjusted timestamps as a second time value;

[0176] In response to detecting that the second error values ​​include an error value greater than or equal to a third preset threshold, a timestamp of previous point cloud data in the same frame is defined as a second time value.

[0177] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0178] In response to detecting that the target multimodal perception data is a frame of multimodal data, based on a point cloud encoder network and an image encoder network, feature extraction is performed on corresponding tensors in the frame of multimodal data to obtain first feature data;

[0179] In response to detecting that the target multimodal perception data is image multimodal data, based on an image encoder network, performing feature extraction on a tensor corresponding to the image multimodal data to obtain second feature data;

[0180] In response to detecting that the target multimodal perception data is point cloud multimodal data, based on a point cloud encoder network, feature extraction is performed on a tensor corresponding to the point cloud multimodal data to obtain third feature data.

[0181] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0182] The target range covered by the multi-source sensor perception is defined as the bird's-eye view space, and the bird's-eye view space is divided into bird's-eye view voxel units of the target number;

[0183] In response to detecting that the feature data is first feature data, respectively acquiring a three-dimensional image feature and a three-dimensional point cloud feature corresponding to the first feature data;

[0184] In response to detecting that the feature data is second feature data, performing multi-scale feature fusion on the second feature data based on a target neural network, obtaining a pixel depth corresponding to the second feature data, and combining an image feature after the multi-scale feature fusion with the pixel depth to obtain a three-dimensional image feature;

[0185] In response to detecting that the feature data is the third feature data, multi-scale feature fusion is performed on the third feature data based on the target neural network to obtain a three-dimensional point cloud feature.

[0186] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0187] Mapping the three-dimensional image features and / or the three-dimensional point cloud features into the bird's-eye view voxel units respectively to obtain target bird's-eye view features;

[0188] Based on the time series queue, historical bird's-eye view features are obtained, and based on the historical bird's-eye view features, feature fusion is performed on the target bird's-eye view features to obtain fusion features containing time series information, and at the same time, the target bird's-eye view features are added to the time series queue, and the historical bird's-eye view features in the time series queue are correspondingly deleted;

[0189] In response to detecting that there is no three-dimensional point cloud feature or three-dimensional image feature in the fused feature, defining the fused feature as the target feature;

[0190] In response to detecting the presence of three-dimensional point cloud features and three-dimensional image features in the fused features, the fused features are input into a bird's-eye view network to extract the bird's-eye view features, and multi-scale feature fusion is performed on the bird's-eye view features to obtain the target features.

[0191] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0192] In response to detecting that the target feature is a feature corresponding to the first feature data, inputting the target feature into the fusion stream detection head for target detection to obtain a target detection result;

[0193] In response to detecting that the target feature is a feature corresponding to the second feature data, inputting the target feature into a camera stream detection head for target detection to obtain a target detection result;

[0194] In response to detecting that the target feature is a feature corresponding to the third feature data, the target feature is input into the radar stream detection head for target detection to obtain a target detection result.

[0195] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0196] S1: Acquire initial multimodal perception data, and preprocess the initial multimodal perception data to obtain target multimodal perception data;

[0197] S2: Selecting a target encoder network to extract feature data from the target multimodal perception data;

[0198] S3: Based on the pre-constructed bird's-eye view space, determine the target features corresponding to the feature data, perform target detection on the target features, and obtain a target detection result.

[0199] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0200] Obtaining a timestamp of multimodal perception data, the multimodal perception data comprising: image data acquired based on a camera and / or point cloud data acquired based on a radar, the timestamp of the multimodal perception data comprising: a first time value corresponding to same-frame image data generated based on a plurality of image data and / or a second time value corresponding to same-frame point cloud data generated based on a plurality of point cloud data;

[0201] In response to detecting that the absolute value of the difference between the first time value and the second time value is less than or equal to a first preset threshold, defining the target multimodal perception data as a frame of multimodal data generated based on the same-frame image data and the same-frame point cloud data, and assigning a new timestamp to the frame of multimodal data;

[0202] In response to detecting that the absolute value of the difference between the first time value and the second time value is greater than a first preset threshold and the timestamp of the same frame image data has been updated, defining the target multimodal perception data as image multimodal data, and assigning a new timestamp to the image multimodal data;

[0203] In response to detecting that the absolute value of the difference between the first time value and the second time value is greater than a first preset threshold and the timestamp of the point cloud data in the same frame has been updated, defining the target multimodal perception data as point cloud multimodal data, and assigning a new timestamp to the point cloud multimodal data;

[0204] The one frame of multimodal data or the image multimodal data or the point cloud multimodal data is encapsulated as a tensor of a target dimension.

[0205] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0206] Acquire a first error value between timestamps corresponding to a plurality of image data and / or a second error value between timestamps corresponding to a plurality of point cloud data;

[0207] In response to detecting that the first error values ​​are all smaller than a second preset threshold, merging the plurality of image data into same-frame image data, adjusting the timestamps of the same-frame image data, and defining the adjusted timestamps as a first time value;

[0208] In response to detecting that there is an error value greater than or equal to a second preset threshold value in the first error values, defining the timestamp of the previous image data of the same frame as a first time value;

[0209] In response to detecting that the second error values ​​are all smaller than a second preset threshold, merging the plurality of point cloud data into same-frame point cloud data, adjusting the timestamps of the same-frame point cloud data, and defining the adjusted timestamps as a second time value;

[0210] In response to detecting that the second error values ​​include an error value greater than or equal to a third preset threshold, a timestamp of previous point cloud data in the same frame is defined as a second time value.

[0211] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0212] In response to detecting that the target multimodal perception data is a frame of multimodal data, based on a point cloud encoder network and an image encoder network, feature extraction is performed on corresponding tensors in the frame of multimodal data to obtain first feature data;

[0213] In response to detecting that the target multimodal perception data is image multimodal data, based on an image encoder network, performing feature extraction on a tensor corresponding to the image multimodal data to obtain second feature data;

[0214] In response to detecting that the target multimodal perception data is point cloud multimodal data, based on a point cloud encoder network, feature extraction is performed on a tensor corresponding to the point cloud multimodal data to obtain third feature data.

[0215] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0216] The target range covered by the multi-source sensor perception is defined as the bird's-eye view space, and the bird's-eye view space is divided into bird's-eye view voxel units of the target number;

[0217] In response to detecting that the feature data is first feature data, respectively acquiring a three-dimensional image feature and a three-dimensional point cloud feature corresponding to the first feature data;

[0218] In response to detecting that the feature data is second feature data, performing multi-scale feature fusion on the second feature data based on a target neural network, obtaining a pixel depth corresponding to the second feature data, and combining an image feature after the multi-scale feature fusion with the pixel depth to obtain a three-dimensional image feature;

[0219] In response to detecting that the feature data is the third feature data, multi-scale feature fusion is performed on the third feature data based on the target neural network to obtain a three-dimensional point cloud feature.

[0220] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0221] Mapping the three-dimensional image features and / or the three-dimensional point cloud features into the bird's-eye view voxel units respectively to obtain target bird's-eye view features;

[0222] Based on the time series queue, historical bird's-eye view features are obtained, and based on the historical bird's-eye view features, feature fusion is performed on the target bird's-eye view features to obtain fusion features containing time series information, and at the same time, the target bird's-eye view features are added to the time series queue, and the historical bird's-eye view features in the time series queue are correspondingly deleted;

[0223] In response to detecting that there is no three-dimensional point cloud feature or three-dimensional image feature in the fused feature, defining the fused feature as the target feature;

[0224] In response to detecting the presence of three-dimensional point cloud features and three-dimensional image features in the fused features, the fused features are input into a bird's-eye view network to extract the bird's-eye view features, and multi-scale feature fusion is performed on the bird's-eye view features to obtain the target features.

[0225] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0226] In response to detecting that the target feature is a feature corresponding to the first feature data, inputting the target feature into the fusion stream detection head for target detection to obtain a target detection result;

[0227] In response to detecting that the target feature is a feature corresponding to the second feature data, inputting the target feature into a camera stream detection head for target detection to obtain a target detection result;

[0228] In response to detecting that the target feature is a feature corresponding to the third feature data, the target feature is input into the radar stream detection head for target detection to obtain a target detection result.

[0229] In one embodiment, a computer program product is provided, the computer program product comprising a computer program, the computer program when executed by a processor implements the following steps:

[0230] S1: Acquire initial multimodal perception data, and preprocess the initial multimodal perception data to obtain target multimodal perception data;

[0231] S2: Selecting a target encoder network to extract feature data from the target multimodal perception data;

[0232] S3: Based on the pre-constructed bird's-eye view space, determine the target features corresponding to the feature data, perform target detection on the target features, and obtain a target detection result.

[0233] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0234] Obtaining a timestamp of multimodal perception data, the multimodal perception data comprising: image data acquired based on a camera and / or point cloud data acquired based on a radar, the timestamp of the multimodal perception data comprising: a first time value corresponding to same-frame image data generated based on a plurality of image data and / or a second time value corresponding to same-frame point cloud data generated based on a plurality of point cloud data;

[0235] In response to detecting that the absolute value of the difference between the first time value and the second time value is less than or equal to a first preset threshold, defining the target multimodal perception data as a frame of multimodal data generated based on the same-frame image data and the same-frame point cloud data, and assigning a new timestamp to the frame of multimodal data;

[0236] In response to detecting that the absolute value of the difference between the first time value and the second time value is greater than a first preset threshold and the timestamp of the same frame image data has been updated, defining the target multimodal perception data as image multimodal data, and assigning a new timestamp to the image multimodal data;

[0237] In response to detecting that the absolute value of the difference between the first time value and the second time value is greater than a first preset threshold and the timestamp of the point cloud data in the same frame has been updated, defining the target multimodal perception data as point cloud multimodal data, and assigning a new timestamp to the point cloud multimodal data;

[0238] The one frame of multimodal data or the image multimodal data or the point cloud multimodal data is encapsulated as a tensor of a target dimension.

[0239] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0240] Acquire a first error value between timestamps corresponding to a plurality of image data and / or a second error value between timestamps corresponding to a plurality of point cloud data;

[0241] In response to detecting that the first error values ​​are all smaller than a second preset threshold, merging the plurality of image data into same-frame image data, adjusting the timestamps of the same-frame image data, and defining the adjusted timestamps as a first time value;

[0242] In response to detecting that there is an error value greater than or equal to a second preset threshold value in the first error values, defining the timestamp of the previous image data of the same frame as a first time value;

[0243] In response to detecting that the second error values ​​are all smaller than a second preset threshold, merging the plurality of point cloud data into same-frame point cloud data, adjusting the timestamps of the same-frame point cloud data, and defining the adjusted timestamps as a second time value;

[0244] In response to detecting that the second error values ​​include an error value greater than or equal to a third preset threshold, a timestamp of previous point cloud data in the same frame is defined as a second time value.

[0245] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0246] In response to detecting that the target multimodal perception data is a frame of multimodal data, based on a point cloud encoder network and an image encoder network, feature extraction is performed on corresponding tensors in the frame of multimodal data to obtain first feature data;

[0247] In response to detecting that the target multimodal perception data is image multimodal data, based on an image encoder network, performing feature extraction on a tensor corresponding to the image multimodal data to obtain second feature data;

[0248] In response to detecting that the target multimodal perception data is point cloud multimodal data, based on a point cloud encoder network, feature extraction is performed on a tensor corresponding to the point cloud multimodal data to obtain third feature data.

[0249] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0250] The target range covered by the multi-source sensor perception is defined as the bird's-eye view space, and the bird's-eye view space is divided into bird's-eye view voxel units of the target number;

[0251] In response to detecting that the feature data is first feature data, respectively acquiring a three-dimensional image feature and a three-dimensional point cloud feature corresponding to the first feature data;

[0252] In response to detecting that the feature data is second feature data, performing multi-scale feature fusion on the second feature data based on a target neural network, obtaining a pixel depth corresponding to the second feature data, and combining an image feature after the multi-scale feature fusion with the pixel depth to obtain a three-dimensional image feature;

[0253] In response to detecting that the feature data is the third feature data, multi-scale feature fusion is performed on the third feature data based on the target neural network to obtain a three-dimensional point cloud feature.

[0254] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0255] Mapping the three-dimensional image features and / or the three-dimensional point cloud features into the bird's-eye view voxel units respectively to obtain target bird's-eye view features;

[0256] Based on the time series queue, historical bird's-eye view features are obtained, and based on the historical bird's-eye view features, feature fusion is performed on the target bird's-eye view features to obtain fusion features containing time series information, and at the same time, the target bird's-eye view features are added to the time series queue, and the historical bird's-eye view features in the time series queue are correspondingly deleted;

[0257] In response to detecting that there is no three-dimensional point cloud feature or three-dimensional image feature in the fused feature, defining the fused feature as the target feature;

[0258] In response to detecting the presence of three-dimensional point cloud features and three-dimensional image features in the fused features, the fused features are input into a bird's-eye view network to extract the bird's-eye view features, and multi-scale feature fusion is performed on the bird's-eye view features to obtain the target features.

[0259] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0260] In response to detecting that the target feature is a feature corresponding to the first feature data, inputting the target feature into the fusion stream detection head for target detection to obtain a target detection result;

[0261] In response to detecting that the target feature is a feature corresponding to the second feature data, inputting the target feature into a camera stream detection head for target detection to obtain a target detection result;

[0262] In response to detecting that the target feature is a feature corresponding to the third feature data, the target feature is input into the radar stream detection head for target detection to obtain a target detection result.

[0263] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0264] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0265] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application.

Claims

1. A target detection method, characterized in that: The method comprises: Acquiring initial multimodal sensing data, and preprocessing the initial multimodal sensing data to obtain target multimodal sensing data; Selecting a target encoder network to extract feature data from the target multimodal perception data; Based on the pre-constructed bird's-eye view space, the target features corresponding to the feature data are determined, and target detection is performed on the target features to obtain a target detection result.

2. The target detection method according to claim 1, characterized in that: The preprocessing methods of multimodal perception data include: Obtaining a timestamp of multimodal perception data, the multimodal perception data comprising: image data acquired based on a camera and / or point cloud data acquired based on a radar, the timestamp of the multimodal perception data comprising: a first time value corresponding to same-frame image data generated based on a plurality of image data and / or a second time value corresponding to same-frame point cloud data generated based on a plurality of point cloud data; In response to detecting that the absolute value of the difference between the first time value and the second time value is less than or equal to a first preset threshold, defining the target multimodal perception data as a frame of multimodal data generated based on the same-frame image data and the same-frame point cloud data, and assigning a new timestamp to the frame of multimodal data; In response to detecting that the absolute value of the difference between the first time value and the second time value is greater than a first preset threshold and the timestamp of the same frame image data has been updated, defining the target multimodal perception data as image multimodal data, and assigning a new timestamp to the image multimodal data; In response to detecting that the absolute value of the difference between the first time value and the second time value is greater than a first preset threshold and the timestamp of the point cloud data in the same frame has been updated, defining the target multimodal perception data as point cloud multimodal data, and assigning a new timestamp to the point cloud multimodal data; The one frame of multimodal data or the image multimodal data or the point cloud multimodal data is encapsulated as a tensor of a target dimension.

3. The target detection method according to claim 2, characterized in that: The method for obtaining the timestamp of the multimodal sensing data includes: Acquire a first error value between timestamps corresponding to a plurality of image data and / or a second error value between timestamps corresponding to a plurality of point cloud data; In response to detecting that the first error values ​​are all smaller than a second preset threshold, merging the plurality of image data into same-frame image data, adjusting the timestamps of the same-frame image data, and defining the adjusted timestamps as a first time value; In response to detecting that there is an error value greater than or equal to a second preset threshold value in the first error values, defining the timestamp of the previous image data of the same frame as a first time value; In response to detecting that the second error values ​​are all smaller than a second preset threshold, merging the plurality of point cloud data into same-frame point cloud data, adjusting the timestamps of the same-frame point cloud data, and defining the adjusted timestamps as a second time value; In response to detecting that the second error values ​​include an error value greater than or equal to a third preset threshold, a timestamp of previous point cloud data in the same frame is defined as a second time value.

4. The target detection method according to claim 1, characterized in that: Selecting a target encoder network to extract feature data from the target multimodal perception data includes: In response to detecting that the target multimodal perception data is a frame of multimodal data, based on a point cloud encoder network and an image encoder network, feature extraction is performed on corresponding tensors in the frame of multimodal data to obtain first feature data; In response to detecting that the target multimodal perception data is image multimodal data, based on an image encoder network, performing feature extraction on a tensor corresponding to the image multimodal data to obtain second feature data; In response to detecting that the target multimodal perception data is point cloud multimodal data, based on a point cloud encoder network, feature extraction is performed on a tensor corresponding to the point cloud multimodal data to obtain third feature data.

5. The target detection method according to claim 1, characterized in that: Based on the pre-constructed bird's-eye view space, determining the target feature corresponding to the feature data includes: The target range covered by the multi-source sensor perception is defined as the bird's-eye view space, and the bird's-eye view space is divided into bird's-eye view voxel units of the target number; In response to detecting that the feature data is first feature data, respectively acquiring a three-dimensional image feature and a three-dimensional point cloud feature corresponding to the first feature data; In response to detecting that the feature data is second feature data, performing multi-scale feature fusion on the second feature data based on a target neural network, obtaining a pixel depth corresponding to the second feature data, and combining an image feature after the multi-scale feature fusion with the pixel depth to obtain a three-dimensional image feature; In response to detecting that the feature data is the third feature data, multi-scale feature fusion is performed on the third feature data based on the target neural network to obtain a three-dimensional point cloud feature.

6. The target detection method according to claim 5, characterized in that: In response to detecting the 3D image feature and / or the 3D point cloud feature, the method further includes: Mapping the three-dimensional image features and / or the three-dimensional point cloud features into the bird's-eye view voxel units respectively to obtain target bird's-eye view features; Based on the time series queue, historical bird's-eye view features are obtained, and based on the historical bird's-eye view features, feature fusion is performed on the target bird's-eye view features to obtain fusion features containing time series information, and at the same time, the target bird's-eye view features are added to the time series queue, and the historical bird's-eye view features in the time series queue are correspondingly deleted; In response to detecting that there is no three-dimensional point cloud feature or three-dimensional image feature in the fused feature, defining the fused feature as the target feature; In response to detecting the presence of three-dimensional point cloud features and three-dimensional image features in the fused features, the fused features are input into a bird's-eye view network to extract the bird's-eye view features, and multi-scale feature fusion is performed on the bird's-eye view features to obtain the target features.

7. The target detection method according to claim 1, characterized in that: The method for obtaining the target detection result includes: In response to detecting that the target feature is a feature corresponding to the first feature data, inputting the target feature into the fusion stream detection head for target detection to obtain a target detection result; In response to detecting that the target feature is a feature corresponding to the second feature data, inputting the target feature into a camera stream detection head for target detection to obtain a target detection result; In response to detecting that the target feature is a feature corresponding to the third feature data, the target feature is input into the radar stream detection head for target detection to obtain a target detection result.

8. A target detection device, characterized in that: The device comprises: A preprocessing module, used to obtain initial multimodal perception data and preprocess the initial multimodal perception data to obtain target multimodal perception data; A feature data extraction module, used to select a target encoder network to extract feature data from the target multimodal perception data; The target detection module is used to determine the target features corresponding to the feature data based on the pre-constructed bird's-eye view space, perform target detection on the target features, and obtain target detection results.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images