Target detection method, device, electronic device and medium

Abstract

The present disclosure proposes a target detection method, device, electronic device and medium, which relate to artificial intelligence fields such as deep learning and computer vision, and can be applied to scenarios such as automatic perception based on large models and autonomous driving. The method includes collecting a detection data set of a detection area, and extracting a first feature sequence of each detection data in the detection data set; obtaining a BEV space of the detection area, projecting the image data collected in the detection area to the BEV space to obtain BEV image data of the image data; extracting a second feature sequence of the BEV image data, and obtaining a three-dimensional fusion feature map obtained by performing feature fusion of each detection data in the detection data set and the image data; performing target detection on the three-dimensional fusion feature map of each detection data respectively to obtain a target detection result of the detection area.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to technical fields such as deep learning and computer vision. Background Art

[0002] With the development of autonomous driving perception, people can perceive obstacles in the vehicle's surroundings through target detection. Point cloud data, image data, and millimeter wave data of the vehicle's surroundings can be obtained, and the collected data can be analyzed to perceive obstacles in the vehicle's surroundings. Summary of the invention

[0003] The present disclosure provides a target detection method, device, electronic device and medium.

[0004] According to a first aspect of the present disclosure, a target detection method is proposed, which includes: collecting a detection data set of a detection area, and extracting a first feature sequence of each detection data in the detection data set, wherein the detection data is multimodal data; acquiring a BEV space of the detection area, and projecting the image data collected in the detection area to the BEV space to obtain BEV image data of the image data; extracting a second feature sequence of the BEV image data, and acquiring a three-dimensional fused feature map by performing feature fusion of each detection data in the detection data set and the image data based on the second feature sequence; performing target detection on the three-dimensional fused feature map of each detection data respectively to obtain a target detection result of the detection area.

[0005] According to a second aspect of the present disclosure, a target detection device is proposed, which includes an acquisition module for acquiring a detection data set of a detection area and extracting a first feature sequence of each detection data in the detection data set, wherein the detection data is multimodal data; a projection module for acquiring a BEV space of the detection area and projecting the image data acquired from the detection area to the BEV space to obtain BEV image data of the image data; a fusion module for extracting a second feature sequence of the BEV image data and acquiring a three-dimensional fused feature map by performing feature fusion of each detection data in the detection data set and the image data based on the second feature sequence; and a detection module for performing target detection on the three-dimensional fused feature map of each detection data respectively to obtain a target detection result of the detection area.

[0006] According to the third aspect of the present disclosure, an electronic device is proposed, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the target detection method proposed in the first aspect above.

[0007] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is proposed, wherein the computer instructions are used to enable the computer to execute the target detection method proposed in the first aspect above.

[0008] According to a fifth aspect of the present disclosure, a computer program product is proposed, including a computer program, and when the computer program is executed by a processor, the computer program implements the target detection method proposed in the first aspect.

[0009] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0011] Figure 1 A schematic diagram of a target detection method according to an embodiment of the present disclosure;

[0012] Figure 2 A schematic diagram of a flow chart of a target detection method according to another embodiment of the present disclosure;

[0013] Figure 3 A schematic diagram of a flow chart of a target detection method according to another embodiment of the present disclosure;

[0014] Figure 4 A schematic diagram of a flow chart of a target detection method according to another embodiment of the present disclosure;

[0015] Figure 5 A schematic diagram of a flow chart of a target detection method according to another embodiment of the present disclosure;

[0016] Figure 6 A schematic diagram of coordinate correction according to an embodiment of the present disclosure;

[0017] Figure 7 is a schematic structural diagram of a target detection device according to an embodiment of the present disclosure;

[0018] Figure 8 A schematic block diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0019] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0020] Deep Learning (DL) is a new research direction in the field of machine learning. Deep learning is to learn the inherent laws and representation levels of sample data. The information obtained in the learning process is very helpful for interpreting data such as text, images and sounds. Its ultimate goal is to enable machines to have analytical learning capabilities like humans and to recognize data such as text, images and sounds.

[0021] Computer vision refers to the use of cameras and computers to replace human eyes to identify, track and measure targets, and further process images so that the computer processes them into images that are more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain "information" from images or multi-dimensional data. Its main task is to obtain three-dimensional information of the corresponding scene by processing the collected pictures or videos. It is a challenging and important research field in both engineering and science.

[0022] Artificial Intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. Artificial Intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing and expert systems. Since the birth of artificial intelligence, the theory and technology of artificial intelligence have become increasingly mature, and the application field has continued to expand. It can be imagined that the technological products brought by artificial intelligence in the future will be the "container" of human wisdom. Artificial Intelligence can simulate the information process of human consciousness and thinking.

[0023] Figure 1 FIG. 1 is a flow chart of a target detection method according to an embodiment of the present disclosure. The method is applicable to a server, such as Figure 1 As shown, the method includes:

[0024] S101, collecting a detection data set of a detection area, and extracting a first feature sequence of each detection data in the detection data set, wherein the detection data is multimodal data.

[0025] In the disclosed embodiment, during the automatic driving process, the vehicle needs to identify obstacles in the vehicle's surrounding environment to ensure the vehicle's driving safety.

[0026] In some implementations, multimodal data of the vehicle's surroundings can be collected, and any fusion method, including front fusion, mid-fusion, or post-fusion, can be adopted to fuse the collected multimodal data, thereby realizing the perception of obstacles in the vehicle's surroundings. In this scenario, the accuracy of the collected data and whether the collected data has data anomalies will have a certain degree of influence on the accuracy of the final obstacle perception and recognition, and the stability is poor.

[0027] In the disclosed embodiment, the area around the vehicle where obstacle identification is required may be marked as the detection area of ​​the vehicle.

[0028] Optionally, a multi-modal detection data acquisition device may be provided on the vehicle, which may include a laser radar device and a millimeter wave radar device, etc.

[0029] Among them, the monitoring data in the detection area can be collected by the multimodal detection data acquisition equipment installed on the vehicle, the data of the detection area can be collected by the laser radar equipment, and the data of the detection area can be collected by the millimeter wave radar equipment installed on the vehicle, so as to obtain the detection data set collected by the detection equipment installed in the detection area.

[0030] In this scenario, feature extraction can be performed on each detection data in the detection data set, thereby realizing obstacle recognition in the detection area.

[0031] Optionally, based on the feature extraction algorithm in the related technology, each detection data in the detection data set can be processed by algorithm to obtain a feature sequence extracted from each detection data, and the extracted feature sequence of each detection data can be marked as the first feature sequence of each detection data.

[0032] S102, acquiring the BEV space of the detection area, and projecting the image data collected in the detection area into the BEV space to obtain BEV image data of the image data.

[0033] In the disclosed embodiment, a corresponding Bird's Eye View (BEV) space exists in the detection area, wherein the detection area can be spatially constructed based on a BEV space construction algorithm in related technologies to obtain the BEV space of the detection area.

[0034] Optionally, a corresponding image acquisition device may be provided on the vehicle to obtain image data collected within the detection area.

[0035] In this scenario, the image data of the detection area can be projected into the BEV space corresponding to the detection area, the collected two-dimensional image data can be converted into three-dimensional image data, and the three-dimensional image data obtained after projecting into the BEV space can be determined as the BEV image data of the image data.

[0036] S103, extracting a second feature sequence of the BEV image data, and acquiring a three-dimensional fusion feature map obtained by performing feature fusion of each detection data and the image data in the detection data set based on the second feature sequence.

[0037] In the embodiments of the present disclosure, the BEV image data can be algorithmically processed based on the feature extraction algorithm in the related technology, so as to achieve feature extraction of the BEV image data, and the feature sequence extracted from the BEV image data can be marked as the second feature sequence of the BEV image data.

[0038] Optionally, the first feature sequences of each detection data included in the detection data set may be subjected to feature fusion processing with the second feature sequence respectively, so as to obtain feature sequences obtained by fusion of each first feature sequence with the second feature sequence respectively.

[0039] It can be understood that the first feature sequence of the detection data A in the detection data set is obtained and the first feature sequence is fused with the second feature sequence extracted based on the BEV image data, so as to obtain the fused feature sequence corresponding to the detection data A.

[0040] And, a first feature sequence of the detection data B in the detection data set is obtained, and the first feature sequence is fused with the second feature sequence extracted based on the BEV image data, so as to obtain a fused feature sequence corresponding to the detection data B.

[0041] Furthermore, based on the three-dimensional feature map generation algorithm in the related technology, algorithmic processing is performed on the fused feature sequence corresponding to the detection data A and the fused feature sequence corresponding to the detection data B, and based on the result of the algorithmic processing, a three-dimensional feature map including each of the detection data A and the detection data B is obtained, and then a three-dimensional fused feature map is obtained by performing feature fusion of each detection data included in the detection data set and the image data.

[0042] S104, performing target detection on the three-dimensional fusion feature map of each detection data to obtain the target detection result of the detection area.

[0043] In the disclosed embodiment, target detection may be performed on the three-dimensional fusion feature map of each detection data respectively, thereby obtaining the detection result of the three-dimensional fusion feature map of each detection data.

[0044] It can be understood that a three-dimensional feature fusion map can be obtained by fusing the first feature sequence and the second feature sequence of any detection data, and target detection can be performed on the three-dimensional feature fusion map to obtain the detection result of the target detection of the three-dimensional feature fusion map corresponding to the detection data.

[0045] And, a three-dimensional feature fusion map is obtained by fusing the first feature sequence and the second feature sequence of another detection data, and target detection is performed on the three-dimensional feature fusion map to obtain a detection result of target detection of the three-dimensional feature fusion map corresponding to the detection data.

[0046] Furthermore, a preset fusion strategy for detection results is obtained, and based on the obtained fusion strategy, the detection results of the three-dimensional fusion feature map corresponding to the point cloud data and the detection results of the three-dimensional fusion feature map corresponding to the millimeter wave data are fused, and the detection results obtained after the fusion are determined as the target detection results of the detection area.

[0047] Among them, the preset fusion strategy of the detection results can be a weighted fusion strategy, which performs weighted fusion on the detection results of the three-dimensional fusion feature map corresponding to the point cloud data and the detection results of the three-dimensional fusion feature map corresponding to the millimeter wave data based on the preset weights, and then obtains the target detection results of the detection area.

[0048] The target detection method proposed in the present disclosure collects a detection data set including point cloud data and millimeter wave data of the detection area, and extracts the first feature sequence of each detection data. Correspondingly, the image data of the detection area is collected, and the image data is projected into the BEV space constructed by the detection area. After the image data BEV image data is obtained, the second feature sequence of the BEV image data is extracted. Furthermore, the first feature sequence of each detection data in the detection data set is respectively fused with the second feature sequence, so as to obtain a three-dimensional fused feature map of each detection data in the detection data set. Target detection is performed on the three-dimensional fused feature map of each detection data respectively, and then the target detection result of the detection area is obtained according to the target detection result of each three-dimensional feature map. In the present disclosure, each detection data in the detection data set is fused with the image data respectively to obtain a three-dimensional fusion feature map of each detection data, and target detection is performed on each three-dimensional fusion feature map respectively, so as to obtain a target detection result of the detection area, and through grouping analysis of each detection data and image data, when each detection data and image data is abnormal, the influence of abnormal data on the accuracy of the target detection result is reduced, the accuracy of the target detection result and the stability of the target detection method are improved, thereby optimizing the accuracy of obstacle recognition during vehicle driving and improving the safety of vehicle driving.

[0049] In the above embodiment, the acquisition of the three-dimensional fusion feature map of each detection data can be combined with Figure 2 Further understanding, Figure 2 FIG. 1 is a flow chart of a target detection method according to another embodiment of the present disclosure. Figure 2 As shown, the method includes:

[0050] S201, collecting a detection data set of a detection area, and extracting a first feature sequence of each detection data in the detection data set, wherein the detection data is multimodal data.

[0051] In the embodiment of the present disclosure, each detection data included in the detection data set includes at least point cloud data and millimeter wave data.

[0052] Optionally, point cloud features of point cloud data in the detection data set and millimeter wave features of millimeter wave data in the detection data set may be extracted.

[0053] like Figure 3 As shown, point cloud data can be input into Figure 3 In the feature extraction network 1 shown, feature extraction is performed on the point cloud data based on the feature extraction network 1, thereby obtaining the point cloud features carried in the point cloud data.

[0054] And, input millimeter wave data into Figure 3In the feature extraction network 3 shown, feature extraction is performed on the millimeter wave data based on the feature extraction network 3, thereby obtaining the millimeter wave features carried in the millimeter wave data.

[0055] Optionally, feature stretching may be performed on the point cloud features and the millimeter wave features respectively to obtain first feature sequences of the point cloud features and the millimeter wave features respectively, so as to obtain first feature sequences of each detection data in the detection data set.

[0056] Among them, the spatial parameters of the BEV space can be obtained, and the feature stretching length required for feature stretching can be determined based on the spatial parameters.

[0057] In the embodiment of the present disclosure, the spatial parameters of the BEV space can be obtained, and the length value, width value and height value of the BEV space can be obtained from the spatial parameters of the BEV space. Further, the product value of the length value, width value and height value can be obtained, and the product value can be determined as the feature stretching length of the feature to be processed.

[0058] Optionally, feature stretching is performed on the feature to be processed based on the feature stretching length to obtain a stretched feature sequence, wherein the feature sequence includes respective first feature sequences obtained after feature stretching of the point cloud feature and the millimeter wave feature.

[0059] In the disclosed embodiment, after determining the feature stretching lengths of the point cloud features and the millimeter wave features, the point cloud features can be feature stretched based on the feature stretching lengths to obtain a feature sequence of the feature stretching lengths, and the feature sequence can be determined as the first feature sequence of the point cloud data.

[0060] Accordingly, the millimeter wave feature can be feature stretched based on the feature stretching length to obtain a feature sequence of the feature stretching length, and the feature sequence can be determined as the first feature sequence of the millimeter wave data.

[0061] As an example, the product value of the length value, the width value and the height value of the BEV space is set to M, and the feature stretching length of the point cloud feature and the millimeter wave feature is M.

[0062] In this example, the point cloud features can be stretched to obtain a feature sequence with a length of M, which is the first feature sequence of the point cloud data, and the millimeter wave features can be stretched to obtain a feature sequence with a length of M, which is the first feature sequence of the millimeter wave data.

[0063] S202, acquiring the BEV space of the detection area, and projecting the image data collected in the detection area into the BEV space to obtain BEV image data of the image data.

[0064] Optionally, image features of the image data may be extracted, and a third projection position of the image features in the BEV space may be determined.

[0065] like Figure 3 As shown, image data can be input into Figure 3 In the feature extraction network 2 shown, the image features of the image data are obtained based on the output results of the feature extraction network 2.

[0066] Among them, a third extraction position of the image feature in the image data can be obtained, a third matching position corresponding to the third extraction position in the BEV space can be determined, and the third matching position can be determined as the third projection position of the image feature in the BEV space.

[0067] In the disclosed embodiment, the extraction position of the image feature in the image data can be obtained and marked as the third extraction position, wherein the corresponding position of the third extraction position in the BEV space can be determined and marked as the third matching position.

[0068] Further, the third matching position is determined as a third projection position of the image feature in the BEV space.

[0069] Among them, the relative position relationship of the third extraction position based on the image data can be obtained, and the position in the BEV space that is in the same relative position relationship with the BEV space is determined as the third matching position corresponding to the third extraction position in the BEV space.

[0070] Further, the third matching position is determined as the projection position of the image feature of the image data at the third extraction position in the BEV space, and the projection position is marked as the third projection position of the image feature of the image data at the third extraction position in the BEV space.

[0071] Optionally, based on the third projection position, the image feature is projected into the BEV space to obtain BEV image data of the image data.

[0072] In this scenario, after determining the third projection position of the image feature at the third extraction position in the BEV space, the image feature at the third extraction position may be projected into the BEV space based on the third projection position.

[0073] Further, based on the projection of all image features, image data obtained after projecting all image features into the BEV space is determined as BEV image data of the image data.

[0074] S203, extracting BEV image features of the BEV image data, and stretching the BEV image features to obtain a second feature sequence of the BEV image data.

[0075] In the embodiments of the present disclosure, the BEV image data may be processed algorithmically based on a feature extraction algorithm in the related art, and then BEV image features in the BEV image data may be extracted based on the algorithm processing results.

[0076] Furthermore, feature stretching is performed on the BEV image features to obtain a second feature sequence of the BEV image data.

[0077] Among them, the BEV image features can be regarded as features to be processed, and the features to be processed are feature stretched based on the feature stretching length to obtain a stretched feature sequence, wherein the feature sequence includes a second feature sequence obtained after feature stretching of the BEV image features.

[0078] As an example, setting the feature stretching length to M, the BEV image feature can be stretched into a feature sequence with a length of M, and the feature sequence is the second feature sequence of the BEV image data.

[0079] S204, extracting image features of the image data and stretching the image features to obtain a third feature sequence of the image data.

[0080] like Figure 3 As shown, image data can be input into Figure 3 In the feature extraction network 2 shown, feature extraction is performed on the image data based on the feature extraction network 2 to obtain image features of the image data.

[0081] Further, feature stretching is performed on the image features based on the feature stretching length, wherein the image features are regarded as features to be processed, and feature stretching is performed on the features to be processed based on the feature stretching length to obtain a stretched feature sequence, wherein the feature sequence includes a third feature sequence obtained after feature stretching of the image features.

[0082] As an example, setting the feature stretching length to M, the image features can be stretched into a feature sequence with a length of M, and the feature sequence is the third feature sequence of the image data.

[0083] S205: Perform feature fusion on the second feature sequence and the third feature sequence to obtain a fourth feature sequence that is feature fused with each of the first feature sequences.

[0084] Optionally, a preconfigured first feature fusion network may be obtained, wherein the first feature fusion network is a two-terminal attention mechanism network.

[0085] like Figure 4 As shown, the first feature fusion network can be a two-end attention mechanism network (cross attention). After obtaining the second feature sequence and the third feature sequence, the second feature sequence and the third feature sequence can be input into Figure 4 In the first feature fusion network shown, the second feature sequence and the third feature sequence are feature fused based on the first feature fusion network, thereby obtaining a fourth feature sequence that is feature fused with the first feature sequence of each detection data.

[0086] Optionally, a first Q value can be obtained based on the second feature sequence, a first K value and a first V value can be obtained based on the third feature sequence, and the first K value, the first V value and the first Q value are input into the first feature fusion network to obtain a fourth feature sequence output by the first feature fusion network.

[0087] In the embodiment of the present disclosure, the first feature fusion network can be a two-end attention mechanism network, wherein the input data of the two-end attention mechanism network includes K value, V value and Q value.

[0088] In this scenario, the second feature sequence can be used as the Q value input into the first feature fusion network and marked as the first Q value. Correspondingly, the third feature sequence can be used as the K value and V value input into the first feature fusion network and marked as the first K value and the first V value.

[0089] like Figure 4 As shown, the first K value, the first V value and the first Q value can be input into Figure 4 In the two-end attention mechanism network shown as the first feature fusion network, based on the output result of the two-end attention mechanism network as the first feature fusion network, a feature sequence that is feature fused with the first feature sequence is obtained and marked as the fourth feature sequence.

[0090] S206, performing feature fusion on each of the first feature sequences and the fourth feature sequence respectively to obtain a first fused feature sequence of each detection data, wherein each detection data includes at least point cloud data and millimeter wave data.

[0091] In the disclosed embodiment, the first feature sequence of each detection data may be feature fused with the fourth feature sequence respectively, wherein each detection data includes at least point cloud data and millimeter wave data.

[0092] Optionally, a preconfigured second feature fusion network may be obtained, wherein the second feature fusion network is a two-terminal attention mechanism network.

[0093] For any first feature sequence, a second K value and a second V value are obtained based on the first feature sequence, a second Q value is obtained based on the fourth feature sequence, and the second K value, the second V value and the second Q value are input into the second feature fusion network to obtain the first fused feature sequence output by the second feature network.

[0094] In the embodiment of the present disclosure, the second feature fusion network can be a two-end attention mechanism network, wherein the input data of the two-end attention mechanism network includes K value, V value and Q value.

[0095] In this scenario, the first feature sequence that needs to be processed by feature fusion can be used as the K value and V value of the input second feature fusion network and marked as the second K value and the second V value, and the fourth feature sequence can be used as the Q value of the input second feature fusion network and marked as the second Q value.

[0096] like Figure 4 As shown, the first feature sequence of the point cloud data can be used as the second K value and the second V value, and input into the second Q value. Figure 4 In the two-end attention mechanism network shown as the second feature fusion network, the first fusion feature sequence between the point cloud data and the image data output by the two-end attention mechanism network is obtained.

[0097] And, the first characteristic sequence of the millimeter wave data can be used as the second K value and the second V value, and input them and the second Q value into Figure 4 In the two-end attention mechanism network shown as the second feature fusion network, the first fusion feature sequence between the millimeter wave data and the image data output by the two-end attention mechanism network is obtained.

[0098] Furthermore, a first fusion feature sequence of each detection data is obtained.

[0099] S207, based on the first fusion feature sequence, obtaining a three-dimensional fusion feature map obtained by performing feature fusion of each detection data in the detection data set and the image data.

[0100] like Figure 3 As shown, the features obtained by feature extraction based on point cloud data and the features obtained by feature extraction based on image data can be fused to obtain a three-dimensional fusion feature map corresponding to the point cloud data.

[0101] Furthermore, the features obtained by extracting features based on the millimeter wave data are fused with the features obtained by extracting features based on the image data, thereby obtaining a three-dimensional fused feature map corresponding to the millimeter wave data.

[0102] Optionally, the first fusion feature sequences of the point cloud data and the millimeter wave data may be projected into the BEV space of the detection area, respectively, to obtain three-dimensional fusion feature maps of the point cloud data and the millimeter wave data, respectively.

[0103] Among them, a first extraction position of the first fusion feature sequence in the point cloud data in the detection data set and a second extraction position in the millimeter wave data in the detection data set can be obtained.

[0104] In the embodiment of the present disclosure, for the point cloud data included in the detection data set, the position of the point cloud feature extracted from the point cloud data can be determined as the first extraction position of the point cloud feature in the point cloud data.

[0105] Furthermore, for the millimeter wave data included in the detection data set, the position where the millimeter wave feature is extracted from the millimeter wave data can be determined as the second extraction position of the millimeter wave feature in the millimeter wave data.

[0106] Optionally, a first matching position corresponding to the first extraction position in the BEV space and a second matching position corresponding to the second extraction position in the BEV space are determined.

[0107] In the embodiment of the present disclosure, for the point cloud data included in the detection data set, the first extraction position of the first fusion feature sequence in the point cloud data is obtained based on the relative position relationship of the point cloud data, and the position in the BEV space that is in this relative position relationship is determined as the first matching position of the first extraction position in the point cloud data in the BEV space.

[0108] Furthermore, for the millimeter wave data included in the detection data set, the second extraction position of the first fusion feature sequence in the millimeter wave data is obtained based on the relative position relationship of the millimeter wave data, and the position in the BEV space that is in this relative position relationship is determined as the second matching position of the second extraction position in the millimeter wave data in the BEV space.

[0109] Optionally, the first matching position is determined as the first projection position of the first fused feature sequence of the point cloud data in the BEV space, and the second matching position is determined as the second projection position of the first fused feature sequence of the millimeter wave data in the BEV space.

[0110] In the embodiment of the present disclosure, for the point cloud data included in the detection data set, the first matching position of the first extraction position of the point cloud feature in the point cloud data in the BEV space can be determined as the first projection position of the first fused feature sequence of the point cloud data at the first extraction position in the BEV space.

[0111] Furthermore, for the millimeter wave data included in the detection data set, the second matching position of the second extraction position of the millimeter wave feature in the millimeter wave data in the BEV space can be determined as the second projection position of the first fused feature sequence of the millimeter wave data at the second extraction position in the BEV space.

[0112] Optionally, based on the first projection position, a first fusion feature sequence of the point cloud data is projected into the BEV space to obtain a three-dimensional fusion feature map corresponding to the point cloud data.

[0113] In the disclosed embodiment, the first fusion feature sequence of the point cloud data at the first extraction position can be projected to the first projection position in the BEV space, thereby realizing the projection of the entire first fusion feature sequence of the point cloud data to the BEV space, so as to obtain a three-dimensional fusion feature map after feature fusion of the point cloud data and the image data.

[0114] like Figure 4 As shown, the feature map formed by projecting all the first fusion feature sequences of the point cloud data into the BEV space can be determined as a three-dimensional fusion feature map obtained after feature fusion of the point cloud data and the image data.

[0115] Optionally, based on the second projection position, the first fusion feature sequence of the millimeter wave data is projected into the BEV space to obtain a three-dimensional fusion feature map corresponding to the millimeter wave data.

[0116] In the disclosed embodiment, the first fusion feature sequence of the millimeter wave data at the second extraction position can be projected to the second projection position in the BEV space, thereby realizing the projection of the entire first fusion feature sequence of the millimeter wave data to the BEV space, so as to obtain a three-dimensional fusion feature map after feature fusion of the millimeter wave data and the image data.

[0117] like Figure 4 As shown, the feature map formed by projecting all the first fusion feature sequences of the millimeter wave data into the BEV space can be determined as a three-dimensional fusion feature map obtained after feature fusion of the millimeter wave data and the image data.

[0118] S208, performing target detection on the three-dimensional fusion feature map of each detection data to obtain the target detection result of the detection area.

[0119] In the disclosed embodiment, target detection may be performed on the three-dimensional fusion feature maps of all the detection data included in the detection data set.

[0120] Among them, the three-dimensional fusion feature maps of each detection data can be respectively input into the 3D detection head for target detection to obtain the candidate detection results of each three-dimensional fusion feature map output by the 3D detection head.

[0121] As an example, Figure 3As shown, the three-dimensional fusion feature map obtained by fusing the point cloud data and the image data included in each detection data can be input into the 3D detection head for target detection, and the three-dimensional fusion feature map obtained by fusing the millimeter wave data and the image data included in each detection data can be input into the 3D detection head for target detection, and the detection results of the three-dimensional fusion feature map obtained by fusing the point cloud data and the image data for target detection and the detection results of the three-dimensional fusion feature map obtained by fusing the millimeter wave data and the image data for target detection are obtained as candidate detection results, respectively.

[0122] As another example, Figure 4 As shown, the three-dimensional fusion feature map of the point cloud data in the detection data set can be input into the 3D detection head, and target detection can be performed on the three-dimensional fusion feature map of the point cloud data based on the 3D detection head, and the result obtained by the target detection is determined as the candidate detection result of the three-dimensional fusion feature map of the point cloud data.

[0123] And, if Figure 4 As shown, the three-dimensional fusion feature map of the millimeter wave data in the detection data set can be input into the 3D detection head, and target detection can be performed on the three-dimensional fusion feature map of the millimeter wave data based on the 3D detection head, and the result obtained by the target detection is determined as the candidate detection result of the three-dimensional fusion feature map of the millimeter wave data.

[0124] Optionally, all candidate detection results may be fused based on Kalman filtering to obtain a target detection result for the detection area.

[0125] As an example, Figure 3 As shown, a Kalman filter is set behind the 3D detection head, which can input all the candidate detection results obtained into Figure 3 In the Kalman filter shown, the final target detection result of the detection area is obtained based on the output result of the Kalman filter.

[0126] As another example, Figure 4 As shown, the candidate detection results obtained by performing target detection on the three-dimensional fusion feature map of the point cloud data and the candidate detection results obtained by performing target detection on the three-dimensional fusion feature map of the millimeter wave data can be input into Figure 4 The Kalman filter shown.

[0127] Furthermore, the candidate detection results are fused based on Kalman filtering, and then the final target detection result of the detection area is output.

[0128] The target detection method proposed in the present disclosure extracts BEV image features of BEV image data and performs feature stretching on them to obtain a second feature sequence, and extracts image features of image data and performs feature stretching on them to obtain a third feature sequence, and performs feature fusion on the second feature sequence and the third feature sequence to obtain a fourth feature sequence that is feature fused with each first feature sequence. Furthermore, the first feature sequence of point cloud data included in the detection data set is fused with the fourth feature sequence to obtain a three-dimensional fused feature map obtained by fusing the point cloud data with the image data, and the first feature sequence of millimeter wave data is fused with the fourth feature sequence to obtain a three-dimensional fused feature map obtained by fusing the millimeter wave data with the image data, and target detection is performed on the three-dimensional fused feature maps of each detection data including the point cloud data and the millimeter wave data, respectively, to obtain candidate detection results of the three-dimensional fused feature maps of each detection data, and then all candidate detection results are fused based on Kalman filtering to obtain target detection results of the detection area. In the present disclosure, by grouping and analyzing point cloud data, millimeter wave data and image data, when abnormalities occur in point cloud data, image data and millimeter wave data, the influence of abnormal data on the accuracy of target detection results is reduced, the accuracy of target detection results and the stability of the target detection method are improved, thereby optimizing the accuracy of obstacle recognition during vehicle driving and improving the safety of vehicle driving.

[0129] The target detection method proposed in the present disclosure needs to identify and correct the coordinates of the feature extraction in the point cloud data, millimeter wave data and image data before extracting features from the point cloud data, millimeter wave data and image data. Figure 5 Further understanding, Figure 5 FIG. 1 is a flow chart of a target detection method according to another embodiment of the present disclosure. Figure 5 As shown, the method includes:

[0130] S501, obtaining a feature extraction object of a detection area and a preconfigured multi-layer perceptron.

[0131] like Figure 4 As shown, when extracting features from point cloud data, millimeter wave data, and image data, it is possible to Figure 4 Multilayer Perceptron (MLP) implementation shown.

[0132] Optionally, the point cloud data, millimeter wave data and image data can be respectively input into respective multilayer perceptrons, and based on the output results of the multilayer perceptron, the point cloud features of the point cloud data, the millimeter wave features of the millimeter wave data and the image features of the image data can be obtained.

[0133] Among them, the objects that need to be feature extracted in the detection area can be marked as feature extraction objects in the detection area.

[0134] S502, inputting the point cloud data, the millimeter wave data and the image data into the multilayer perceptron respectively, and obtaining the first perception coordinates of the feature extraction object output by the multilayer perceptron in the point cloud data, the second perception coordinates of the millimeter wave data and the third perception coordinates of the image data.

[0135] like Figure 4 As shown, the point cloud data can be input into a preconfigured MLP to obtain the coordinates of the corresponding position of the feature extraction object recognized by the MLP in the point cloud data as the first perceived coordinates of the feature extraction object output by the MLP in the point cloud data.

[0136] Accordingly, the millimeter wave data can be input into a preconfigured MLP to obtain the coordinates of the corresponding position of the feature extraction object identified by the MLP in the millimeter wave data as the second perceived coordinates of the feature extraction object output by the MLP in the millimeter wave data.

[0137] Furthermore, the image data may be input into a preconfigured MLP to obtain the coordinates of the corresponding position of the feature extraction object identified by the MLP in the image data as the third perceptual coordinates of the feature extraction object output by the MLP in the image data.

[0138] S503, identifying whether there is an error in the output result of the multilayer perceptron.

[0139] Optionally, in response to the point cloud object of the point cloud data at the first perception coordinate being different from the feature extraction object, and / or the millimeter wave object of the millimeter wave data at the second perception coordinate being different from the feature extraction object, and / or the image object of the image data at the third perception coordinate being different from the feature extraction object, it is determined that an error exists in the output result of the multilayer perceptron.

[0140] Among them, the object corresponding to the point cloud data on the first perception coordinate collected by the point cloud data in the detection area can be obtained, and the object can be determined as the point cloud object of the point cloud data on the first perception coordinate.

[0141] Accordingly, the object corresponding to the millimeter wave data at the second perception coordinate collected by the millimeter wave data within the detection area can be obtained, and the object can be determined as the millimeter wave object of the millimeter wave data at the first perception coordinate, and the object corresponding to the image data at the third perception coordinate collected by the image data within the detection area can be obtained, and the object can be determined as the image object of the image data at the third perception coordinate.

[0142] In this scenario, the point cloud object of the point cloud data at the first perception coordinate, the millimeter wave object of the millimeter wave data at the second perception coordinate, and the image object of the image data at the third perception coordinate can be compared with the feature extraction objects in the detection area, and then the output result of the MLP can be identified whether there is an error based on the comparison result.

[0143] Optionally, when the point cloud object of the point cloud data at the first perception coordinate is different from the feature extraction object within the detection area, and / or the millimeter wave object of the millimeter wave data at the second perception coordinate is different from the feature extraction object within the detection area, and / or the image object of the image data at the third perception coordinate is different from the feature extraction object within the detection area, it can be determined that there is an error in the output result of the current MLP.

[0144] S504, in response to identifying that there is an error in the output result of the multilayer perceptron, obtaining the error coordinates in the first perception coordinates, the second perception coordinates, and the third perception coordinates, and performing coordinate correction on the error coordinates to obtain corrected first target feature extraction coordinates.

[0145] Optionally, an error object different from the feature extraction object in the point cloud object, the millimeter wave object, and the image object may be acquired, and the perception coordinates corresponding to the error object may be determined as the error coordinates.

[0146] In the disclosed embodiment, objects among the point cloud objects on the first perception coordinates, the millimeter wave objects on the second perception coordinates, and the image objects on the third perception coordinates that are different from the feature extraction objects in the detection area can be marked as error objects.

[0147] In this scenario, the perception coordinates corresponding to the error object may be acquired and determined as the error coordinates among the first perception coordinates, the second perception coordinates, and the third perception coordinates.

[0148] It can be understood that if the perceived coordinates corresponding to the error object are the first perceived coordinates, then the first perceived coordinates can be determined as the error coordinates, if the perceived coordinates corresponding to the error object are the second perceived coordinates, then the second perceived coordinates can be determined as the error coordinates, and if the perceived coordinates corresponding to the error object are the third perceived coordinates, then the third perceived coordinates can be determined as the error coordinates.

[0149] Optionally, the relative coordinate relationship between the point cloud data, the millimeter wave data and the image data may be acquired.

[0150] In the disclosed embodiment, the relative coordinate relationship among the point cloud data, millimeter wave data and image data within a historical time range can be obtained.

[0151] Optionally, historical reference objects of the historical detection area, as well as historical point cloud data, historical millimeter wave data, and historical image data collected in the historical detection area may be acquired.

[0152] In the disclosed embodiment, the relative coordinate relationship between point cloud data, millimeter wave data and image data can be obtained based on historical data collected from the historical detection area of ​​the vehicle, wherein the historical reference object of the historical detection area, as well as the historical point cloud data, historical millimeter wave data and historical image data collected from the historical detection area can be obtained.

[0153] Among them, the detection area of ​​the vehicle within the historical time range can be obtained and marked as the historical detection area of ​​the vehicle.

[0154] Furthermore, data is collected for the historical detection area based on the laser radar equipment, millimeter wave radar equipment and image acquisition equipment installed on the vehicle, and the point cloud data collected by the laser radar equipment is marked as historical point cloud data of the historical detection area, the millimeter wave data collected by the millimeter wave radar is marked as historical millimeter wave data of the historical detection area, and the image data collected by the image acquisition equipment is marked as historical image data of the historical detection area.

[0155] In this scenario, objects that require feature extraction within the detection area may be obtained as historical reference objects of the historical detection area.

[0156] Optionally, the historical point cloud data, historical millimeter wave data and historical image data are respectively input into a multilayer perceptron to obtain the first historical perception coordinates of the historical reference object output by the multilayer perceptron in the historical point cloud data, the second historical perception coordinates in the historical millimeter wave data and the third historical perception coordinates in the historical image data.

[0157] In the disclosed embodiment, historical point cloud data, historical millimeter wave data, and historical image data can be respectively input into a multilayer perceptron to obtain the corresponding position coordinates of the historical reference object identified by the multilayer perceptron in the historical point cloud data as the first historical perception coordinates, the corresponding position coordinates of the historical reference object in the historical millimeter wave data as the second historical perception coordinates, and the corresponding position coordinates of the historical reference object in the historical image data as the third historical perception coordinates.

[0158] Optionally, first historically extracted coordinates of the historical reference object in the historical point cloud data, second historically extracted coordinates in the millimeter wave data, and second historically extracted coordinates in the historical image data are acquired.

[0159] In the disclosed embodiment, the coordinates of the actual position of the historical reference object in the historical point cloud data can be obtained as the first historical extracted coordinates of the historical reference object in the historical point cloud data, as well as the coordinates of the actual position of the historical reference object in the historical millimeter wave data as the second historical perceived coordinates of the historical reference object in the historical millimeter wave data, and as the third historical perceived coordinates of the historical reference object in the historical image data.

[0160] Furthermore, based on the first historical perception coordinates, the first historical extraction coordinates, the second historical perception coordinates, the second historical extraction coordinates, the third historical perception coordinates and the third historical extraction coordinates, the relative coordinate relationship between the point cloud data, the millimeter wave data and the image data is obtained.

[0161] Optionally, based on the learning algorithm of coordinate relationships in related technologies, the first historical perceived coordinates, the first historical extracted coordinates, the second historical perceived coordinates, the second historical extracted coordinates, the third historical perceived coordinates and the third historical extracted coordinates can be algorithmically processed, and the relative coordinate relationship between the point cloud data, millimeter wave data and image data can be obtained based on the results of the algorithm processing.

[0162] Optionally, correction parameters of the error coordinates are determined based on the relative coordinate relationship, and the error coordinates are adjusted based on the correction parameters to obtain corrected first target feature extraction coordinates.

[0163] In the disclosed embodiment, the error coordinates may be corrected based on the relative coordinate relationship, and the coordinates obtained after the correction may be determined as the target feature extraction coordinates of the feature extraction object and marked as the first target feature extraction coordinates.

[0164] Among them, relevant parameters such as the correction distance and correction direction of the error coordinates can be determined according to the relative coordinate relationship, and these parameters can be determined as the correction parameters of the error coordinates.

[0165] In this scenario, the error coordinates may be moved by the correction distance and the correction direction in the error coordinates based on the acquired correction parameters, thereby obtaining coordinates after the error coordinates are corrected, and the coordinates are determined as the first target feature extraction coordinates.

[0166] In the scenario where the error coordinate is the first perception coordinate, such as Figure 6 As shown, set Figure 6 If the first perception coordinates corresponding to point 1 in the point cloud are error coordinates, the first perception coordinates as error coordinates can be corrected based on the correction parameters, and the coordinates corresponding to point 1' obtained after correction are determined as the first target feature extraction coordinates of the feature extraction object in the point cloud data.

[0167] And, set Figure 6 If the first perception coordinates corresponding to point 2 in the point cloud are error coordinates, the first perception coordinates as error coordinates can be corrected based on the correction parameters, and the coordinates corresponding to point 2' obtained after correction are determined as the first target feature extraction coordinates of the feature extraction object in the point cloud data.

[0168] In the scenario where the error coordinate is the second perception coordinate, such as Figure 6 As shown, set Figure 6 If the second perception coordinates corresponding to point 1 in the image are error coordinates, the second perception coordinates as error coordinates can be corrected based on the correction parameters, and the coordinates corresponding to point 1' obtained after correction are determined as the first target feature extraction coordinates of the feature extraction object in the millimeter wave data.

[0169] And, set Figure 6 If the second perception coordinates corresponding to point 2 in the image are error coordinates, the second perception coordinates as error coordinates can be corrected based on the correction parameters, and the coordinates corresponding to point 2' obtained after correction are determined as the first target feature extraction coordinates of the feature extraction object in the millimeter wave data.

[0170] In the scenario where the error coordinate is the third perception coordinate, such as Figure 6 As shown, set Figure 6 If the third perception coordinates corresponding to point 1 in the image are error coordinates, the third perception coordinates as error coordinates can be corrected based on the correction parameters, and the coordinates corresponding to point 1' obtained after correction can be determined as the first target feature extraction coordinates of the feature extraction object in the image data.

[0171] And, set Figure 6 If the third perceived coordinates corresponding to point 2 in the image are error coordinates, the third perceived coordinates as error coordinates can be corrected based on the correction parameters, and the coordinates corresponding to point 2' obtained after correction can be determined as the first target feature extraction coordinates of the feature extraction object in the image data.

[0172] It should be noted that after the error coordinates are corrected, the remaining perception coordinates other than the error coordinates in the first perception coordinates, the second perception coordinates and the third perception coordinates can also be obtained, the sampling data corresponding to the remaining perception coordinates can be determined, and the remaining perception coordinates can be determined as the second target feature extraction coordinates of the feature extraction object in the sampling data, wherein the sampling data includes at least one of point cloud data, millimeter wave data and image data of the detection area.

[0173] In the disclosed embodiment, the perceived coordinates that are not identified as error coordinates among the first perceived coordinates, the second perceived coordinates, and the third perceived coordinates may be marked as remaining perceived coordinates.

[0174] In this scenario, when the remaining perception coordinates are the first perception coordinates, the corresponding sampling data is point cloud data, when the remaining perception coordinates are the second perception coordinates, the corresponding sampling data is millimeter wave data, and when the remaining perception coordinates are the third perception coordinates, the corresponding sampling data is image data.

[0175] In the disclosed embodiment, the remaining perception coordinates are the perception coordinates that have not been determined as error coordinates. It can be seen that the object on the remaining perception coordinates output by the MLP is the same as the feature extraction object. In this scenario, the remaining perception coordinates can be determined as the target feature extraction coordinates of the feature extraction object in its corresponding sampling data and marked as the second target feature extraction coordinates.

[0176] It can be understood that when the remaining perception coordinates are the first perception coordinates, the first perception coordinates can be determined as the second target feature extraction coordinates of the feature extraction object in the point cloud data, and the point cloud features of the feature extraction object can be obtained from the point cloud data based on the second target feature extraction coordinates.

[0177] Accordingly, when the remaining perception coordinate is the second perception coordinate, the second perception coordinate can be determined as the second target feature extraction coordinate of the feature extraction object in the millimeter wave data, and the millimeter wave feature of the feature extraction object can be obtained from the millimeter wave data based on the second target feature extraction coordinate.

[0178] And, when the remaining perception coordinate is the third perception coordinate, the third perception coordinate can be determined as the second target feature extraction coordinate of the feature extraction object in the image data, and the image feature of the feature extraction object is obtained from the image data based on the second target feature extraction coordinate.

[0179] The target detection method proposed in the present disclosure inputs point cloud data, millimeter wave data and image data into a multilayer perceptron respectively, obtains the first perception coordinates of the feature extraction object output by the multilayer perceptron in the point cloud data, the second perception coordinates in the millimeter wave data and the third perception coordinates in the image data, and performs error recognition on the output result of the multilayer perceptron. When it is recognized that there is an error in the output result of the multilayer perceptron, the error coordinates in the first perception coordinates, the second perception coordinates and the third perception coordinates are obtained, and the error coordinates are corrected to obtain the corrected first target feature extraction coordinates. In the present disclosure, by performing error recognition and correction on the first perception coordinates in the point cloud data, the second perception coordinates in the millimeter wave data and the third perception coordinates in the image data, the accuracy of the features obtained by feature extraction of the point cloud data, the millimeter wave data and the image data, as well as the accuracy of the subsequent projection of the BEV space, thereby improving the accuracy of the target detection results, optimizing the stability of the target detection method and the accuracy of obstacle recognition during vehicle driving, and improving the safety of vehicle driving.

[0180] Corresponding to the target detection methods proposed in the above-mentioned embodiments, an embodiment of the present disclosure further proposes a target detection device applicable to a server. Since the target detection device applicable to a server proposed in the embodiment of the present disclosure corresponds to the target detection methods applicable to a server proposed in the above-mentioned embodiments, the implementation methods of the above-mentioned target detection methods applicable to a server are also applicable to the target detection device applicable to a server proposed in the embodiment of the present disclosure, and will not be described in detail in the following embodiments.

[0181] Figure 7 FIG. 1 is a schematic diagram of a target detection device according to an embodiment of the present disclosure. Figure 7 As shown, the target detection device 700 includes a collection module 71, a projection module 72, a fusion module 73 and a detection module 74, wherein:

[0182] The acquisition module 71 is used to acquire a detection data set of the detection area and extract a first feature sequence of each detection data in the detection data set, wherein the detection data is multimodal data;

[0183] A projection module 72 is used to obtain the BEV space of the detection area, and project the image data collected in the detection area to the BEV space to obtain BEV image data of the image data;

[0184] A fusion module 73 is used to extract a second feature sequence of the BEV image data, and obtain a three-dimensional fusion feature map obtained by performing feature fusion of each detection data and the image data in the detection data set based on the second feature sequence;

[0185] The detection module 74 is used to perform target detection on the three-dimensional fusion feature map of each detection data to obtain the target detection result of the detection area.

[0186] In the disclosed embodiment, the fusion module 73 is also used to: extract BEV image features of BEV image data, and stretch the BEV image features to obtain a second feature sequence of BEV image data; extract image features of image data and stretch the image features to obtain a third feature sequence of image data; perform feature fusion on the second feature sequence and the third feature sequence to obtain a fourth feature sequence that is feature fused with each first feature sequence; perform feature fusion on each first feature sequence and the fourth feature sequence respectively to obtain a first fused feature sequence of each detection data, wherein each detection data includes at least point cloud data and millimeter wave data; based on the first fused feature sequence, obtain a three-dimensional fused feature map obtained by feature fusion of each detection data and the image data in the detection data set.

[0187] In the disclosed embodiment, the fusion module 73 is also used to: obtain a preconfigured first feature fusion network, wherein the first feature fusion network is a two-end attention mechanism network; obtain a first Q value based on the second feature sequence, and obtain a first K value and a first V value based on the third feature sequence; input the first K value, the first V value and the first Q value into the first feature fusion network to obtain a fourth feature sequence output by the first feature fusion network.

[0188] In the disclosed embodiment, the fusion module 73 is also used to: obtain a preconfigured second feature fusion network, wherein the second feature fusion network is a two-end attention mechanism network; for any first feature sequence, obtain a second K value and a second V value based on the first feature sequence, and obtain a second Q value based on the fourth feature sequence; input the second K value, the second V value and the second Q value into the second feature fusion network to obtain a first fused feature sequence output by the second feature network.

[0189] In the disclosed embodiment, the fusion module 73 is also used to: obtain a first extraction position of the first fusion feature sequence in the point cloud data in the detection data set, and a second extraction position in the millimeter wave data in the detection data set; determine a first matching position corresponding to the first extraction position in the BEV space, and a second matching position corresponding to the second extraction position in the BEV space; determine the first matching position as the first projection position of the first fusion feature sequence of the point cloud data in the BEV space, and determine the second matching position as the second projection position of the first fusion feature sequence of the millimeter wave data in the BEV space; based on the first projection position, project the first fusion feature sequence of the point cloud data to the BEV space to obtain a three-dimensional fusion feature map corresponding to the point cloud data; based on the second projection position, project the first fusion feature sequence of the millimeter wave data to the BEV space to obtain a three-dimensional fusion feature map corresponding to the millimeter wave data.

[0190] In the disclosed embodiment, the acquisition module 71 is also used to: extract point cloud features of point cloud data in the detection data set, and millimeter wave features of millimeter wave data in the detection data set; perform feature stretching on the point cloud features and millimeter wave features respectively to obtain the first feature sequences of the point cloud features and the millimeter wave features respectively, so as to obtain the first feature sequence of each detection data in the detection data set.

[0191] In the disclosed embodiment, the projection module 72 is also used to: extract image features of the image data and determine a third projection position of the image features in the BEV space; based on the third projection position, project the image features into the BEV space to obtain BEV image data of the image data.

[0192] In the disclosed embodiment, the projection module 72 is also used to: obtain a third extraction position of the image feature in the image data; determine a third matching position corresponding to the third extraction position in the BEV space, and determine the third matching position as the third projection position of the image feature in the BEV space.

[0193] In the disclosed embodiment, the detection module 74 is also used to: input the three-dimensional fusion feature maps of each detection data into the 3D detection head for target detection, and obtain candidate detection results of each three-dimensional fusion feature map output by the 3D detection head; fuse all the candidate detection results based on Kalman filtering to obtain the target detection result of the detection area.

[0194] In the disclosed embodiment, the device also includes a stretching module, which is used to: obtain spatial parameters of the BEV space, and determine the feature stretching length required for feature stretching based on the spatial parameters; perform feature stretching on the feature to be processed based on the feature stretching length to obtain a feature sequence after stretching, wherein the feature sequence includes respective first feature sequences obtained after feature stretching of point cloud features and millimeter wave features, a second feature sequence obtained after feature stretching of BEV image features, and a third feature sequence obtained after feature stretching of image features.

[0195] In the disclosed embodiment, the stretching module is also used to: obtain the length value, width value and height value of the BEV space from the spatial parameters of the BEV space; obtain the product value of the length value, width value and height value, and determine the product value as the feature stretching length of the feature to be processed.

[0196] In the embodiment of the present disclosure, the device also includes a correction module, which is used to: obtain a feature extraction object of the detection area and a preconfigured multilayer perceptron; input point cloud data, millimeter wave data, and image data into the multilayer perceptron respectively to obtain the feature extraction object output by the multilayer perceptron in the first perception coordinates of the point cloud data, the second perception coordinates of the millimeter wave data, and the third perception coordinates of the image data; identify whether there is an error in the output result of the multilayer perceptron; in response to identifying that there is an error in the output result of the multilayer perceptron, obtain the error coordinates in the first perception coordinates, the second perception coordinates, and the third perception coordinates, and perform coordinate correction on the error coordinates to obtain the corrected first target feature extraction coordinates.

[0197] In the disclosed embodiment, the correction module is also used to: in response to the point cloud object of the point cloud data at the first perception coordinate being different from the feature extraction object, and / or the millimeter wave object of the millimeter wave data at the second perception coordinate being different from the feature extraction object, and / or the image object of the image data at the third perception coordinate being different from the feature extraction object, determine and recognize that there is an error in the output result of the multilayer perceptron.

[0198] In the disclosed embodiment, the correction module is also used to: obtain error objects that are different from the feature extraction objects in the point cloud objects, millimeter wave objects and image objects, and determine the perceived coordinates corresponding to the error objects as error coordinates; obtain the relative coordinate relationship between the point cloud data, millimeter wave data and image data; determine the correction parameters of the error coordinates based on the relative coordinate relationship, and adjust the error coordinates based on the correction parameters to obtain the corrected first target feature extraction coordinates.

[0199] In the disclosed embodiment, the correction module is also used to: obtain historical reference objects in the historical detection area, as well as historical point cloud data, historical millimeter wave data and historical image data collected in the historical detection area; input the historical point cloud data, historical millimeter wave data and historical image data into the multilayer perceptron respectively to obtain the first historical perception coordinates of the historical reference object output by the multilayer perceptron in the historical point cloud data, the second historical perception coordinates in the historical millimeter wave data and the third historical perception coordinates in the historical image data; obtain the first historical extraction coordinates of the historical reference object in the historical point cloud data, the second historical extraction coordinates in the millimeter wave data and the second historical extraction coordinates in the historical image data; based on the first historical perception coordinates, the first historical extraction coordinates, the second historical perception coordinates, the second historical extraction coordinates, the third historical perception coordinates and the third historical extraction coordinates, obtain the relative coordinate relationship between the point cloud data, the millimeter wave data and the image data.

[0200] In the embodiment of the present disclosure, the correction module is also used to: obtain the remaining perception coordinates other than the error coordinates in the first perception coordinates, the second perception coordinates and the third perception coordinates; determine the sampling data corresponding to the remaining perception coordinates, and determine the remaining perception coordinates as the second target feature extraction coordinates of the feature extraction object in the sampling data, wherein the sampling data includes at least one of point cloud data, millimeter wave data and image data of the detection area.

[0201] The target detection device proposed in the present disclosure collects a detection data set including point cloud data and millimeter wave data of the detection area, and extracts the first feature sequence of each detection data. Correspondingly, the image data of the detection area is collected, and the image data is projected into the BEV space constructed by the detection area. After the image data BEV image data is obtained, the second feature sequence of the BEV image data is extracted. Furthermore, the first feature sequence of each detection data in the detection data set is respectively fused with the second feature sequence, so as to obtain the three-dimensional fusion feature map of each detection data in the detection data set. Target detection is performed on the three-dimensional fusion feature map of each detection data respectively, and then the target detection result of the detection area is obtained according to the target detection result of each three-dimensional feature map. In the present invention, each detection data in the detection data set is fused with the image data respectively to obtain a three-dimensional fusion feature map of each detection data in the detection data set, and target detection is performed on each three-dimensional fusion feature map respectively, so as to obtain the target detection result of the detection area, and through grouping analysis of each detection data and image data in the detection data set, when each detection data and image data in the detection data set is abnormal, the influence of the abnormal data on the accuracy of the target detection result is reduced, the accuracy of the target detection result and the stability of the target detection method are improved, thereby optimizing the accuracy of obstacle recognition during vehicle driving and improving the safety of vehicle driving.

[0202] According to an embodiment of the present disclosure, the present disclosure also proposes an electronic device, a readable storage medium, and a computer program product.

[0203] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0204] like Figure 8As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0205] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 806, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0206] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as the target detection method. For example, in some embodiments, the target detection method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the target detection method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the target detection method in any other appropriate manner (e.g., by means of firmware).

[0207] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0208] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be presented to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, implements the functions / operations specified in the flow chart and / or block diagram. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0209] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0210] To propose interactions with a user account, the systems and techniques described herein may be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user account; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user account can propose input to the computer. Other types of devices may also be used to propose interactions with the user account; for example, feedback proposed to the user account may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user account may be received in any form (including acoustic input, voice input, or tactile input).

[0211] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user account computer with a graphical user account interface or a web browser through which a user account can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0212] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0213] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0214] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A target detection method, wherein: The method comprises: Collecting a detection data set of the detection area, and extracting a first feature sequence of each detection data in the detection data set, wherein the detection data is multimodal data; Acquire the BEV space of the detection area, and project the image data collected by the detection area to the BEV space to obtain BEV image data of the image data; Extracting a second feature sequence of the BEV image data, and acquiring a three-dimensional fusion feature map obtained by performing feature fusion of each detection data and the image data in the detection data set based on the second feature sequence; Target detection is performed on the three-dimensional fusion feature map of each detection data to obtain the target detection result of the detection area.

2. The method according to claim 1, wherein: The extracting the second feature sequence of the BEV image data, and acquiring a three-dimensional fusion feature map obtained by performing feature fusion of each detection data and the image data in the detection data set based on the second feature sequence, includes: extracting BEV image features of the BEV image data, and stretching the BEV image features to obtain the second feature sequence of the BEV image data; Extracting image features of the image data and stretching the image features to obtain a third feature sequence of the image data; Performing feature fusion on the second feature sequence and the third feature sequence to obtain a fourth feature sequence that is feature fused with each of the first feature sequences; Performing feature fusion on each first feature sequence and the fourth feature sequence respectively to obtain a first fused feature sequence of each detection data, wherein each detection data includes at least point cloud data and millimeter wave data; Based on the first fusion feature sequence, a three-dimensional fusion feature map is obtained by performing feature fusion of each detection data and image data in the detection data set.

3. The method according to claim 2, wherein: The step of fusing the second feature sequence and the third feature sequence to obtain a fourth feature sequence that is fused with the first feature sequences includes: Obtain a preconfigured first feature fusion network, wherein the first feature fusion network is a two-end attention mechanism network; Obtain a first Q value based on the second characteristic sequence, and obtain a first K value and a first V value based on the third characteristic sequence; The first K value, the first V value and the first Q value are input into the first feature fusion network to obtain the fourth feature sequence output by the first feature fusion network.

4. The method according to claim 2, wherein: The step of fusing the first feature sequences with the fourth feature sequences to obtain the first fused feature sequences of the detection data comprises: Obtain a preconfigured second feature fusion network, wherein the second feature fusion network is a two-end attention mechanism network; For any first characteristic sequence, a second K value and a second V value are obtained based on the first characteristic sequence, and a second Q value is obtained based on the fourth characteristic sequence; The second K value, the second V value and the second Q value are input into the second feature fusion network to obtain the first fusion feature sequence output by the second feature network.

5. The method according to claim 2, wherein: The step of obtaining, based on the first fusion feature sequence, a three-dimensional fusion feature map obtained by performing feature fusion of each detection data in the detection data set and the image data includes: Acquire a first extraction position of the first fused feature sequence in the point cloud data in the detection data set, and a second extraction position in the millimeter wave data in the detection data set; Determine a first matching position corresponding to the first extraction position in the BEV space, and a second matching position corresponding to the second extraction position in the BEV space; Determine the first matching position as the first projection position of the first fused feature sequence of the point cloud data in the BEV space, and determine the second matching position as the second projection position of the first fused feature sequence of the millimeter wave data in the BEV space; Based on the first projection position, projecting the first fusion feature sequence of the point cloud data to the BEV space to obtain the three-dimensional fusion feature map corresponding to the point cloud data; Based on the second projection position, the first fusion feature sequence of the millimeter wave data is projected to the BEV space to obtain the three-dimensional fusion feature map corresponding to the millimeter wave data.

6. The method according to claim 1, wherein: The collecting of a detection data set of the detection area and extracting a first feature sequence of each detection data in the detection data set, wherein the detection data is multimodal data, includes: Extracting point cloud features of the point cloud data in the detection data set and millimeter wave features of the millimeter wave data in the detection data set; Feature stretching is performed on the point cloud features and the millimeter wave features respectively to obtain first feature sequences of the point cloud features and the millimeter wave features respectively, so as to obtain first feature sequences of each detection data in the detection data set.

7. The method according to claim 1, wherein: The step of acquiring the BEV space of the detection area and projecting the image data collected by the detection area to the BEV space to obtain the BEV image data of the image data includes: extracting an image feature of the image data, and determining a third projection position of the image feature in the BEV space; Based on the third projection position, the image feature is projected into the BEV space to obtain the BEV image data of the image data.

8. The method according to claim 7, wherein: The extracting the image feature of the image data and determining the third projection position of the image feature in the BEV space includes: Acquire a third extraction position of the image feature in the image data; A third matching position corresponding to the third extraction position in the BEV space is determined, and the third matching position is determined as the third projection position of the image feature in the BEV space.

9. The method according to claim 1, wherein: The performing target detection on the three-dimensional fusion feature map of each detection data to obtain the target detection result of the detection area includes: Inputting the 3D fusion feature maps of each detection data into the 3D detection head for target detection, and obtaining candidate detection results of each 3D fusion feature map output by the 3D detection head; All candidate detection results are fused based on Kalman filtering to obtain the target detection result of the detection area.

10. The method according to any one of claims 1 to 9, wherein: The method further comprises: Acquiring spatial parameters of the BEV space, and determining a feature stretching length required for feature stretching based on the spatial parameters; Based on the feature stretching length, feature stretching is performed on the feature to be processed to obtain a stretched feature sequence, wherein the feature sequence includes respective first feature sequences obtained after feature stretching of point cloud features and millimeter wave features, a second feature sequence obtained after feature stretching of BEV image features, and a third feature sequence obtained after feature stretching of image features.

11. The method according to claim 10, wherein: The acquiring of the spatial parameters of the BEV space and determining the characteristic stretching length required for characteristic stretching based on the spatial parameters includes: Acquire a length value, a width value, and a height value of the BEV space from the space parameters of the BEV space; The product value of the length value, the width value and the height value is obtained, and the product value is determined as the feature stretching length of the feature to be processed.

12. The method according to any one of claims 1 to 9, wherein: The method further comprises: Acquire a feature extraction object and a preconfigured multi-layer perceptron of the detection area; Inputting the point cloud data, the millimeter wave data and the image data into the multilayer perceptron respectively, and obtaining the first perception coordinates of the point cloud data, the second perception coordinates of the millimeter wave data and the third perception coordinates of the image data of the feature extraction object output by the multilayer perceptron; Identify whether there is an error in the output result of the multilayer perceptron; In response to identifying that an output result of the multilayer perceptron has an error, the error coordinates among the first perception coordinates, the second perception coordinates, and the third perception coordinates are acquired, and coordinate correction is performed on the error coordinates to obtain corrected first target feature extraction coordinates.

13. The method according to claim 12, wherein: The step of identifying whether an output result of the multilayer perceptron has an error comprises: In response to the point cloud object of the point cloud data at the first perception coordinate being different from the feature extraction object, and / or the millimeter wave object of the millimeter wave data at the second perception coordinate being different from the feature extraction object, and / or the image object of the image data at the third perception coordinate being different from the feature extraction object, it is determined that an error exists in the output result of the multilayer perceptron.

14. The method according to claim 12, wherein: In response to identifying that an output result of the multilayer perceptron has an error, acquiring error coordinates among the first perception coordinates, the second perception coordinates, and the third perception coordinates, and performing coordinate correction on the error coordinates to obtain corrected first target feature extraction coordinates, including: Acquire an error object different from the feature extraction object in a point cloud object, a millimeter wave object, and an image object, and determine the perception coordinates corresponding to the error object as the error coordinates; Acquiring a relative coordinate relationship between the point cloud data, the millimeter wave data, and the image data; A correction parameter of the error coordinate is determined based on the relative coordinate relationship, and the error coordinate is adjusted based on the correction parameter to obtain a corrected first target feature extraction coordinate.

15. The method according to claim 14, wherein: The acquiring the relative coordinate relationship among the point cloud data, the millimeter wave data and the image data comprises: Acquire historical reference objects of the historical detection area, as well as historical point cloud data, historical millimeter wave data, and historical image data collected in the historical detection area; Inputting the historical point cloud data, the historical millimeter wave data and the historical image data into the multilayer perceptron respectively, obtaining the first historical perception coordinates of the historical reference object in the historical point cloud data, the second historical perception coordinates in the historical millimeter wave data and the third historical perception coordinates in the historical image data output by the multilayer perceptron; Acquire a first historical extraction coordinate of the historical reference object in the historical point cloud data, a second historical extraction coordinate in the millimeter wave data, and a second historical extraction coordinate in the historical image data; Based on the first historical perception coordinates, the first historical extraction coordinates, the second historical perception coordinates, the second historical extraction coordinates, the third historical perception coordinates and the third historical extraction coordinates, the relative coordinate relationship between the point cloud data, the millimeter wave data and the image data is acquired.

16. The method according to claim 12, wherein: In response to identifying that an output result of the multilayer perceptron has an error, obtaining error coordinates among the first perception coordinates, the second perception coordinates, and the third perception coordinates, and performing coordinate correction on the error coordinates to obtain corrected first target feature extraction coordinates, comprising: Acquire remaining perception coordinates, excluding the error coordinates, among the first perception coordinates, the second perception coordinates, and the third perception coordinates; Determine the sampling data corresponding to the remaining perception coordinates, and determine the remaining perception coordinates as the second target feature extraction coordinates of the feature extraction object in the sampling data, wherein the sampling data includes at least one of the point cloud data, the millimeter wave data and the image data of the detection area.

17. A target detection device, wherein: The device comprises: A collection module, used for collecting a detection data set of a detection area and extracting a first feature sequence of each detection data in the detection data set, wherein the detection data is multimodal data; A projection module, used for acquiring the BEV space of the detection area, and projecting the image data collected by the detection area to the BEV space to obtain BEV image data of the image data; a fusion module, used for extracting a second feature sequence of the BEV image data, and obtaining a three-dimensional fusion feature map obtained by performing feature fusion of each detection data and the image data in the detection data set based on the second feature sequence; The detection module is used to perform target detection on the three-dimensional fusion feature map of each detection data to obtain the target detection result of the detection area.

18. The device according to claim 17, wherein: The fusion module is further used for: extracting BEV image features of the BEV image data, and stretching the BEV image features to obtain the second feature sequence of the BEV image data; Extracting image features of the image data and stretching the image features to obtain a third feature sequence of the image data; Performing feature fusion on the second feature sequence and the third feature sequence to obtain a fourth feature sequence that is feature fused with each of the first feature sequences; Performing feature fusion on each first feature sequence and the fourth feature sequence respectively to obtain a first fused feature sequence of each detection data, wherein each detection data includes at least point cloud data and millimeter wave data; Based on the first fusion feature sequence, a three-dimensional fusion feature map is obtained by performing feature fusion of each detection data and image data in the detection data set.

19. The device according to claim 18, wherein: The fusion module is further used for: Obtain a preconfigured first feature fusion network, wherein the first feature fusion network is a two-end attention mechanism network; Obtain a first Q value based on the second characteristic sequence, and obtain a first K value and a first V value based on the third characteristic sequence; The first K value, the first V value and the first Q value are input into the first feature fusion network to obtain the fourth feature sequence output by the first feature fusion network.

20. The device according to claim 18, wherein The fusion module is further used for: Obtain a preconfigured second feature fusion network, wherein the second feature fusion network is a two-end attention mechanism network; For any first characteristic sequence, a second K value and a second V value are obtained based on the first characteristic sequence, and a second Q value is obtained based on the fourth characteristic sequence; The second K value, the second V value and the second Q value are input into the second feature fusion network to obtain the first fusion feature sequence output by the second feature network.

21. The device according to claim 18, wherein The fusion module is further used for: Acquire a first extraction position of the first fused feature sequence in the point cloud data in the detection data set, and a second extraction position in the millimeter wave data in the detection data set; Determine a first matching position corresponding to the first extraction position in the BEV space, and a second matching position corresponding to the second extraction position in the BEV space; Determine the first matching position as the first projection position of the first fused feature sequence of the point cloud data in the BEV space, and determine the second matching position as the second projection position of the first fused feature sequence of the millimeter wave data in the BEV space; Based on the first projection position, projecting the first fusion feature sequence of the point cloud data to the BEV space to obtain the three-dimensional fusion feature map corresponding to the point cloud data; Based on the second projection position, the first fusion feature sequence of the millimeter wave data is projected to the BEV space to obtain the three-dimensional fusion feature map corresponding to the millimeter wave data.

22. The device according to claim 17, wherein: The acquisition module is also used for: Extracting point cloud features of the point cloud data in the detection data set and millimeter wave features of the millimeter wave data in the detection data set; Feature stretching is performed on the point cloud features and the millimeter wave features respectively to obtain first feature sequences of the point cloud features and the millimeter wave features respectively, so as to obtain first feature sequences of each detection data in the detection data set.

23. The device according to claim 17, wherein: The projection module is further used for: extracting an image feature of the image data, and determining a third projection position of the image feature in the BEV space; Based on the third projection position, the image feature is projected into the BEV space to obtain the BEV image data of the image data.

24. The device according to claim 23, wherein: The projection module is further used for: Acquire a third extraction position of the image feature in the image data; A third matching position corresponding to the third extraction position in the BEV space is determined, and the third matching position is determined as the third projection position of the image feature in the BEV space.

25. The device according to claim 17, wherein: The detection module is further used for: Inputting the 3D fusion feature maps of each detection data into the 3D detection head for target detection, and obtaining candidate detection results of each 3D fusion feature map output by the 3D detection head; All candidate detection results are fused based on Kalman filtering to obtain the target detection result of the detection area.

26. The device according to any one of claims 17 to 25, wherein: The device also includes a stretching module, which is used to: Acquiring spatial parameters of the BEV space, and determining a feature stretching length required for feature stretching based on the spatial parameters; Based on the feature stretching length, feature stretching is performed on the feature to be processed to obtain a stretched feature sequence, wherein the feature sequence includes respective first feature sequences obtained after feature stretching of point cloud features and millimeter wave features, a second feature sequence obtained after feature stretching of BEV image features, and a third feature sequence obtained after feature stretching of image features.

27. The device according to claim 26, wherein: The stretching module is also used for: Acquire a length value, a width value, and a height value of the BEV space from the space parameters of the BEV space; The product value of the length value, the width value and the height value is obtained, and the product value is determined as the feature stretching length of the feature to be processed.

28. The device according to any one of claims 17 to 25, wherein: The device also includes a correction module, which is used to: Acquire a feature extraction object and a preconfigured multi-layer perceptron of the detection area; Inputting the point cloud data, the millimeter wave data and the image data into the multilayer perceptron respectively, and obtaining the first perception coordinates of the point cloud data, the second perception coordinates of the millimeter wave data and the third perception coordinates of the image data of the feature extraction object output by the multilayer perceptron; Identify whether there is an error in the output result of the multilayer perceptron; In response to identifying that an output result of the multilayer perceptron has an error, the error coordinates among the first perception coordinates, the second perception coordinates, and the third perception coordinates are acquired, and coordinate correction is performed on the error coordinates to obtain corrected first target feature extraction coordinates.

29. The device according to claim 28, wherein The correction module is further used for: In response to the point cloud object of the point cloud data at the first perception coordinate being different from the feature extraction object, and / or the millimeter wave object of the millimeter wave data at the second perception coordinate being different from the feature extraction object, and / or the image object of the image data at the third perception coordinate being different from the feature extraction object, it is determined that an error exists in the output result of the multilayer perceptron.

30. The device according to claim 28, wherein The correction module is further used for: Acquire an error object different from the feature extraction object in a point cloud object, a millimeter wave object, and an image object, and determine the perception coordinates corresponding to the error object as the error coordinates; Acquiring a relative coordinate relationship between the point cloud data, the millimeter wave data, and the image data; A correction parameter of the error coordinate is determined based on the relative coordinate relationship, and the error coordinate is adjusted based on the correction parameter to obtain a corrected first target feature extraction coordinate.

31. The device according to claim 30, wherein The correction module is further used for: Acquire historical reference objects of the historical detection area, as well as historical point cloud data, historical millimeter wave data, and historical image data collected in the historical detection area; Inputting the historical point cloud data, the historical millimeter wave data and the historical image data into the multilayer perceptron respectively, obtaining the first historical perception coordinates of the historical reference object in the historical point cloud data, the second historical perception coordinates in the historical millimeter wave data and the third historical perception coordinates in the historical image data output by the multilayer perceptron; Acquire a first historical extraction coordinate of the historical reference object in the historical point cloud data, a second historical extraction coordinate in the millimeter wave data, and a second historical extraction coordinate in the historical image data; Based on the first historical perception coordinates, the first historical extraction coordinates, the second historical perception coordinates, the second historical extraction coordinates, the third historical perception coordinates and the third historical extraction coordinates, the relative coordinate relationship between the point cloud data, the millimeter wave data and the image data is acquired.

32. The apparatus of claim 28, wherein: The correction module is further used for: Acquire remaining perception coordinates, excluding the error coordinates, among the first perception coordinates, the second perception coordinates, and the third perception coordinates; Determine the sampling data corresponding to the remaining perception coordinates, and determine the remaining perception coordinates as the second target feature extraction coordinates of the feature extraction object in the sampling data, wherein the sampling data includes at least one of the point cloud data, the millimeter wave data and the image data of the detection area.

33. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 16.

34. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-16.

35. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 16.

Images