Target detection method and device, electronic equipment and vehicle
Through the feature extraction and alignment processing of multimodal data, combined with the two-way fusion strategy of attention mechanism, the accuracy and robustness of target detection under changes in weather conditions are solved, and the detection accuracy and stability of the environmental perception system are improved. .
Patent Information
- Application Number
- CN202411091293.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-09
- Publication Date
- 2024-12-03
AI Technical Summary
In the prior art, target detection lacks accuracy and robustness in environmental perception, especially when weather conditions change, it is difficult to effectively improve detection accuracy and robustness of the millimeter wave radar and the camera.
By obtaining multimodal data of the target scene, including camera data and millimeter wave radar data, feature extraction and alignment processing are adopted, combined with the two-way fusion strategy of attention mechanism, different combination methods are selected according to weather conditions for information fusion, and target detection is improved accuracy.
It improves the robustness of the perception system for environmental changes and abnormal situations, and enhances the accuracy and accuracy of target detection, especially in poor weather conditions.
Smart Images

Figure CN119068172A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing technology, and in particular to a target detection method, device, electronic equipment and vehicle. Background Art
[0002] Since there are various sensors in target detection and environmental perception, but each sensor has its own advantages and disadvantages, generally speaking, one sensor cannot achieve accurate perception of the required environmental information. Therefore, multi-sensor fusion technology has become the focus and hotspot of current research and has become an indispensable part of the vehicle environmental perception system. Millimeter-wave radar has a long detection distance, short response time, and low cost, but poor directionality and low accuracy. Visual sensors such as cameras can provide a large amount of target information at a low cost, but are easily affected by weather conditions and changes in lighting. Summary of the invention
[0003] The present invention aims to solve the problem of related technical limitations at least to a certain extent. To this end, the present invention provides a target detection method, device, electronic equipment and vehicle, which can accurately detect targets.
[0004] On the one hand, an embodiment of the present invention provides a target detection method, including:
[0005] Acquire multimodal data and weather conditions of the target scene; multimodal data includes camera data and millimeter wave radar data;
[0006] Extract features from the multimodal data to obtain feature data; the feature data includes camera features and millimeter wave radar features;
[0007] When the weather condition is good, performing a first alignment process on the camera feature and the millimeter-wave radar feature based on the first data distribution of the camera feature;
[0008] When the weather condition is bad, the camera feature is converted into a bird's-eye view and a stereoscopic image feature is obtained through a stitching operation, and a second alignment process is performed on the stereoscopic image feature and the millimeter-wave radar feature based on a second data distribution of the stereoscopic image feature;
[0009] Based on the feature results of the first alignment process or the second alignment process, feature fusion is performed through a bidirectional fusion strategy based on an attention mechanism to obtain target features;
[0010] Based on the target features, target detection is performed through the preset regression head to obtain the target detection result in the target scene.
[0011] In some embodiments, obtaining multimodal data and weather conditions of a target scene includes the following steps:
[0012] Acquire camera data of a target scene through a camera device; the camera data is image data;
[0013] The millimeter-wave radar data of the target scene is obtained through the millimeter-wave radar; the millimeter-wave radar data is point cloud data;
[0014] The weather conditions are acquired in response to input information of the target object, or the weather conditions are acquired by performing classification processing based on the camera data through a pre-trained classification model.
[0015] In some embodiments, performing a first alignment process on the camera feature and the millimeter-wave radar feature based on the first data distribution of the camera feature includes the following steps:
[0016] Performing a third alignment process on the image pixels of the camera features and the depth points of the millimeter-wave radar features through a noise filtering network;
[0017] Based on the result of the third alignment processing, the attention alignment method is used to project the depth points to the front view where the camera features are located, and the depth points are depth enhanced and corrected, and the first alignment processing of the camera features and the millimeter-wave radar features is completed on the first data distribution of the camera features.
[0018] In some embodiments, performing a second alignment process on the stereoscopic image features and the millimeter-wave radar features based on the second data distribution of the stereoscopic image features includes the following steps:
[0019] Image feature fusion of stereo image features through continuous convolution blocks;
[0020] Back-projection extraction is performed based on the result of image feature fusion to obtain a pseudo point cloud target;
[0021] Based on the pseudo point cloud target, a second alignment process is performed on the stereo image features and the millimeter wave radar features on a second data distribution of the stereo image features.
[0022] In some embodiments, based on the feature results of the first alignment process or the second alignment process, feature fusion is performed through a bidirectional fusion strategy based on an attention mechanism to obtain target features, including the following steps:
[0023] When the weather condition is good, the camera feature is used as the first feature and the millimeter wave radar feature is used as the second feature;
[0024] When the weather condition is bad, the stereo image feature is used as the first feature and the millimeter wave radar feature is used as the second feature;
[0025] According to the first feature and the second feature, feature fusion is performed through bidirectional attention communication and multi-layer local spatial feature association to obtain the target feature.
[0026] In some embodiments, according to the first feature and the second feature, feature fusion is performed through bidirectional attention communication and multi-layer local spatial feature association to obtain a target feature, including the following steps:
[0027] Performing self-attention processing on the first feature to obtain a first self-attention feature; performing self-attention processing on the second feature to obtain a second self-attention feature;
[0028] A first cross-attention feature is obtained by cross-attention processing according to the first self-attention feature and the second self-attention feature; a second cross-attention feature is obtained by cross-attention processing according to the second self-attention feature and the first self-attention feature;
[0029] Perform feature multiplication on the first cross-attention feature and the second cross-attention feature to obtain a third feature;
[0030] Performing feature post-processing on the third feature to obtain a fourth feature;
[0031] Based on the number of channels of the fourth feature, the second cross-attention feature is expanded through a linear layer to obtain the fifth feature;
[0032] The fourth feature is taken as the first feature, the fifth feature is taken as the second feature, and the step of performing self-attention processing on the first feature to obtain the first self-attention feature is returned until the number of feature processing cycles reaches the preset number of feature fusion levels, and the fourth feature finally obtained is taken as the target feature.
[0033] In some embodiments, performing feature post-processing on the third feature comprises the following steps:
[0034] The third feature is processed by convolution, normalization and activation function in sequence.
[0035] On the other hand, an embodiment of the present invention provides a target detection device, including:
[0036] The first module is used to obtain multimodal data and weather conditions of the target scene; the multimodal data includes camera data and millimeter wave radar data;
[0037] The second module is used to extract features from the multimodal data to obtain feature data; the feature data includes camera features and millimeter wave radar features;
[0038] A third module is used for performing a first alignment process on the camera feature and the millimeter wave radar feature based on the first data distribution of the camera feature when the weather condition is good;
[0039] A fourth module is used for converting the camera features into a bird's-eye view and obtaining a stereoscopic image feature through a stitching operation when the weather condition is bad, and performing a second alignment process on the stereoscopic image feature and the millimeter-wave radar feature based on a second data distribution of the stereoscopic image feature;
[0040] A fifth module is used to perform feature fusion based on the feature results of the first alignment process or the second alignment process through a bidirectional fusion strategy based on an attention mechanism to obtain target features;
[0041] The sixth module is used to perform target detection based on target features through a preset regression head to obtain target detection results in the target scene.
[0042] On the other hand, an embodiment of the present invention provides an electronic device, including: a processor and a memory; the memory is used to store programs; the processor executes the program to implement the above-mentioned target detection method.
[0043] On the other hand, an embodiment of the present invention provides a computer storage medium, in which a program executable by a processor is stored. When the program executable by the processor is executed by the processor, it is used to implement the above-mentioned target detection method.
[0044] On the other hand, an embodiment of the present invention provides a vehicle, which includes the above-mentioned target detection device or the above-mentioned electronic device.
[0045] The embodiment of the present invention obtains multimodal data and weather conditions of the target scene; the multimodal data includes camera data and millimeter wave radar data; feature extraction is performed on the multimodal data to obtain feature data; the feature data includes camera features and millimeter wave radar features; when the weather conditions are good, the camera features and the millimeter wave radar features are first aligned based on the first data distribution of the camera features; when the weather conditions are poor, the camera features are converted to a bird's-eye view and stereo image features are obtained through a splicing operation, and the stereo image features and the millimeter wave radar features are second aligned based on the second data distribution of the stereo image features; based on the feature results of the first alignment processing or the second alignment processing, feature fusion is performed through a bidirectional fusion strategy based on an attention mechanism to obtain target features; based on the target features, target detection is performed through a preset regression head to obtain the target detection result in the target scene. The target detection method of the present invention using the fusion of millimeter wave radar and camera can improve the robustness of the perception system to environmental changes and abnormal conditions, and according to different weather conditions, different combinations are selected in the information fusion process, which can more effectively detect the target object and improve the detection precision and accuracy. The present invention can accurately perform target detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] The accompanying drawings are used to provide a further understanding of the technical solution of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the technical solution of the present invention and do not constitute a limitation on the technical solution of the present invention.
[0047] Figure 1 This is a schematic diagram of an implementation environment for target detection provided by an embodiment of the present invention;
[0048] Figure 2 is a flow chart of a target detection method provided by an embodiment of the present invention;
[0049] Figure 3 A schematic diagram of a data acquisition process according to an embodiment of the present invention;
[0050] Figure 4 A schematic diagram of a first alignment process provided by an embodiment of the present invention;
[0051] Figure 5 A schematic diagram of a second alignment process according to an embodiment of the present invention;
[0052] Figure 6 A schematic diagram of the structure of multi-level two-way communication provided by an embodiment of the present invention;
[0053] Figure 7 A schematic diagram of the process of FV-Align provided by an embodiment of the present invention;
[0054] Figure 8 A schematic diagram of the BEV-Align process provided by an embodiment of the present invention;
[0055] Fig. 9 A schematic diagram of the structure of a target detection device provided by an embodiment of the present invention;
[0056] Fig.10 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0057] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0058] It should be noted that, although the functional modules are divided in the system schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the system or the order in the flowchart. The terms "first / S100", "second / S200", etc. in the specification, claims and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.
[0059] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.
[0060] It is understandable that the target detection method provided in the embodiment of the present invention can be applied to any computer device with data processing and computing capabilities, and this computer device can be various types of terminals or servers. When the computer device in the embodiment is a server, the server is an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. In some embodiments, the terminal is a smart phone, a tablet computer, a laptop computer, and a desktop computer, but is not limited to this.
[0061] like Figure 1 FIG. 1 is a schematic diagram of an implementation environment provided by an embodiment of the present invention. Figure 1 The implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected to a network wirelessly or wired to complete data transmission and exchange.
[0062] Server 101 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.
[0063] In addition, the server 101 can also be a node server in the blockchain network. Blockchain is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, and encryption algorithm.
[0064] The terminal 102 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc. The terminal 102 may also be a vehicle-mounted terminal of various device types described above, but is not limited thereto. The terminal 102 and the server 101 may be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment of the present invention.
[0065] Based on the example Figure 1 In the implementation environment shown, an embodiment of the present invention provides a target detection method. The target detection method is applied to the server 101 as an example for explanation below. It can be understood that the target detection method can also be applied to the terminal 102.
[0066] Reference Figure 2 , Figure 2 The flowchart of the target detection method applied to the server provided by the embodiment of the present invention, the execution subject of the target detection method can be any of the aforementioned computer devices (including servers or terminals). Figure 2 , the method comprises the following steps:
[0067] S100, acquiring multimodal data and weather conditions of a target scene;
[0068] Among them, multimodal data includes camera data and millimeter wave radar data;
[0069] It should be noted that in some embodiments, Figure 3 As shown, step S100 may include the following steps: S101, acquiring camera data of the target scene through a camera device; the camera data is image data; S102, acquiring millimeter-wave radar data of the target scene through a millimeter-wave radar; the millimeter-wave radar data is point cloud data; S103, acquiring weather conditions in response to input information of the target object, or obtaining weather conditions by performing classification processing based on the camera data through a pre-trained classification model.
[0070] For example, in some specific implementations, the classification model can be trained by a conventional classifier based on scene images with labeled weather condition results, and then used for classification processing of weather conditions; at the same time, weather data can also be directly obtained based on manual input from the user.
[0071] S200, extracting features from the multimodal data to obtain feature data;
[0072] Among them, the feature data includes camera features and millimeter wave radar features;
[0073] Exemplarily, in some specific implementations, the camera feature is obtained by feature extraction based on the camera data, and the millimeter-wave radar feature is obtained by feature extraction based on the millimeter-wave radar data.
[0074] S300: When the weather condition is good, perform a first alignment process on the camera feature and the millimeter-wave radar feature based on the first data distribution of the camera feature;
[0075] It should be noted that in some embodiments, Figure 4 As shown, performing a first alignment process on the camera features and the millimeter-wave radar features based on the first data distribution of the camera features may include the following steps: S301, performing a third alignment process on the image pixels of the camera features and the depth points of the millimeter-wave radar features through a noise filtering network; S302, based on the result of the third alignment process, using an attention alignment method to project the depth points to the front view where the camera features are located, and performing depth enhancement and depth correction on the depth points, and completing the first alignment process on the camera features and the millimeter-wave radar features on the first data distribution of the camera features.
[0076] For example, in some specific implementations, when weather conditions are good, fusion can be performed after alignment based on the front view. This experiment uses FCOS3D as the basic network, first aligns the projected depth points and image pixels through a noise filtering network, and then uses the attention alignment method to perform depth enhancement and depth correction on the millimeter-wave radar points projected to the front view, and aligns them on the data distribution of the strong modality.
[0077] S400, when the weather condition is not good, converting the camera feature into a bird's-eye view and obtaining a stereoscopic image feature through a stitching operation, and performing a second alignment process on the stereoscopic image feature and the millimeter-wave radar feature based on a second data distribution of the stereoscopic image feature;
[0078] It should be noted that the stitching operation is used to stitch camera features of different viewing angles into a feature image of a bird's-eye view (surround view).
[0079] It should be noted that in some embodiments, Figure 5 As shown, performing a second alignment process on the stereoscopic image features and the millimeter-wave radar features based on the second data distribution of the stereoscopic image features may include the following steps: S401, performing image feature fusion on the stereoscopic image features through continuous convolution blocks; S402, performing back-projection extraction based on the result of the image feature fusion to obtain a pseudo point cloud target; S403, completing a second alignment process on the stereoscopic image features and the millimeter-wave radar features on the second data distribution of the stereoscopic image features based on the pseudo point cloud target.
[0080] For example, in some specific implementations, when weather conditions are bad, fusion can be performed based on BEV alignment. This experiment uses BEVDet as the basic network and uses the LSS method to convert the front view features of the camera RGB image to BEV. Then, a simple splicing operation and continuous convolution blocks are used to fuse the image features to extract the millimeter-wave radar pseudo point cloud targets with columnar features, and align the two modes in the weak modal data distribution.
[0081] Among them, the purpose of LSS: to propose a model called 'Lift-Splat-Shooti' to learn consistent representations from multi-view images and achieve end-to-end autonomous driving planning. The model converts images into 3D point cloud representations and performs predictive planning from BEV to achieve the fusion and utilization of contextual features within the camera's field of view.
[0082] Lift (feature extraction): By processing the image, a series of points of all possible depths are generated for each pixel in the image, and features are generated for each point. For the depth with the highest confidence for each pixel, a conical point cloud is generated to represent the contextual features within the camera's field of view, generating a depth estimate.
[0083] Splat (projection): Project each cone point cloud onto a reference plane to obtain a bird's-eye view representation. This process converts the cone point cloud into a tensor of fixed dimension that can be processed by subsequent convolutional neural networks.
[0084] Shoot: Use the inferred cost map for motion planning. By "shooting" different trajectories, calculating their costs, and selecting the trajectory with the lowest cost as the final planning result.
[0085] Pseudo LiDAR (pseudo point cloud) can generate a depth map by estimating the depth of the surround view (bird's eye view) image, and then back-project the depth map to generate a 3D pseudo point cloud.
[0086] S500, based on the feature results of the first alignment process or the second alignment process, perform feature fusion through a bidirectional fusion strategy based on an attention mechanism to obtain target features;
[0087] It should be noted that, in some embodiments, step S500 may include the following steps: when the weather condition is good, the camera feature is used as the first feature and the millimeter-wave radar feature is used as the second feature; when the weather condition is poor, the stereo image feature is used as the first feature and the millimeter-wave radar feature is used as the second feature; based on the first feature and the second feature, feature fusion is performed through two-way attention communication and multi-layer local spatial feature association to obtain the target feature.
[0088] Among them, in some embodiments, according to the first feature and the second feature, feature fusion is performed through two-way attention communication and multi-layer local spatial feature association to obtain the target feature, which may include the following steps: performing self-attention processing on the first feature to obtain the first self-attention feature; performing self-attention processing on the second feature to obtain the second self-attention feature; obtaining the first cross-attention feature through cross-attention processing according to the first self-attention feature and the second self-attention feature; obtaining the second cross-attention feature through cross-attention processing according to the second self-attention feature and the first self-attention feature; performing feature multiplication on the first cross-attention feature and the first cross-attention feature to obtain the third feature; performing feature post-processing on the third feature to obtain the fourth feature; based on the number of channels of the fourth feature, expanding the number of channels of the second cross-attention feature through a linear layer to obtain the fifth feature; taking the fourth feature as the first feature and the fifth feature as the second feature, returning to execute the step of performing self-attention processing on the first feature to obtain the first self-attention feature, until the number of feature processing cycles reaches the preset number of feature fusion levels, and taking the fourth feature finally obtained as the target feature.
[0089] In some embodiments, post-processing the third feature may include the following steps: sequentially performing convolution processing, normalization processing, and activation function processing on the third feature. Specifically, the aforementioned processing steps may be sequentially performed through a convolution block, a normalization block, and an activation function.
[0090] For example, in some specific implementations, for the dense versus sparse but spatially corresponding modalities such as camera-millimeter wave radar, the present invention proposes a bidirectional fusion strategy BSF based on the attention mechanism that is suitable for spatial relationships in the fusion process. BSF is divided into two sub-modules: bidirectional attention communication and multi-layer local spatial feature association. Given the camera feature F C , millimeter wave radar feature F R , the algorithm in BSF is:
[0091]
[0092] F R =G[C Attn (S Attn (norm(F R )), S Attn (norm(F C )))];
[0093] Among them, C Attn represents cross attention, S Attn represents self-attention, H and G represent the post-processing networks for two features respectively.
[0094] The first part of BSF consists of self-attention and cross-attention. First, the features inside the sample are extracted in the self-attention structure, and then the cross-attention is used to communicate with another modality, so that the expression domain of the feature construction tends to be similar. The process of the attention mechanism can actually be abstracted into two stages: first, the question and the keyword are determined, then the weight coefficient of the similarity between the question and the keyword is calculated, and then the value is weighted and summed according to the coefficient. The attention calculation in the present invention adopts the deformable attention proposed by Deformable DETR, that is, it is calculated by sampling the learning attention feature points. On this basis, since the questions and keywords of the self-attention mechanism come from the input itself, its attention calculation is equivalent to capturing the internal correlation of the features in the same expression. The biggest difference from CNN is that the attention calculation is global, while the usual two-dimensional convolution calculation is limited by the fixed parameters of the convolution kernel, and can only convolve the local correlation of the internal area.
[0095] After the cross attention, the BSF module C and F R Convert back to 2D shape and expand on the number of channels to get F RC ∈R N×2C×H×W , passing it through a module H consisting of a 3×3 convolution block, a normalization block BatchNorm2D, and a ReLU activation function to extract the unique local features of the image. In order to increase the communication between modalities and retain the original information of the error at the same time, and reduce the difficulty of model training, BSF uses short connections in the multi-level communication process, such as Figure 6 (The camera features in the figure correspond to the aforementioned camera features or the stereo image features after conversion to BEV in different application scenarios). The direction of the arrow indicates the direction of information flow. In order to make the input size and channels of the feature maps of the same stage the same, the camera features are converted to the stereo image features before each stage ends. Through an additional linear layer G, the number of channels is equal to Keeping it consistent, the formula becomes:
[0096]
[0097] Among them, F i represents the features of the i-th layer, F i-1 Represents the features of the i-1th layer. The features of the first layer come from their respective feature base networks.
[0098] S600: Based on the target features, target detection is performed through a preset regression head to obtain a target detection result in the target scene.
[0099] Exemplarily, in some specific implementations, the regression head plays a key role in the target detection model. It predicts the precise position and / or other relevant attributes of the target by learning the mapping relationship between the input data and the target output. The output of the regression head usually includes the center coordinates, width, height, etc. of the target. These parameters are learned through the training process to minimize the gap between the predicted box and the actual target box. It should be noted that the technical focus of the present invention is on the feature processing flow for multimodal data, and the regression head is not specifically limited. In some other implementation scenarios, the regression head can also be used together with the classification head based on the target features of the aforementioned feature processing results, where the classification head is responsible for identifying the target category, and the regression head is responsible for adjusting the position and size of the target.
[0100] In order to explain the principle of the technical solution of the present invention in detail, the overall process of the present invention is described below in combination with some specific embodiments. It is easy to understand that the following is an explanation of the technical principle of the present invention and cannot be regarded as a limitation of the present invention.
[0101] First of all, it should be noted that there are currently several solutions for sensor fusion target detection methods using millimeter wave radar and camera. 1. Joint calibration is used to synchronize the data of radar and camera in time and space, and then the radar data is transformed into the image to generate the region of interest, and the frame difference method is used to determine the area of the object. The judgment and distance are made by comparing whether the two regions of interest overlap. This fusion method is more difficult to set the threshold of the region of interest. When the region of interest is set large, multiple vehicle targets will be included in the region of interest. When the region of interest is set small, the vehicle targets will be included incompletely. 2. After the time and space calibration, the sensor fusion is mainly carried out by evidence reasoning. First, the recognition framework is established, and the basic probability distribution function of each evidence is calculated based on the information obtained by each sensor. The basic probability distribution function under all evidence is calculated according to the combination rule of DS evidence reasoning. Finally, according to the specified decision criteria, the hypothesis with the highest confidence is selected as the fusion function. This method has a low detection rate for vehicles on both sides of the lane. 3. Obtain millimeter-wave radar data through millimeter-wave radar, extract effective targets, remove static targets and random noise, obtain image data through the camera set on the roadside pole, perform target detection and multi-target tracking based on YOLOv3 and Deep SORT tracking algorithms, obtain camera data, and perform multi-sensor fusion of millimeter-wave radar data and camera data based on fusion algorithms to achieve real-time detection, tracking and positioning of targets by roadside equipment. The above solutions are relatively complex, with low target recognition accuracy and precision, and poor feasibility.
[0102] In view of this, the embodiments of the present invention provide a target detection method using millimeter wave radar and camera fusion with different technical concepts, which is specifically implemented as follows:
[0103] According to the fusion stage, the fusion process can be divided into pre-fusion and post-fusion. Pre-fusion refers to the fusion of features, which enables the framework to adaptively adjust the fused features, so that the output fusion features can be naturally applied to the target task. Generally, the fusion of lidar point cloud and camera RGB image uses the pre-fusion fusion method. Post-fusion refers to the assumption that each modality can complete the target detection task independently, or even if a certain modality cannot complete the target detection task, it can also contribute to a certain indicator of the task, and then combine and balance the results of each modality, obtain correction and then output. Since the performance effects of the commonly used modalities cannot fully meet the assumptions of post-fusion, the fusion features obtained in the pre-fusion stage can more adaptively improve the prediction accuracy. In the target detection method, the accuracy of pre-fusion is better than that of post-fusion.
[0104] In the fusion process, the modality that contributes more to the fusion result is called a strong modality, and the modality that contributes less to the fusion result is called a weak modality. The present invention proposes an information fusion method that selects different combinations in the information fusion process according to different weather conditions, and switches the selection of strong modalities and weak modalities. Since the camera's detection of targets and environmental perception are easily affected by weather conditions and light intensity, in the process of target detection and environmental perception by the fusion of millimeter-wave radar and camera, when the weather conditions are good, the camera is selected as the strong modality in the fusion process, and the millimeter-wave radar is selected as the weak modality for target detection. The fusion method at this time is to use the camera as the dominant modality. During fusion, the information received by the camera will be given priority, and the information detected by the millimeter-wave radar will be used as a supplement to distance, size, etc., to contribute one or two points to improve the accuracy of the final data. When weather conditions are bad, such as rain, snow, or fog, the millimeter-wave radar is selected as the strong mode and the camera is selected as the weak mode for target detection during the fusion process. The fusion method at this time is to use the millimeter-wave radar as the dominant mode. The information detected by the millimeter-wave radar will be given priority during fusion, and the information received by the camera will be used as a supplement to the color, type, etc., contributing one or two points to improve the accuracy of the final data.
[0105] The two methods provided by the present invention are as follows: Figure 7 and Figure 8As shown, FV-Align performs target detection after the data distribution of the strong mode is aligned; BEV-Align performs target detection after the data distribution of the weak mode is aligned, and then the two modal data distributions are aligned. The influence of the distribution form of the initial data on the feature extraction and the accuracy of the final result is proved. 1. FV-Align: The present invention introduces the projection depth points of the millimeter-wave radar on the classic framework FCOS3D of the front view, and performs two types of filtering enhancement by complementing the adjacent area and the depth value of the noise point. Finally, the 3D target detection task is performed under the distribution of the front view for comparison; 2. BEV-Align: Based on the data distribution under the BEV (Bird's-Eye-View) perspective, the present invention has made a 3D target detection method fused under BEV based on BEVDet. A comparative analysis was made in terms of multi-modality versus single modality and front view versus BEV.
[0106] Among them, some specific application scenarios, such as Figure 7 As shown in the figure, FV-Align: Based on the front view alignment and then fusion, this experiment uses FCOS3D as the basic network. First, the projected depth points and image pixels are aligned through the noise filtering network, and then the attention alignment method is used to perform depth enhancement and depth correction on the millimeter-wave radar points projected to the front view, and the depth feature points and image feature points are fused; alignment is performed on the strong modal data distribution, and then the target detection task is completed.
[0107] FCOS3D projects the 7DoF 3D position of the 3D object detection task onto the 2D image, and uses the projection center as the 3D center, so the center actually contains 2.5D information of depth. By using the unique 2D attribute of the center normalized between different feature levels, the 3D object detection task can be decoupled into 2.5D centrality and 3D attributes. The entire FCOS3D process can be classified into the standard framework of object detection, namely: basic framework, neck framework, and predicted head.
[0108] like Figure 8 As shown in the figure, BEV-Align: Based on BEV alignment and then fusion, this experiment uses BEVDet as the basic network and uses the LSS method to convert the front view features of the camera RGB image to BEV. Then, a simple splicing operation and continuous convolution blocks are used to fuse the image features, extract the columnar features of the millimeter-wave radar pseudo point cloud targets, align the two modes in the weak modal data distribution, and then complete the target detection task.
[0109] BEVDet proposes to transfer the 3D detection target based only on the camera to BEV under the condition of comparable accuracy and detection speed, so as to improve the accuracy of BEV position and reach consistency with the paradigm of BEV semantic segmentation, which is conducive to the integration of the entire perception framework, that is, the converted BEV features are not only suitable for 3D target detection, but also for BEV semantic segmentation. BEVDet is a simple and general BEV framework. When using the LSS method to convert image features from the front view distribution to the BEV distribution, the BEV data integrates the six surround cameras of the nuScenes dataset into one scene as supervision. Therefore, its input data is much less than that of single-purpose data, which is prone to overfitting. To address this problem, BEVDet proposes a method for data enhancement decoupling. In the data preprocessing stage, the framework pre-stores the parameters of image cropping, rotation, flipping, etc. After the feature extraction network of the shared image, the parameters are mapped back to the original space, thereby enhancing the input image data and avoiding disconnection between scenes. Secondly, he updated the NMS algorithm to improve its adaptability to 3D object detection scenarios and removed unnecessary sequential execution operators to speed up its execution process.
[0110] Specifically, for the dense and sparse modalities with spatial correspondence such as camera and millimeter wave radar, a bidirectional fusion strategy BSF based on the attention mechanism suitable for spatial relationships is proposed in the fusion process of FV-Align and BEV-Align. For tasks such as target detection that emphasize the detection of spatial relationships, in addition to similarity weighted judgment, bidirectional fusion solves the comparison between dense and sparse data, and uses local convolution to better align the two modalities, which greatly improves the accuracy of target detection.
[0111] The present invention uses BSF, a spatial feature fusion module based on a bidirectional attention mechanism. BSF is a fusion module that fully communicates dense strong modalities and sparse weak modalities, and takes into account the local relationship of the two modal spatial alignment. The present invention cites the concept of bidirectional attention, that is, the calculation process is not separated, and the two-way comparison of parallel queries is carried out. The purpose is to hope that the two modalities can be converted to the same semantic space through bidirectional communication during the calculation process of feature similarity, thereby enhancing the semantic consistency in the feature alignment process.
[0112] In summary, the present invention selects different combinations in the information fusion process according to different weather conditions, and converts the information fusion mode of the selection of strong modalities and weak modalities. When the weather conditions are not good, such as rain, snow, foggy weather, etc., the millimeter wave radar is selected as the strong modality in the fusion process, and the camera is selected as the weak modality for target detection; when the weather conditions are good, the camera is selected as the strong modality in the fusion process, and the millimeter wave radar is selected as the weak modality for target detection; and the present invention uses BSF, a spatial feature fusion module based on the bidirectional attention mechanism. BSF is a fusion module that fully communicates dense strong modalities and sparse weak modalities, and takes into account the local relationship of the spatial alignment of the two modalities. The concept of bidirectional attention is cited, that is, the calculation process is not separated, and the bidirectional comparison of parallel queries is used. The purpose is to hope that the two modalities are converted to the same semantic space through bidirectional communication during the calculation of feature similarity, and the semantic consistency in the feature alignment process is enhanced. Specifically, the target detection method of the present invention using the fusion of millimeter wave radar and camera can improve the robustness of the perception system to environmental changes and abnormal conditions. The present invention solves the problem of low accuracy and precision in target recognition. The novel bidirectional fusion strategy BSF (Bidirectional Spatial Fusion) introduced in this invention is applicable to multimodal features with spatial relationships. Compared with the single-frame mode, BEV-Radar has a lower cost and solves the problem of speed prediction. It also has an advantage over other camera-millimeter-wave radar methods. At the same time, according to different weather conditions, different combinations are selected in the information fusion process, and the selection of strong and weak modes is switched, which can more effectively detect the target and improve the precision and accuracy of the detection.
[0113] On the other hand, Fig. 9 As shown, an embodiment of the present invention provides a target detection device 900, which may include:
[0114] The first module 910 is used to obtain multimodal data and weather conditions of the target scene; the multimodal data includes camera data and millimeter wave radar data;
[0115] The second module 920 is used to extract features from the multimodal data to obtain feature data; the feature data includes camera features and millimeter wave radar features;
[0116] The third module 930 is used for performing a first alignment process on the camera feature and the millimeter wave radar feature based on the first data distribution of the camera feature when the weather condition is good;
[0117] A fourth module 940 is used for, when the weather condition is bad, converting the camera feature into a bird's-eye view and obtaining a stereoscopic image feature through a stitching operation, and performing a second alignment process on the stereoscopic image feature and the millimeter-wave radar feature based on a second data distribution of the stereoscopic image feature;
[0118] A fifth module 950 is used to perform feature fusion based on the feature result of the first alignment process or the second alignment process by a bidirectional fusion strategy based on an attention mechanism to obtain a target feature;
[0119] The sixth module 960 is used to perform target detection based on target features through a preset regression head to obtain a target detection result in a target scene.
[0120] The contents of the method embodiments of the present invention are all applicable to the device embodiments. The functions specifically implemented by the device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.
[0121] On the other hand, an embodiment of the present invention further provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above target detection method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.
[0122] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.
[0123] like Fig.10 As shown, Fig.10 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:
[0124] The processor 1001 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention.
[0125] The memory 1002 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1002 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1002, and the processor 1001 calls and executes the network node population optimization method of the embodiment of the present invention;
[0126] Input / output interface 1003, used to implement information input and output;
[0127] The communication interface 1004 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);
[0128] A bus 1005 , which transmits information between various components of the device (e.g., the processor 1001 , the memory 1002 , the input / output interface 1003 , and the communication interface 1004 );
[0129] The processor 1001 , the memory 1002 , the input / output interface 1003 and the communication interface 1004 are connected to each other in communication within the device via the bus 1005 .
[0130] The electronic device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0131] The contents of the method embodiments of the present invention are all applicable to the electronic device embodiments. The functions specifically implemented by the electronic device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.
[0132] Another aspect of an embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores a program, and the program is executed by a processor to implement the above method.
[0133] It should be noted that the computer-readable medium shown in the embodiment of the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (Compact Disc Read to Only Memory, CD to ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.
[0134] The contents of the method embodiments of the present invention are all applicable to the computer-readable storage medium embodiments. The functions specifically implemented by the computer-readable storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.
[0135] The embodiment of the present invention also discloses a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device can read the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the above method.
[0136] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0137] It should be noted that, although several modules of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to an embodiment of the present invention, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.
[0138] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD to ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation of the present invention.
[0139] In some selectable embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Selectable embodiments are expected, wherein the order of various operations is changed and the sub-operation of a part for which is described as a larger operation is performed independently.
[0140] In addition, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise specified, one or more of the functions and / or features can be integrated into a single physical device and / or software module, or one or more functions and / or features can be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present invention. More specifically, in view of the properties, functions and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional skills of the engineer. Therefore, those skilled in the art can implement the present invention set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.
[0141] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.
[0142] The logic and / or steps represented in the flowchart or otherwise described herein, for example, may be considered as an ordered list of executable instructions for implementing logical functions, and may be embodied in any computer-readable medium for use by an instruction execution device, apparatus or device (such as a computer-based device, a device including a processor, or other device that can fetch instructions from an instruction execution device, apparatus or device and execute the instructions), or in conjunction with such instruction execution device, apparatus or device. For purposes of this specification, "computer-readable medium" may be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution device, apparatus or device, or in conjunction with such instruction execution device, apparatus or device.
[0143] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.
[0144] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution device. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0145] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.
[0146] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.
[0147] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present invention.
Claims
1. A target detection method, It is characterized in that The following steps are involved: Acquire multimodal data and weather conditions of a target scene; the multimodal data includes camera data and millimeter wave radar data; Extracting features from the multimodal data to obtain feature data; the feature data includes camera features and millimeter wave radar features; When the weather condition is good, performing a first alignment process on the camera feature and the millimeter-wave radar feature based on the first data distribution of the camera feature; When the weather condition is bad, converting the camera feature into a bird's-eye view and obtaining a stereoscopic image feature through a stitching operation, and performing a second alignment process on the stereoscopic image feature and the millimeter-wave radar feature based on a second data distribution of the stereoscopic image feature; Based on the feature results of the first alignment process or the second alignment process, performing feature fusion through a bidirectional fusion strategy based on an attention mechanism to obtain target features; Based on the target features, target detection is performed through a preset regression head to obtain a target detection result in the target scene.
2. The target detection method according to claim 1, It is characterized in that The step of acquiring multimodal data and weather conditions of the target scene comprises the following steps: Acquire the camera data of the target scene through a camera device; the camera data is image data; Acquire the millimeter-wave radar data of the target scene through a millimeter-wave radar; the millimeter-wave radar data is point cloud data; The weather condition is acquired in response to input information of the target object, or the weather condition is acquired by performing classification processing based on the camera data through a pre-trained classification model.
3. The target detection method according to claim 1, It is characterized in that The first alignment processing of the camera feature and the millimeter-wave radar feature based on the first data distribution of the camera feature comprises the following steps: Performing a third alignment process on the image pixels of the camera feature and the depth points of the millimeter wave radar feature through a noise filtering network; Based on the result of the third alignment processing, the attention alignment method is used to project the depth point to the front view where the camera feature is located, and the depth point is depth enhanced and corrected, and the first alignment processing is completed on the camera feature and the millimeter-wave radar feature on the first data distribution of the camera feature.
4. The target detection method according to claim 1, It is characterized in that The performing a second alignment process on the stereoscopic image feature and the millimeter-wave radar feature based on the second data distribution of the stereoscopic image feature comprises the following steps: Performing image feature fusion on the stereo image features through continuous convolution blocks; Perform back-projection extraction based on the result of the image feature fusion to obtain a pseudo point cloud target; Based on the pseudo point cloud target, the second alignment process is performed on the stereo image feature and the millimeter wave radar feature on the second data distribution of the stereo image feature.
5. The target detection method according to claim 1, It is characterized in that The feature results based on the first alignment processing or the second alignment processing are subjected to feature fusion through a bidirectional fusion strategy based on an attention mechanism to obtain target features, including the following steps: When the weather condition is good, the camera feature is used as the first feature, and the millimeter wave radar feature is used as the second feature; When the weather condition is bad, taking the stereoscopic image feature as the first feature and taking the millimeter wave radar feature as the second feature; According to the first feature and the second feature, the feature fusion is performed through two-way attention communication and multi-layer local space feature association to obtain the target feature.
6. The target detection method according to claim 5, It is characterized in that The step of fusing the features according to the first feature and the second feature through bidirectional attention communication and multi-layer local spatial feature association to obtain the target feature includes the following steps: Performing self-attention processing on the first feature to obtain a first self-attention feature; performing the self-attention processing on the second feature to obtain a second self-attention feature; A first cross-attention feature is obtained by cross-attention processing according to the first self-attention feature and the second self-attention feature; a second cross-attention feature is obtained by cross-attention processing according to the second self-attention feature and the first self-attention feature; Performing feature multiplication on the first cross-attention feature and the second cross-attention feature to obtain a third feature; Performing feature post-processing on the third feature to obtain a fourth feature; Based on the number of channels of the fourth feature, expanding the number of channels of the second cross-attention feature through a linear layer to obtain a fifth feature; The fourth feature is used as the first feature, and the fifth feature is used as the second feature. The step of performing self-attention processing on the first feature to obtain the first self-attention feature is returned to the step, until the number of feature processing cycles reaches the preset number of feature fusion levels, and the fourth feature finally obtained is used as the target feature.
7. The target detection method according to claim 6, It is characterized in that The post-processing of the third feature comprises the following steps: The third feature is sequentially subjected to convolution processing, normalization processing and activation function processing.
8. A target detection device, It is characterized in that include: The first module is used to obtain multimodal data and weather conditions of the target scene; the multimodal data includes camera data and millimeter wave radar data; The second module is used to extract features from the multimodal data to obtain feature data; the feature data includes camera features and millimeter wave radar features; A third module is configured to perform a first alignment process on the camera feature and the millimeter-wave radar feature based on the first data distribution of the camera feature when the weather condition is good; A fourth module is configured to, when the weather condition is bad, convert the camera feature into a bird's-eye view and obtain a stereoscopic image feature through a stitching operation, and perform a second alignment process on the stereoscopic image feature and the millimeter-wave radar feature based on a second data distribution of the stereoscopic image feature; A fifth module is used to perform feature fusion based on the feature results of the first alignment processing or the second alignment processing through a bidirectional fusion strategy based on an attention mechanism to obtain a target feature; The sixth module is used to perform target detection based on the target features through a preset regression head to obtain the target detection result in the target scene.
9. An electronic device, It is characterized in that including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 7.
10. A vehicle, It is characterized in that The vehicle includes the target detection device according to claim 8 or the electronic device according to claim 9.
Citation Information
Patent Citations
Target detection method and device, storage medium, chip and vehicle
CN115082886A
Multi-modal information space-time fusion 3D target detection method and system in foggy day scene
CN115393680A
Data fusion method and device, vehicle and storage medium
CN116363615A
4D millimeter wave radar and visual adaptive fusion target identification system
CN118155174A
