Target detection method and device, electronic equipment, storage medium and program product

Abstract

The invention relates to the technical field of target detection, in particular to a target detection method and device, electronic equipment, a storage medium and a program product, and the method comprises the steps: obtaining a training data set; the training data set is utilized to train a multi-modal target detection model, the multi-modal target detection model outputs a target detection result, the multi-modal target detection model comprises a first processing module, a second processing module, a Munban model and a third processing module, the first processing module is used for extracting features and carrying out sparse fusion, and the second processing module is used for carrying out sparse fusion on the features; the second processing module is used for determining a state point and creating a scene point, and the Mangbar model updates the state point and the scene point based on a state point self-attention mechanism and a decoding layer differentiation serialization strategy; the third processing module is used for fusing the features after sparse fusion and the updated features of the state points and the scene points and then outputting a target detection result; and performing target detection by using the trained multi-modal target detection model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of target detection, and in particular relates to a target detection method and device, electronic equipment, a storage medium and a program product. BACKGROUND

[0002] With the continuous development of deep learning and computer vision technology, multi-modal fusion target detection has become a key research direction in intelligent perception systems. Multi-modal data (such as images, point clouds, radar data, etc.) contains rich environmental information and has wide application value in automatic driving, robot perception, security monitoring, etc.

[0003] The DETR (Detection Transformer) method based on the Transformer in the related art has achieved good results in the target detection task, and in particular has shown strong performance in the three-dimensional target detection task. However, the Transformer decoder in the DETR method uses fixed scene point features for query refinement, which limits the contribution of subsequent decoder layers to the final detection results and restricts the improvement of the final detection performance. SUMMARY

[0004] The present application provides a target detection method, device, electronic equipment, storage medium and program product to solve the problem of fixed scene point features in the related art target detection method, which limits the detection performance.

[0005] The first aspect of the present application provides a target detection method, comprising the following steps: obtaining a training data set, wherein the training data set comprises surround view image data, radar data and corresponding label information; training a pre-constructed multi-modal target detection model using the training data set, the multi-modal target detection model outputs a target detection result, and the multi-modal target detection model comprises a first processing module, a second processing module, a Mamba model and a third processing module, wherein the first processing module is used to extract the features of the surround view image data and the radar data and perform sparse fusion, the second processing module is used to determine a state point based on the prior position data and the speed data of the surround view image data, and create a scene point based on the features of the radar data, the Mamba model updates the state point and the scene point based on the state point self-attention mechanism and the decoding layer differential serialization strategy, and the third processing module is used to fuse the features of the sparse fusion and the updated state point and scene point, and then output the target detection result; in the training process, the model parameters of the multi-modal target detection model are updated based on the target detection result and the label information, and the training is stopped until a preset condition is met, and the trained multi-modal target detection model is used for target detection.

[0006] Optionally, in an embodiment of the present application, the first processing module comprises: a point cloud backbone network, a feature pyramid network and a feature fusion module, wherein the point cloud backbone network is configured to extract features of the radar data; the feature pyramid network is configured to extract features of the surround view image data; and the feature fusion module is configured to fuse the extracted features of the radar data and the features of the surround view image data.

[0007] Optionally, in an embodiment of the present application, the third processing module comprises: a space-time sampling module, a mixing module, a gated feed-forward network and a detection head, wherein the space-time sampling module is configured to determine a target sampling range based on the features of the updated state points, extract spatial key features in the target sampling range based on the sparsely fused features, and fuse the multi-frame sparsely fused features based on a convolution operation and extract time key features; the mixing module is configured to mix the spatial key features and the time key features; the gated feed-forward network is configured to process the mixed spatial key features and the time key features to obtain target key features; and the detection head is configured to output the target detection result based on the target key features.

[0008] Optionally, in an embodiment of the present application, before training the pre-constructed multi-modal target detection model using the training data set, the method further comprises: performing sequence processing on the radar data.

[0009] Optionally, in an embodiment of the present application, the Mamba model is further configured to: convert the features of the state points and the features of the scene points into high-dimensional sequence vectors respectively; fuse the high-dimensional sequence vectors based on a linear state conversion matrix to obtain a fused state sequence; filter the fused state sequence based on a nonlinear activation function to obtain a target state sequence; in the multi-layer decoder of the Mamba model, each layer of the decoder adopts a different sequence processing strategy to process the target state sequence to obtain a multi-layer state sequence; process the multi-layer state sequence based on a state point self-attention mechanism to obtain a correlation state sequence; and extract the correlation state sequence to obtain the features of the updated state points and the features of the scene points.

[0010] Optionally, in an embodiment of the present application, the target detection result comprises at least one of a category, a geometric parameter and a motion state of the target.

[0011] The second aspect embodiment of the application provides a target detection device, comprising: an acquisition module configured to acquire a training data set, wherein the training data set comprises surround view image data, radar data, and corresponding annotation information; a training module configured to train a pre-constructed multi-modal target detection model using the training data set, wherein the multi-modal target detection model outputs a target detection result, and the multi-modal target detection model comprises a first processing module, a second processing module, a Mamba model, and a third processing module, wherein the first processing module is configured to extract features of the surround view image data and the radar data and perform sparse fusion, the second processing module is configured to determine state points based on prior position data and velocity data of the surround view image data, and create scene points based on features of the radar data, the Mamba model updates the state points and the scene points based on a state point self-attention mechanism and a decoding layer differential serialization strategy, and the third processing module is configured to fuse the features of the state points and the scene points after being updated and the features after being fused to output the target detection result; and an updating module configured to update model parameters of the multi-modal target detection model based on the target detection result and the annotation information during the training process, and stop the training until a preset condition is met, and use the trained multi-modal target detection model to perform target detection.

[0012] Optionally, in an embodiment of the application, the first processing module comprises a point cloud backbone network, a feature pyramid network, and a feature fusion module, wherein the point cloud backbone network is configured to extract features of the radar data, the feature pyramid network is configured to extract features of the surround view image data, and the feature fusion module is configured to fuse the extracted features of the radar data and the surround view image data.

[0013] Optionally, in an embodiment of the application, the third processing module comprises a space-time sampling module, a mixing module, a gated feed-forward network, and a detection head, wherein the space-time sampling module is configured to determine a target sampling range based on features of the updated state points, extract spatial key features in the target sampling range based on the features after being fused, and fuse multiple frames of features after being fused based on a convolution operation and extract time key features, the mixing module is configured to mix the spatial key features and the time key features, the gated feed-forward network is configured to process the mixed spatial key features and the time key features to obtain target key features, and the detection head is configured to output the target detection result based on the target key features.

[0014] Optionally, in an embodiment of the application, the device further comprises a processing module configured to perform serialization processing on the radar data before training the pre-constructed multi-modal target detection model using the training data set.

[0015] Optionally, in an embodiment of the present application, the Mamba model is further used to: convert the feature of the state point and the feature of the scene point into high-dimensional sequence vectors respectively; fuse the high-dimensional sequence vectors based on a linear state transition matrix to obtain a fused state sequence; filter the fused state sequence based on a nonlinear activation function to obtain a target state sequence; in the multi-layer decoder of the Mamba model, each layer of the decoder adopts a different serialization processing strategy to process the target state sequence to obtain a multi-layer state sequence; process the multi-layer state sequence based on a state point self-attention mechanism to obtain a related state sequence; and extract the related state sequence to obtain the feature of the updated state point and the feature of the scene point.

[0016] Optionally, in an embodiment of the present application, the target detection result includes at least one of a category, a geometric parameter and a motion state of the target.

[0017] The third aspect of the embodiments of the present application provides an electronic device, comprising: a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor executes the program to perform the target detection method of the above-mentioned embodiments.

[0018] The fourth aspect of the embodiments of the present application provides a computer readable storage medium, which stores a computer program or instructions, and the computer program or instructions are executed by a processor to perform the target detection method of the above-mentioned embodiments.

[0019] The fifth aspect of the embodiments of the present application provides a computer program product, comprising a computer program or instructions, and the computer program or instructions are executed to implement the target detection method of the above-mentioned embodiments.

[0020] Therefore, the present application has at least the following beneficial effects: The embodiments of the present application can train a multi-modal target detection model using a training data set, the multi-modal target detection model includes a first processing module, a second processing module, a Mamba model and a third processing module, the first processing module can extract features and perform sparse feature fusion, filter redundant features of invalid regions, and avoid computational waste caused by full region fusion, the second processing module is used to determine state points and scene points, and then update the state points and the scene points based on a state point self-attention mechanism and a decoding layer differential serialization strategy through the Mamba model, update the features of the scene points and the state points dynamically, let the decoder layer contribute to the detection result, and improve the detection performance of the multi-modal target model, and the third processing module fuses the features after sparse fusion and the features of the updated state points and scene points to output a target detection result, and uses the trained multi-modal target detection model to perform target detection. Therefore, the technical problems of fixed scene point feature, limited detection performance and the like in the related art are solved.

[0021] Additional aspects and advantages of the present application will be made apparent by the following description and the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0022] The above and / or additional aspects and advantages of the present application will become apparent and be readily appreciated from the following description, including the accompanying drawings. Figure 1 A flowchart of a target detection method according to an embodiment of the present application is provided. Figure 2 A complete execution flowchart of a target detection method according to an embodiment of the present application is provided. Figure 3 An example diagram of a target detection device according to an embodiment of the present application is provided. Figure 4 A structural schematic diagram of an electronic device according to an embodiment of the present application is provided. DETAILED DESCRIPTION

[0023] Embodiments of the present application are described in detail below with reference to the accompanying drawings, in which the same or similar components have the same or similar designations and functions throughout. The embodiments described below are examples in which reference numerals in the drawings denote like or similar elements or elements having the same or similar functions. The embodiments described below by reference to the accompanying drawings are exemplary and are intended to explain the present application, and are not to be understood as limiting the present application.

[0024] Before describing the method of the present application, the related art of the present application is introduced.

[0025] The three-dimensional target detection method in the related art is mainly divided into three categories: voting-based method, extension-based method and DETR-based method.

[0026] 1. Voting-based method: This type of method moves surface points to the center of the object through a voting mechanism, and clusters points moved to the same area to generate candidate points. These methods can better preserve the original geometric information of 3D point clouds, but due to the class-agnostic voting mechanism, adjacent but different class points are easily misgrouped.

[0027] 2. Extension-based method: This type of method uses a generative sparse decoder to generate high-quality target proposals from object surface voxel features with the same semantic prediction. Although this method is superior to the voting-based method in detection accuracy, it is highly dependent on the proposal generation module, and involves a large number of manual hyperparameter settings, affecting the generalization ability of the model.

[0028] 3. DETR-based methods: The DETR method uses a Transformer decoder to iteratively refine the query points and optimizes object detection using scene point features. However, the Transformer decoder in the DETR method uses fixed scene point features for query refinement, resulting in limited contributions from subsequent decoder layers to the final detection result. Furthermore, the computational complexity of the Transformer is quadratic, making it difficult to apply efficiently in large-scale point cloud scenarios.

[0029] Based on the aforementioned technologies, the following problems exist: 1. Fixed scene point features limit detection performance: Existing DETR methods fail to update scene point features between different decoder layers, resulting in a smaller contribution from subsequent decoder layers, which in turn limits the improvement of the final detection performance.

[0030] 2. High computational complexity, making it difficult to handle large-scale data: The computational complexity of the Transformer decoder is quadratic. When processing large-scale point cloud or multimodal data, the computational overhead is huge, making it difficult to meet the needs of real-time applications.

[0031] 3. Lack of efficient multimodal information fusion mechanism: Current methods are unable to fully utilize the complementarity of multimodal data, resulting in insufficient robustness of target detection in complex environments.

[0032] 4. Inaccurate depth prediction affects target localization: Traditional methods usually rely on single-frame prediction or limited feature mapping for depth estimation, which makes it difficult to accurately model the 3D information in the scene, resulting in large errors in target depth estimation.

[0033] Therefore, this application provides a target detection method to solve at least one of the above-mentioned technical problems.

[0034] Specifically, Figure 1 This is a schematic flowchart of a target detection method provided in an embodiment of this application.

[0035] like Figure 1 As shown, the target detection method includes the following steps: In step S101, a training dataset is obtained, which includes panoramic image data, radar data, and corresponding annotation information.

[0036] The annotation information includes the target's category, geometric parameters, and motion state.

[0037] It is understood that the embodiments of this application can obtain a training dataset for subsequent training of a multimodal target detection model.

[0038] The training data set of the embodiment of the application can be collected from various real scenes. The data can come from different sensors and perspectives, thereby constructing a multi-modal and rich training data set. Through the construction of such a multi-source data set, the subsequent multi-modal target detection can draw information from different perspectives and sensor signals, thereby obtaining more comprehensive target recognition and detection capabilities in the subsequent training process, and improving the perception and understanding capabilities of the model in complex environments.

[0039] In step S102, the pre-constructed multi-modal target detection model is trained using the training data set. The multi-modal target detection model outputs a target detection result. The multi-modal target detection model includes a first processing module, a second processing module, a Mamba model, and a third processing module. The first processing module is configured to extract features of the surround view image data and the radar data and perform sparse fusion. The second processing module is configured to determine state points based on prior position data and velocity data of the surround view image data, and create scene points based on features of the radar data. The Mamba model is configured to update the state points and the scene points based on a state point self-attention mechanism and a decoding layer differential serialization strategy. The third processing module is configured to fuse the features after sparse fusion, the features of the updated state points, and the features of the updated scene points, and output the target detection result.

[0040] The multi-modal target detection model is a model that fuses two types of different modal data, i.e., surround view image data and radar data, to achieve 3D target detection. The state points are dynamic query points initialized by radar prior position (such as the distance and azimuth angle of the target from the radar) and velocity data, and are used to track the real-time state of the target. The scene points are environmental feature points created based on radar features, and are used to represent the scene information around the target.

[0041] The embodiment of the application can train a pre-constructed multi-modal target detection model using a training data set. The multi-modal target detection model outputs a target detection result. The multi-modal target detection model includes a first processing module, a second processing module, a Mamba model, and a third processing module, Specifically, the first processing module extracts features of the surround view image data and the radar data and performs sparse fusion. By fusing only the radar features in the visible area of the surround view image with the image features of the corresponding area, redundant features in invalid areas are filtered, and the computational waste caused by full-area fusion is avoided. The second processing module determines state points based on prior position data and velocity data of the surround view image data, and creates scene points based on features of the radar data. The Mamba model updates the state points and the scene points based on a state point self-attention mechanism and a decoding layer differential serialization strategy. The third processing module fuses the features after sparse fusion, the features of the updated state points, and the features of the updated scene points, and outputs the target detection result.

[0042] The mamba model of the embodiment of the application has multiple layers of decoders, each layer adopts a different feature serialization method, and through dynamic updating of scene point and state point features, the decoder layers can fully contribute to the detection result to improve the detection performance of the subsequent multi-modal target model.

[0043] In addition, it should be noted that radar data naturally has three-dimensional spatial properties (such as target and radar distance, azimuth angle, and other prior location information), which is different from the method in the related art which only relies on a single frame of image (which lacks direct depth information) or limited feature mapping. In the present application, the three-dimensional prior information of the radar is integrated into the state point initialization, providing an accurate initial reference for subsequent depth estimation and reducing the error of depth estimation from the data source.

[0044] In an embodiment of the present application, the first processing module includes a point cloud backbone network, a feature pyramid network, and a feature fusion module, wherein the point cloud backbone network is used to extract features of the radar data; the feature pyramid network is used to extract features of the surround view image data; and the feature fusion module is used to fuse the extracted features of the radar data and the features of the surround view image data.

[0045] It can be understood that the first processing module of the embodiment of the present application includes a point cloud backbone network, a feature pyramid network, and a feature fusion module, wherein, The point cloud backbone is used to extract effective features in the radar data; The FPN (Feature Pyramid Network) is used to extract features of the surround view image data, wherein the feature pyramid network can perform multi-scale feature extraction, thereby covering the feature requirements of targets of different sizes and avoiding the problem of missing small targets or misjudging large targets with single-scale features; The feature fusion module is used to fuse the extracted features of the radar data and the features of the surround view image data, and through the integration of radar features and surround view image features, complementary multi-modal features are formed, thereby improving the detection robustness of the multi-modal target detection model in complex environments.

[0046] In an embodiment of the present application, the third processing module includes a space-time sampling module, a mixing module, a gated feed-forward network, and a detection head, wherein the space-time sampling module is used to determine a target sampling range based on the updated features of the state points, extract spatial key features within the target sampling range based on the sparsely fused features, and fuse the multi-frame sparsely fused features based on a convolution operation and extract temporal key features; the mixing module is used to mix the spatial key features and the temporal key features; the gated feed-forward network is used to process the mixed spatial key features and the temporal key features to obtain target key features; and the detection head is used to output a target detection result based on the target key features.

[0047] The fused features can be BEV (Bird's-Eye View) features, and the sparse fusion is BEV feature fusion.

[0048] It can be understood that the third processing module of the embodiment of the application includes a space-time sampling module, a mixing module, a GFFN (Gated Feed-Forward Network) and a detection head, wherein, The space-time sampling module is a module that combines spatial dimensions (state point positions) and time dimensions (multi-frame data) to sample features, realizes the dual functions of focusing on a target region and capturing a motion trend, specifically, determines a target sampling range based on the updated state point features, extracts spatial key features in the target sampling range based on the sparse fused features, focuses on the region where the target is located to avoid irrelevant region feature interference and improve positioning accuracy, and fuses the multi-frame sparse fused features based on a convolution operation and extracts time key features, which captures the motion trend of the target by fusing the multi-frame features through convolution, so as to capture the dynamic target. The mixing module is used to mix the spatial key features and the time key features. The GFFN (Gated Feed-Forward Network) can filter redundant features through a gating unit and retain key features, and enhance the discriminability of the features through a nonlinear transformation, specifically, obtains target key features by processing the mixed spatial key features and time key features. The detection head can be understood as the output layer of the model, usually includes a classification branch and a regression branch, and can convert the target key features output by the GFFN into specific detection results.

[0049] In an embodiment of the application, the Mamba model is further used to: convert the features of the state points and the features of the scene points into high-dimensional sequence vectors respectively; fuse the high-dimensional sequence vectors based on a linear state transition matrix to obtain a fused state sequence; filter the fused state sequence based on a nonlinear activation function to obtain a target state sequence; in the multi-layer decoder of the Mamba model, each layer of the decoder adopts a different serialization processing strategy to process the target state sequence to obtain a multi-layer state sequence; process the multi-layer state sequence based on a state point self-attention mechanism to obtain an associated state sequence; and extract the associated state sequence to obtain the features of the updated state points and the features of the scene points.

[0050] The nonlinear activation function is a gating activation function used for feature filtering, can be Sigmoid or Swish, and the output value range is between 0 and 1, which can dynamically assign different importance weights to different features; and the associated state sequence is a sequence containing the association relationship between the state points and the scene points.

[0051] It can be understood that the Mamba model of the embodiment of the present application is further used for: Converting the features of the state points and the features of the scene points into high-dimensional sequence vectors respectively; Fusing the high-dimensional sequence vectors based on a linear state transition matrix to obtain a fused state sequence, and realizing efficient fusion of features through a linear complex calculation method based on the Mamba model; Filtering the fused state sequence based on a nonlinear activation function to obtain a target state sequence; In the multi-layer decoder of the Mamba model, each layer of the decoder adopts a different serialization processing strategy to process the target state sequence to obtain a multi-layer state sequence, so as to realize step-by-step optimization of features from local to global; Processing the multi-layer state sequence based on a state point self-attention mechanism to obtain a related state sequence; Extracting the related state sequence to obtain updated features of the state points and the features of the scene points.

[0052] In an embodiment of the present application, before training the pre-constructed multi-modal target detection model using the training data set, it further includes: performing serialization processing on the radar data.

[0053] It can be understood that the embodiment of the present application can convert discrete radar point cloud data into ordered sequence vectors, which meets the processing requirements of the Mamba model for data and avoids model training failure or feature extraction failure due to data format mismatch.

[0054] In step S103, in the training process, the model parameters of the multi-modal target detection model are updated based on the target detection result and the annotation information, and the training is stopped until a preset condition is met, and the trained multi-modal target detection model is used for target detection.

[0055] The target detection result includes at least one of the category, the geometric parameter and the motion state of the target; the preset condition can be that a preset round is reached, or the prediction accuracy reaches a certain threshold, etc., which is not limited.

[0056] It can be understood that in the training process, the model parameters of the multi-modal target model are updated based on the target detection result and the annotation information, and the training is stopped until a preset condition is met, the model parameters are iteratively updated, the model is gradually approximated to the real result, the detection accuracy is improved, and the trained target detection model is used for target detection.

[0057] In summary, the target detection method of the present application, the specific implementation process includes: through the enhancement of scene point feature update mechanism, different serialization strategies are applied in each decoding layer; the standard state point self-attention mechanism is adopted in the Mamba model, and the linear complexity calculation method of the Mamba model is adopted; the multi-frame BEV feature fusion method is adopted, and the fusion is carried out through convolution operation; the time sequence correlation method is adopted, and the target motion trend is predicted through the feature change between adjacent frames.

[0058] The target detection method of the present application is described below through a specific embodiment, and the complete flow is as shown in Figure 2 The target detection method of the present application is described below through a specific embodiment, and the complete flow is as shown in 1. Collection and annotation of data set.

[0059] The surround view image, Radar data and corresponding annotation information are collected from various real scenes. These data come from different sensors and perspectives, and a multi-modal, rich and diverse training and evaluation data set is constructed. Through the construction of such a multi-source data set, the model can extract information from different perspectives and sensor signals, thereby obtaining more comprehensive target recognition capability in the training process, and improving the model's perception and understanding ability in complex environments.

[0060] 2. Data feature preprocessing.

[0061] In the data preprocessing stage, first, the Radar point cloud data is serialized, and the effective features in the Radar data are extracted through a special point cloud backbone network. At the same time, the surround view image data is subjected to multi-scale feature extraction through a feature pyramid network, obtaining hierarchical image features. In this process, the fusion of Radar and image features provides multi-dimensional perception information for subsequent 3D target detection, ensuring the richness and accuracy of the information.

[0062] 3. Initialization query and SSM (State Space Model).

[0063] Using the prior position information and velocity data provided in the Radar data, a number of query points are initialized as state points, and scene points are created according to the extracted Radar features. The state points and scene points are input into the state space model, which finely adjusts the matching and relationship between the scene points and state points on this basis, so as to ensure that the model can more accurately capture the dynamic characteristics of the targets in the scene, and also can further correct the depth estimation bias.

[0064] In addition, it should be noted that the state space model of the present application is the Mamba model.

[0065] 4. Multi-modal feature fusion.

[0066] In this stage, the image features are sparsely fused with the Radar features from the surround view area. This fusion not only preserves the unique information of each data source, but also enhances the model's perception of targets from different perspectives. Based on the location information of the query point, a spatio-temporal sampling operation is performed to adaptively mix the feature information from images and Radar. The updated query point is processed by the GFFN and finally input into the detection head, outputting accurate 3D target detection results.

[0067] In summary, the embodiment introduces an interactive state space model, dynamically updates scene point features and query point features between different decoder layers, makes the decoder fully capture scene point features, improves target detection performance, fully utilizes the linear complexity calculation advantage of the Mamba model, improves calculation efficiency, uses a multi-frame BEV feature fusion method, combines convolution operation for efficient fusion, and improves target detection robustness. Stacking BEV features in more historical frames improves the model's ability to capture complex patterns.

[0068] According to the target detection method proposed in the embodiment of the application, the multi-modal target detection model can be trained using the training data set. The multi-modal target detection model includes a first processing module, a second processing module, a Mamba model, and a third processing module. The first processing module can extract features and perform sparse feature fusion to filter redundant features in invalid regions, avoiding the calculation waste caused by full-area fusion. The second processing module is used to determine state points and scene points, and then update the state points and scene points based on the state point self-attention mechanism and the decoding layer differential serialization strategy of the Mamba model. By dynamically updating the scene point and state point features, the decoder layer can fully contribute to the detection results to improve the detection performance of the multi-modal target model. The third processing module fuses the sparse fused features and the updated state point and scene point features to output the target detection results. The trained multi-modal target detection model is used for target detection.

[0069] Secondly, the target detection device according to the embodiment of the application is described with reference to the accompanying drawings.

[0070] Figure 3 is a block schematic diagram of the target detection device according to the embodiment of the application.

[0071] As shown in Figure 3 , the target detection device 10 includes an acquisition module 100, a training module 200, and an updating module 300.

[0072] The acquisition module 100 is configured to acquire a training data set, wherein the training data set comprises surround view image data, radar data and corresponding annotation information.

[0073] In an embodiment of the present application, the first processing module comprises a point cloud backbone network, a feature pyramid network and a feature fusion module, wherein the point cloud backbone network is configured to extract features of the radar data; the feature pyramid network is configured to extract features of the surround view image data; and the feature fusion module is configured to fuse the extracted features of the radar data and the surround view image data.

[0074] In an embodiment of the present application, the third processing module comprises a space-time sampling module, a mixing module, a gated feed-forward network and a detection head, wherein the space-time sampling module is configured to determine a target sampling range based on the features of the updated state points, extract spatial key features in the target sampling range based on the sparsely fused features, and fuse multiple frames of the sparsely fused features based on a convolution operation and extract time key features; the mixing module is configured to mix the spatial key features and the time key features; the gated feed-forward network is configured to process the mixed spatial key features and the time key features to obtain target key features; and the detection head is configured to output the target detection result based on the target key features.

[0075] In an embodiment of the present application, the device 10 of the embodiment of the present application further comprises a processing module.

[0076] Before training the pre-constructed multi-modal target detection model using the training data set, the processing module is configured to perform sequence processing on the radar data.

[0077] In an embodiment of the present application, the Mamba model is further used to: convert the feature of the state point and the feature of the scene point into high-dimensional sequence vectors respectively; fuse the high-dimensional sequence vectors based on a linear state transition matrix to obtain a fused state sequence; filter the fused state sequence based on a nonlinear activation function to obtain a target state sequence; in the multi-layer decoder of the Mamba model, each layer of the decoder adopts a different serialization processing strategy to process the target state sequence to obtain a multi-layer state sequence; process the multi-layer state sequence based on a state point self-attention mechanism to obtain an associated state sequence; and extract the associated state sequence to obtain the feature of the updated state point and the feature of the scene point.

[0078] In an embodiment of the present application, the target detection result includes at least one of a category, a geometric parameter and a motion state of the target.

[0079] It should be noted that the foregoing explanation and description of the target detection method embodiments also apply to the target detection device of this embodiment, which will not be described here again.

[0080] According to the target detection device provided in the embodiments of the present application, the multi-modal target detection model can be trained by using the training data set, the multi-modal target detection model includes the first processing module, the second processing module, the Mamba model and the third processing module, the first processing module can extract features and perform sparse feature fusion, filter redundant features of invalid regions, and avoid calculation waste caused by full-area fusion, the second processing module is used to determine the state point and the scene point, and then update the state point and the scene point based on the state point self-attention mechanism and the decoding layer differential serialization strategy through the Mamba model, update the features of the scene point and the state point dynamically, so that the decoder layer can fully contribute to the detection result, so as to improve the detection performance of the multi-modal target model, and the third processing module fuses the sparse fused features and the features of the updated state point and scene point, and outputs the target detection result, and the multi-modal target detection model trained is used for target detection.

[0081] Figure 4 The structure schematic diagram of the electronic device provided in the embodiments of the present application is shown. The electronic device can include: The memory 401, the processor 402 and the computer program stored in the memory 401 and executable on the processor 402.

[0082] The processor 402 implements the target detection method provided in the above embodiments when executing the program.

[0083] Further, the electronic device further includes: The communication interface 403 is used for communication between the memory 401 and the processor 402.

[0084] The memory 401 is used to store the computer program executable on the processor 402.

[0085] The memory 401 can include a high-speed RAM memory and can also include a non-volatile memory, such as at least one disk memory.

[0086] If the memory 401, the processor 402 and the communication interface 403 are implemented independently, the communication interface 403, the memory 401 and the processor 402 can be connected to each other through a bus and complete communication between each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For convenience of representation, Figure 4 In the figure, only one thick line is used to represent, but it does not mean that there is only one bus or one type of bus.

[0087] Optionally, in a specific implementation, if the memory 401, the processor 402 and the communication interface 403 are integrated on a chip, the memory 401, the processor 402 and the communication interface 403 can complete communication between each other through an internal interface.

[0088] The processor 402 can be a Central Processing Unit (CPU), or an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement one or more embodiments of the present application.

[0089] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program or instructions, and the computer program or instructions are executed by a processor to implement the target detection method.

[0090] The embodiments of the present application also provide a computer program product, which includes a computer program or instructions, and the computer program or instructions are executed to implement the target detection method.

[0091] In the description of the application, reference to "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" etc. means that a particular feature, structure, material or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the application. The illustrative appearances of the above-mentioned terms in various places in the specification are not necessarily referred to the same embodiment or example. Moreover, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples. Furthermore, in non-contradictory cases, those skilled in the art can combine and combine the features of different embodiments or examples described in the specification and the features of different embodiments or examples.

[0092] In addition, the terms "first", "second" are only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the application, the meaning of "N" is at least two, such as two, three, etc., unless otherwise specifically limited.

[0093] Any process or method descriptions or descriptions of the flow diagrams in the specification can be understood as representing code modules, segments or portions of code which include one or more executable instructions for implementing the specified logic function or process. It should also be understood that the preferred embodiments of the application include the combination of hardware and software configured to effect the functions described in the specification and illustrated in the drawings.

[0094] It should be understood that portions of the application can be implemented in hardware, software, firmware or combinations thereof. In the above embodiments, the N steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. As in another embodiment implemented in hardware, any one or more of the following technologies known in the art can be used: discrete logic circuit with logic gates for implementing logic functions on data signals, application specific integrated circuits with appropriate combinational logic gates, programmable gate arrays, field programmable gate arrays, etc.

[0095] Those skilled in the art of the present technology can understand that all or part of the steps carried out by the above-mentioned embodiment method can be completed by program instructions to the relevant hardware, and the program can be stored in a computer readable storage medium. The program includes one or a combination of steps of the method embodiments when executed.

Claims

1. A target detection method, characterized in that, Includes the following steps: Obtain a training dataset, wherein the training dataset includes panoramic image data, radar data, and corresponding annotation information; A pre-built multimodal object detection model is trained using the training dataset. The multimodal object detection model outputs object detection results. The multimodal object detection model includes a first processing module, a second processing module, a Mamba model, and a third processing module. The first processing module is used to extract features from the surround view image data and the radar data and perform sparse fusion. The second processing module is used to determine state points based on the prior position data and velocity data of the surround view image data and create scene points based on the features of the radar data. The Mamba model updates the state points and the scene points based on the state point self-attention mechanism and the decoding layer differential serialization strategy. The third processing module is used to fuse the sparsely fused features and the updated state points and scene points and output the target detection result. During training, the model parameters of the multimodal target detection model are updated based on the target detection results and the annotation information until the preset conditions are met, at which point training stops and the trained multimodal target detection model is used for target detection.

2. The target detection method according to claim 1, characterized in that, The first processing module includes: a point cloud backbone network, a feature pyramid network, and a feature fusion module, wherein, The point cloud backbone network is used to extract features from the radar data; The feature pyramid network is used to extract features from the panoramic image data; The feature fusion module is used to fuse the features extracted from the radar data and the features from the surround view image data.

3. The target detection method according to claim 1, characterized in that, The third processing module includes: a spatiotemporal sampling module, a hybrid module, a gated feedforward network, and a detection head, wherein... The spatiotemporal sampling module is used to determine the target sampling range based on the features of the updated state points, extract key spatial features within the target sampling range based on the sparsely fused features, and fuse the sparsely fused features of multiple frames based on convolution operations to extract key temporal features. The mixing module is used to mix the spatial key features and the temporal key features; The gated feedforward network is used to process the mixed spatial and temporal key features to obtain the target key features; The detection head is used to output the target detection result based on the target's key features.

4. The target detection method according to claim 1, characterized in that, Before training the pre-built multimodal object detection model using the training dataset, the following steps are also included: The radar data is then serialized.

5. The target detection method according to claim 1, characterized in that, The Mamba model is further used for: The features of the state points and the features of the scene points are respectively converted into high-dimensional sequence vectors; The high-dimensional sequence vectors are fused based on the linear state transition matrix to obtain a fused state sequence; The target state sequence is obtained by filtering the fused state sequence based on a nonlinear activation function; In the multi-layer decoder of the Mamba model, each layer decoder uses a different serialization processing strategy to process the target state sequence, resulting in a multi-layer state sequence; The associated state sequence is obtained by processing the multi-layer state sequence based on the state point self-attention mechanism. The updated features of the state points and scene points are obtained by extracting the associated state sequence.

6. The target detection method according to claim 1, characterized in that, The target detection result includes at least one of the target's category, geometric parameters, and motion state.

7. A target detection device, characterized in that, include: The acquisition module is used to acquire the training dataset, wherein the training dataset includes panoramic image data, radar data and corresponding annotation information; The training module is used to train a pre-built multimodal object detection model using the training dataset. The multimodal object detection model outputs object detection results. The multimodal object detection model includes a first processing module, a second processing module, a Mamba model, and a third processing module. The first processing module is used to extract features from the surround view image data and the radar data and perform sparse fusion. The second processing module is used to determine state points based on the prior position data and velocity data of the surround view image data and create scene points based on the features of the radar data. The Mamba model updates the state points and the scene points based on the state point self-attention mechanism and the decoding layer differential serialization strategy. The third processing module is used to fuse the sparsely fused features and the updated state points and scene points and output the target detection result. The update module is used to update the model parameters of the multimodal object detection model based on the object detection results and the annotation information during the training process, until the training stops when the preset conditions are met, and the trained multimodal object detection model is used for object detection.

8. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to implement the target detection method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, The computer program or instructions are executed by a processor to implement the target detection method as described in any one of claims 1-6.

10. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed, they implement the target detection method as described in any one of claims 1-6.

Images