Training methods, target object detection methods, electronic equipment

Abstract

The present application provides a training method, a target object detection method, and an electronic device, relating to the field of autonomous driving technology. The training method includes: determining image features, point cloud features, and a first instance feature of an object sample, wherein the first instance feature is determined based on the point cloud feature; processing the first instance feature and the point cloud feature based on a first point cloud decoder to determine a second instance feature of the object sample; processing the first instance feature and the image feature based on an image decoder to determine a third instance feature of the object sample; fusing the second instance feature and the third instance feature to obtain a fourth instance feature; and adjusting a target detection model based on the fourth instance feature.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and specifically to a training method, a target object detection method, and an electronic device. Background Art

[0002] In the field of autonomous driving, target detection is a key perception technology. There are currently three main methods: based on pure vision, based on pure point cloud, and based on multimodal fusion of visual point cloud.

[0003] The first two single-modal methods each have their own advantages and disadvantages. Specifically, while visual data has rich color and texture features, it lacks accurate depth information, and the data quality is significantly affected by lighting, weather, and viewing angle. Point cloud data can directly obtain accurate depth information and is insensitive to lighting changes, but the data is sparse, especially in long-distance areas, resulting in the loss of effective information. In contrast, multimodal fusion methods combine the color and texture advantages of vision with the depth advantages of point clouds, resulting in higher detection accuracy. However, when training existing multimodal fusion models, they usually train the point cloud processing branch first and then the visual processing branch. Although this can reduce the CPU's video memory usage, this method of optimizing the single-modal branches separately ignores the characteristics of multimodal fusion and affects the final performance of the model during fusion. Summary of the Invention

[0004] In view of this, embodiments of the present application provide a training method, a target object detection method, and an electronic device.

[0005] In a first aspect, an embodiment of the present application provides a training method for training a target detection model, the target detection model including a first point cloud decoder and an image decoder, the method including: determining image features, point cloud features and first instance features of an object sample, wherein the first instance feature is determined based on the point cloud feature; processing the first instance feature and the point cloud feature based on the first point cloud decoder to determine the second instance feature of the object sample; processing the first instance feature and the image feature based on the image decoder to determine the third instance feature of the object sample; fusing the second instance feature and the third instance feature to obtain a fourth instance feature; and adjusting the target detection model based on the fourth instance feature.

[0006] In combination with the first aspect, in certain implementations of the first aspect, adjusting the target detection model based on the fourth instance feature includes: using the fourth instance feature as the new first instance feature, repeatedly executing the steps of determining the second instance feature of the object sample to adjusting the target detection model based on the fourth instance feature, until the fourth instance feature meets the preset convergence condition or the number of repeated executions reaches the set number of cycles.

[0007] In combination with the first aspect, in certain implementations of the first aspect, the first instance feature and the point cloud feature are processed based on the first point cloud decoder to determine the second instance feature of the object sample, including: projecting the first instance feature to the two-dimensional space where the image data is located based on the first point cloud decoder to obtain a feature matrix, wherein the image feature is determined based on the image data; and performing feature interaction on the feature matrix and the point cloud feature based on the first point cloud decoder to obtain the second instance feature of the object sample.

[0008] In combination with the first aspect, in certain implementations of the first aspect, the first point cloud decoder includes a multi-head self-attention module, and before projecting the first instance feature to the two-dimensional space where the image data is located based on the first point cloud decoder, it also includes: processing the first instance feature based on the multi-head self-attention module to obtain a fifth instance feature; the first point cloud decoder also includes a projection module; based on the first point cloud decoder, the first instance feature is projected to the two-dimensional space where the image data is located to obtain a feature matrix, including: projecting the fifth instance feature to the two-dimensional space where the image data is located based on the projection module to obtain a feature matrix.

[0009] In combination with the first aspect, in certain implementations of the first aspect, the fifth instance feature is projected into the two-dimensional space where the image data is located based on the projection module to obtain a feature matrix, including: based on the projection module, selecting key points in the detection box of each object sample contained in the fifth instance feature; projecting the key points into the two-dimensional space where the image data is located to obtain the coordinates of the key points in the two-dimensional space; based on the coordinates of the key points in the two-dimensional space, querying the features of the point cloud features at the corresponding coordinates to form a feature matrix.

[0010] In combination with the first aspect, in certain implementations of the first aspect, the image decoder includes a projection module; processing the first instance feature and the image feature based on the image decoder to determine the third instance feature of the object sample includes: projecting the first instance feature to the image coordinate system where the image data is located based on the projection module, wherein the image feature is determined based on the image data; and performing feature interaction between the first instance feature in the image coordinate system and the image feature to obtain the third instance feature of the object sample.

[0011] In combination with the first aspect, in some implementations of the first aspect, the target detection model also includes an image encoder, a point cloud encoder and a second point cloud decoder, the image encoder is a front module of the image decoder, and the point cloud encoder serves as a front module of the first point cloud decoder and the second point cloud decoder respectively; determining the image features, point cloud features and first instance features of the object sample includes: processing the image data based on the image encoder to obtain the image features of the object sample; processing the point cloud data based on the point cloud encoder to obtain the point cloud features of the object sample; processing the point cloud features of the object sample based on the second point cloud decoder to obtain the first instance features of the object sample.

[0012] In combination with the first aspect, in some implementations of the first aspect, a point cloud sparse query module is provided between the point cloud encoder and the first point cloud decoder, and between the point cloud encoder and the second point cloud decoder; an image sparse query module is provided between the image encoder and the image decoder; image data is processed based on the image encoder to obtain image features of the object sample, including: processing the image data based on the image encoder to obtain initial image features; performing attention-weighted sampling on the initial image features based on the image sparse query module to generate image features; processing the point cloud data based on the point cloud encoder to obtain point cloud features of the object sample, including: processing the point cloud data based on the point cloud encoder to obtain initial point cloud features; performing key point sampling on the initial point cloud features based on the point cloud sparse query module to generate point cloud features.

[0013] In combination with the first aspect, in certain implementations of the first aspect, processing the point cloud features of the object sample based on the second point cloud decoder to obtain the first instance feature of the object sample includes: determining the first initial instance feature and the random learnable instance feature; processing the first initial instance feature and the random learnable instance feature based on the second point cloud decoder to obtain the second initial instance feature; processing the second initial instance feature and the point cloud feature of the object sample based on the second point cloud decoder to obtain the first instance feature of the object sample.

[0014] In combination with the first aspect, in certain implementations of the first aspect, determining the first initial instance feature includes: determining a potential detection frame within a detection range in the point cloud sample data; clustering and / or screening the potential detection frames based on the distribution data of the potential detection frames to obtain the first initial instance feature; the method also includes: setting a detection range, the detection range including at least one of the forward distance, backward distance, left distance and right distance of the vehicle; generating a spatial mask based on the detection range, and filtering the point cloud sample data based on the spatial mask, and determining the filtered range as the detection range; the distribution data includes the distance between the potential detection frames and / or the target category.

[0015] In the second aspect, an embodiment of the present application provides a method for detecting a target object, including: acquiring point cloud data and image data of the target object; processing the point cloud data and image data based on a target detection model to obtain a detection result of the target object, wherein the target detection model is trained based on the training method mentioned in the first aspect.

[0016] In a third aspect, an embodiment of the present application provides a training device for training a target detection model, the target detection model including a first point cloud decoder and an image decoder, and the device including: a first determination module for determining the image features, point cloud features and first instance features of an object sample, wherein the first instance feature is determined based on the point cloud feature; a second determination module for determining the second instance feature of the object sample by processing the first instance feature and the point cloud feature based on the first point cloud decoder; a third determination module for determining the third instance feature of the object sample by processing the first instance feature and the image feature based on the image decoder; a fusion module for fusing the second instance feature and the third instance feature to obtain a fourth instance feature; and an adjustment module for adjusting the target detection model based on the fourth instance feature.

[0017] In the second aspect, an embodiment of the present application provides a target object detection device, including: an acquisition module for acquiring point cloud data and image data of the target object; a processing module for processing the point cloud data and image data based on a target detection model to obtain a detection result of the target object, wherein the target detection model is trained based on the training method mentioned in the first aspect.

[0018] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program for executing the methods of the first and second aspects.

[0019] In a sixth aspect, an embodiment of the present application provides an electronic device, comprising: a processor; a memory for storing processor-executable instructions; and the processor is configured to execute the methods described in the first and second aspects.

[0020] In a seventh aspect, an embodiment of the present application provides a computer program product, which includes instructions. When the instructions are executed on an electronic device, the electronic device implements the method described in the first and second aspects.

[0021] In this application, a joint training method is adopted to determine the image features, point cloud features and first instance features determined based on the point cloud features of the object sample in sequence, and then use the first point cloud decoder and image decoder to process the first instance features and point cloud features, and the first instance features and image features respectively to obtain the second instance features and the third instance features, and fuse the two to obtain the fourth instance features. Finally, the target detection model is adjusted based on the fourth instance features. This joint training method can give full play to the advantages of the depth information of the point cloud and the appearance information of the image to improve the detection accuracy. At the same time, it avoids the problems caused by training single-modal branches separately, enhances the multimodal fusion effect, and improves the training efficiency. In addition, the model fully considers the interaction of multimodal features during the training process, further enhances the adaptability to different environments and conditions, and improves the robustness of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The above and other purposes, features, and advantages of the present application will become more apparent through a more detailed description of the embodiments of the present application in conjunction with the accompanying drawings. The accompanying drawings are intended to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the drawings, the same reference numerals generally represent the same components or steps.

[0023] Figure 1 The figure is a flow chart of a training method provided in one embodiment of the present application.

[0024] Figure 2 The figure shows a flow chart of determining the image features, point cloud features and first instance features of an object sample provided by an embodiment of the present application.

[0025] Figure 3 FIG2 is a flow chart of determining a second instance feature of an object sample provided by an embodiment of the present application.

[0026] Figure 4 FIG2 is a flow chart of determining a third instance feature of an object sample provided by an embodiment of the present application.

[0027] Figure 5 Shown is a schematic diagram of the framework of the training method provided in one embodiment of the present application.

[0028] Figure 6 FIG2 is a flow chart of a method for detecting a target object according to an embodiment of the present application.

[0029] Figure 7 Shown is a schematic structural diagram of a training device provided in one embodiment of the present application.

[0030] Figure 8Shown is a schematic structural diagram of a target object detection device provided in one embodiment of the present application.

[0031] Figure 9 Shown is a structural schematic diagram of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0033] Figure 1 FIG. 1 is a flow chart of a training method provided by an embodiment of the present application. Exemplarily, the method is used to train a target detection model, which includes a first point cloud decoder and an image decoder. Figure 1 As shown, the method includes the following steps.

[0034] Step S110 , determining image features, point cloud features, and first instance features of the object sample.

[0035] Image features are features obtained by processing the image data of an object sample. These features can characterize information such as the object sample's appearance. Point cloud features are features obtained by processing the point cloud data of an object sample. These features can reflect characteristics such as the object sample's three-dimensional shape and spatial position. First instance features are determined based on point cloud features and can characterize the characteristics of a specific instance in the object sample.

[0036] In one implementation, an image feature extraction network is first used to process the image data of an object sample to obtain image features. Simultaneously, a point cloud feature extraction network is used to process the point cloud data to obtain point cloud features. Based on the point cloud features, first instance features are then determined to help the object detection model better understand and distinguish between different instances of objects.

[0037] Step S120 : Processing the first instance feature and the point cloud feature based on the first point cloud decoder to determine a second instance feature of the object sample.

[0038] The first point cloud decoder is a module within the object detection model. Optionally, it takes the first instance features and point cloud features as inputs and, through its internal network structure and algorithmic mechanisms, interacts, fuses, and transforms these two features. For example, the first point cloud decoder mines the specific instance information of the object sample contained in the first instance features, while combining it with the three-dimensional structural information in the point cloud features to further enhance and optimize the expression of these features, thereby generating more accurate and richer second instance features.

[0039] Step S130 : determining a third instance feature of the object sample based on processing the first instance feature and the image feature by the image decoder.

[0040] The image decoder is a module within the object detection model that processes image features. Optionally, the image decoder uses the first instance features and image features as input. Through its internal network structure and algorithmic mechanisms, the decoder interacts, fuses, and transforms these two features, further enhancing and optimizing their representation to produce more accurate and richer third instance features.

[0041] Step S140: Fusing the second instance feature and the third instance feature to obtain a fourth instance feature.

[0042] Specifically, the fourth instance feature is the result of integrating the high-level instance features of object samples in both point cloud and image modalities, and can characterize the characteristics of object samples more comprehensively and accurately.

[0043] It's understandable that the second instance feature has a strong advantage in 3D structural information in point clouds, while the third instance feature has a strong advantage in rich appearance information in images. By combining these two features using fusion strategies such as concatenation, weighted summation, or more complex attention fusion methods, the resulting fourth instance feature can simultaneously capture the 3D structural details and appearance feature details of the object sample, thereby more comprehensively describing the characteristics of the object sample.

[0044] Step S150: Adjust the target detection model based on the fourth instance feature.

[0045] The target detection model is a model used to automatically detect object samples in images or point cloud data and determine their location and category. The model parameters are continuously adjusted through training to improve the accuracy and robustness of detection.

[0046] Optionally, the fourth instance features, as comprehensive features containing rich object sample information, are input into the object detection model. The model calculates a loss function based on the differences between these features and the true object sample information (such as the annotated location and category). Then, through the backpropagation algorithm, the gradient of the loss function with respect to the model parameters is propagated backwards, thereby updating the values ​​of each trainable parameter in the model. This process aims to enable the model to better utilize the fourth instance features to predict the location and category of object samples, continuously improving the model's detection performance and enabling the model to more accurately detect new object samples.

[0047] In this embodiment, a joint training method is adopted to sequentially determine the image features, point cloud features, and first instance features determined based on the point cloud features of the object sample. Then, the first instance features and point cloud features, and the first instance features and image features are processed by the first point cloud decoder and the image decoder, respectively, to obtain the second instance features and the third instance features. The two are fused to obtain the fourth instance features, and finally, the target detection model is adjusted based on the fourth instance features. This joint training method can give full play to the advantages of the depth information of the point cloud and the appearance information of the image, thereby improving the detection accuracy. At the same time, it avoids the problems caused by separately training single-modal branches, enhances the multimodal fusion effect, and improves the training efficiency. In addition, the model fully considers the interaction of multimodal features during the training process, further enhances the adaptability to different environments and conditions, and improves the robustness of the model.

[0048] exist Figure 1 On the basis of the illustrated embodiment, the target detection model is adjusted based on the fourth instance feature, including: taking the fourth instance feature as the new first instance feature, repeatedly executing the steps of determining the second instance feature of the object sample to adjusting the target detection model based on the fourth instance feature, until the fourth instance feature meets the preset convergence condition or the number of repeated executions reaches the set number of cycles.

[0049] The preset convergence condition refers to a pre-set criterion for determining whether the fourth instance feature has reached an ideal state. For example, if the change in the fourth instance feature over multiple consecutive iterations is less than a preset threshold, or if the loss function value of the object detection model based on this feature on the validation set is lower than a threshold, the preset convergence condition is considered to be met. The set number of loops refers to the predetermined number of loop executions; when the set number of loop executions is reached, the loop process terminates.

[0050] Optionally, the fourth instance feature is used as the new first instance feature, replacing the first instance feature from the previous iteration. Then, steps S120 through S150 are repeated. This loop is repeated until the fourth instance feature meets a preset convergence condition or the loop has been executed a set number of times (e.g., 100 times). The loop then stops.

[0051] In this embodiment, in this way, the model can continuously use the new fourth instance feature to learn and adjust, gradually improve the detection performance of object samples, better mine the information in the point cloud and image features, enhance the expression ability of features, and ultimately enable the target detection model to perform target detection more accurately when facing new object samples.

[0052] Figure 2 The figure shows a flow chart of determining the image features, point cloud features and first instance features of an object sample provided by an embodiment of the present application. Figure 1 Based on the embodiment shown, Figure 2 The embodiment shown is described below in detail. Figure 2 The embodiment shown and Figure 1 The differences and similarities between the illustrated embodiments are not described in detail.

[0053] like Figure 2 As shown, in this embodiment, the target detection model also includes an image encoder, a point cloud encoder and a second point cloud decoder. The image encoder is a front module of the image decoder, and the point cloud encoder serves as a front module of the first point cloud decoder and the second point cloud decoder respectively.

[0054] Specifically, determining the image features, point cloud features, and first instance features of the object sample includes the following steps.

[0055] Step S210 : Processing the image data based on the image encoder to obtain image features of the object sample.

[0056] Image encoders are usually composed of deep learning architectures such as convolutional neural networks. Their main function is to extract and encode features from original image data, and convert spatial, texture, color and other information in the image into higher-level feature representations so that subsequent decoders can more effectively use these features for tasks such as target detection.

[0057] In one implementation, image data is first input into an image encoder, which uses its multi-layered neural network structure (such as convolutional layers, pooling layers, and activation layers) to extract and abstract features layer by layer. This process gradually transforms the image's raw pixel information into higher-level semantic features. The resulting output, the image features of the object sample, more compactly and effectively represent the key visual information of the object sample in the image, providing a foundation for the subsequent decoding and detection processes of the object detection model.

[0058] In another implementation, an image sparse query module is provided between the image encoder and the image decoder. The image encoder processes image data to obtain image features of an object sample, including: processing the image data using the image encoder to obtain initial image features; and performing attention-weighted sampling on the initial image features using the image sparse query module to generate image features.

[0059] It can be understood that the image sparse query module is a module used to perform attention-weighted sampling of initial image features. Through the attention mechanism, it can automatically learn and focus on the key parts of image features related to the object sample, thereby screening and weighting the features to generate more representative and targeted image features.

[0060] The initial image features contain the original feature representation of various information in the image data, but have not yet undergone further screening and weighting. The information is relatively redundant and not very targeted.

[0061] Optionally, in this embodiment, the image encoder first processes the image data to obtain initial image features. Next, the image sparse query module uses an attention mechanism to assign weights to different components of the initial image features. It is understood that the image sparse query module can focus on features relevant to the object sample and suppress irrelevant or minor feature information.

[0062] In this way, the generated image features are more focused on the key information of the object sample, improving the expressiveness and pertinence of the features, and providing more effective feature support for subsequent object detection tasks. Furthermore, through sparse querying, this solution can reduce model complexity without compromising model accuracy, shortening training time and cycle time, and reducing GPU memory usage.

[0063] Step S220 : Processing the point cloud data based on the point cloud encoder to obtain point cloud features of the object sample.

[0064] The point cloud encoder is a front-end module used in object detection models to process point cloud data. It consists of a series of neural network layers, such as convolutional layers and fully connected layers. Its main function is to encode the 3D coordinate information and other related attribute information in the point cloud data and extract point cloud features that can represent the shape, structure, and other characteristics of the object samples in the point cloud data.

[0065] In one implementation, point cloud data is first input into a point cloud encoder, which, through its internal multi-layer neural network structure, extracts and abstracts the point cloud data layer by layer. During this process, the original coordinate information and other attribute information of the point cloud data are gradually converted into higher-level semantic features. The resulting output, the point cloud features of the object sample, more compactly and effectively represent the key 3D information of the object sample in the point cloud data, providing a foundation for the subsequent decoding and detection processes of the object detection model.

[0066] In another implementation, a point cloud sparse query module is provided between the point cloud encoder and the first point cloud decoder, and between the point cloud encoder and the second point cloud decoder; the point cloud data is processed based on the point cloud encoder to obtain point cloud features of the object sample, including: processing the point cloud data based on the point cloud encoder to obtain initial point cloud features; and performing key point sampling on the initial point cloud features based on the point cloud sparse query module to generate point cloud features.

[0067] Optionally, the point cloud encoder is used to first process the point cloud data to generate initial point cloud features. These features contain essential 3D information about the object samples in the point cloud data. Next, the point cloud sparse query module is used to sample key points from the initial point cloud features. For example, a specific sampling strategy, such as clustering or eigenvalue-based methods, is used to select the most representative key points. These key points are then subjected to feature extraction and encoding, generating point cloud features that are more focused on the key information of the object samples.

[0068] This solution helps remove redundant information, improving feature quality and processing efficiency, enabling subsequent object detection models to more efficiently utilize point cloud features for object identification and localization. Similarly, through sparse querying, this solution can reduce model complexity without sacrificing accuracy, shortening training time and cycles, and reducing GPU memory usage.

[0069] Step S230 : Processing the point cloud features of the object sample based on the second point cloud decoder to obtain a first instance feature of the object sample.

[0070] Similar to the first point cloud decoder, the second point cloud decoder is also a module in the object detection model, which is specifically used to process point cloud features to generate instance features related to object samples.

[0071] In one implementation, point cloud features are fed as input to a second point cloud decoder. The second point cloud decoder further decodes and extracts features from the point cloud features through its internal multi-layer neural network structure. For example, the second point cloud decoder utilizes the three-dimensional information in the point cloud features to generate higher-level first instance features related to the object sample. These first instance features can more accurately describe key information such as the shape, size, and position of the object sample, providing a more effective feature representation for subsequent object detection models.

[0072] In another implementation, the point cloud features of the object sample are processed based on the second point cloud decoder to obtain the first instance features of the object sample, including: determining the first initial instance features and the random learnable instance features; processing the first initial instance features and the random learnable instance features based on the second point cloud decoder to obtain the second initial instance features; processing the second initial instance features and the point cloud features of the object sample based on the second point cloud decoder to obtain the first instance features of the object sample.

[0073] Random learnable instance features are trainable feature vectors whose initial values ​​are randomly set. They can be continuously learned and adjusted during training. Their purpose is to capture key instance information of object samples and enhance the model's ability to express the characteristics of object samples.

[0074] Optionally, first, a first initial instance feature and a random learnable instance feature are determined. Then, a second point cloud decoder combines the first initial point cloud feature and the random learnable instance feature to obtain a second initial instance feature. Then, the second initial instance feature and the point cloud feature of the object sample are processed again based on the second point cloud decoder to obtain a first instance feature of the object sample.

[0075] This approach effectively improves feature representation by introducing random learnable instance features and gradually optimizing them based on the initial instance features. This allows the model to more accurately describe the characteristics of object samples, thereby improving object detection accuracy. Furthermore, this progressive optimization process enhances the model's robustness, enabling more stable detection performance in the face of fluctuating data quality or complex scenarios.

[0076] The solution in steps S210 to S230 constructs an efficient feature extraction and processing flow by introducing an image encoder, a point cloud encoder, and a second point cloud decoder. Specifically, the image encoder first preprocesses the image data to extract the image features of the object sample, providing rich appearance information for subsequent target recognition. The point cloud encoder preprocesses the point cloud data to extract the point cloud features of the object sample, providing accurate three-dimensional structural information for the model. On this basis, the second point cloud decoder further processes the point cloud features to generate first instance features, which can more accurately describe key information such as the shape, size, and position of the object sample. This staged feature extraction and processing method not only improves the quality and expression ability of the features, but also enhances the model's ability to recognize and locate object samples, thereby improving the accuracy and robustness of target detection.

[0077] Furthermore, in some embodiments, determining the first initial instance feature includes: determining a potential detection box within a detection range in the point cloud sample data; and clustering and / or screening the potential detection box based on distribution data of the potential detection box to obtain the first initial instance feature.

[0078] Point cloud sample data refers to a collection of points collected from three-dimensional space to represent an object or scene. Each point contains three-dimensional coordinate information and possibly other attribute information (such as reflectivity). For example, point cloud sample data is point cloud data of objects such as vehicles and pedestrians, and their surroundings, obtained by LiDAR scanning.

[0079] The detection range refers to a predefined spatial range within the point cloud sample data. It is used to limit the model's focus area, eliminate interference from irrelevant areas, and improve detection efficiency and accuracy. For example, in autonomous driving, the detection range can be determined based on the vehicle's direction of travel and the sensor's effective detection range. A potential detection box is a three-dimensional spatial box within the detection range that is preliminarily determined based on certain rules or algorithms to potentially contain object samples.

[0080] In one implementation, regions potentially containing object samples are initially identified within the point cloud sample data, generating a series of potential detection frames. Next, these potential detection frames are clustered and / or filtered based on their distribution data. Specifically, clustering aims to group similar potential detection frames together to reduce the amount of data required for subsequent processing and to preliminarily determine the number and approximate location of object samples. Filtering involves removing obviously unreasonable detection frames or those that do not conform to the characteristics of the object samples based on predefined criteria (such as their size, shape, and point density), retaining those that are more likely to contain the object samples. Distribution data includes the distance between potential detection frames and / or the target category. For example, based on the distance between potential detection frames, similar detection frames are clustered and merged into a more accurate detection frame, thereby reducing redundancy and improving detection efficiency. Furthermore, by incorporating target category information, objects of different categories are distinguished and filtered, retaining detection frames that are more consistent with the target category characteristics, improving detection accuracy and reducing false detections and missed detections. Ultimately, the first initial instance features are obtained, providing a foundation for further feature extraction and object recognition in the subsequent object detection model.

[0081] This solution can effectively narrow the search scope, improve detection efficiency, avoid blind searches of the entire point cloud sample data, and reduce computational complexity. Secondly, the clustering and screening process can remove potential detection boxes that do not meet the target characteristics, reduce the amount of data for subsequent processing, highlight key target information, improve the accuracy and relevance of features, enhance the model's ability to recognize object samples, and reduce false detections and missed detections. In addition, the traditional first initial instance feature is randomly generated, which can easily lead to slow model convergence and low training. This embodiment uses a detection box instead of a random initial query, making the first instance feature relatively more accurate.

[0082] Furthermore, in some embodiments, a detection range is set; based on the detection range, a spatial mask is generated, and the point cloud sample data is filtered based on the spatial mask, and the filtered range is determined as the detection range.

[0083] A spatial mask is a tool or rule used to spatially filter point cloud sample data. It is generated based on a set detection range, marking areas within the detection range as valid and areas outside the detection range as invalid. In other words, when a spatial mask is applied to point cloud sample data, only point cloud data within the detection range is retained, while point cloud data outside the detection range is filtered out. Optionally, the detection range includes at least one of the following: forward distance, rearward distance, leftward distance, and rightward distance to the vehicle.

[0084] For example, first, the detection range is set according to the specific task requirements. For example, in an autonomous driving scenario, the forward distance to the vehicle is set to 50 meters, the backward distance to the vehicle is set to 20 meters, the left distance to the vehicle is set to 10 meters, and the right distance to the vehicle is set to 10 meters. Then, a spatial mask is generated based on this detection range. The spatial mask marks the areas within the detection range as valid and the areas outside the detection range as invalid. Next, the spatial mask is applied to the point cloud sample data to filter the point cloud sample data, retaining only the point cloud data within the detection range and filtering out the point cloud data outside the detection range.

[0085] In this embodiment, by setting a detection range that includes at least one of the forward, backward, left, and right distances of the vehicle, and generating a spatial mask to filter the point cloud sample data, the focus area of ​​target detection is limited, the detection efficiency is improved, the interference of irrelevant data is reduced, and the computational complexity is reduced, so that the model can quickly analyze key areas, meet real-time requirements, and at the same time, enhance detection accuracy.

[0086] Figure 3 The figure shows a flow chart of determining the second instance feature of an object sample provided by an embodiment of the present application. Figure 1 Based on the embodiment shown, Figure 3 The embodiment shown is described below in detail. Figure 3 The embodiment shown and Figure 1 The differences and similarities between the illustrated embodiments are not described in detail.

[0087] like Figure 3 As shown, in this embodiment, determining the second instance feature of the object sample based on the first point cloud decoder processing the first instance feature and the point cloud feature includes the following steps.

[0088] Step S310 : Projecting the first instance feature into the two-dimensional space where the image data is located based on the first point cloud decoder to obtain a feature matrix.

[0089] Specifically, image features are determined based on image data. The two-dimensional space in which the image data resides refers to the planar space in which the image data resides, typically represented by the image's pixel coordinate system. In this embodiment, the first instance feature is projected into the two-dimensional space to achieve alignment and fusion with the image feature. Furthermore, the feature matrix refers to the two-dimensional feature representation formed by projecting the first instance feature.

[0090] In one implementation, the first point cloud decoder uses spatial transformation and interpolation techniques to map the three-dimensional information in the first instance feature to a two-dimensional plane, so that the point cloud features can interact and fuse with the image features in the same space, thereby better combining the information of the two modalities and improving the accuracy of target detection.

[0091] In another implementation, the first point cloud decoder includes a multi-head self-attention module and a projection module. Furthermore, the multi-head self-attention module processes the first instance feature to obtain a fifth instance feature, and the projection module projects the fifth instance feature into the two-dimensional space where the image data resides to obtain a feature matrix.

[0092] The multi-head self-attention module is a neural network module used to process features. It can simultaneously calculate attention in multiple different subspaces, thereby capturing complex relationships and long-range dependencies between features. The projection module is mainly used to convert high-dimensional features to a specified spatial dimension, such as projecting three-dimensional features into two-dimensional space.

[0093] Optionally, first, the first instance feature is processed using a multi-head self-attention module to obtain a fifth instance feature. It is understandable that through the multi-head self-attention mechanism, the model can capture the complex relationships and dependencies within the first instance feature in different subspaces, thereby extracting a richer feature representation. Next, the fifth instance feature is projected into the two-dimensional space where the image data resides using a projection module to obtain a feature matrix, allowing it to interact and fuse with the image features in the same space.

[0094] Furthermore, based on the projection module, the fifth instance feature is projected into the two-dimensional space where the image data is located to obtain a feature matrix, including: based on the projection module, selecting key points in the detection box of each object sample contained in the fifth instance feature; projecting the key points into the two-dimensional space where the image data is located to obtain the coordinates of the key points in the two-dimensional space; based on the coordinates of the key points in the two-dimensional space, querying the features of the point cloud features at the corresponding coordinates to form a feature matrix.

[0095] Key points are representative points in point cloud data.

[0096] Optionally, first, key points are selected in the detection box of each object sample contained in the fifth instance feature. For example, the selection of key points should be based on the geometric features or semantic information of the object sample to ensure that the selected points can accurately represent the shape and position of the object sample. Then, the selected key points are projected into the two-dimensional space where the image data is located to obtain the coordinates of the key points in the two-dimensional space. Finally, based on the coordinates of the key points in the two-dimensional space, the features of the point cloud features at the corresponding coordinates are queried through methods such as spatial mapping and interpolation, and the queried features are combined into a feature matrix for subsequent feature interaction and fusion.

[0097] In this implementation, selecting key points reduces data volume and focuses on key information about the object sample, improving processing efficiency. Secondly, projecting the key points into two-dimensional space aligns point cloud features with image features in the same space, enhancing feature relevance and complementarity. Finally, by querying the features of the point cloud at corresponding coordinates and forming a feature matrix, the characteristics of the object sample can be more accurately described, improving the accuracy and robustness of target detection.

[0098] Step S320: performing feature interaction on the feature matrix and the point cloud features based on the first point cloud decoder to obtain a second instance feature of the object sample.

[0099] Feature interaction refers to the process of fusing and co-processing feature matrices and point cloud features, allowing them to complement and enhance each other, thereby generating more comprehensive and richer instance features. Furthermore, the second instance feature obtained through feature interaction combines the advantages of both the image feature matrix and point cloud features. It incorporates both the appearance information (such as color and texture) of the object sample as well as its depth and shape information, enabling a more accurate description of the object sample's characteristics and providing stronger support for subsequent object detection.

[0100] In one implementation, the first point cloud decoder can combine the appearance information in the image feature matrix with the depth and shape information in the point cloud features through element-level weighted summation; or concatenate the two into a higher-dimensional feature vector, and then further extract and transform the features through a neural network layer to obtain a second instance feature.

[0101] In another implementation, the first point cloud decoder further includes a deformable crisscross attention module, a feedforward neural network, and a normalization layer. After obtaining the feature matrix, the feature matrix and point cloud features are sequentially input into the deformable crisscross attention module. The feature matrix and point cloud features are then processed by the feedforward neural network and the normalization layer to perform feature interaction on the feature matrix and point cloud features to obtain the second instance features.

[0102] In this embodiment shown in steps S310 and S320, feature fusion is enhanced by projecting the first instance feature into the two-dimensional space where the image data resides to generate a feature matrix, and using the first point cloud decoder to interact with the feature matrix and the point cloud features. This cross-modal feature interaction enables the appearance information of the image features and the depth and shape information of the point cloud features to complement each other, generating a more comprehensive second instance feature, thereby improving the accuracy of target detection. At the same time, feature interaction enhances the robustness and adaptability of the model to complex scenes, improves the model's ability to distinguish, and enables it to more clearly distinguish different object samples.

[0103] Figure 4The figure shows a flow chart of determining the third instance feature of an object sample provided by an embodiment of the present application. Figure 1 Based on the embodiment shown, Figure 4 The embodiment shown is described below in detail. Figure 4 The embodiment shown and Figure 1 The differences and similarities between the illustrated embodiments are not described in detail.

[0104] like Figure 4 As shown, in this embodiment, the image decoder includes a projection module; based on the image decoder processing the first instance feature and the image feature, determining the third instance feature of the object sample includes the following steps.

[0105] Step S410: Projecting the first instance feature to the image coordinate system where the image data is located based on a projection module.

[0106] The image coordinate system is usually a two-dimensional coordinate system with the upper left corner of the image data as the origin and pixels as the unit, with the horizontal x-axis and the vertical y-axis. It is used to accurately locate pixels or feature points in the image data.

[0107] Optionally, the projection module maps the coordinates and eigenvalues ​​of the first instance feature to the image coordinate system through specific mathematical transformations (such as matrix multiplication, coordinate mapping, etc.) so that the first instance feature and the image feature are in the same coordinate system, providing a basis for subsequent feature interaction and fusion, ensuring that the two features can be accurately corresponded and combined, thereby improving the model's detection performance for object samples.

[0108] Step S420 : performing feature interaction between the first instance feature in the image coordinate system and the image feature to obtain a third instance feature of the object sample.

[0109] Feature interaction refers to the process of fusing the first instance feature in the image coordinate system with the image feature. It can be understood that feature interaction can fully utilize the advantages of both features to generate richer third instance features.

[0110] In one implementation, while ensuring that the first instance feature and the image feature are in the same coordinate system, the information of the two features is fused through feature interaction operations (such as element-level weighted summation and feature concatenation). In this way, the third instance feature contains both the depth and shape information from the first instance feature and the color and texture information from the image feature, thereby more comprehensively describing the characteristics of the object sample and improving the accuracy of object detection.

[0111] In another implementation, the image decoder further includes a deformable cross-attention module, a feedforward neural network, and a normalization layer. After obtaining the first instance feature in the image coordinate system, the first instance feature in the image coordinate system and the image feature are sequentially input into the deformable cross-attention module. The first instance feature in the image coordinate system and the image feature are then processed by the feedforward neural network and the normalization layer to achieve feature interaction between the first instance feature in the image coordinate system and the image feature, thereby obtaining a third instance feature of the object sample.

[0112] In this embodiment, a projection module is used to ensure spatial alignment of features, laying the foundation for subsequent interaction. Feature interaction then integrates depth, shape, color, and texture information to generate a more comprehensive third-instance feature. This fusion improves object detection accuracy, enabling the model to more accurately identify objects.

[0113] Figure 5 The figure shows a schematic diagram of the framework of the training method provided in one embodiment of the present application. Figure 5 As shown, first, image data is input into an image encoder, and point cloud data is input into a point cloud encoder to obtain image features and point cloud features of the object sample. Then, a second point cloud decoder processes the point cloud features and initial instances (i.e., second initial instance features) to obtain first instance features of the object sample. Next, an image decoder processes these first instance features and image features to obtain third instance features of the object sample. Simultaneously, the first point cloud decoder processes the first instance features to obtain second instance features. Finally, the second and third instance features are fused to obtain fourth instance features, and the object detection model is adjusted based on the fourth instance features.

[0114] In the next round of training, the fourth instance feature is used as the new first instance feature, and decoding and fusion operations are continued to gradually optimize the performance of the model.

[0115] Figure 6 FIG. 1 is a flow chart of a method for detecting a target object according to an embodiment of the present application. Figure 6 As shown, the method includes the following steps.

[0116] Step S610: Acquire point cloud data and image data of the target object.

[0117] For example, point cloud data is acquired through sensors such as lidar. This point cloud data consists of a series of points with three-dimensional coordinates, representing information such as the shape and position of an object. Image data is obtained through camera capture, which contains rich color and texture information but lacks accurate depth information.

[0118] Step S620: Process the point cloud data and image data based on the target detection model to obtain a detection result of the target object.

[0119] Optionally, the target detection model is trained based on the training method mentioned in the above embodiment. The detection result of the target object includes information such as the category, position, direction and dimension of the target object.

[0120] In this embodiment, the target detection model can give full play to the depth information of the point cloud and the appearance information of the image, improve the detection accuracy, and enhance the adaptability to different environments and conditions. Therefore, the object detection results obtained are more accurate and faster.

[0121] Combined with the above Figures 1 to 6 , describes the method embodiment of the present application in detail, and the following is combined with Figure 7 and Figure 8 , the device embodiment of the present application is described in detail. It should be understood that the description of the method embodiment corresponds to the description of the device embodiment, so for parts not described in detail, reference can be made to the previous method embodiment.

[0122] Figure 7 The figure shows a schematic diagram of the structure of a training device provided in one embodiment of the present application. Figure 7 As shown, the training device 70 provided in the embodiment of the present application includes:

[0123] A first determining module 710 is configured to determine an image feature, a point cloud feature, and a first instance feature of an object sample, wherein the first instance feature is determined based on the point cloud feature;

[0124] A second determining module 720 is configured to process the first instance feature and the point cloud feature based on the first point cloud decoder to determine a second instance feature of the object sample;

[0125] A third determining module 730 is configured to determine a third instance feature of the object sample based on processing the first instance feature and the image feature by the image decoder;

[0126] a fusion module 740 for fusing the second instance feature and the third instance feature to obtain a fourth instance feature;

[0127] The adjustment module 750 is used to adjust the target detection model based on the fourth instance feature.

[0128] In one embodiment of the present application, the adjustment module 750 is also used to use the fourth instance feature as the new first instance feature, and repeatedly execute the steps of determining the second instance feature of the object sample to adjusting the target detection model based on the fourth instance feature until the fourth instance feature meets the preset convergence condition or the number of repeated executions reaches the set number of cycles.

[0129] In one embodiment of the present application, the second determination module 720 is also used to project the first instance feature to the two-dimensional space where the image data is located based on the first point cloud decoder to obtain a feature matrix, wherein the image feature is determined based on the image data; and perform feature interaction on the feature matrix and the point cloud feature based on the first point cloud decoder to obtain a second instance feature of the object sample.

[0130] In one embodiment of the present application, the first point cloud decoder includes a multi-head self-attention module and a projection module, and the second determination module 720 is also used to process the first instance feature based on the multi-head self-attention module to obtain the fifth instance feature; and project the fifth instance feature to the two-dimensional space where the image data is located based on the projection module to obtain a feature matrix.

[0131] In one embodiment of the present application, the second determination module 720 is also used to select key points in the detection box of each object sample contained in the fifth instance feature based on the projection module; project the key points to the two-dimensional space where the image data is located to obtain the coordinates of the key points in the two-dimensional space; and query the features of the point cloud features at the corresponding coordinates based on the coordinates of the key points in the two-dimensional space to form a feature matrix.

[0132] In one embodiment of the present application, the image decoder includes a projection module; the third determination module 730 is also used to project the first instance feature to the image coordinate system where the image data is located based on the projection module, wherein the image feature is determined based on the image data; and perform feature interaction between the first instance feature in the image coordinate system and the image feature to obtain the third instance feature of the object sample.

[0133] In one embodiment of the present application, the target detection model also includes an image encoder, a point cloud encoder and a second point cloud decoder, the image encoder is a front module of the image decoder, and the point cloud encoder serves as a front module of the first point cloud decoder and the second point cloud decoder respectively; the first determination module 710 is also used to process image data based on the image encoder to obtain image features of the object sample; process point cloud data based on the point cloud encoder to obtain point cloud features of the object sample; and process point cloud features of the object sample based on the second point cloud decoder to obtain first instance features of the object sample.

[0134] In one embodiment of the present application, a point cloud sparse query module is provided between the point cloud encoder and the first point cloud decoder, and between the point cloud encoder and the second point cloud decoder; an image sparse query module is provided between the image encoder and the image decoder; the first determination module 710 is also used to process image data based on the image encoder to obtain initial image features; perform attention-weighted sampling on the initial image features based on the image sparse query module to generate image features; process point cloud data based on the point cloud encoder to obtain point cloud features of the object sample, including: processing point cloud data based on the point cloud encoder to obtain initial point cloud features; performing key point sampling on the initial point cloud features based on the point cloud sparse query module to generate point cloud features.

[0135] In one embodiment of the present application, the first determination module 710 is also used to determine the first initial instance feature and the random learnable instance feature; process the first initial instance feature and the random learnable instance feature based on the second point cloud decoder to obtain the second initial instance feature; and process the second initial instance feature and the point cloud feature of the object sample based on the second point cloud decoder to obtain the first instance feature of the object sample.

[0136] In one embodiment of the present application, the first determination module 710 is also used to determine potential detection frames within the detection range in the point cloud sample data; cluster and / or filter the potential detection frames based on the distribution data of the potential detection frames to obtain first initial instance features; the method also includes: setting a detection range, the detection range includes at least one of the forward distance, backward distance, left distance and right distance of the vehicle; based on the detection range, generating a spatial mask, and filtering the point cloud sample data based on the spatial mask, and determining the filtered range as the detection range; the distribution data includes the distance between potential detection frames and / or the target category.

[0137] Figure 8 The figure shows a schematic diagram of the structure of a target object detection device provided by an embodiment of the present application. Figure 8 As shown, the target object detection device 80 provided in the embodiment of the present application includes:

[0138] An acquisition module 810 is used to acquire point cloud data and image data of a target object;

[0139] The processing module 820 is used to process the point cloud data and image data based on the target detection model to obtain the detection result of the target object.

[0140] Below, reference Figure 9 To describe the electronic device according to the embodiment of the present application. Figure 9 Shown is a schematic structural diagram of an electronic device provided by an exemplary embodiment of the present application.

[0141] like Figure 9As shown, the electronic device 90 includes one or more processors 901 and a memory 902 .

[0142] The processor 901 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 90 to perform desired functions.

[0143] The memory 902 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), a hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 901 may execute the program instructions to implement the methods of the various embodiments of the present application described above and / or other desired functions.

[0144] In one example, the electronic device 90 may further include an input device 903 and an output device 904 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0145] The input device 903 may include, for example, a keyboard, a mouse, and the like.

[0146] The output device 904 can output various information to the outside, and can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto.

[0147] Of course, to simplify, Figure 9 Only some of the components related to the present application in the electronic device 90 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, the electronic device 90 may further include any other appropriate components according to specific application scenarios.

[0148] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the method according to various embodiments of the present application described above in this specification.

[0149] The computer program product may be written in any combination of one or more programming languages ​​to implement the program code for performing the operations of the embodiments of the present application, including object-oriented programming languages ​​such as Java, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0150] In addition, an embodiment of the present application may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, enables the processor to execute the steps of the method according to various embodiments of the present application described above in this specification.

[0151] The computer-readable storage medium may be any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may include, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0152] The basic principles of the present application have been described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, and effects mentioned in this application are merely illustrative and not restrictive, and it should not be assumed that these advantages, strengths, and effects are required of each embodiment of this application. In addition, the specific details disclosed above are merely illustrative and facilitating understanding, and are not restrictive. The above details do not limit this application to necessarily being implemented using the above specific details.

[0153] The block diagrams of the devices, devices, equipment, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, devices, equipment, and systems can be connected, arranged, or configured in any manner. Words such as "include," "comprise," "have," and the like are open-ended words, meaning "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or" and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0154] It should also be noted that in the apparatus, device, and method of the present application, each component or each step can be decomposed and / or recombined, and such decomposition and / or recombination should be regarded as equivalent solutions of the present application.

[0155] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present application. Therefore, the present application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0156] The above description has been provided for the purpose of illustration and description. Furthermore, this description is not intended to limit the embodiments of the present application to the forms disclosed herein. Although a number of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A training method, characterized in that: For training a target detection model, the target detection model includes a first point cloud decoder and an image decoder, the method comprising: Determine an image feature, a point cloud feature, and a first instance feature of the object sample, wherein the first instance feature is determined based on the point cloud feature; Determine a second instance feature of the object sample based on processing the first instance feature and the point cloud feature by the first point cloud decoder; determining a third instance feature of the object sample based on processing the first instance feature and the image feature by the image decoder; fusing the second instance feature and the third instance feature to obtain a fourth instance feature; Adjusting the object detection model based on the fourth instance feature; The object detection model further includes an image encoder, a point cloud encoder, and a second point cloud decoder. The image encoder is a pre-module of the image decoder, and the point cloud encoder serves as a pre-module of the first point cloud decoder and the second point cloud decoder respectively. The image features, point cloud features, and first instance features of the object sample are determined, including: Processing image data based on the image encoder to obtain image features of the object sample; Processing the point cloud data based on the point cloud encoder to obtain point cloud features of the object sample; The point cloud features of the object sample are processed based on the second point cloud decoder to obtain a first instance feature of the object sample.

2. The training method according to claim 1, characterized in that The adjusting the target detection model based on the fourth instance feature includes: The fourth instance feature is used as a new first instance feature, and the steps of determining the second instance feature of the object sample to adjusting the target detection model based on the fourth instance feature are repeated until the fourth instance feature meets the preset convergence condition or the number of repeated executions reaches the set number of cycles.

3. The training method according to claim 1, characterized in that The processing of the first instance feature and the point cloud feature based on the first point cloud decoder to determine a second instance feature of the object sample includes: Projecting the first instance feature into a two-dimensional space where the image data is located based on the first point cloud decoder to obtain a feature matrix, wherein the image feature is determined based on the image data; Perform feature interaction on the feature matrix and the point cloud features based on the first point cloud decoder to obtain a second instance feature of the object sample.

4. The training method according to claim 3, characterized in that The first point cloud decoder includes a multi-head self-attention module, and before projecting the first instance feature to the two-dimensional space where the image data is located based on the first point cloud decoder, further includes: Processing the first instance feature based on the multi-head self-attention module to obtain a fifth instance feature; The first point cloud decoder also includes a projection module; the first instance feature is projected to the two-dimensional space where the image data is located based on the first point cloud decoder to obtain a feature matrix, including: based on the projection module, the fifth instance feature is projected to the two-dimensional space where the image data is located to obtain the feature matrix.

5. The training method according to claim 4, characterized in that The projecting the fifth instance feature to the two-dimensional space where the image data is located based on the projection module to obtain the feature matrix includes: Selecting key points in the detection box of each object sample included in the fifth instance feature based on the projection module; Projecting the key point to the two-dimensional space where the image data is located to obtain the coordinates of the key point in the two-dimensional space; Based on the coordinates of the key points in the two-dimensional space, the features of the point cloud features at the corresponding coordinates are queried to form the feature matrix.

6. The training method according to any one of claims 1 to 5, characterized in that: The image decoder includes a projection module; and determining a third instance feature of the object sample based on processing the first instance feature and the image feature by the image decoder includes: Projecting the first instance feature to an image coordinate system where image data is located based on the projection module, wherein the image feature is determined based on the image data; Perform feature interaction between the first instance feature in the image coordinate system and the image feature to obtain a third instance feature of the object sample.

7. The training method according to claim 1, characterized in that A point cloud sparse query module is provided between the point cloud encoder and the first point cloud decoder, and between the point cloud encoder and the second point cloud decoder; an image sparse query module is provided between the image encoder and the image decoder; The processing of the image data based on the image encoder to obtain the image features of the object sample includes: Processing the image data based on the image encoder to obtain initial image features; Performing attention-weighted sampling on the initial image features based on the image sparse query module to generate the image features; The step of processing the point cloud data based on the point cloud encoder to obtain point cloud features of the object sample includes: Processing the point cloud data based on the point cloud encoder to obtain initial point cloud features; Key point sampling is performed on the initial point cloud features based on the point cloud sparse query module to generate the point cloud features.

8. The training method according to claim 1, characterized in that: The step of processing the point cloud features of the object sample based on the second point cloud decoder to obtain a first instance feature of the object sample includes: determining a first initial instance feature and a random learnable instance feature; Processing the first initial instance feature and the random learnable instance feature based on the second point cloud decoder to obtain a second initial instance feature; The second initial instance feature and the point cloud feature of the object sample are processed based on the second point cloud decoder to obtain a first instance feature of the object sample.

9. The training method according to claim 8, characterized in that The determining of the first initial instance feature comprises: Determine potential detection boxes within the detection range in the point cloud sample data; Clustering and / or screening the potential detection boxes based on the distribution data of the potential detection boxes to obtain the first initial instance features; The method further comprises: Setting the detection range, wherein the detection range includes at least one of the forward distance, the rearward distance, the leftward distance, and the rightward distance of the vehicle; generating a spatial mask based on the detection range, filtering the point cloud sample data based on the spatial mask, and determining the filtered range as the detection range; The distribution data includes distances between the potential detection boxes and / or target categories.

10. A method for detecting a target object, characterized in that: include: Obtain point cloud data and image data of the target object; The point cloud data and the image data are processed based on a target detection model to obtain a detection result of the target object, wherein the target detection model is trained based on the training method mentioned in any one of claims 1 to 9.

11. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is used to execute the training method described in any one of claims 1 to 9 and / or the target object detection method described in claim 10.

Images