Three-dimensional target detection method, model training method, equipment and storage medium

Abstract

The embodiment of the invention provides a three-dimensional target detection method, a model training method, equipment and a storage medium. The method comprises the following steps: inputting first modal training data into a three-dimensional training reference model, and inputting second modal training data into a three-dimensional target detection model; constructing a similarity matrix under at least one feature dimension according to a first modal feature determined by the three-dimensional training reference model and a second modal feature determined by the three-dimensional target detection model, and determining feature loss information according to the similarity matrix under the at least one feature dimension; determining a quality score of a first detection result output by the three-dimensional training reference model, and determining response loss information according to the quality score, the first detection result and a second detection result output by the three-dimensional target detection model; the three-dimensional target detection model is trained according to the feature loss information and the response loss information, the trained three-dimensional target detection model is obtained, and the detection precision is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and particularly relates to a three-dimensional target detection method, a model training method, equipment and a storage medium. BACKGROUND

[0002] With the continuous improvement of the intelligent level of power system operation, the three-dimensional target detection technology is more and more widely used in the power system due to its accurate and comprehensive capture ability, for example, it can be applied to inspection and operation, construction, equipment manufacturing and other scenarios.

[0003] The traditional three-dimensional target detection technology mainly performs three-dimensional target detection based on LiDAR (Light Detection and Ranging).

[0004] This three-dimensional target detection method has high detection accuracy, but the hardware device cost is high and the data acquisition is limited, which is difficult to meet the large-scale deployment requirements. SUMMARY

[0005] The embodiments of the present application provide a three-dimensional target detection method, a model training method, equipment and a storage medium, which can improve the training of a three-dimensional target detection model through multiple loss information, thereby improving the detection accuracy of the three-dimensional target detection model.

[0006] In a first aspect, the embodiments of the present application provide a three-dimensional target detection model training method, comprising:

[0007] obtaining first modality training data and second modality training data of a to-be-detected object, inputting the first modality training data into a three-dimensional training reference model, and inputting the second modality training data into a three-dimensional target detection model;

[0008] constructing a similarity matrix in at least one feature dimension according to the first modality feature of the first modality training data determined by the three-dimensional training reference model and the second modality feature of the second modality training data determined by the three-dimensional target detection model, and determining feature loss information according to the similarity matrix in the at least one feature dimension;

[0009] obtaining a first detection result output by the three-dimensional training reference model and a second detection result output by the three-dimensional target detection model, and determining response loss information according to the quality score, the first detection result and the second detection result after determining the quality score of the first detection result;

[0010] The three-dimensional target detection model is trained based on the feature loss information and the response loss information to obtain the trained three-dimensional target detection model.

[0011] In one possible implementation, the at least one feature dimension includes a global feature dimension and a temporal feature dimension; based on the similarity matrix under the at least one feature dimension, feature loss information is determined, including:

[0012] The first loss information is determined based on the global similarity matrix under the global feature dimension; wherein the global similarity matrix is ​​determined based on the similarity between the first modal feature and the second modal feature;

[0013] The second loss information is determined based on the inter-modal temporal similarity mapping matrix and the intra-modal temporal similarity mapping matrix under the temporal feature dimension; wherein, the inter-modal temporal similarity mapping matrix is ​​determined based on the similarity between the historical frame features of the first modality feature and the current frame features of the second modality feature; the intra-modal temporal similarity mapping matrix is ​​determined based on the similarity between the historical frame features of the first modality feature and the current frame features of the first modality feature;

[0014] The feature loss information is determined based on the first loss information and the second loss information.

[0015] In one possible implementation, the method further includes:

[0016] Determine the first feature mean of the first modal feature, and determine the second feature mean of the second modal feature;

[0017] The first modal feature is mean-centered based on the mean of the first feature to obtain the processed first modal feature; and the second modal feature is mean-centered based on the mean of the second feature to obtain the processed second modal feature.

[0018] The global similarity matrix is ​​determined based on the similarity between the processed first modal features and the processed second modal features.

[0019] In one possible implementation, it includes:

[0020] Determine the diagonal matrix of the global similarity matrix;

[0021] The first sub-loss is determined by the square of the difference between the identity matrix and the diagonal matrix.

[0022] The second sub-loss is determined based on the square of the diagonal matrix and a preset adjustment parameter;

[0023] The first loss information is determined based on the first sub-loss and the second sub-loss.

[0024] In one possible implementation, the second loss information is determined based on the inter-modal temporal similarity mapping matrix and the intra-modal temporal similarity mapping matrix under the temporal feature dimension, including:

[0025] Determine the first probability distribution of the intermodal temporal similarity mapping matrix, and determine the second probability distribution of the intramodal temporal similarity mapping matrix;

[0026] The second loss information is determined based on the probability distribution difference between the first probability distribution and the second probability distribution.

[0027] In one possible implementation, the first detection result includes a predicted regression result and a predicted classification result; determining the quality score of the first detection result includes:

[0028] Determine the intersection-union ratio (IUU) of the predicted regression results with the corresponding actual regression results;

[0029] The quality score is determined based on the crossover ratio of the regression results and the predicted classification results.

[0030] In one possible implementation, the method further includes:

[0031] Determine the matching cost of the second detection result;

[0032] The 3D target detection model is trained based on the matching cost, the feature loss information, and the response loss information to obtain the trained 3D target detection model.

[0033] Secondly, embodiments of this application provide a three-dimensional target detection method, including:

[0034] Acquire target modal data including the object to be detected;

[0035] The target modal data is input into the trained three-dimensional target detection model to obtain the three-dimensional target detection result; wherein the trained three-dimensional target detection model is determined according to the training method of the three-dimensional target detection model according to any one of the first aspects above.

[0036] Thirdly, embodiments of this application provide a training apparatus for a three-dimensional target detection model, comprising:

[0037] The first acquisition unit is used to acquire the first modality training data and the second modality training data of the object to be detected, and input the first modality training data into the three-dimensional training reference model, and input the second modality training data into the three-dimensional target detection model;

[0038] The first determining unit is used to construct a similarity matrix in at least one feature dimension based on the first modal features of the first modal training data determined by the three-dimensional training reference model and the second modal features of the second modal training data determined by the three-dimensional target detection model, and to determine feature loss information based on the similarity matrix in the at least one feature dimension.

[0039] The second determining unit is used to obtain the first detection result output by the training reference model of the three dimensions and the second detection result output by the target detection model of the three dimensions, and after determining the quality score of the first detection result, determine the response loss information based on the quality score, the first detection result and the second detection result;

[0040] The training unit is used to train the three-dimensional target detection model based on the feature loss information and the response loss information to obtain the trained three-dimensional target detection model.

[0041] In one possible implementation, the at least one feature dimension includes a global feature dimension and a temporal feature dimension; in this case, the first determining unit is configured to:

[0042] The first loss information is determined based on the global similarity matrix under the global feature dimension; wherein the global similarity matrix is ​​determined based on the similarity between the first modal feature and the second modal feature;

[0043] The second loss information is determined based on the inter-modal temporal similarity mapping matrix and the intra-modal temporal similarity mapping matrix under the temporal feature dimension; wherein, the inter-modal temporal similarity mapping matrix is ​​determined based on the similarity between the historical frame features of the first modality feature and the current frame features of the second modality feature; the intra-modal temporal similarity mapping matrix is ​​determined based on the similarity between the historical frame features of the first modality feature and the current frame features of the first modality feature;

[0044] The feature loss information is determined based on the first loss information and the second loss information.

[0045] In one possible implementation, the device is also used for:

[0046] Determine the first feature mean of the first modal feature, and determine the second feature mean of the second modal feature;

[0047] The first modal feature is mean-centered based on the mean of the first feature to obtain the processed first modal feature; and the second modal feature is mean-centered based on the mean of the second feature to obtain the processed second modal feature.

[0048] The global similarity matrix is ​​determined based on the similarity between the processed first modal features and the processed second modal features.

[0049] In one possible implementation, the first determining unit is configured to:

[0050] Determine the diagonal matrix of the global similarity matrix;

[0051] The first sub-loss is determined by the square of the difference between the identity matrix and the diagonal matrix.

[0052] The second sub-loss is determined based on the square of the diagonal matrix and a preset adjustment parameter;

[0053] The first loss information is determined based on the first sub-loss and the second sub-loss.

[0054] In one possible implementation, the first determining unit is configured to:

[0055] Determine the first probability distribution of the intermodal temporal similarity mapping matrix, and determine the second probability distribution of the intramodal temporal similarity mapping matrix;

[0056] The second loss information is determined based on the probability distribution difference between the first probability distribution and the second probability distribution.

[0057] In one possible implementation, the first detection result includes a predicted regression result and a predicted classification result; in this case, the second determining unit is configured to:

[0058] Determine the intersection-union ratio (IUU) of the predicted regression results with the corresponding actual regression results;

[0059] The quality score is determined based on the crossover ratio of the regression results and the predicted classification results.

[0060] In one possible implementation, the device is also used for:

[0061] Determine the matching cost of the second detection result;

[0062] The 3D target detection model is trained based on the matching cost, the feature loss information, and the response loss information to obtain the trained 3D target detection model.

[0063] Fourthly, embodiments of this application provide a three-dimensional target detection device, comprising:

[0064] The second acquisition unit is used to acquire target modal data including the object to be detected;

[0065] A detection unit is used to input the target modal data into a trained three-dimensional target detection model to obtain a three-dimensional target detection result; wherein the trained three-dimensional target detection model is determined by the training method of the three-dimensional target detection model according to any one of the first aspects above.

[0066] Fifthly, embodiments of this application provide a computer device, including: a memory and a processor;

[0067] The memory stores computer-executed instructions;

[0068] The processor executes computer execution instructions stored in the memory, causing the processor to perform the various possible implementations of the first aspect above, or the various possible implementations of the second aspect above.

[0069] In a sixth aspect, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement various possible implementations as described in the first aspect above, or to implement various possible implementations as described in the second aspect above.

[0070] In a seventh aspect, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the various possible implementations described in the first aspect above, or implements the various possible implementations described in the second aspect above.

[0071] The three-dimensional object detection method, model training method, device, and storage medium provided in this application can acquire first-modal training data and second-modal training data of the object to be detected, input the first-modal training data into a three-dimensional training reference model, and input the second-modal training data into a three-dimensional object detection model. Then, based on the first-modal features of the first-modal training data determined by the three-dimensional training reference model and the second-modal features of the second-modal training data determined by the three-dimensional object detection model, a similarity matrix under at least one feature dimension is constructed, and feature loss information is determined based on the similarity matrix under at least one feature dimension. This implementation can determine refined and comprehensive feature loss information from at least one feature dimension, thereby considering global correlation, local correlation, and temporal correlation in multimodal features, thus improving knowledge transfer efficiency. Afterwards, the first detection result output by the three-dimensional training reference model and the second detection result output by the three-dimensional object detection model can be acquired, and after determining the quality score of the first detection result, response loss information is determined based on the quality score, the first detection result, and the second detection result. At this point, distillation can be performed based on the high-quality initial detection results, allowing the 3D object detection model to focus on learning effective information, thereby improving the accuracy and robustness of knowledge distillation. Finally, the 3D object detection model is trained using feature loss information and response loss information to obtain a trained 3D object detection model. This allows for improvements in the object detection performance of the 3D object detection model based on the 3D training reference model, thereby reducing hardware costs in 3D object detection scenarios and meeting the needs of large-scale deployment. Attached Figure Description

[0072] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0073] Figure 1 A flowchart illustrating a training method for a three-dimensional target detection model provided in an embodiment of this application;

[0074] Figure 2 A flowchart illustrating another training method for a three-dimensional target detection model provided in this application embodiment;

[0075] Figure 3 A schematic diagram illustrating the training process of a three-dimensional target detection model provided in an embodiment of this application;

[0076] Figure 4 A flowchart illustrating a three-dimensional target detection method provided in an embodiment of this application;

[0077] Figure 5A schematic diagram of the structure of a training device for a three-dimensional target detection model provided in an embodiment of this application;

[0078] Figure 6 This is a schematic diagram of the structure of a three-dimensional target detection device provided in an embodiment of this application;

[0079] Figure 7 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application.

[0080] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0081] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0082] With the continuous improvement of the intelligence level of power system operation, three-dimensional target detection technology, due to its accurate and comprehensive capture capabilities, is being used more and more widely in power systems. For example, it can be applied to various scenarios such as inspection and maintenance, construction, and equipment manufacturing.

[0083] Traditional 3D target detection technology is mainly based on LiDAR (Light Detection and Ranging) for 3D target detection.

[0084] This three-dimensional target detection method has high detection accuracy, but the hardware cost is high and data acquisition is limited, making it difficult to meet the needs of large-scale deployment.

[0085] In contrast, 3D object detection methods based on multi-view images have become a research hotspot due to their lower hardware costs, higher availability, and rich semantic information. However, 3D object detection methods based on multi-view images are susceptible to occlusion, lighting, and viewpoint differences when lacking precise spatial cues, resulting in limited detection performance.

[0086] Based on this, in order to improve the performance of 3D target detection methods based on multi-view images, a cross-modal knowledge distillation method is proposed. This method can transfer the geometric information of the LiDAR mode to the multi-view image mode.

[0087] However, the non-homogeneity of features across different modalities (e.g., the sparsity of radar point cloud features determined by LiDAR and the density of image features based on multi-view images make direct alignment of features difficult) and the large differences in feature space distribution make cross-modal knowledge distillation challenging. Existing cross-modal knowledge distillation methods focus only on minimizing the global distance between cross-modal features, ignoring modality-specific feature structure knowledge, resulting in poor knowledge distillation performance and impacting the detection performance of the final model.

[0088] Furthermore, temporal information integration is crucial for 3D target detection in dynamic scenes, but existing methods often focus on static features in a single frame, neglecting the impact of cross-frame motion continuity on detection accuracy.

[0089] The training method for the 3D object detection model provided in this application can optimize the student model (i.e., the 3D object detection model) based on multi-view images using a LiDAR-based teacher model (i.e., the 3D training reference model), thereby improving the object detection performance of the 3D object detection model. Specifically, feature loss information can be constructed based on a similarity matrix in at least one feature dimension, thereby achieving alignment processing between LiDAR features and multi-view image features. Based on the quality score, the detection results of the 3D training reference model, and the detection results of the 3D object detection model, selective distillation is performed at the response level to optimize the training of the 3D object detection model, thus improving the object detection accuracy of the trained 3D object detection model.

[0090] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0091] Figure 1 A flowchart illustrating a training method for a three-dimensional target detection model provided in this application embodiment is shown below. Figure 1 As shown, the method includes:

[0092] S101. Obtain the first modality training data and the second modality training data of the object to be detected, and input the first modality training data into the three-dimensional training reference model, and input the second modality training data into the three-dimensional target detection model.

[0093] In one example, the first modality training data could indicate radar point cloud data acquired from LiDAR; the second modality training data could indicate multi-view image data acquired from a camera.

[0094] In one example, a 3D training reference model can instruct a high-precision model for processing radar point cloud data; a 3D target detection model can instruct a lightweight model for processing multi-view image data.

[0095] S102. Based on the first modal features of the first modal training data determined by the three-dimensional training reference model and the second modal features of the second modal training data determined by the three-dimensional target detection model, construct a similarity matrix under at least one feature dimension, and determine the feature loss information based on the similarity matrix under at least one feature dimension.

[0096] In one example, the first modal features of the first modal training data can indicate radar point cloud features.

[0097] In one example, a three-dimensional training reference model can obtain low-level point cloud features from radar point cloud data based on a three-dimensional encoder, and convert these low-level point cloud features into BEV (Bird's Eye View) features with spatial ranges of X, Y, and Z and number of channels D, thereby obtaining the first modal features.

[0098] In one example, the second modality features of the second modality training data can indicate multi-view image features.

[0099] In one example, after determining low-level image features from multi-view image data based on a 3D object detection model, these low-level image features can be converted into BEV features based on estimated depth cues. Alternatively, the depth can be implicitly learned using a cross-attention mechanism to convert the low-level image features into BEV features.

[0100] At this point, spatial alignment between radar point cloud features and multi-view image features can be achieved by determining the first modal feature and the second modal feature.

[0101] In one example, the feature dimension can indicate the alignment dimension of the first modality feature and the second modality feature. For example, the feature dimension can be any one of the structural dimension, global dimension, semantic dimension, and temporal dimension. There is no restriction on the type of feature dimension here, and the choice is based on actual needs.

[0102] Optionally, in this embodiment, feature loss information can be determined based on the similarity matrix under at least one feature dimension. In this case, the model parameters can be adjusted based on the feature loss information to align the first modality feature and the second modality feature.

[0103] S103. Obtain the first detection result output by the three-dimensional training reference model and the second detection result output by the three-dimensional object detection model. After determining the quality score of the first detection result, determine the response loss information based on the quality score, the first detection result, and the second detection result.

[0104] In one example, both the first and second detection results include at least a regression result and a classification result, where the regression result indicates the object candidate box of the object to be detected, and the classification result indicates the object category of the object to be detected.

[0105] In one example, a quality score can indicate the detection performance of the first detection result.

[0106] S104. Based on the feature loss information and response loss information, train the three-dimensional target detection model to obtain the trained three-dimensional target detection model.

[0107] In one example, the optimization objective of the 3D object detection model can be determined based on the feature loss information and the response loss information, and the 3D object detection model can be trained based on the optimization objective to obtain the trained 3D object detection model.

[0108] As described above, this embodiment of the application can acquire first modality training data and second modality training data of the object to be detected, input the first modality training data into a three-dimensional training reference model, and input the second modality training data into a three-dimensional object detection model. Then, based on the first modality features of the first modality training data determined by the three-dimensional training reference model and the second modality features of the second modality training data determined by the three-dimensional object detection model, a similarity matrix under at least one feature dimension is constructed, and feature loss information is determined based on the similarity matrix under at least one feature dimension. This implementation can determine refined and comprehensive feature loss information from at least one feature dimension, thereby considering global correlation, local correlation, and temporal correlation in multimodal features, thus improving knowledge transfer efficiency. Subsequently, the first detection result output by the three-dimensional training reference model and the second detection result output by the three-dimensional object detection model can be acquired, and after determining the quality score of the first detection result, response loss information is determined based on the quality score, the first detection result, and the second detection result. At this time, distillation can be performed based on the high-quality first detection result, so that the three-dimensional object detection model can focus on learning effective information, thereby improving the accuracy and robustness of knowledge distillation. Finally, based on the feature loss information and response loss information, the 3D object detection model is trained to obtain the trained 3D object detection model. This allows the object detection performance of the 3D object detection model to be improved based on the 3D training reference model, thereby reducing the hardware cost in 3D object detection scenarios and meeting the needs of large-scale deployment.

[0109] Figure 2 A flowchart illustrating another training method for a three-dimensional object detection model provided in this application embodiment is shown below. Figure 2 As shown, in this embodiment... Figure 1 Based on the examples, the training method for the 3D target detection model is described in detail, including:

[0110] S201. Obtain the first modality training data and the second modality training data of the object to be detected, and input the first modality training data into the three-dimensional training reference model, and input the second modality training data into the three-dimensional target detection model.

[0111] In one example, this step can be referred to the content described in S101 above, and will not be repeated in detail here.

[0112] S202. Based on the first modal features of the first modal training data determined by the three-dimensional training reference model and the second modal features of the second modal training data determined by the three-dimensional target detection model, construct a similarity matrix under at least one feature dimension.

[0113] Optionally, at least one feature dimension may include a global feature dimension and a time feature dimension.

[0114] At this point, for the global feature dimension, the global similarity matrix can be determined based on the similarity between the first modality feature and the second modality feature.

[0115] At this point, regarding the global feature dimension, on the one hand, the inter-modal temporal similarity mapping matrix under the temporal feature dimension can be determined based on the similarity between the historical frame features of the first modality feature and the current frame features of the second modality feature; on the other hand, the intra-modal temporal similarity mapping matrix under the temporal feature dimension can be determined based on the similarity between the historical frame features of the first modality feature and the current frame features of the first modality feature.

[0116] In one possible implementation, to eliminate the global bias between the first modal feature and the second modal feature, in this embodiment of the application, before determining the global similarity matrix under the global feature dimension based on the similarity between the first and second modal features, a first feature mean of the first modal feature and a second feature mean of the second modal feature can be determined firstly. Then, the first modal feature is mean-centered based on the first feature mean to obtain the processed first modal feature; and the second modal feature is mean-centered based on the second feature mean to obtain the processed second modal feature. Finally, the global similarity matrix is ​​determined based on the similarity between the processed first modal feature and the processed second modal feature.

[0117] In specific implementation, the mean of the first feature can be determined by the following formula (1), the mean of the second feature can be determined by formula (2), the mean of the first modal feature can be centered by formula (3) to obtain the processed first modal feature, and the mean of the second modal feature can be centered by formula (4) to obtain the processed second modal feature.

[0118] (1)

[0119] (2)

[0120] (3)

[0121] (4)

[0122] Where F represents the first modal feature, G represents the second modal feature. ; This represents the mean of the first characteristic; This represents the mean of the second characteristic; This represents the first modal feature after processing. This represents the processed second modal feature.

[0123] Optionally, when determining the global similarity matrix based on the similarity between the processed first modality features and the processed second modality features, you can refer to the following formula (5).

[0124] (5)

[0125] Where C represents the global similarity matrix, express The transpose of .

[0126] Then, feature loss information can be determined based on the determined similarity matrix, as described in the process below.

[0127] S203. Determine the first loss information based on the global similarity matrix under the global feature dimension.

[0128] In related technologies, element-level distance loss or weighted foreground masking is generally used to make the 3D object detection model mimic the features of the 3D training reference model, while ignoring the inherent structural knowledge in different modal features, thus failing to effectively distill the rich information contained in different modal features.

[0129] Based on this, the embodiments of this application can propose a correlation normalization distillation method by constructing first loss information to maximize the similarity between different modal features.

[0130] In practice, the diagonal matrix of the global similarity matrix can be determined first; then, the first sub-loss can be determined based on the square of the difference between the identity matrix and the diagonal matrix; next, the second sub-loss can be determined based on the square of the diagonal matrix and the preset adjustment parameter; finally, the first loss information can be determined based on the first and second sub-losses.

[0131] Optionally, the diagonal matrix of the global similarity matrix can be used to reflect the importance of feature channels.

[0132] At this point, the first loss information can be found in the following formula (6).

[0133] (6)

[0134] in, This indicates the first loss information; This represents the i-th element in the diagonal matrix; This indicates the preset adjustment parameters, for example, The value can be preset to 0.01.

[0135] In the above implementation, the similarity between the feature components of the first modality and the second modality can be encouraged by the square of the difference between the identity matrix and the diagonal matrix, and the correlation between misaligned feature components can be reduced by the square of the diagonal matrix and a preset adjustment parameter. At this time, based on the first loss information, the 3D object detection model can learn the global structural pattern of the 3D training reference model. Simultaneously, by introducing the preset adjustment parameter, the diversity of features is balanced, preventing overfitting of the noise of the first modality.

[0136] S204. Determine the second loss information based on the inter-modal temporal similarity mapping matrix and the intra-modal temporal similarity mapping matrix under the temporal feature dimension.

[0137] Since sensors can acquire data containing the object to be detected in real time in practical application scenarios, this application embodiment can integrate time information and combine it with information from historical frame data to perform three-dimensional target detection processing of the current frame, so as to improve the accuracy of three-dimensional target detection of the current frame.

[0138] Research has shown that due to the non-uniformity of spatial and temporal alignment, it is impossible to directly learn features from historical frame data. Therefore, this application proposes a temporal consistency distillation method, which indirectly learns features from historical frame data by introducing temporal similarity.

[0139] In practice, the similarity matrix under the time feature dimension can be determined first.

[0140] In one example, suppose the current frame feature of the first modality is denoted as... The current frame features of the second modality are denoted as The historical frame features of the first modality (e.g., the kth previous frame) are denoted as .

[0141] Therefore, the intermodal temporal similarity mapping matrix determined based on the similarity between the historical frame features of the first modality and the current frame features of the second modality can be seen in the following formula (7).

[0142] (7)

[0143] Based on the similarity between the historical frame features of the first modality and the current frame features of the first modality, the intramodal temporal similarity mapping matrix under the temporal feature dimension is determined, as shown in the formula (8) below.

[0144] (8)

[0145] in, Represents the intermodal temporal similarity mapping matrix; K represents the maximum number of historical frames to reference; This represents the intramodal temporal similarity mapping matrix.

[0146] In one example, after determining the inter-modal temporal similarity mapping matrix and the intra-modal temporal similarity mapping matrix in the time dimension, the second loss information can be determined.

[0147] In practice, the first probability distribution of the inter-modal temporal similarity mapping matrix can be determined first, and the second probability distribution of the intra-modal temporal similarity mapping matrix can be determined second, based on the probability distribution difference between the first and second probability distributions.

[0148] Optionally, the first probability distribution and the second probability distribution can be determined by applying a Softmax function to each row of the inter-modal temporal similarity mapping matrix and the intra-modal temporal similarity mapping matrix. Then, the second loss information is determined based on the probability distribution difference between the first probability distribution and the second probability distribution, as shown in formula (9) below.

[0149] (9)

[0150] in, This indicates the second loss information. This represents the Kulbeck-Leibler divergence.

[0151] S205. Determine the characteristic loss information based on the first loss information and the second loss information.

[0152] In the above implementation, the importance distribution of feature channels can be constrained using the first loss information, enabling the 3D object detection model to learn the geometric structure pattern of the 3D training reference model, thus achieving cross-modal structural alignment of features. Simultaneously, the first modality features and the second modality features are aligned with temporal consistency using the second loss information. At this point, the performance of the 3D object detection model in learning the feature information of the 3D training reference model can be improved based on the feature loss information determined by multiple feature dimensions, thereby enhancing the detection performance of the trained 3D object detection model.

[0153] S206. Obtain the first detection result output by the three-dimensional training reference model and the second detection result output by the three-dimensional object detection model.

[0154] S207. Determine the quality score of the first test result.

[0155] Optionally, the first detection result includes the predicted regression result and the predicted classification result.

[0156] At this point, since the first and second detection results may yield a large number of candidate targets, some of which are meaningless, the 3D target detection model can selectively learn the prediction results of the 3D training reference model. Therefore, in this embodiment, a quality score is introduced and assigned to the first detection result of the 3D training reference model, thereby achieving selective distillation.

[0157] Based on this, when determining the quality score of the first detection result, we can first determine the crossover ratio (CRO) between the predicted regression result and the corresponding true regression result; then, based on the CRO and the predicted classification result, we can determine the quality score.

[0158] For example, the process of determining the quality score can be seen in the following formula (10).

[0159] (10)

[0160] in, Indicates quality score; This represents the regression result of the prediction of the i-th detection candidate box in the first detection result; This represents the true regression result of the i-th detection candidate box; This represents the classification result of the i-th candidate detection box in the first detection result; This indicates the calculation of the intersection-union ratio; This represents the preset hyperparameters.

[0161] S208. Determine the response loss information based on the quality score, the first test result, and the second test result.

[0162] For example, the process of determining response loss information can be seen in the following formula (11).

[0163] (11)

[0164] in, Indicates response loss information; This function returns the index of the first detection result that matches the i-th second detection result. This represents the regression result of the prediction of the i-th detection candidate box in the second detection result; This represents the classification result of the i-th candidate detection box in the second detection result.

[0165] In one possible implementation, in addition to the feature loss information and response loss information mentioned above, the embodiments of this application may also combine the task loss information of the three-dimensional target detection model to train the three-dimensional target detection model, as described below.

[0166] S209. Determine the matching cost of the second detection result.

[0167] In one example, the matching cost of the second detection result can be determined using the Hungarian algorithm.

[0168] S210. Based on the matching cost, feature loss information, and response loss information, train the 3D target detection model to obtain the trained 3D target detection model.

[0169] At this point, let's assume the matching cost is denoted as... Therefore, the total loss information used for training the 3D object detection model can be determined. As shown in the following formula (12).

[0170] (12)

[0171] As described above, the embodiments of this application can achieve multi-level collaborative optimization of the three-dimensional target detection model through feature loss information under the global feature dimension, feature loss information under the time feature dimension, response loss information, and task loss information, thereby improving the training accuracy of the three-dimensional target detection model, reducing the dependence on high-cost sensors, and thus meeting the needs of large-scale deployment.

[0172] Figure 3 This is a schematic diagram of the training process of a three-dimensional target detection model provided in an embodiment of this application, as shown below. Figure 3 As shown, assuming that the first modality training data is radar point cloud data and the second modality training data is multi-view image data, the radar point cloud data is input into the three-dimensional training reference model, and the multi-view image data is input into the three-dimensional target detection model, so as to train the three-dimensional target detection model according to the three-dimensional training reference model.

[0173] Specifically, we can first determine the radar BEV features corresponding to the radar point cloud data based on the 3D training reference model, and then determine the camera BEV features corresponding to the multi-view image data based on the 3D target detection model. Next, based on the radar and camera BEV features, we determine the first loss information in the global feature dimension and the second loss information in the temporal feature dimension. Based on these first and second loss information, we can then determine the feature loss information. At this point, feature-level distillation can be performed based on the feature loss information.

[0174] Next, response loss information can be constructed based on the radar point cloud detection results output by the 3D training reference model, the corresponding quality score, and the camera image detection results output by the 3D target detection model. At this point, response-level distillation can be achieved based on the response loss information.

[0175] At this point, by combining feature loss information and response loss information, the 3D target detection model can be trained to obtain the trained 3D target detection model.

[0176] Figure 4 This is a flowchart illustrating a three-dimensional target detection method provided in an embodiment of this application, as shown below. Figure 4 As shown, the method includes:

[0177] S401. Obtain target modal data including the object to be detected.

[0178] In one example, target modal data can be understood as the multi-view image data mentioned above.

[0179] S402. Input the target modal data into the trained 3D target detection model to obtain the 3D target detection result; wherein, the trained 3D target detection model is determined according to the training method of the 3D target detection model in any of the above items.

[0180] Figure 5 This is a schematic diagram of the structure of a training device for a three-dimensional target detection model provided in an embodiment of this application, as shown below. Figure 5 As shown, the training device 50 for the three-dimensional target detection model provided in this embodiment includes:

[0181] The first acquisition unit 501 is used to acquire the first modality training data and the second modality training data of the object to be detected, input the first modality training data into the three-dimensional training reference model, and input the second modality training data into the three-dimensional target detection model.

[0182] The first determining unit 502 is used to construct a similarity matrix in at least one feature dimension based on the first modal features of the first modal training data determined by the three-dimensional training reference model and the second modal features of the second modal training data determined by the three-dimensional target detection model, and to determine feature loss information based on the similarity matrix in at least one feature dimension.

[0183] The second determining unit 503 is used to acquire the first detection result output by the three-dimensional training reference model and the second detection result output by the three-dimensional target detection model, and after determining the quality score of the first detection result, determine the response loss information based on the quality score, the first detection result and the second detection result.

[0184] Training unit 504 is used to train the 3D target detection model based on feature loss information and response loss information to obtain the trained 3D target detection model.

[0185] In one possible implementation, at least one feature dimension includes a global feature dimension and a temporal feature dimension; in this case, the first determining unit 502 is configured to:

[0186] The first loss information is determined based on the global similarity matrix under the global feature dimension; wherein, the global similarity matrix is ​​determined based on the similarity between the first modality feature and the second modality feature;

[0187] The second loss information is determined based on the inter-modal temporal similarity mapping matrix and the intra-modal temporal similarity mapping matrix under the temporal feature dimension; wherein, the inter-modal temporal similarity mapping matrix is ​​determined based on the similarity between the historical frame features of the first modality and the current frame features of the second modality; the intra-modal temporal similarity mapping matrix is ​​determined based on the similarity between the historical frame features of the first modality and the current frame features of the first modality.

[0188] Based on the first loss information and the second loss information, the characteristic loss information is determined.

[0189] In one possible implementation, the device is also used for:

[0190] Determine the first feature mean of the first modality feature, and determine the second feature mean of the second modality feature;

[0191] The first modal feature is mean-centered based on the mean of the first feature to obtain the processed first modal feature; and the second modal feature is mean-centered based on the mean of the second feature to obtain the processed second modal feature.

[0192] The global similarity matrix is ​​determined based on the similarity between the processed first modality features and the processed second modality features.

[0193] In one possible implementation, the first determining unit 502 is configured to:

[0194] Determine the diagonal matrix of the global similarity matrix;

[0195] The first sub-loss is determined by the square of the difference between the identity matrix and the diagonal matrix;

[0196] The second sub-loss is determined based on the square of the diagonal matrix and the preset adjustment parameters;

[0197] The first loss information is determined based on the first sub-loss and the second sub-loss.

[0198] In one possible implementation, the first determining unit 502 is configured to:

[0199] Determine the first probability distribution of the intermodal temporal similarity mapping matrix and the second probability distribution of the intramodal temporal similarity mapping matrix;

[0200] The second loss information is determined based on the probability distribution difference between the first probability distribution and the second probability distribution.

[0201] In one possible implementation, the first detection result includes the predicted regression result and the predicted classification result; in this case, the second determining unit 503 is used to:

[0202] Determine the crossover ratio (CRO) between the predicted regression results and the corresponding actual regression results;

[0203] The quality score is determined based on the crossover ratio of the regression results and the predicted classification results.

[0204] In one possible implementation, the device is also used for:

[0205] Determine the matching cost of the second detection result;

[0206] The 3D target detection model is trained based on the matching cost, feature loss information, and response loss information to obtain the trained 3D target detection model.

[0207] The training device for the three-dimensional target detection model provided in this embodiment can execute the training method for the three-dimensional target detection model provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0208] Figure 6 This is a schematic diagram of the structure of a three-dimensional target detection device provided in an embodiment of this application, as shown below. Figure 6 As shown, the three-dimensional target detection device 60 provided in this embodiment includes:

[0209] The second acquisition unit 601 is used to acquire target modal data including the object to be detected;

[0210] The detection unit 602 is used to input target modal data into the trained three-dimensional target detection model to obtain the three-dimensional target detection result; wherein, the trained three-dimensional target detection model is determined according to the training method of any of the above three-dimensional target detection models.

[0211] The three-dimensional target detection device provided in this embodiment can execute the three-dimensional target detection method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0212] Figure 7 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Figure 7 As shown, the computer device 70 provided in this embodiment includes at least one processor 701 and a memory 702. Optionally, the computer device 70 further includes a communication component 703. The processor 701, memory 702, and communication component 703 are connected via a bus 704.

[0213] In a specific implementation, at least one processor 701 executes computer execution instructions stored in memory 702, causing at least one processor 701 to perform the above-described method.

[0214] The specific implementation process of processor 701 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0215] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0216] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0217] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0218] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0219] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0220] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0221] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0222] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0223] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0224] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0225] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0226] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0227] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A training method for a three-dimensional target detection model, characterized in that, include: Acquire the first modality training data and the second modality training data of the object to be detected, and input the first modality training data into the three-dimensional training reference model, and input the second modality training data into the three-dimensional target detection model; Based on the first modal features of the first modal training data determined by the three-dimensional training reference model and the second modal features of the second modal training data determined by the three-dimensional object detection model, a similarity matrix under at least one feature dimension is constructed, and feature loss information is determined based on the similarity matrix under the at least one feature dimension. Obtain the first detection result output by the training reference model of the three dimensions and the second detection result output by the target detection model of the three dimensions. After determining the quality score of the first detection result, determine the response loss information based on the quality score, the first detection result and the second detection result. The three-dimensional target detection model is trained based on the feature loss information and the response loss information to obtain the trained three-dimensional target detection model.

2. The method according to claim 1, characterized in that, The at least one feature dimension includes a global feature dimension and a temporal feature dimension; Based on the similarity matrix under at least one feature dimension, feature loss information is determined, including: The first loss information is determined based on the global similarity matrix under the global feature dimension; wherein the global similarity matrix is ​​determined based on the similarity between the first modal feature and the second modal feature; The second loss information is determined based on the inter-modal temporal similarity mapping matrix and the intra-modal temporal similarity mapping matrix under the temporal feature dimension; wherein, the inter-modal temporal similarity mapping matrix is ​​determined based on the similarity between the historical frame features of the first modality feature and the current frame features of the second modality feature; the intra-modal temporal similarity mapping matrix is ​​determined based on the similarity between the historical frame features of the first modality feature and the current frame features of the first modality feature; The feature loss information is determined based on the first loss information and the second loss information.

3. The method according to claim 2, characterized in that, The method further includes: Determine the first feature mean of the first modal feature, and determine the second feature mean of the second modal feature; The first modal feature is mean-centered based on the mean of the first feature to obtain the processed first modal feature; and the second modal feature is mean-centered based on the mean of the second feature to obtain the processed second modal feature. The global similarity matrix is ​​determined based on the similarity between the processed first modal features and the processed second modal features.

4. The method according to claim 2, characterized in that, Based on the global similarity matrix under the global feature dimension, the first loss information is determined, including: Determine the diagonal matrix of the global similarity matrix; The first sub-loss is determined by the square of the difference between the identity matrix and the diagonal matrix. The second sub-loss is determined based on the square of the diagonal matrix and a preset adjustment parameter; The first loss information is determined based on the first sub-loss and the second sub-loss.

5. The method according to claim 2, characterized in that, Based on the inter-modal temporal similarity mapping matrix and the intra-modal temporal similarity mapping matrix under the aforementioned temporal feature dimension, the second loss information is determined, including: Determine the first probability distribution of the intermodal temporal similarity mapping matrix, and determine the second probability distribution of the intramodal temporal similarity mapping matrix; The second loss information is determined based on the probability distribution difference between the first probability distribution and the second probability distribution.

6. The method according to claim 1, characterized in that, The first detection result includes the predicted regression result and the predicted classification result; Determining the quality score of the first test result includes: Determine the intersection-union ratio (IUU) of the predicted regression results with the corresponding actual regression results; The quality score is determined based on the crossover ratio of the regression results and the predicted classification results.

7. The method according to any one of claims 1-6, characterized in that, The method further includes: Determine the matching cost of the second detection result; The 3D target detection model is trained based on the matching cost, the feature loss information, and the response loss information to obtain the trained 3D target detection model.

8. A three-dimensional target detection method, characterized in that, include: Acquire target modal data including the object to be detected; The target modal data is input into the trained three-dimensional target detection model to obtain the three-dimensional target detection result; wherein the trained three-dimensional target detection model is determined by the training method of the three-dimensional target detection model according to any one of claims 1-7.

9. A computer device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform a training method for a three-dimensional target detection model as described in any one of claims 1-7, or to perform a three-dimensional target detection method as described in claim 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the training method of the three-dimensional target detection model as described in any one of claims 1-7, or to execute the three-dimensional target detection method as described in claim 8.

Images