Target detection model optimization method and device, electronic equipment and storage medium

Abstract

This invention provides a method, apparatus, electronic device, and storage medium for optimizing a target detection model. The method relates to the field of artificial intelligence technology and includes: determining an initial target segmentation result based on an image dataset; jointly optimizing a clustering network model and a localization network model based on the initial target segmentation result and an alternating training strategy to obtain a first clustering network model and a first localization network model; performing temporal optimization on a two-dimensional candidate target set and a three-dimensional candidate target set respectively to obtain a first candidate target set and a second candidate target set; the two-dimensional candidate target set and the three-dimensional candidate target set are obtained during the joint optimization process of the clustering network model and the localization network model, respectively; and optimizing the first clustering network model and the first localization network model based on the first candidate target set and the second candidate target set to obtain a target detection model, thereby achieving target detection in different application scenarios and improving the accuracy of target detection.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus, electronic device, and storage medium for optimizing a target detection model. Background Technology

[0002] Object detection has wide applications in autonomous driving, industrial inspection, and security monitoring. Existing 2D and 3D object detection methods rely on manually labeled bounding boxes. These manually labeled boxes are then used to train deep learning networks, which are then used to perform object detection. However, this method typically requires significant manpower and time for labeling, and when extended to new scenarios, additional labeled data is needed to fine-tune the model parameters. Therefore, unsupervised object detection methods are of great significance and have broad application prospects. Unsupervised object detection methods aim to extract visual cues from raw data to supervise model training, thereby reducing reliance on manual labeling.

[0003] In related technologies, unsupervised target detection methods only use two-dimensional images or only three-dimensional point clouds to detect targets. However, the target detection results obtained by these technologies are significantly different from those using manually labeled supervised data. In a single modality, the raw data of a single frame lacks constraint information, making it difficult to extract effective clues and thus failing to detect the target. Summary of the Invention

[0004] This invention provides a target detection model optimization method, apparatus, electronic device, and storage medium to address the defect that existing unsupervised target detection methods cannot detect targets, and to achieve accurate detection of targets in different application scenarios without labeled data, i.e., without supervision.

[0005] This invention provides a method for optimizing a target detection model, comprising:

[0006] Based on the image dataset, an initial target segmentation result is determined; the image dataset includes a two-dimensional image sequence and a three-dimensional point cloud sequence; there is a correspondence between the two-dimensional image sequence and the three-dimensional point cloud sequence.

[0007] Based on the initial target segmentation results and the alternating training strategy, the clustering network model and the localization network model are jointly optimized to obtain a first clustering network model and a first localization network model; the clustering network model is used to detect targets in the images of the 3D point cloud sequence; the localization network model is used to detect targets in the images of the 2D image sequence.

[0008] Temporal optimization is performed on the two-dimensional candidate target set and the three-dimensional candidate target set respectively to obtain the first candidate target set and the second candidate target set; the two-dimensional candidate target set and the three-dimensional candidate target set are obtained by the clustering network model and the localization network model respectively during the joint optimization process;

[0009] Based on the first candidate target set and the second candidate target set, the first clustering network model and the first localization network model are optimized respectively to obtain the target detection model.

[0010] According to the object detection model optimization method provided by the present invention, the method involves jointly optimizing a clustering network model and a localization network model based on the initial object segmentation result and an alternating training strategy to obtain a first clustering network model and a first localization network model, comprising:

[0011] The point cloud data in the three-dimensional point cloud sequence is input into the clustering network model to obtain the first segmentation result of the point cloud data output by the clustering network model; the clustering network model is trained based on the sample point cloud data and the initial target segmentation result.

[0012] The images in the two-dimensional image sequence are input into the localization network model to obtain the second segmentation result output by the localization network; the localization network model is trained based on the sample images and the first segmentation result.

[0013] Based on the first segmentation result, the second segmentation result, and the alternating training strategy, the clustering network model and the localization network model are jointly optimized to obtain the first clustering network model and the first localization network model.

[0014] According to the target detection model optimization method provided by the present invention, the step of inputting point cloud data from the three-dimensional point cloud sequence into the clustering network model to obtain a first segmentation result of the point cloud data output by the clustering network model includes:

[0015] The point cloud data in the three-dimensional point cloud sequence is input into the feature extraction module in the clustering network model to obtain the voxel feature data output by the feature extraction module.

[0016] The voxel feature data is input into the voting module in the clustering network model to obtain the category and deviation value corresponding to the point cloud data output by the voting module;

[0017] The category and the deviation value are input into the clustering module in the clustering network model to obtain the first segmentation result output by the clustering module.

[0018] According to the target detection model optimization method provided by the present invention, the loss function adopted by the clustering network model is expressed by formula (1), wherein:

[0019]

[0020] Where M represents the total number of foreground points, and i represents the i-th point. This represents the monitoring signal corresponding to the i-th point. Δx represents the true deviation at the i-th point. i This represents the prediction bias at the i-th point.

[0021] According to the object detection model optimization method provided by the present invention, the method involves jointly optimizing the clustering network model and the localization network model based on the first segmentation result, the second segmentation result, and the alternating training strategy to obtain the first clustering network model and the first localization network model, comprising:

[0022] Based on the first segmentation result, the first supervision information of the localization network model is determined;

[0023] Based on the first supervision information, the localization network model is trained to obtain the first localization network model.

[0024] According to the object detection model optimization method provided by the present invention, the method involves jointly optimizing the clustering network model and the localization network model based on the first segmentation result, the second segmentation result, and the alternating training strategy to obtain the first clustering network model and the first localization network model, comprising:

[0025] Based on the second segmentation result, the second supervision information of the clustering network model is determined;

[0026] Based on the second supervision information, the clustering network model is trained to obtain the first clustering network model.

[0027] According to the target detection model optimization method provided by the present invention, temporal optimization is performed on a two-dimensional candidate target set and a three-dimensional candidate target set respectively to obtain a first candidate target set and a second candidate target set, including:

[0028] Kalman filtering is used to perform time-series tracking on the two-dimensional candidate target set and the three-dimensional candidate target set, respectively.

[0029] Based on the tracking results, the first candidate target set and the second candidate target set are determined.

[0030] According to a target detection model optimization method provided by the present invention, determining the first candidate target set and the second candidate target set based on the tracking results includes:

[0031] Candidate targets whose number of consecutively matched image frames is less than a first preset threshold are judged as incorrect segmentation results;

[0032] The erroneous segmentation results are filtered out from the two-dimensional candidate target set to obtain the first candidate target set.

[0033] According to a target detection model optimization method provided by the present invention, determining the first candidate target set and the second candidate target set based on the tracking results includes:

[0034] Candidate targets whose point cloud data in a continuously matched 3D point cloud sequence is less than a second preset threshold are judged as incorrect segmentation results;

[0035] The erroneous segmentation results are filtered out from the three-dimensional candidate target set;

[0036] If there are any missing segmentation results, the segmentation results are added to the three-dimensional candidate target set to obtain the second candidate target set.

[0037] The present invention also provides a target detection model optimization device, comprising:

[0038] The determination module is used to determine the initial target segmentation result based on the image dataset; the image dataset includes a two-dimensional image sequence and a three-dimensional point cloud sequence; there is a correspondence between the two-dimensional image sequence and the three-dimensional point cloud sequence;

[0039] A joint optimization module is used to jointly optimize a clustering network model and a localization network model based on the initial target segmentation results and an alternating training strategy, to obtain a first clustering network model and a first localization network model; the clustering network model is used to detect targets in the images of the 3D point cloud sequence; the localization network model is used to detect targets in the images of the 2D image sequence.

[0040] The temporal optimization module is used to perform temporal optimization on the two-dimensional candidate target set and the three-dimensional candidate target set respectively to obtain the first candidate target set and the second candidate target set; the two-dimensional candidate target set and the three-dimensional candidate target set are obtained by the clustering network model and the localization network model respectively in the process of joint optimization.

[0041] The target optimization module is used to optimize the first clustering network model and the first localization network model based on the first candidate target set and the second candidate target set, respectively, to obtain the target detection model.

[0042] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the target detection model optimization method as described above.

[0043] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the target detection model optimization method as described above.

[0044] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the target detection model optimization method as described above.

[0045] The present invention provides a target detection model optimization method, apparatus, electronic device, and storage medium. It determines an initial target segmentation result using an image dataset. The image dataset includes a two-dimensional image sequence and a three-dimensional point cloud sequence. A correspondence exists between the two-dimensional image sequence and the three-dimensional point cloud sequence. Based on the initial target segmentation result and an alternating training strategy, a clustering network model and a localization network model are jointly optimized to obtain a first clustering network model and a first localization network model. The clustering network model is used to detect targets in the images of the three-dimensional point cloud sequence. The localization network model is used to detect targets in the images of the two-dimensional image sequence. Temporal optimization is performed on the two-dimensional candidate target set and the three-dimensional candidate target set to obtain a first candidate target set and a second candidate target set. The two-dimensional candidate target set and the three-dimensional candidate target set are obtained during the joint optimization process of the clustering network model and the localization network model, respectively. Based on the first candidate target set and the second candidate target set, the first clustering network model and the first localization network model are optimized to obtain a target detection model. This target detection model can accurately detect targets in the images corresponding to the two-dimensional image sequence and the three-dimensional point cloud sequence, achieving accurate target detection in different application scenarios without labeled data, i.e., without supervision, thus improving the accuracy and efficiency of target detection. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0047] Figure 1 This is one of the flowcharts illustrating the target detection model optimization method provided by the present invention;

[0048] Figure 2This is a schematic diagram of the clustering network model provided by the present invention;

[0049] Figure 3 This is the second flowchart illustrating the target detection model optimization method provided by the present invention;

[0050] Figure 4 This is the third flowchart illustrating the target detection model optimization method provided by this invention;

[0051] Figure 5 This is a schematic diagram of the target detection model optimization device provided by the present invention;

[0052] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0054] The target detection model optimization method provided by the present invention will be described in detail below with reference to the accompanying drawings, through some embodiments and application scenarios.

[0055] This invention provides a target detection model optimization method applicable to target detection scenarios. The method involves determining an initial target segmentation result based on an image dataset. The image dataset includes a two-dimensional image sequence and a three-dimensional point cloud sequence, with a correspondence between them. Based on the initial target segmentation result and an alternating training strategy, a clustering network model and a localization network model are jointly optimized to obtain a first clustering network model and a first localization network model. The clustering network model is used to detect targets in the images of the three-dimensional point cloud sequence, and the localization network model is used to detect targets in the images of the two-dimensional image sequence. Temporal optimization is performed on the two-dimensional candidate target set and the three-dimensional candidate target set to obtain a first candidate target set and a second candidate target set. The two-dimensional candidate target set and the three-dimensional candidate target set are obtained during the joint optimization process of the clustering network model and the localization network model, respectively. Based on the first candidate target set and the second candidate target set, the first clustering network model and the first localization network model are optimized to obtain the target detection model. The method provided by this invention can accurately detect targets in images corresponding to two-dimensional image sequences and three-dimensional point cloud sequences using a target detection model. It can accurately detect targets in different application scenarios without labeled data, i.e., without supervision, thus improving the accuracy and efficiency of target detection.

[0056] The following is combined Figures 1-4 The target detection model optimization method of the present invention is described.

[0057] Figure 1 This is one of the flowcharts illustrating the target detection model optimization method provided by the present invention, such as... Figure 1 As shown, the method includes steps 101-104, wherein:

[0058] Step 101: Determine the initial target segmentation result based on the image dataset; the image dataset includes a two-dimensional image sequence and a three-dimensional point cloud sequence; there is a correspondence between the two-dimensional image sequence and the three-dimensional point cloud sequence.

[0059] It should be noted that the target detection model optimization method provided by this invention is applicable to target detection scenarios, such as autonomous driving scenarios. The executing entity of this method can be a target detection model optimization device, such as an electronic device, or a control module in the target detection model optimization device for executing the target detection model optimization method.

[0060] Specifically, an image dataset for unsupervised object detection is pre-constructed. This image dataset includes two-dimensional image sequences and three-dimensional point cloud sequences. The images in the two-dimensional image sequences are RGB images, and the point cloud sequences in the three-dimensional point cloud sequences are calibrated three-dimensional point cloud sequences. There is a correspondence between the two-dimensional image sequences and the three-dimensional point cloud sequences; for example, the two-dimensional image sequence is represented as {I1,…,I...}. t The three-dimensional point cloud sequence is represented as {P1,…,P}. t}, where t represents time, I1 represents the image acquired at time 1, I t Let Pt represent the image acquired at time t, and P1 represent the point cloud data acquired at time 1. t This represents the point cloud data collected at time t.

[0061] Based on the constructed image dataset, initial segmentation results can be obtained from the 2D image sequence and the 3D point cloud sequence. The specific implementation process for obtaining the initial segmentation results is described below.

[0062] First, ground points are removed from the 3D point cloud sequence, and point cloud data P for each frame is estimated from the 3D point cloud sequence. j 3D scene flow S j The motion cues can be estimated using optical flow networks or by using accurate velocity information obtained from special sensors, without any restrictions.

[0063] Secondly, based on the 3D scene flow, each frame of point cloud data in the application scenario is clustered to obtain multiple cluster instances corresponding to the point cloud data; among them, the 3D position information (x, y, z) and 3D scene flow information (v) of the point cloud data are used during clustering. x v y v z The data includes the location information (u, v) of the image in the two-dimensional image sequence corresponding to each frame of point cloud data. It should be noted that the clustering method used for each frame of point cloud data in the application scenario can be a hierarchical density clustering algorithm or other methods, without any restrictions.

[0064] Next, motion cues are used to filter foreground target instances from multiple clustered instances. Here, it is assumed that the moving target in the application scenario belongs to the foreground target, specifically using the following formula (2), where:

[0065]

[0066] Where, ξ j Denotes each cluster instance, |ξ j | represents the number of points in the point cloud data contained in each cluster instance, ||S t (pi )||2 represents point p i The corresponding velocity scalar values ​​are δ, which represents the velocity threshold, and η, which represents the motion point ratio threshold. For example, δ = 0.05 and η = 0.8.

[0067] It should be noted that the above formula (2) is used to determine whether point pi in each frame of point cloud data meets the motion condition (i.e., when the proportion of moving points segmented by the candidate cluster instance is greater than η, all points contained in the candidate cluster instance are considered moving points). Points that meet the motion condition are considered foreground target points, and points that do not meet the motion condition are considered background points. Represented as the foreground target point, Then it becomes the background point.

[0068] Step 102: Based on the initial target segmentation results and the alternating training strategy, jointly optimize the clustering network model and the localization network model to obtain the first clustering network model and the first localization network model; the clustering network model is used to detect targets in the images of the three-dimensional point cloud sequence; the localization network model is used to detect targets in the images of the two-dimensional image sequence.

[0069] It should be noted that the clustering network model is a three-dimensional clustering network model, used to detect targets in images in a three-dimensional point cloud sequence, that is, the clustering network model can detect targets in three-dimensional space; the localization network model is a two-dimensional localization network model, used to detect targets in images in a two-dimensional image sequence.

[0070] Specifically, based on the determined initial target segmentation results and the alternating training strategy, the clustering network model and the localization network model can be jointly optimized. After joint optimization, the first clustering network model and the first localization network model can be obtained.

[0071] Step 103: Perform temporal optimization on the two-dimensional candidate target set and the three-dimensional candidate target set respectively to obtain the first candidate target set and the second candidate target set; the two-dimensional candidate target set and the three-dimensional candidate target set are obtained by the clustering network model and the localization network model respectively in the process of joint optimization.

[0072] It should be noted that the two-dimensional candidate target set and the three-dimensional candidate target set are obtained by the clustering network model and the localization network model respectively during the joint optimization process. The two-dimensional candidate target set includes multiple two-dimensional candidate target segmentation results corresponding to the two-dimensional image sequence, and the three-dimensional candidate target set includes multiple three-dimensional candidate target segmentation results corresponding to the three-dimensional point cloud sequence.

[0073] Specifically, by performing temporal optimization on the segmentation results of multiple two-dimensional candidate targets in the two-dimensional candidate target set and the segmentation results of multiple three-dimensional candidate targets in the three-dimensional candidate target set, optimized segmentation results of multiple candidate targets corresponding to the two-dimensional candidate target set and optimized segmentation results of multiple candidate targets corresponding to the three-dimensional candidate target set can be obtained, thereby obtaining the first candidate target set and the second candidate target set.

[0074] Step 104: Based on the first candidate target set and the second candidate target set, optimize the first clustering network model and the first localization network model respectively to obtain the target detection model.

[0075] Specifically, the object detection model includes a target clustering network model corresponding to the first clustering network model and a target localization network model corresponding to the first localization network model. Using the first candidate target set and the second candidate target set as target supervision information for the first clustering network model and the first localization network model, respectively, and optimizing and training the first clustering network model and the first localization network model, the target clustering network model corresponding to the first clustering network model and the target localization network model corresponding to the first localization network model can be obtained, thus yielding the object detection model.

[0076] By inputting any image into the target localization network model, the object detection result output by the target localization network model is obtained; by inputting any single frame point cloud data into the target clustering network model, the point cloud instance segmentation result output by the target clustering network model is obtained; the target localization network model and the target clustering network model can achieve target detection without labeled data, that is, without supervised information, thus obtaining target detection results in any application scenario.

[0077] The target detection model optimization method provided by this invention determines the initial target segmentation result through an image dataset. The image dataset includes a two-dimensional image sequence and a three-dimensional point cloud sequence. There is a correspondence between the two-dimensional image sequence and the three-dimensional point cloud sequence. Then, based on the initial target segmentation result and an alternating training strategy, a clustering network model and a localization network model are jointly optimized to obtain a first clustering network model and a first localization network model. The clustering network model is used to detect targets in the images of the three-dimensional point cloud sequence, that is, it can detect targets in three-dimensional space. The localization network model is used to detect targets in the images of the two-dimensional image sequence. Temporal optimization is performed on the two-dimensional candidate target set and the three-dimensional candidate target set respectively to obtain a first candidate target set and a second candidate target set. The two-dimensional candidate target set and the three-dimensional candidate target set are obtained by the clustering network model and the localization network model respectively during the joint optimization process. Based on the first candidate target set and the second candidate target set, the first clustering network model and the first localization network model are optimized respectively to obtain a target detection model. The target detection model can accurately detect targets in the images corresponding to the two-dimensional image sequence and the three-dimensional point cloud sequence, achieving accurate target detection in different application scenarios without labeled data, that is, without supervision, thus improving the accuracy and efficiency of target detection.

[0078] Optionally, step 102 above can be implemented by the following steps:

[0079] Step 1) Input the point cloud data in the three-dimensional point cloud sequence into the clustering network model to obtain the first segmentation result of the point cloud data output by the clustering network model; the clustering network model is trained based on the sample point cloud data and the initial target segmentation result.

[0080] It should be noted that a clustering network model is pre-built. During the training of this clustering network model, the initial target segmentation results are used as the initial supervision information for the clustering network model. By training this clustering network model, a trained clustering network model can be obtained. The trained clustering network model can achieve target detection in a 3D point cloud sequence without labeled data.

[0081] Specifically, by inputting the point cloud data from the 3D point cloud sequence into the clustering network model, the first segmentation result of the point cloud data output by the clustering network model can be obtained.

[0082] Optionally, the step of inputting the point cloud data in the three-dimensional point cloud sequence into the clustering network model to obtain the first segmentation result of the point cloud data output by the clustering network model includes:

[0083] The point cloud data in the 3D point cloud sequence is input into the feature extraction module in the clustering network model to obtain the voxel feature data output by the feature extraction module; the voxel feature data is input into the voting module in the clustering network model to obtain the category and deviation value corresponding to the point cloud data output by the voting module; the category and the deviation value are input into the clustering module in the clustering network model to obtain the first segmentation result output by the clustering module.

[0084] Specifically, the constructed clustering network model includes a feature extraction module, a voting module, and a clustering module.

[0085] First, the feature extraction module uses a backbone network to voxelize the point cloud data of each frame in the 3D point cloud sequence. Then, the voxelized point cloud data is input into the feature extraction module, which can extract voxel features. The feature extraction module outputs voxel feature data and then projects the voxel feature data onto each point to obtain voxel feature data with (C+3) dimensions at each point, where C represents the dimension of the latent variables of the backbone network used by the feature extraction module, and 3 represents the 3D position information of each point.

[0086] Secondly, the voxel feature data is input into the voting module, which uses a multilayer perceptron network. The voting module includes two modules: category prediction and center point regression. Category prediction is a binary classification prediction used to distinguish between foreground target points and background points. The loss function for category prediction is the focus loss function. Center point regression is used to predict the deviation value between the current point and the center of the cluster instance to which the point belongs.

[0087] Finally, based on the category and bias value predicted by the voting module, the category and bias value are input into the clustering module, which uses the connective component algorithm to obtain the final first segmentation result.

[0088] Figure 2 This is a schematic diagram of the clustering network model provided by the present invention, as shown below. Figure 2 As shown, the clustering network model includes a feature extraction module, a voting module, and a clustering module. The point cloud data in the 3D point cloud sequence is voxelized, and then the voxelized point cloud data is input into the feature extraction module. The feature extraction module uses a backbone network to extract 3D voxel feature data, which is then input into the voting module. The voting module uses a multilayer perceptron and includes two modules: category prediction and center point regression. Category prediction is a binary classification prediction used to distinguish between foreground target points and background points. Center point regression is used to predict the deviation value from the current point to the center of the cluster instance to which the point belongs. Finally, based on the category and deviation value predicted by the voting module, the category and deviation value are input into the clustering module, which outputs the final first segmentation result.

[0089] Optionally, the loss function used in the clustering network model is expressed as in formula (1), where:

[0090]

[0091] Where M represents the total number of foreground points, and i represents the i-th point. This represents the monitoring signal corresponding to the i-th point. Δx represents the true deviation at the i-th point. i This represents the prediction bias at the i-th point.

[0092] Specifically, the loss function used for centroid regression in the voting module of the clustering network model is shown in formula (1) above. During the training of the clustering network model, based on the loss value obtained by formula (1) above and the loss value obtained by the focus loss function used for category prediction, it is determined whether the clustering network model has reached the convergence condition. Before reaching the convergence condition, the parameters of the clustering network model are continuously changed until the clustering network model converges.

[0093] Step 2) Input the images in the two-dimensional image sequence into the localization network model to obtain the second segmentation result output by the localization network; the localization network model is trained based on the sample images and the first segmentation result.

[0094] It should be noted that the localization network model is trained based on the sample images and the first segmentation result output by the clustering network model in step 1) above. Utilizing the correspondence between the 2D image sequence and the 3D point cloud sequence, the first segmentation result output by the clustering network model, i.e., the point cloud instance segmentation result, is projected onto the 2D image to obtain the target detection box in the 2D image. This target detection box is used as the initial supervision information for the localization network model. The localization network model is trained using sample images from the 2D image sequence. The localization network model can employ either a Faster Region-based Convolutional Neural Network (Faster R-CNN) or a Mask Region-based Convolutional Neural Network (Mask R-CNN). The main difference is that Faster R-CNN uses the point cloud projected onto the 2D image to obtain the outer rectangular box as the initial supervision information, while Mask R-CNN uses the convex hull obtained from the point cloud projected onto the 2D image as the initial supervision information.

[0095] Specifically, after the localization network model is trained, the images in the two-dimensional image sequence are input into the localization network model to obtain the second segmentation result output by the localization network. That is, the localization network model can achieve target detection in the two-dimensional image sequence without labeling data.

[0096] Step 3) Based on the first segmentation result, the second segmentation result, and the alternating training strategy, jointly optimize the clustering network model and the localization network model to obtain the first clustering network model and the first localization network model.

[0097] Specifically, based on the first segmentation result output by the clustering network model, the second segmentation result output by the localization network model, and the alternating training strategy, the clustering network model and the localization network model can be jointly optimized, resulting in the first clustering network model and the first localization network model.

[0098] The target detection model optimization method provided by this invention involves inputting point cloud data from a 3D point cloud sequence into a clustering network model to obtain a first segmentation result of the point cloud data output by the clustering network model. The clustering network model is trained based on sample point cloud data and the initial target segmentation result. Then, images from a 2D image sequence are input into a localization network model to obtain a second segmentation result output by the localization network. The localization network model is trained based on sample images and the first segmentation result. Based on the first segmentation result, the second segmentation result, and an alternating training strategy, the clustering network model and the localization network model are jointly optimized to obtain a first clustering network model and a first localization network model. These models are then further optimized to obtain a target detection model. This method enables accurate detection of targets in images corresponding to 2D image sequences and 3D point cloud sequences without the need for labeled data. Furthermore, it can detect targets in different application scenarios, improving the accuracy and efficiency of target detection.

[0099] Optionally, the specific implementation of step 3) above includes:

[0100] Based on the first segmentation result, the first supervision information of the localization network model is determined; based on the first supervision information, the localization network model is trained to obtain the first localization network model.

[0101] It should be noted that the alternating training strategy for jointly optimizing the clustering network model and the localization network model can be used in multiple rounds of iterative optimization, such as three rounds of iterative optimization, depending on the actual needs.

[0102] The following section describes the specific process of each round of iterative optimization (two-dimensional (2D) optimization) in the joint optimization localization network model.

[0103] The parameters of the clustering network model are fixed, and the first segmentation result output by the clustering network model is projected onto a two-dimensional image as the first supervision information of the localization network model. Then, based on the first supervision information, the localization network model is trained using a sequence of two-dimensional images. During the training process, the first supervision information is used as pseudo-label information for the images in the two-dimensional image sequence. The second segmentation result output by the localization network model is then compared with the first supervision information, and the loss value of the localization network model is calculated. Based on the calculated loss value, it is determined whether the localization network model has reached the convergence condition. If the localization network model has reached the convergence condition, the training of the localization network model ends, and the first localization network model is obtained.

[0104] It should be noted that, since the localization network model is optimized through multiple rounds of iteration, the first localization network model obtained in the last round of iteration is used as the final first localization network model.

[0105] The target detection model optimization method provided by this invention determines the first supervision information of the localization network model based on the first segmentation result output by the clustering network model; based on the first supervision information, the localization network model is trained to obtain the first localization network model, thereby optimizing the localization network model. This enables the localization network model to accurately detect targets in two-dimensional image sequences of different scenes without labeled data, i.e. without supervision, thus improving the accuracy and efficiency of target detection.

[0106] Optionally, the specific implementation of step 3) above includes:

[0107] Based on the second segmentation result, the second supervision information of the clustering network model is determined; based on the second supervision information, the clustering network model is trained to obtain the first clustering network model.

[0108] Specifically, the process of each round of iterative optimization (3D optimization) in the joint optimization clustering network model will be described.

[0109] The parameters of the localization network model are fixed. The second segmentation result output by the localization network model, i.e., the two-dimensional pixels in the target detection box, is projected onto the three-dimensional point cloud space and used as the second supervision information of the clustering network model, i.e., the pseudo-label information of the clustering network model. The first segmentation result output by the clustering network model is then compared with the second supervision information to calculate the loss value of the clustering network model. The calculated loss value is used to determine whether the clustering network model has reached the convergence condition. If the clustering network model has reached the convergence condition, the training of the clustering network model ends, and the first clustering network model can be obtained.

[0110] It should be noted that, since the clustering network model is optimized through multiple rounds of iteration, the first clustering network model obtained in the last round of iteration is used as the final first clustering network model.

[0111] The target detection model optimization method provided by this invention determines the second supervision information of the clustering network model based on the second segmentation result output by the localization network model; based on the second supervision information, the clustering network model is trained to obtain the first clustering network model, thereby optimizing the clustering network model. This enables the clustering network model to accurately detect targets in 3D point cloud sequence images of different scenes without labeled data, i.e. without supervision, thus improving the accuracy and efficiency of target detection.

[0112] Optionally, step 103 above can be implemented by the following steps:

[0113] Step a) Use Kalman filtering to perform time-series tracking on the two-dimensional candidate target set and the three-dimensional candidate target set respectively;

[0114] Step b) Based on the tracking results, determine the first candidate target set and the second candidate target set.

[0115] Specifically, the consistency constraints of the time series are used to optimize the two-dimensional and three-dimensional candidate target sets in order to discover missed foreground targets and filter out erroneous foreground targets.

[0116] In practice, Kalman filtering is used to perform time-series tracking on the two-dimensional candidate target set and the three-dimensional candidate target set respectively. Based on the tracking results of the two-dimensional candidate target set and the three-dimensional candidate target set, the first candidate target set corresponding to the two-dimensional candidate target set and the second candidate target set corresponding to the three-dimensional candidate target set can be determined.

[0117] The target detection model optimization method provided by this invention uses Kalman filtering to perform temporal tracking on two-dimensional and three-dimensional candidate target sets respectively; then, based on the tracking results, a first candidate target set and a second candidate target set are determined, thereby optimizing the two-dimensional and three-dimensional candidate target sets. This allows for the optimization of the first clustering network model and the first localization network model, thus enabling accurate detection of targets in two-dimensional image sequences and three-dimensional point cloud sequences in different application scenarios without labeled data, i.e., without supervision, improving the accuracy and efficiency of target detection.

[0118] Optionally, the specific implementation of step b) above includes:

[0119] Candidate targets with fewer than a first preset threshold number of consecutively matched image frames are identified as incorrect segmentation results; the incorrect segmentation results are filtered out from the two-dimensional candidate target set to obtain the first candidate target set.

[0120] Specifically, candidate targets with fewer than a first preset threshold number of consecutively matched image frames are determined to be incorrect segmentation results; for example, if candidate targets with fewer than 5 consecutively matched image frames are determined to be incorrect segmentation results, then the incorrect segmentation results are filtered out from the two-dimensional candidate target set to obtain the first candidate target set.

[0121] The target detection model optimization method provided by this invention determines candidate targets with fewer than a first preset threshold number of consecutively matched image frames as erroneous segmentation results; filters out erroneous segmentation results from the two-dimensional candidate target set to obtain the first candidate target set, thereby optimizing the two-dimensional candidate target set. This optimizes the first localization network model, enabling accurate detection of targets in two-dimensional image sequences of different application scenarios without labeled data, i.e., without supervision, thus improving the accuracy and efficiency of target detection.

[0122] Optionally, the specific implementation of step b) above includes:

[0123] Candidate targets whose point cloud data in a continuously matched 3D point cloud sequence is less than a second preset threshold are identified as erroneous segmentation results; the erroneous segmentation results are filtered out from the 3D candidate target set; in the case of missing segmentation results, the segmentation results are added to the 3D candidate target set to obtain the second candidate target set.

[0124] Specifically, candidate targets whose point cloud data in a continuously matched 3D point cloud sequence is less than a second preset threshold are judged as incorrect segmentation results. For example, if candidate targets whose point cloud data in a continuously matched 3D point cloud sequence is less than 5 frames are judged as incorrect segmentation results, then the incorrect segmentation results are filtered out from the 3D candidate target set. In the case of missing segmentation results, the segmentation results are added to the 3D candidate target set, that is, bidirectional bounding is performed in time according to the estimation of Kalman filtering, and the missing foreground targets are added to the 3D candidate target set to obtain the second candidate target set.

[0125] The target detection model optimization method provided by this invention determines candidate targets whose point cloud data in a continuously matched 3D point cloud sequence is less than a second preset threshold as erroneous segmentation results; filters out erroneous segmentation results from the 3D candidate target set; and, in the case of missing segmentation results, supplements the segmentation results into the 3D candidate target set to obtain a second candidate target set. This optimizes the 3D candidate target set, which in turn optimizes the first clustering network model. As a result, it can accurately detect targets in images of 3D point cloud sequences in different application scenarios without labeled data, i.e., without supervision, thus improving the accuracy and efficiency of target detection.

[0126] Figure 3 This is the second flowchart illustrating the target detection model optimization method provided by this invention, as shown below. Figure 3 As shown, there is a correspondence between the two-dimensional image sequence and the three-dimensional point cloud sequence, i.e., 2D to 3D cues. Based on the motion cues in the three-dimensional point cloud sequence, the initial target segmentation result can be obtained, distinguishing between the moving foreground and the stationary background in the point cloud data. This initial target segmentation result is then used as the initial supervision information for the clustering network model, and the point cloud data from the three-dimensional point cloud sequence is input into the clustering network model. Clustering network model Training yields the first segmentation output by the clustering network model, representing the image object detection result. This first segmentation result is then projected onto a two-dimensional image, serving as initial supervision information for the localization network model and as input to the localization network model from the sequence of two-dimensional images. For the localization network model By training, a localization network model can be obtained. The output of the second segmentation result is the segmentation result of the point cloud instance. Then, the object instances in the target detection box of the second segmentation result are back-projected into the three-dimensional space as the first supervision information of the clustering network model. The clustering network model is further optimized to obtain the first clustering network model. Then, the first segmentation result output by the clustering network model is projected into the two-dimensional image space as the second supervision information of the localization network model. The localization network model is further optimized to obtain the first localization network model. It should be noted that the same optimization process needs to be performed three times in the process of further optimizing the localization network model and the clustering network model.

[0127] During the optimization of the clustering network model and the localization network model, a two-dimensional candidate target set composed of the first segmentation results output by the clustering network model and a three-dimensional candidate target set composed of the second segmentation results output by the localization network model are obtained. The two-dimensional candidate target set includes multiple two-dimensional candidate target segmentation results, and the three-dimensional candidate target set includes multiple three-dimensional candidate target segmentation results. Using temporal cues and temporal consistency constraints, temporal optimization is performed on the two-dimensional and three-dimensional candidate target sets to obtain the first candidate target set corresponding to the two-dimensional candidate target set and the second candidate target set corresponding to the three-dimensional candidate target set. Finally, the first clustering network model and the first localization network model are trained and optimized again using the first and second candidate target sets to obtain the target clustering network model and the target localization network model.

[0128] The target detection model optimization method provided by this invention combines 3D point cloud sequences and 2D image sequences to optimize clustering network models and localization network models, achieving unsupervised target detection. It utilizes 2D-3D cues (correspondence between 3D point cloud sequences and 2D image sequences), motion cues (motion between each frame of point cloud data in the 3D point cloud sequence), and temporal cues (temporal relationships between consecutive image frames and consecutive point cloud data frames) between the two modalities of 3D point cloud sequences and 2D image sequences to jointly optimize the localization network and clustering network, resulting in a target localization network and a target clustering network. This allows the target localization network and target clustering network to accurately detect targets in 2D image sequences and 3D point cloud sequences in any application scenario without labeled data, i.e., without supervision, thus improving the accuracy and efficiency of target detection.

[0129] Figure 4 This is the third flowchart of the target detection model optimization method provided by the present invention, as shown in Figure 4. The method includes steps 401-408; wherein,

[0130] Step 401, Construct an image dataset. Construct an image dataset for unsupervised object detection, which includes a two-dimensional image sequence and a three-dimensional point cloud sequence; there is a correspondence between the two-dimensional image sequence and the three-dimensional point cloud sequence.

[0131] Step 402: Determine the initial target segmentation result. Perform 3D scene flow estimation on the 3D point cloud sequence, i.e., estimate the motion cues of the 3D point cloud sequence. Then, perform density clustering on each frame of point cloud data corresponding to the 3D scene flow estimation to obtain multiple cluster instances corresponding to the point cloud data. Select foreground target instances from multiple cluster instances to obtain the initial target segmentation result.

[0132] Step 403: Train the clustering network model. Using the initial target segmentation results as initial supervision information for the clustering network model, the model is trained using point cloud data from the 3D point cloud sequence, yielding the first segmentation result output by the clustering network model.

[0133] Step 404: Train the localization network model. Project the first segmentation result onto the two-dimensional image to obtain a two-dimensional object detection box on the two-dimensional image. Use this two-dimensional object detection box as the initial supervision information for the localization network model. Train the localization network model using images from the two-dimensional image sequence to obtain the second segmentation result output by the localization network model.

[0134] Step 405: Using an alternating training strategy, the clustering network model and the localization network model are jointly optimized to obtain the first clustering network model and the first localization network model.

[0135] Step 406 involves performing temporal optimization on the two-dimensional and three-dimensional candidate target sets respectively, resulting in the first and second candidate target sets. The two-dimensional and three-dimensional candidate target sets are obtained through the joint optimization process of the clustering network model and the localization network model, respectively.

[0136] Step 407: Based on the first candidate target set and the second candidate target set, optimize the first clustering network model and the first localization network model respectively to obtain the target detection model, wherein the target detection model includes the target clustering network model and the target localization network model.

[0137] Step 408: Based on the target clustering network model and the target localization network model, input any two-dimensional image into the target localization network model to obtain the image target detection result output by the target localization network model; input any single frame point cloud data into the target clustering network model to obtain the point cloud instance segmentation result output by the target clustering network model, thereby obtaining the target detection result.

[0138] The target detection model optimization method provided by this invention only relies on 3D point cloud sequences and 2D image sequences during the training phase of the clustering network model and the localization network. After the clustering network model and the localization network are trained and optimized, i.e., after obtaining the target clustering network model and the target localization network, target detection can be performed on any 2D image and single-frame point cloud data. This method requires no manual annotation, can detect different targets in complex application scenarios, and has strong robustness to scene complexity. Without relying on manual annotation, it can simultaneously solve the tasks of 2D image detection and 3D point cloud sequence instance segmentation. Furthermore, compared with the detection performance in application scenarios with manual annotation, it has broad application prospects.

[0139] The target detection model optimization device provided by the present invention is described below. The target detection model optimization device described below and the target detection model optimization method described above can be referred to in correspondence.

[0140] Figure 5 This invention provides a target detection model optimization device, such as... Figure 5 As shown, the target detection model optimization device 500 includes: a determination module 501, a joint optimization module 502, a temporal optimization module 503, and a target optimization module 504; wherein,

[0141] The determination module 501 is used to determine the initial target segmentation result based on the image dataset; the image dataset includes a two-dimensional image sequence and a three-dimensional point cloud sequence; there is a correspondence between the two-dimensional image sequence and the three-dimensional point cloud sequence;

[0142] The joint optimization module 502 is used to jointly optimize the clustering network model and the localization network model based on the initial target segmentation result and the alternating training strategy to obtain a first clustering network model and a first localization network model; the clustering network model is used to detect targets in the images of the three-dimensional point cloud sequence; the localization network model is used to detect targets in the images of the two-dimensional image sequence.

[0143] The temporal optimization module 503 is used to perform temporal optimization on the two-dimensional candidate target set and the three-dimensional candidate target set respectively to obtain a first candidate target set and a second candidate target set; the two-dimensional candidate target set and the three-dimensional candidate target set are respectively obtained by the clustering network model and the localization network model in the joint optimization process.

[0144] The target optimization module 504 is used to optimize the first clustering network model and the first localization network model based on the first candidate target set and the second candidate target set, respectively, to obtain a target detection model.

[0145] The target detection model optimization device provided by this invention determines the initial target segmentation result through an image dataset. The image dataset includes a two-dimensional image sequence and a three-dimensional point cloud sequence. There is a correspondence between the two-dimensional image sequence and the three-dimensional point cloud sequence. Then, based on the initial target segmentation result and an alternating training strategy, a clustering network model and a localization network model are jointly optimized to obtain a first clustering network model and a first localization network model. The clustering network model is used to detect targets in the images of the three-dimensional point cloud sequence. The localization network model is used to detect targets in the images of the two-dimensional image sequence. Temporal optimization is performed on the two-dimensional candidate target set and the three-dimensional candidate target set respectively to obtain a first candidate target set and a second candidate target set. The two-dimensional candidate target set and the three-dimensional candidate target set are obtained by the clustering network model and the localization network model respectively during the joint optimization process. Based on the first candidate target set and the second candidate target set, the first clustering network model and the first localization network model are optimized respectively to obtain a target detection model. The target detection model can accurately detect targets in the images corresponding to the two-dimensional image sequence and the three-dimensional point cloud sequence, realizing target detection in different scenarios and improving the accuracy and efficiency of target detection.

[0146] Optionally, the joint optimization module 502 is specifically used for:

[0147] The point cloud data in the three-dimensional point cloud sequence is input into the clustering network model to obtain the first segmentation result of the point cloud data output by the clustering network model; the clustering network model is trained based on the sample point cloud data and the initial target segmentation result.

[0148] The images in the two-dimensional image sequence are input into the localization network model to obtain the second segmentation result output by the localization network; the localization network model is trained based on the sample images and the first segmentation result.

[0149] Based on the first segmentation result, the second segmentation result, and the alternating training strategy, the clustering network model and the localization network model are jointly optimized to obtain the first clustering network model and the first localization network model.

[0150] Optionally, the joint optimization module 502 is specifically used for:

[0151] The point cloud data in the three-dimensional point cloud sequence is input into the feature extraction module in the clustering network model to obtain the voxel feature data output by the feature extraction module.

[0152] The voxel feature data is input into the voting module in the clustering network model to obtain the category and deviation value corresponding to the point cloud data output by the voting module;

[0153] The category and the deviation value are input into the clustering module in the clustering network model to obtain the first segmentation result output by the clustering module.

[0154] Optionally, the loss function used in the clustering network model is expressed as in formula (1), where:

[0155]

[0156] Where M represents the total number of foreground points, and i represents the i-th point. This represents the monitoring signal corresponding to the i-th point. Δx represents the true deviation at the i-th point. i This represents the prediction bias at the i-th point.

[0157] Optionally, the joint optimization module 502 is specifically used for:

[0158] Based on the first segmentation result, the first supervision information of the localization network model is determined;

[0159] Based on the first supervision information, the localization network model is trained to obtain the first localization network model.

[0160] Optionally, the joint optimization module 502 is specifically used for:

[0161] Based on the second segmentation result, the second supervision information of the clustering network model is determined;

[0162] Based on the second supervision information, the clustering network model is trained to obtain the first clustering network model.

[0163] Optionally, the timing optimization module 503 is specifically used for:

[0164] Kalman filtering is used to perform time-series tracking on the two-dimensional candidate target set and the three-dimensional candidate target set, respectively.

[0165] Based on the tracking results, the first candidate target set and the second candidate target set are determined.

[0166] Optionally, the timing optimization module 503 is specifically used for:

[0167] Candidate targets whose number of consecutively matched image frames is less than a first preset threshold are judged as incorrect segmentation results;

[0168] The erroneous segmentation results are filtered out from the two-dimensional candidate target set to obtain the first candidate target set.

[0169] Optionally, the timing optimization module 503 is specifically used for:

[0170] Candidate targets whose point cloud data in a continuously matched 3D point cloud sequence is less than a second preset threshold are judged as incorrect segmentation results;

[0171] The erroneous segmentation results are filtered out from the three-dimensional candidate target set;

[0172] If there are any missing segmentation results, the segmentation results are added to the three-dimensional candidate target set to obtain the second candidate target set.

[0173] Figure 6 This is a schematic diagram of the physical structure of an electronic device provided by the present invention, such as... Figure 6 As shown, the electronic device 600 may include: a processor 610, a communication interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a target detection model optimization method, which includes:

[0174] Based on the image dataset, an initial target segmentation result is determined; the image dataset includes a two-dimensional image sequence and a three-dimensional point cloud sequence; there is a correspondence between the two-dimensional image sequence and the three-dimensional point cloud sequence.

[0175] Based on the initial target segmentation results and the alternating training strategy, the clustering network model and the localization network model are jointly optimized to obtain a first clustering network model and a first localization network model; the clustering network model is used to detect targets in the images of the 3D point cloud sequence; the localization network model is used to detect targets in the images of the 2D image sequence.

[0176] Temporal optimization is performed on the two-dimensional candidate target set and the three-dimensional candidate target set respectively to obtain the first candidate target set and the second candidate target set; the two-dimensional candidate target set and the three-dimensional candidate target set are obtained by the clustering network model and the localization network model respectively during the joint optimization process;

[0177] Based on the first candidate target set and the second candidate target set, the first clustering network model and the first localization network model are optimized respectively to obtain the target detection model.

[0178] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0179] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is able to execute the target detection model optimization method provided by the above methods, the method comprising:

[0180] Based on the image dataset, an initial target segmentation result is determined; the image dataset includes a two-dimensional image sequence and a three-dimensional point cloud sequence; there is a correspondence between the two-dimensional image sequence and the three-dimensional point cloud sequence.

[0181] Based on the initial target segmentation results and the alternating training strategy, the clustering network model and the localization network model are jointly optimized to obtain a first clustering network model and a first localization network model; the clustering network model is used to detect targets in the images of the 3D point cloud sequence; the localization network model is used to detect targets in the images of the 2D image sequence.

[0182] Temporal optimization is performed on the two-dimensional candidate target set and the three-dimensional candidate target set respectively to obtain the first candidate target set and the second candidate target set; the two-dimensional candidate target set and the three-dimensional candidate target set are obtained by the clustering network model and the localization network model respectively during the joint optimization process;

[0183] Based on the first candidate target set and the second candidate target set, the first clustering network model and the first localization network model are optimized respectively to obtain the target detection model.

[0184] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the target detection model optimization method provided by the methods described above, the method comprising:

[0185] Based on the image dataset, an initial target segmentation result is determined; the image dataset includes a two-dimensional image sequence and a three-dimensional point cloud sequence; there is a correspondence between the two-dimensional image sequence and the three-dimensional point cloud sequence.

[0186] Based on the initial target segmentation results and the alternating training strategy, the clustering network model and the localization network model are jointly optimized to obtain a first clustering network model and a first localization network model; the clustering network model is used to detect targets in the images of the 3D point cloud sequence; the localization network model is used to detect targets in the images of the 2D image sequence.

[0187] Temporal optimization is performed on the two-dimensional candidate target set and the three-dimensional candidate target set respectively to obtain the first candidate target set and the second candidate target set; the two-dimensional candidate target set and the three-dimensional candidate target set are obtained by the clustering network model and the localization network model respectively during the joint optimization process;

[0188] Based on the first candidate target set and the second candidate target set, the first clustering network model and the first localization network model are optimized respectively to obtain the target detection model.

[0189] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0190] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0191] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for optimizing a target detection model, characterized in that, include: Based on the image dataset, determine the initial target segmentation result; The image dataset includes two-dimensional image sequences and three-dimensional point cloud sequences; There is a correspondence between the two-dimensional image sequence and the three-dimensional point cloud sequence; Based on the initial target segmentation results and the alternating training strategy, the clustering network model and the localization network model are jointly optimized to obtain a first clustering network model and a first localization network model; the clustering network model is used to detect targets in the images of the 3D point cloud sequence; the localization network model is used to detect targets in the images of the 2D image sequence. Temporal optimization is performed on the two-dimensional candidate target set and the three-dimensional candidate target set respectively to obtain the first candidate target set and the second candidate target set; the two-dimensional candidate target set and the three-dimensional candidate target set are obtained by the clustering network model and the localization network model respectively during the joint optimization process; Based on the first candidate target set and the second candidate target set, the first clustering network model and the first localization network model are optimized respectively to obtain the target detection model; The step of jointly optimizing the clustering network model and the localization network model based on the initial target segmentation result and the alternating training strategy to obtain the first clustering network model and the first localization network model includes: The point cloud data in the three-dimensional point cloud sequence is input into the clustering network model to obtain the first segmentation result of the point cloud data output by the clustering network model; the clustering network model is trained based on the sample point cloud data and the initial target segmentation result. The images in the two-dimensional image sequence are input into the localization network model to obtain the second segmentation result output by the localization network; the localization network model is trained based on the sample images and the first segmentation result. Based on the first segmentation result, the second segmentation result, and the alternating training strategy, the clustering network model and the localization network model are jointly optimized to obtain the first clustering network model and the first localization network model.

2. The target detection model optimization method of claim 1, wherein, The step of inputting point cloud data from the 3D point cloud sequence into the clustering network model to obtain the first segmentation result of the point cloud data output by the clustering network model includes: The point cloud data in the three-dimensional point cloud sequence is input into the feature extraction module in the clustering network model to obtain the voxel feature data output by the feature extraction module. The voxel feature data is input into the voting module in the clustering network model to obtain the category and deviation value corresponding to the point cloud data output by the voting module; The category and the deviation value are input into the clustering module in the clustering network model to obtain the first segmentation result output by the clustering module.

3. The target detection model optimization method of claim 1 or 2, wherein, The loss function used in the clustering network model is expressed by formula (1), where: （1） Where M represents the total number of foreground points, and i represents the i-th point. This represents the monitoring signal corresponding to the i-th point. This represents the true deviation at the i-th point; This represents the prediction bias at the i-th point.

4. The target detection model optimization method of claim 1, wherein, The step of jointly optimizing the clustering network model and the localization network model based on the first segmentation result, the second segmentation result, and the alternating training strategy to obtain the first clustering network model and the first localization network model includes: Based on the first segmentation result, the first supervision information of the localization network model is determined; Based on the first supervision information, the localization network model is trained to obtain the first localization network model.

5. The target detection model optimization method of claim 1, wherein, The step of jointly optimizing the clustering network model and the localization network model based on the first segmentation result, the second segmentation result, and the alternating training strategy to obtain the first clustering network model and the first localization network model includes: Based on the second segmentation result, the second supervision information of the clustering network model is determined; Based on the second supervision information, the clustering network model is trained to obtain the first clustering network model.

6. The target detection model optimization method of claim 1, wherein, Temporal optimization is performed on the two-dimensional candidate target set and the three-dimensional candidate target set respectively to obtain the first candidate target set and the second candidate target set, including: Kalman filtering is used to perform time-series tracking on the two-dimensional candidate target set and the three-dimensional candidate target set, respectively. Based on the tracking results, the first candidate target set and the second candidate target set are determined.

7. The target detection model optimization method of claim 6, wherein, The step of determining the first candidate target set and the second candidate target set based on the tracking results includes: Candidate targets whose number of consecutively matched image frames is less than a first preset threshold are judged as incorrect segmentation results; The erroneous segmentation results are filtered out from the two-dimensional candidate target set to obtain the first candidate target set.

8. The target detection model optimization method of claim 6, wherein, Determining the first candidate target set and the second candidate target set based on the tracking results includes: Candidate targets whose point cloud data in a continuously matched 3D point cloud sequence is less than a second preset threshold are judged as incorrect segmentation results; The erroneous segmentation results are filtered out from the three-dimensional candidate target set; If there are any missing segmentation results, the segmentation results are added to the three-dimensional candidate target set to obtain the second candidate target set. 9.A target detection model optimization apparatus, characterized in that, include: The determination module is used to determine the initial target segmentation result based on the image dataset; The image dataset includes two-dimensional image sequences and three-dimensional point cloud sequences; There is a correspondence between the two-dimensional image sequence and the three-dimensional point cloud sequence; A joint optimization module is used to jointly optimize a clustering network model and a localization network model based on the initial target segmentation results and an alternating training strategy, to obtain a first clustering network model and a first localization network model; the clustering network model is used to detect targets in the images of the 3D point cloud sequence; the localization network model is used to detect targets in the images of the 2D image sequence. The temporal optimization module is used to perform temporal optimization on the two-dimensional candidate target set and the three-dimensional candidate target set respectively to obtain the first candidate target set and the second candidate target set; the two-dimensional candidate target set and the three-dimensional candidate target set are obtained by the clustering network model and the localization network model respectively in the process of joint optimization. The target optimization module is used to optimize the first clustering network model and the first localization network model based on the first candidate target set and the second candidate target set, respectively, to obtain a target detection model; The joint optimization module is used to jointly optimize the clustering network model and the localization network model based on the initial target segmentation result and the alternating training strategy to obtain a first clustering network model and a first localization network model, including: The point cloud data in the three-dimensional point cloud sequence is input into the clustering network model to obtain the first segmentation result of the point cloud data output by the clustering network model; the clustering network model is trained based on the sample point cloud data and the initial target segmentation result. The images in the two-dimensional image sequence are input into the localization network model to obtain the second segmentation result output by the localization network; the localization network model is trained based on the sample images and the first segmentation result. Based on the first segmentation result, the second segmentation result, and the alternating training strategy, the clustering network model and the localization network model are jointly optimized to obtain the first clustering network model and the first localization network model.

10. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the target detection model optimization method as described in any one of claims 1 to 8.

11. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, When the computer program is executed by the processor, it implements the target detection model optimization method as described in any one of claims 1 to 8.

12. A computer program product comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the target detection model optimization method as described in any one of claims 1 to 8.

Images