Three-dimensional target detection model training method, electronic device, and computer storage medium

Abstract

Embodiments of the present application provide a three-dimensional target detection model training method, an electronic device and a computer storage medium, wherein the three-dimensional target detection model training method comprises: obtaining a current point cloud frame training sample; in the process of training the three-dimensional target detection model using the current point cloud frame training sample, obtaining historical data corresponding to a historical point cloud frame training sample from a memory area set for the three-dimensional target detection model and independent of the three-dimensional target detection model; fusing the historical data and training data corresponding to the current point cloud frame training sample, and performing model training of the current round according to the fusion result. Through the embodiments of the present application, the effect of lightweight model training is achieved.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular to a three-dimensional target detection model training method, electronic device, and computer storage medium. Background Technology

[0002] 3D target detection is a core function of autonomous driving devices such as vehicles, aircraft, and mobile robots. Based on this function, autonomous driving devices can perform tasks such as driving planning, motion prediction, and obstacle avoidance.

[0003] Typically, these autonomous driving devices use 3D object detection models for 3D object detection. These models take the point cloud of their environment as input and process it to obtain information about target objects, point cloud semantic segmentation results, scene optical flow, and so on. During 3D object detection, the temporal information of point cloud frames can effectively improve the model's detection performance. Therefore, more and more 3D detection models are incorporating the temporal information of point cloud frames into their training process. For example, in a 3D object model training scheme using an LSTM (Long Short-Term Memory) network, the model's hidden state variables are used to record historical point cloud frame information and iteratively update it. However, this requires the LSTM model to perform N forward inferences during training if N historical point cloud frames are used, and the intermediate variable values ​​of the historical inputs need to be recorded. This results in the model training process consuming a large amount of memory, leading to slow training speed; furthermore, the backpropagation path is long, easily causing problems such as vanishing or exploding gradients. Summary of the Invention

[0004] In view of this, embodiments of this application provide a three-dimensional target detection model training scheme to at least partially solve the above problems.

[0005] According to a first aspect of the embodiments of this application, a method for training a three-dimensional object detection model is provided, comprising: acquiring a current point cloud frame training sample; during the process of training a three-dimensional object detection model using the current point cloud frame training sample, acquiring historical data corresponding to historical point cloud frame training samples from a memory region set for the three-dimensional object detection model and independent of the three-dimensional object detection model; fusing the historical data with the training data corresponding to the current point cloud frame training sample, and performing model training for the current round based on the fusion result.

[0006] According to a second aspect of the present application, an electronic device is provided, including: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus; the memory is used to store at least one executable instruction, which causes the processor to perform an operation corresponding to the method described in the first aspect.

[0007] According to a third aspect of the embodiments of this application, a computer storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0008] According to the solution provided in this application, during the training of the 3D object detection model, an independent memory region is used to store historical data corresponding to training samples of historical point cloud frames. When training the model for the current point cloud frame training sample, the model does not need to perform forward inference and record historical intermediate variable values; these values ​​are directly obtained from this memory region, effectively improving the training speed of fusing training samples from multiple point cloud frames. Furthermore, compared to the memory occupied and used by the LSTM solution during training, the memory size of this region is significantly reduced. In addition, when using a memory region to store historical data corresponding to training samples of historical point cloud frames, the backward gradient propagation of the 3D object detection model will not be passed to this memory region, thus avoiding the difficulty of model optimization and preventing gradient vanishing or exploding, achieving the effect of lightweight model training. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0010] Figure 1 A schematic diagram of an exemplary system for the verification code generation method applicable to embodiments of this application;

[0011] Figure 2A This is a flowchart illustrating the steps of a three-dimensional target detection model training method according to Embodiment 1 of this application;

[0012] Figure 2B for Figure 2A A schematic diagram of an example training framework in the illustrated embodiment;

[0013] Figure 2C for Figure 2A A schematic diagram of the data storage structure corresponding to training data of different data types in the embodiment shown.

[0014] Figure 2D for Figure 2A A schematic diagram of the fusion and update method of training data for the first image type in the illustrated embodiment;

[0015] Figure 2E for Figure 2A The diagram shows a fusion and update method for training data of the second image type in the embodiment shown.

[0016] Figure 2F for Figure 2A The diagram illustrates the fusion and update method of training data for the third image type in the embodiment shown.

[0017] Figure 2G for Figure 2A The diagram shows a fusion and update method for training data of the fourth image type in the embodiment shown.

[0018] Figure 2H for Figure 2A A schematic diagram illustrating a method for updating point-type training data in the illustrated embodiment;

[0019] Figure 2I for Figure 2A A schematic diagram of the training process of a three-dimensional target detection model in the illustrated embodiment;

[0020] Figure 3 This is a flowchart illustrating the steps of a three-dimensional target detection model training method according to Embodiment 2 of this application;

[0021] Figure 4 This is a schematic diagram of the structure of an electronic device according to Embodiment 3 of this application. Detailed Implementation

[0022] To enable those skilled in the art to better understand the technical solutions in the embodiments of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art should fall within the protection scope of the embodiments of this application.

[0023] The specific implementation of the embodiments of this application will be further described below with reference to the accompanying drawings.

[0024] Figure 1 An exemplary system applicable to embodiments of this application is shown. For example... Figure 1 As shown, the system 100 may include a cloud server 102, a communication network 104, and / or one or more user devices 106. Figure 1The example in the text shows multiple user devices.

[0025] The cloud server 102 can be any suitable device for storing information, data, programs, and / or any other suitable type of content, including but not limited to distributed storage system devices, server clusters, computing cloud server clusters, etc. In some embodiments, the cloud server 102 can perform any suitable function. For example, in some embodiments, a 3D object detection model can be set up in the cloud server 102. As an optional example, in some embodiments, the cloud server 102 can be used to train the 3D object detection model. As another example, in some embodiments, the cloud server 102 can set up a dedicated memory area for the 3D object detection model during training, so that during the training of the 3D object detection model using the current point cloud frame training samples, historical data corresponding to historical point cloud frame training samples can be obtained from this memory area; then, the historical data can be fused with the training data corresponding to the current point cloud frame training samples to train the 3D object detection model for the current round. In this embodiment of the application, a training round refers to a training session using a training sample. For example, if the model is trained 10,000 times, one of the training sessions is a training round. For example, the 20th training session is the 20th training round. If the 20th training round is ongoing, it is the current training round.

[0026] In some embodiments, communication network 104 may be any suitable combination of one or more wired and / or wireless networks. For example, communication network 104 may include any one or more of the following: the Internet, intranet, wide area network (WAN), local area network (LAN), wireless network, digital subscriber line (DSL) network, frame relay network, asynchronous transfer mode (ATM) network, virtual private network (VPN), and / or any other suitable communication network. User equipment 106 may be connected to communication network 104 via one or more communication links (e.g., communication link 112), and communication network 104 may be linked to cloud server 102 via one or more communication links (e.g., communication link 114). Communication links may be any communication link suitable for transmitting data between user equipment 106 and cloud server 102, such as network links, dial-up links, wireless links, hardwired links, any other suitable communication links, or any suitable combination of such links.

[0027] User device 106 may include any one or more user devices suitable for interacting with the user and cloud server 102. In some embodiments, user device 106 may initiate a 3D object detection model training request to the cloud server to trigger the cloud server 102 to train the 3D object detection model. In some embodiments, user device 106 may include any suitable type of device. For example, in some embodiments, user device 106 may include mobile devices, tablet computers, laptop computers, desktop computers, wearable computers, game consoles, media players, vehicle entertainment systems, and / or any other suitable type of user device.

[0028] Based on the above system, the training scheme for the three-dimensional target detection model of this application will be described below through several embodiments.

[0029] Example 1

[0030] Reference Figure 2A The diagram shows a flowchart of the steps of a three-dimensional target detection model training method according to Embodiment 1 of this application.

[0031] For ease of explanation, the following describes one training framework from an embodiment of this application as an example, such as... Figure 2B As shown. Figure 2B In this paper, the three-dimensional target detection model is simply illustrated as a single-frame detector. However, those skilled in the art should understand that in practical applications, any three-dimensional target detection model with the function of three-dimensional target detection based on point cloud can be applied to the scheme of this application embodiment. This application embodiment does not limit the specific implementation structure of the three-dimensional target detection model.

[0032] Depend on Figure 2B As can be seen, this embodiment of the application sets up a memory bank for the 3D object detection model, which is set up independently of the 3D object detection model. It is mainly used to store historical data corresponding to some point cloud frame training samples during the training process of the 3D object detection model, including point cloud data, intermediate feature data, predicted feature data, etc. These point cloud frame training samples, relative to the current input, that is, the point cloud frame training samples used to train the 3D object detection model (i.e., the current point cloud frame training samples), are called historical data corresponding to historical point cloud frame training samples. This historical data is fused with the training data corresponding to the current point cloud frame training samples in the training of the 3D object detection model, and is continuously iteratively updated.

[0033] The 3D object detection model is trained based on the fused data. After obtaining the prediction result (output), it calculates the corresponding loss value based on the ground truth value corresponding to the current point cloud frame training sample and a preset loss function (set by those skilled in the art according to actual needs, such as cross-entropy loss function). Then, backpropagation is performed based on this loss value to adjust the model parameters, and the next round of training is performed based on the new model parameters until the training termination condition is reached, such as reaching a preset number of training iterations or the loss value reaching a preset threshold.

[0034] During backpropagation, adjustments to model parameters are not propagated to the aforementioned memory region set up for the 3D object detection model. In other words, the backpropagation gradient is not propagated to the aforementioned memory region, and historical data in this memory region is not affected by backpropagation, thus avoiding the problem of model optimization difficulties caused by historical data.

[0035] Based on the above exemplary 3D object detection model training framework, the 3D object detection model training method of this embodiment includes the following steps:

[0036] Step S202: Obtain the current point cloud frame training sample.

[0037] Training a 3D object detection model typically requires a large number of training samples. These samples form a training sample set, from which one sample is selected for each training iteration. This training sample used for the current iteration is called the current point cloud frame training sample. This point cloud frame training sample includes both point cloud data and corresponding label data, such as annotations of 3D objects.

[0038] Step S204: During the training of the 3D object detection model using the current point cloud frame training samples, historical data corresponding to the historical point cloud frame training samples are obtained from the memory region set for the 3D object detection model and independent of the 3D object detection model.

[0039] As mentioned earlier, 3D object detection models require a large number of training samples for training. Therefore, for the current point cloud frame training sample, all previous training samples can be referred to as historical point cloud frame training samples. However, in practical applications, a certain number of training samples within that range before the current point cloud frame training sample are usually set as historical point cloud frame training samples that the current point cloud frame training sample can refer to. This number of training samples can be flexibly set by those skilled in the art.

[0040] During model training, various training data are involved, including the number of vectors corresponding to training samples received by the input layer, feature data generated by feature extraction in the hidden layer, and prediction data generated by detection and prediction in the prediction or output layer. To fully integrate historical training data and avoid memory and optimization issues, this embodiment sets up a memory region for the 3D object detection model during training. This memory region can be of a fixed size or a variable size within a certain range. For simplicity, this embodiment uses a fixed size; however, those skilled in the art should understand that the solution of this embodiment can be implemented with reference to the fixed-size memory region for variable-size memory regions.

[0041] In one feasible approach, certain model layer positions can be preset in the 3D object detection model, such as the input layer and the positions of certain hidden layers (i.e., hidden layers), to fuse historical data and training data corresponding to the current point cloud frame training samples at these positions. That is, fixed-point fusion is performed to improve the model's data fusion and overall training efficiency. In this case, obtaining the historical data corresponding to the historical point cloud frame training samples can be achieved by obtaining the historical data corresponding to the historical point cloud frame training samples at multiple preset model layer positions. Optionally, these multiple preset model layer positions may include: the input layer, the 3D point cloud feature extraction layer, and the 3D object prediction layer. The 3D point cloud feature extraction layer may include all feature extraction layers, or only some of them, such as the 3D feature extraction layer when transitioning from 3D features to 2D features. However, this is not limited to this; in practical applications, those skilled in the art can also choose other model layers, or use all model layers of the 3D object detection model as multiple preset model layers, performing data fusion at all of these model layers.

[0042] Furthermore, the training data corresponding to the training samples during the training process can be a data vector corresponding to the training samples, such as the point cloud data vector corresponding to the point cloud frame training samples; it can also be an intermediate feature, such as the three-dimensional intermediate feature vector generated after the model extracts three-dimensional features based on the point cloud data vector; or it can be a prediction vector, such as the feature vector corresponding to the prediction result obtained by the model after converting the three-dimensional intermediate feature vector into a two-dimensional intermediate feature vector and then predicting the target object. Since different training data have different data structures and processing methods, in order to facilitate effective processing of different training data and improve processing efficiency, in one feasible approach, the above-mentioned acquisition of historical data corresponding to multiple preset model layer positions of historical point cloud frame training samples can be implemented as follows: determining the data type of the training data corresponding to the current point cloud frame training sample at the current model layer position; and acquiring the historical data of historical point cloud frame training samples from the data storage structure of the memory region corresponding to that data type.

[0043] In this embodiment, the training data is categorized into point type and image type. The point type is irregular and indicates unordered data. Point-type training data includes, but is not limited to, point clouds, bounding boxes, 3D object detection and segmentation results, motion velocities, etc., all of which can be subtypes of the point type. The image type, on the other hand, is regular and indicates regularized data. Image-type training data includes, but is not limited to, input images, intermediate feature maps, final prediction maps, etc., all of which can be subtypes of the image type.

[0044] Based on this, in the memory region corresponding to the 3D object detection model, memory blocks can be allocated for storing training data based on the data type of each model layer position, or for subtypes of the data type. To improve the efficiency of data storage and retrieval, in this embodiment, one memory block is assigned to each subtype. For example, as shown... Figure 2C As shown, at the input layer of the model, there is a memory block (represented as memory block 1) for storing the point cloud data vector corresponding to the input point cloud; at a hidden layer of the model, there is a memory block (represented as memory block i) for storing intermediate feature maps; at the prediction output layer of the model, there is a memory block (represented as memory block N-2) for storing the prediction map, a memory block (represented as memory block N-1) for storing the target bounding boxes of the marked target objects, and a memory block (represented as memory block N) for indicating the motion speed of the target objects.

[0045] In specific storage implementation, different data storage structures can be used for training data of different data types. For example, point-type training data can use a queue structure, while image-type training data can use a data storage structure that facilitates the storage of matrix data, such as an array queue or array linking. However, this is not the only limitation. In practical applications, other data storage structures that facilitate the storage and updating of data are also applicable to the solutions in the embodiments of this application.

[0046] In this case, when the current point cloud frame training sample is processed to a certain layer by the 3D object detection model, if data fusion is required at that layer, the data storage structure containing the same data type as the training data of the current point cloud frame training sample at that layer can be determined from the data storage structure of the corresponding memory block, and the corresponding historical data can be obtained from it for subsequent fusion.

[0047] Step S206: Fuse the historical data corresponding to the historical point cloud frame training samples in the memory region with the training data corresponding to the current point cloud frame training samples, and perform model training for the current round based on the fusion result.

[0048] This includes fusing historical data with training data corresponding to the current point cloud frame training sample at multiple preset model layer locations.

[0049] As mentioned earlier, when the current point cloud frame training sample is processed to a certain layer by the 3D object detection model, if data fusion is required at that layer, the data storage structure containing the same data type as the training data of the current point cloud frame training sample at that layer can be determined from the corresponding memory block's data storage structure, and the corresponding historical data can be retrieved from it for fusion. For example, with Figure 2C The example shown illustrates the following: In the input layer, the point cloud data vector corresponding to the current point cloud frame training sample is fused with the point cloud data vectors corresponding to multiple historical point cloud frame training samples stored in memory block 1. In the model's preset hidden layer, the intermediate feature vector corresponding to the current point cloud frame training sample is fused with the feature vector of the intermediate feature map corresponding to the historical point cloud frame training sample stored in memory block i. In the prediction output layer, the feature vector of the prediction map corresponding to the current point cloud frame training sample is fused with the feature vector of the prediction map corresponding to the historical point cloud frame training sample stored in memory block N-2. The feature vector of the target prediction box in the prediction map corresponding to the current point cloud frame training sample is fused with the feature vector of the target box corresponding to the historical point cloud frame training sample stored in memory block N-1. The feature vector of the predicted motion velocity of the target object corresponding to the current point cloud frame training sample is fused with the feature vector of the motion velocity of the target object corresponding to the historical point cloud frame training sample stored in memory block N. Therefore, throughout the entire training process of the 3D object detection model, the historical data stored in the corresponding memory regions can effectively fuse historical training information into the current training process, improving the model's training effect and prediction performance.

[0050] In the above process, a feasible method for fusing point-type training data, which involves combining historical data in the memory region with the training data corresponding to the current point cloud frame training sample, can be achieved by concatenating the point-type training data corresponding to the current point cloud frame training sample with the corresponding historical data of the same point type. This method effectively achieves the fusion of point-type training data, is simple to implement, and is cost-effective and fast.

[0051] For training data of image type, one feasible approach is to fuse historical data in the memory region with the training data corresponding to the current point cloud frame training sample. This can be achieved by: summing, splicing, or fusing the training data of the image type corresponding to the current point cloud frame training sample with the historical data of the corresponding image type, or by gating.

[0052] Furthermore, since each round of training generates new training data, it is also necessary to update the historical data stored in the memory area. For example, based on the training data corresponding to the current point cloud frame training sample, the historical data corresponding to the historical point cloud frame training sample in the memory area is iteratively updated to ensure the validity and timeliness of the historical data.

[0053] The following, combined with Figure 2D-2G This paper explains the fusion and updating of training data for the above image types.

[0054] Figure 2D This illustrates an example of data fusion using summation for training data of image type. As shown in the figure, the training data X corresponding to the current point cloud frame training sample... t First, it is labeled to indicate that it is the current training data. This labeling guides the subsequent training of the model to be mainly based on the current training data. Then, the historical data H corresponding to the historical point cloud frame training samples obtained from the image-type data storage structure... t-1 First, spatial transformation is used to avoid the influence of sensor motion (such as translation and rotation). Then, the historical data H after spatial transformation... t-1 and the current training data X t The two data sets are summed (shown as "+" in the diagram) to merge them into a new training data H. t The new training data H t On one hand, it will be used as input data for training the model layers following the current model layer; on the other hand, it will replace the original historical data H. t-1 As historical data for the next fusion, that is, as historical data H for the next fusion. t-1 This method is simple to implement and allows for rapid integration.

[0055] Figure 2E This illustrates another example of data fusion using image-based training data, achieved through data stitching. As shown in the figure, the training data X corresponding to the current point cloud frame training sample... t First, a 1×1 convolutional layer is used to perform convolution to obtain the convolution result of the current training data; then, the historical data H corresponding to the historical point cloud frame training samples is obtained from the data storage structure that stores image types. t-1 First, spatial transformation is used to avoid the influence of the sensor's own motion (such as translation and rotation). Then, the historical data H after spatial transformation... t-1A 1×1 convolutional layer is also used to perform convolution to obtain the convolution results of the historical training data. Next, the convolution results of the current training data and the historical training data are concatenated (symbolized as "C" in the diagram) to merge the two data sets into new training data H. t The new training data H t On one hand, it will be used as input data for training the model layers following the current model layer; on the other hand, it will replace the original historical data H. t-1 As historical data for the next fusion, that is, as historical data H for the next fusion. t-1 This approach allows for better feature fusion.

[0056] Figure 2F This illustrates yet another example of training data for image types, using data fusion based on maximum values. As shown in the figure, the historical data H corresponding to the historical point cloud frame training samples obtained from the data storage structure storing image types... t-1 First, spatial transformation is used to avoid the influence of sensor motion (such as translation and rotation). Then, the historical data H after spatial transformation... t-1 The data will be labeled with the maximum value corresponding to different training epochs to eliminate the influence of possible false positive features on model training. After labeling, it will be compared with the current training data X. t Together, from historical data H t-1 and the current training data X t The maximum value corresponding to each feature is selected (symbolized as "M" in the diagram). Then, the maximum values ​​corresponding to each feature are combined to form new training data H. t The new training data H t On one hand, it will be used as input data for training the model layers following the current model layer; on the other hand, it will replace the original historical data H. t-1 As historical data for the next fusion, that is, as historical data H for the next fusion. t-1 This method can effectively filter out the features that best represent the characteristics of point clouds, providing a valid basis for feature extraction and prediction in subsequent model layers.

[0057] Figure 2G This illustrates another example of data fusion based on gating processing for training data of image types. As shown in the figure, the historical data H corresponding to the historical point cloud frame training samples obtained from the data storage structure storing image types... t-1 First, spatial transformation is used to avoid the influence of the sensor's own motion (such as translation and rotation). Then, the historical data H after spatial transformation... t-1 Will it be combined with the current training data Xt They are processed together using gating. The training of these two parts uses a reset gate and an update gate to determine the historical data H. t-1 and the current training data X t How much data should be fused from each dataset? Then, using gated fusion (Candidate), these selected datasets are fused to obtain new training data H. t The new training data H t On one hand, it will be used as input data for training the model layers following the current model layer; on the other hand, it will replace the original historical data H. t-1 As historical data for the next fusion, that is, as historical data H for the next fusion. t-1 This approach can more accurately identify the features that need to be fused, thus providing a more accurate basis for feature extraction and prediction in subsequent model layers.

[0058] For point-type training data, historical data can be directly retrieved from the data storage structure in the corresponding memory area and then concatenated with the current training data to achieve point-type training data fusion. Taking a queue structure as an example, due to the FIFO (First-In, First-Out) characteristic of the queue, when the queue is full, the historical data stored therein can be dequeued by adding the current training data to the tail of the queue, thereby updating the point-type training data. An exemplary point-type training data update is as follows: Figure 2H As shown in the figure, before the current training data arrives, the data in the original queue consists of historical training data 275-290. After retrieving training data 275-290 from the queue and concatenating it with the current training data 291 corresponding to the current point cloud frame training sample, on the one hand, the newly generated training data is passed to the subsequent model layers for processing; on the other hand, training data 291 is added to the queue, while training data 275 in the original queue is dequeued from the head of the queue, and the new queue includes training data 276-291.

[0059] The following example, using the training process of a specific 3D object detection model, illustrates the above process. Figure 2I As shown.

[0060] from Figure 2IAs shown in the example, the current point cloud frame training sample is input into the 3D object detection model and received by its input layer. Upon receiving the point cloud data vector corresponding to the training sample, the input layer retrieves the point cloud data vectors corresponding to historical point cloud frame training samples from its corresponding memory block. Then, it concatenates the point cloud data vector corresponding to the current point cloud frame training sample with multiple point cloud data vectors corresponding to historical point cloud frame training samples to generate a new point cloud data vector. This new point cloud data vector is then passed to the model's voxelization layer for voxelization. The voxelized vector is then input to multiple 3D feature extraction layers for 3D feature extraction, with each 3D feature extraction layer outputting a 3D feature map. In this example, data fusion is performed only in the last extraction layer of the multiple 3D feature extraction layers. Therefore, after the last 3D feature extraction layer outputs its corresponding 3D feature map, the historical 3D feature maps output by the historical point cloud frame training samples at that layer are retrieved from the memory block corresponding to that layer. Then, the current 3D feature map and the historical 3D feature map output by the last 3D feature extraction layer are fused. Any of the four fusion methods mentioned above for image types can be used to fuse them to obtain a new fused 3D feature map.

[0061] The newly fused 3D feature map is transmitted to multiple 2D feature extraction layers for 2D feature extraction. After the last 2D feature extraction layer outputs its corresponding 2D feature map, the prediction layer uses this 2D feature map to perform target prediction, generating a prediction map. In this example, if data fusion is also performed at the prediction layer, the historical prediction map corresponding to the training samples of historical point cloud frames will be obtained from the memory block corresponding to the prediction layer. Furthermore, any of the four fusion methods mentioned above for image types can be used to fuse the prediction map output by the current prediction layer and the historical prediction map obtained from the memory block to obtain a new fused prediction map.

[0062] The newly fused prediction map can be passed through a detection head to obtain the corresponding 3D target detection results.

[0063] Furthermore, based on the 3D object detection result and the label data corresponding to the training samples in the current point cloud frame, a loss value is calculated using a preset loss function. Backpropagation is then performed based on this loss value to adjust the model parameters of the 3D object detection model. The backpropagation data does not pass to the memory region corresponding to the 3D object detection model, i.e., it is not passed to the aforementioned memory blocks, thus achieving lightweight model training.

[0064] As can be seen, through this embodiment, during the training of the 3D object detection model, a separate memory region is used to store the historical data corresponding to the training samples of historical point cloud frames. When training the model for the current point cloud frame training sample, there is no need for the model to perform forward inference and record historical intermediate variable values; these values ​​can be directly obtained from this memory region, effectively improving the training speed of fusing training samples from multiple point cloud frames. Furthermore, compared to the memory occupied and used by the LSTM scheme during training, the memory size of this region is significantly reduced. In addition, when using a memory region to store the historical data corresponding to the training samples of historical point cloud frames, the backward gradient propagation of the 3D object detection model will not be passed to this memory region, thus avoiding the difficulty of model optimization and preventing gradient vanishing or exploding, achieving the effect of lightweight model training.

[0065] Example 2

[0066] Reference Figure 3 The flowchart illustrates the steps of a three-dimensional target detection model training method according to Embodiment 2 of this application.

[0067] The training method for the 3D target detection model in this embodiment is based on the method shown in Embodiment 1 above, and its training mode is further improved so that the length of its training sequence can be expanded.

[0068] Specifically, the three-dimensional target detection model training method in this embodiment includes the following steps:

[0069] Step S302: Obtain the point cloud frame sequence data stream; divide the point cloud frame sequence data stream into multiple sub-sequences according to the preset sequence length; randomly recombine the multiple sub-sequences to generate a new point cloud frame sequence data stream; obtain the current point cloud frame training sample from the new point cloud frame sequence data stream.

[0070] Unlike the direct acquisition of the current point cloud frame training samples in Embodiment 1, this method, considering actual point cloud acquisition scenarios, acquires the current point cloud frame training samples from the point cloud frame sequence data stream. This allows the 3D object detection model to incorporate more effective historical information, and also enables the trained 3D object detection model to be better suited for actual 3D object detection scenarios. The preset sequence length can be flexibly set by those skilled in the art according to actual needs.

[0071] Since the 3D object detection model is trained on a point cloud frame sequence data stream basis, in one feasible approach, before training the model using point cloud frame training samples from the current point cloud frame sequence data stream, the memory region corresponding to the 3D object detection model is cleared. This cleared memory region, along with point cloud frame training samples from the new point cloud frame sequence data stream, is then used to train the 3D object detection model. In other words, when training the model using the current point cloud frame training samples, data fusion is performed entirely using the training data corresponding to historical point cloud frame training samples from its respective point cloud frame sequence data stream. Because the corresponding point cloud frame sequence data stream is typically for the same scene, fusing information from the same scene and then training the model based on this allows the model to learn more effective and accurate information, resulting in better training performance.

[0072] Similarly, since the training data corresponding to the point cloud frame training samples in the same point cloud frame sequence data stream need to be fused together, during the training process, the same data augmentation processing needs to be performed on the point cloud frame training samples obtained from the same point cloud frame sequence data stream to avoid fusion bias.

[0073] For example, firstly, the point cloud data D is organized into sequences S1, ..., Sn; secondly, S1, ..., Sn is segmented according to a preset sequence length to obtain more smaller sequences s1, ..., sk; then, these smaller sequences are randomly recombinated to form new sequences. When training begins using the new sequences, the contents of the memory bank corresponding to the 3D object detection model are cleared. During training, the same data augmentation method is used for training data corresponding to the same sequence to maintain consistency in the random state and ensure the consistency of data augmentation.

[0074] The above process achieves model training on a per-point cloud frame sequence basis, but it is essentially still single-frame training. Therefore, the model parameters used for training each point cloud frame may differ, potentially leading to inconsistencies between the features used in the training and inference phases. To address this, in one feasible approach, a point cloud frame sequence of a first sequence length is used as the current point cloud frame training sample during initial training. Subsequently, based on the difference between the model's loss value and a preset threshold, or based on the number of training iterations, the sequence length of the point cloud frame sequence corresponding to the training sample is increased according to a preset sequence growth interval. This is because, as training progresses, the changes in model parameters may become smaller. Therefore, using a longer point cloud frame sequence for model training will not produce significant bias and is more consistent with actual 3D object detection scenarios. The first sequence length, preset threshold, number of training iterations, and preset sequence growth interval can all be set by those skilled in the art according to actual needs.

[0075] For example, at the start of training, the sequence length is set to 1, and as training progresses, the sequence length gradually increases. In a simple example, assuming the model uses 10,000 training samples and iterates 100 times (10,000 * 100 rounds), the sequence length for the first 1-25 iterations can be 1, meaning single-frame training; the sequence length for iterations 26-75 can be 10, meaning 10 point cloud training samples are trained together in one round; and the sequence length for iterations 76-100 can be 30-50, meaning 30-50 point cloud training samples are trained together in one round.

[0076] Through the above process, the model can be trained with infinite frame data expansion and with dynamic training sequence length.

[0077] Step S304: During the training of the 3D object detection model using the current point cloud frame training samples, historical data corresponding to the historical point cloud frame training samples are obtained from the memory region set for the 3D object detection model and independent of the 3D object detection model.

[0078] Step S306: Fuse the historical data with the training data corresponding to the current point cloud frame training sample, and perform model training for the current round based on the fusion result.

[0079] The specific implementation of steps S304-S306 can be referred to the description of the corresponding part of the aforementioned embodiment one, and will not be described in detail here.

[0080] As can be seen, this embodiment uses a pre-defined memory region for the 3D object detection model to store the training data corresponding to the historical point cloud frame training samples and performs iterative updates. This method has a small memory footprint and does not require repeated calculations of historical training data. Furthermore, during backpropagation, the gradient is not transferred to the memory region, avoiding optimization difficulties and achieving lightweight training. In addition, by generating a point cloud frame sequence data stream and training the model in units of point cloud frame sequences, the length of the training sequence can be infinitely expanded to achieve infinite frame data fusion. Moreover, by training with a dynamic training sequence length, the length of the point cloud frame sequence of the training samples can be variable, thereby solving the problem of inconsistency between training and inference caused by model autoregression.

[0081] Example 3

[0082] Reference Figure 4 The diagram shows a structural schematic of an electronic device according to Embodiment 3 of this application. The specific embodiments of this application do not limit the specific implementation of the electronic device.

[0083] like Figure 4As shown, the electronic device may include: a processor 402, a communications interface 404, a memory 406, and a communications bus 408.

[0084] in:

[0085] The processor 402, communication interface 404, and memory 406 communicate with each other via communication bus 408.

[0086] Communication interface 404 is used to communicate with other electronic devices or servers.

[0087] The processor 402 is used to execute program 410, specifically the relevant steps in the above method embodiments.

[0088] Specifically, program 410 may include program code that includes computer operation instructions.

[0089] Processor 402 may be a CPU, an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. The smart device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0090] Memory 406 is used to store program 410. Memory 406 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0091] Specifically, program 410 can be used to cause processor 4502 to perform the operation corresponding to the method described in any of the foregoing multiple method embodiments.

[0092] The specific implementation of each step in procedure 410 can be found in the corresponding descriptions of the steps and units in the above method embodiments, and has corresponding beneficial effects, which will not be repeated here. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding process descriptions in the foregoing method embodiments, and will not be repeated here.

[0093] This application also provides a computer program product, including computer instructions that instruct a computing device to perform an operation corresponding to any of the methods in the above-described multiple method embodiments.

[0094] It should be noted that, depending on the implementation needs, the various components / steps described in the embodiments of this application can be broken down into more components / steps, or two or more components / steps or parts of the operation of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of this application.

[0095] The methods described in the embodiments of this application can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code downloaded over a network that is originally stored in a remote recording medium or a non-transitory machine-readable medium and will be stored in a local recording medium. Thus, the methods described herein can be processed by software stored on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or FPGA). It is understood that the computer, processor, microprocessor controller, or programmable hardware includes storage components (e.g., RAM, ROM, flash memory, etc.) capable of storing or receiving software or computer code that, when accessed and executed by the computer, processor, or hardware, implements the methods described herein. Furthermore, when a general-purpose computer accesses code used to implement the methods shown herein, the execution of the code transforms the general-purpose computer into a dedicated computer for executing the methods shown herein.

[0096] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application.

[0097] The above embodiments are only used to illustrate the embodiments of this application, and are not intended to limit the embodiments of this application. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of this application. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of this application, and the patent protection scope of the embodiments of this application should be defined by the claims.

Claims

1. A method for training a three-dimensional object detection model, comprising: Obtain the current point cloud frame training sample; During the training of the 3D target detection model using the current point cloud frame training samples, historical data corresponding to multiple preset model layer positions of the historical point cloud frame training samples are obtained from a memory region set for the 3D target detection model and independent of the 3D target detection model. The historical data is fused with the training data corresponding to the current point cloud frame training sample, and the model is trained in the current round based on the fusion result. The step of acquiring historical data corresponding to multiple preset model layer positions of historical point cloud frame training samples includes: Determine the data type of the training data corresponding to the current point cloud frame training sample at the current model layer position; Historical data of historical point cloud frame training samples are obtained from the data storage structure of the memory region corresponding to the data type.

2. The method according to claim 1, wherein, The step of fusing the historical data with the training data corresponding to the current point cloud frame training sample includes: fusing the corresponding historical data with the training data corresponding to the current point cloud frame training sample at the preset multiple model layer positions.

3. The method according to claim 1, wherein, The data types include: point types for indicating unordered data and image types for indicating regularized data.

4. The method according to claim 3, wherein, When the data type is a point type, the step of fusing the historical data with the training data corresponding to the current point cloud frame training sample includes: The training data of the point type corresponding to the current point cloud frame training sample is concatenated with the historical data of the corresponding point type.

5. The method according to claim 3, wherein, When the data type is an image, the step of fusing the historical data with the training data corresponding to the current point cloud frame training sample includes: The training data of the image type corresponding to the current point cloud frame training sample is summed, concatenated, or fused with the historical data of the corresponding image type by means of maximum value or gating.

6. The method according to any one of claims 1-5, wherein, The method further includes: Based on the training data corresponding to the current point cloud frame training sample, the historical data corresponding to the historical point cloud frame training sample in the memory region is iteratively updated.

7. The method according to any one of claims 1-5, wherein, The preset multiple model layer positions include: input layer, 3D point cloud feature extraction layer, and 3D target prediction layer.

8. The method according to any one of claims 1-5, wherein, The step of obtaining the current point cloud frame training sample includes: Obtain the point cloud frame sequence data stream; The point cloud frame sequence data stream is segmented according to a preset sequence length to obtain multiple sub-sequences; Multiple subsequences are randomly recombinated to generate a new point cloud frame sequence data stream; Obtain the current point cloud frame training sample from the new point cloud frame sequence data stream.

9. The method according to claim 8, wherein, The method further includes: The memory region is cleared so that the three-dimensional target detection model can be trained using the cleared memory region and the point cloud frame training samples in the new point cloud frame sequence data stream.

10. The method according to claim 8, wherein, The method further includes: For point cloud frame training samples obtained from the same point cloud frame sequence data stream, the same data augmentation processing is performed during the training process.

11. The method according to any one of claims 1-5, wherein, The step of obtaining the current point cloud frame training sample includes: During initial training, a sequence of point cloud frames of the first sequence length is used as the training sample for the current point cloud frame. Based on the difference between the loss value and the preset threshold, or based on the number of training iterations, the sequence length of the point cloud frame sequence corresponding to the point cloud frame training samples used to train the model is increased according to the preset sequence growth interval.

12. An electronic device, comprising: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction that causes the processor to perform the operation corresponding to the method as described in any one of claims 1-11.

13. A computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any one of claims 1-11.

Images