Point cloud data processing method and device, equipment and storage medium

Abstract

The disclosure provides a point cloud data processing method, device and equipment and a storage medium, which can be applied to the fields of artificial intelligence, image processing and intelligent driving. The point cloud data processing method comprises: inputting an initial point cloud data set representing a target object into a data enhancement strategy prediction model, and outputting a target strategy identifier, wherein the data enhancement strategy prediction model is matched with a target object detection model used for detecting the target object; selecting a target data enhancement strategy corresponding to the target strategy identifier from N data enhancement strategies, wherein the target data enhancement strategy is used for data enhancement on the initial point cloud data set, and N is a positive integer; and processing the initial point cloud data set based on a target data enhancement operation in the target data enhancement strategy and a target data enhancement parameter corresponding to the target data enhancement operation, to obtain a target point cloud data set.

Description

Technical Field

[0001] This disclosure relates to the fields of artificial intelligence, image processing, and intelligent driving, and more specifically, to a point cloud data processing method, apparatus, device, storage medium, and program product. Background Technology

[0002] With the rapid development of technology, more and more vehicles are equipped with autonomous driving assistance systems to enhance the convenience of driving. This places higher demands on the reliability of these systems. In related technologies, autonomous driving assistance systems can process point cloud data detected by relevant detection devices based on intelligent algorithm models such as neural network models. This allows them to identify target objects around the vehicle based on the processing results of the point cloud data, thereby achieving related assisted driving functions.

[0003] In realizing the concept disclosed herein, the inventors discovered that the related technologies have at least the following problems: the quantity and / or quality of data used to train the related intelligent algorithm model will have a significant impact on the recognition accuracy of the intelligent algorithm model, and the related technologies usually cannot provide data that can meet the training requirements, which makes it difficult for the trained intelligent algorithm model to effectively identify the target object. Summary of the Invention

[0004] In view of this, the present disclosure provides a point cloud data processing method, apparatus, device, storage medium, and program product.

[0005] One aspect of this disclosure provides a point cloud data processing method, comprising:

[0006] The initial point cloud dataset representing the target object is input into the data augmentation strategy prediction model, and the target strategy identifier is output. The data augmentation strategy prediction model is matched with the target object detection model used to detect the target object.

[0007] Select the target data augmentation strategy corresponding to the target strategy identifier from N data augmentation strategies, where the target data augmentation strategy is used to augment the initial point cloud dataset, and N is a positive integer; and

[0008] Based on the target data augmentation operation in the above target data augmentation strategy, and the target data augmentation parameters corresponding to the above target data augmentation operation, the above initial point cloud dataset is processed to obtain the target point cloud dataset.

[0009] According to embodiments of this disclosure, the above-mentioned data augmentation strategy prediction model is trained using a collaborative training method, which includes:

[0010] The sample point cloud dataset is input into the untrained candidate data augmentation strategy prediction model so that, based on the sample candidate augmentation strategy identifier output by the above candidate data augmentation strategy prediction model, a candidate data augmentation strategy corresponding to the above sample candidate augmentation strategy identifier is selected from N data augmentation strategies.

[0011] The above data augmentation strategy is used to process the above sample point cloud dataset to obtain a data-augmented sample candidate point cloud dataset.

[0012] Using the aforementioned candidate point cloud dataset, a candidate object detection model matching the aforementioned candidate data augmentation strategy prediction model is trained, resulting in a new trained candidate object detection model; and

[0013] Based on the evaluation metrics of the new candidate target object detection model and the difference information between the evaluation metrics of the sample target, the model parameters of the candidate data augmentation strategy prediction model are adjusted to obtain the trained data augmentation strategy prediction model.

[0014] According to embodiments of this disclosure, the aforementioned sample point cloud dataset includes L subsets of sample point cloud data, where L is a positive integer. These L subsets are input into the aforementioned candidate data augmentation strategy prediction model in batches. The aforementioned collaborative training method further includes:

[0015] Iteratively input the m-th sample point cloud data subset of the m-th batch into the current candidate data augmentation strategy prediction model from the L subsets of the above sample point cloud data to obtain the sample candidate augmentation strategy identifier output by the current candidate data augmentation strategy prediction model, where L≥M≥1;

[0016] Using the candidate data augmentation strategy corresponding to the current sample candidate augmentation strategy identifier from among the N data augmentation strategies described above, the above-mentioned m-th sample point cloud data subset is processed to obtain the data-augmented m-th sample candidate point cloud dataset.

[0017] Based on the above m-th sample candidate point cloud dataset, train the current candidate target object detection model to obtain the new candidate target object detection model after training.

[0018] Based on the m-th evaluation index of the new candidate target object detection model and the evaluation index difference information between the sample target evaluation index, the model parameters of the current candidate data augmentation strategy prediction model are iteratively adjusted to obtain a new candidate data prediction model.

[0019] The candidate data prediction model corresponding to the convergence of the above evaluation index difference information is determined as the above data augmentation strategy prediction model.

[0020] According to embodiments of this disclosure, the target data augmentation operation described above includes at least one of the following:

[0021] Rotate, scale, move, flip, and mirror operations.

[0022] According to embodiments of this disclosure, the above-described data augmentation strategy prediction model includes a model built based on a neural network algorithm.

[0023] According to embodiments of this disclosure, the model constructed based on the neural network algorithm described above includes at least one of the following:

[0024] Recurrent Neural Network (RNN) Model, Long Short-Term Memory (LSTM) Model, Gated Recurrent Neural Network (GRN) Model, Backpropagation Network Model, Fully Connected Neural Network Model.

[0025] According to embodiments of this disclosure, the point cloud data processing method further includes:

[0026] Based on the aforementioned target point cloud dataset, target detection is performed on the aforementioned target objects to obtain the detection results for the aforementioned target objects.

[0027] Another aspect of this disclosure provides a method for training a data augmentation strategy prediction model, comprising:

[0028] Obtain training samples, wherein the training samples include a sample point cloud dataset, sample labels corresponding to the sample point cloud dataset, and sample target evaluation metrics, wherein the sample point cloud dataset is used to represent part or all of the sample target objects; and

[0029] An untrained candidate data augmentation strategy prediction model is trained using a collaborative training method to obtain a trained data augmentation strategy prediction model. The collaborative training method includes:

[0030] The sample point cloud dataset is input into the untrained candidate data augmentation strategy prediction model so that, based on the sample candidate augmentation strategy identifier output by the above candidate data augmentation strategy prediction model, a candidate data augmentation strategy corresponding to the above sample candidate augmentation strategy identifier is selected from N data augmentation strategies, where N is a positive integer;

[0031] The above data augmentation strategy is used to process the above sample point cloud dataset to obtain the data-augmented sample candidate point cloud dataset.

[0032] Using the aforementioned candidate point cloud dataset and sample labels, a candidate object detection model matching the aforementioned candidate data augmentation strategy prediction model is trained, resulting in a new, trained candidate object detection model; and

[0033] Based on the evaluation metrics of the new candidate target object detection model and the difference information between the evaluation metrics of the sample target, the model parameters of the candidate data augmentation strategy prediction model are adjusted to obtain the trained data augmentation strategy prediction model.

[0034] The aforementioned data augmentation strategy prediction model is used based on the point cloud data processing method described above.

[0035] According to embodiments of this disclosure, the aforementioned sample point cloud dataset includes L subsets of sample point cloud data, where L is a positive integer. These L subsets are input into the aforementioned candidate data augmentation strategy prediction model in batches. The aforementioned collaborative training method further includes:

[0036] Iteratively input the m-th sample point cloud data subset of the m-th batch into the current candidate data augmentation strategy prediction model from the L subsets of the above sample point cloud data to obtain the sample candidate augmentation strategy identifier output by the current candidate data augmentation strategy prediction model, where L≥M≥1;

[0037] Using the candidate data augmentation strategy corresponding to the current sample candidate augmentation strategy identifier from among the N data augmentation strategies described above, the above-mentioned m-th sample point cloud data subset is processed to obtain the data-augmented m-th sample candidate point cloud dataset.

[0038] Based on the above m-th sample candidate point cloud dataset, train the current candidate target object detection model to obtain the new candidate target object detection model after training.

[0039] Based on the m-th evaluation index of the new candidate target object detection model and the evaluation index difference information between the sample target evaluation index, the model parameters of the current candidate data augmentation strategy prediction model are iteratively adjusted to obtain a new candidate data prediction model.

[0040] The candidate data prediction model corresponding to the convergence of the above evaluation index difference information is determined as the above data augmentation strategy prediction model.

[0041] According to embodiments of this disclosure, the model parameters of the current candidate data augmentation strategy prediction model are iteratively adjusted based on the difference information between the m-th evaluation index of the new candidate target object detection model and the evaluation index of the sample target:

[0042] The evaluation index m above is processed according to the preset training algorithm, and the evaluation index difference information between the above sample target evaluation index is processed, so as to iteratively adjust the model parameters of the current candidate data prediction model and obtain a new candidate data prediction model.

[0043] The aforementioned preset training algorithm includes at least one of the following: policy gradient algorithm, near-end policy optimization algorithm, and confidence region policy optimization algorithm.

[0044] According to embodiments of this disclosure, the above evaluation metrics include at least one of the following:

[0045] Mean precision, accuracy, and recall.

[0046] Another aspect of this disclosure also provides a point cloud data processing apparatus, comprising:

[0047] The strategy label prediction module is used to input the initial point cloud dataset representing the target object into the data augmentation strategy prediction model and output the target strategy label, wherein the data augmentation strategy prediction model is matched with the target object detection model used to detect the target object.

[0048] The strategy selection module is used to select a target data augmentation strategy corresponding to the aforementioned target strategy identifier from N data augmentation strategies, wherein the aforementioned target data augmentation strategy is used to augment the aforementioned initial point cloud dataset, and N is a positive integer; and

[0049] The data augmentation module is used to process the initial point cloud dataset based on the target data augmentation operation in the target data augmentation strategy and the target data augmentation parameters corresponding to the target data augmentation operation, to obtain the target point cloud dataset.

[0050] Another aspect of this disclosure provides a training apparatus for a data augmentation strategy prediction model, comprising:

[0051] The sample acquisition module is used to acquire training samples, wherein the training samples include a sample point cloud dataset, sample labels corresponding to the sample point cloud dataset, and sample target evaluation metrics. The sample point cloud dataset is used to represent part or all of the sample target objects, and L is a positive integer; and

[0052] The training module is used to train an untrained candidate data augmentation strategy prediction model using a collaborative training method, resulting in a trained data augmentation strategy prediction model. The collaborative training method includes:

[0053] The sample point cloud dataset is input into the untrained candidate data augmentation strategy prediction model so that, based on the sample candidate augmentation strategy identifier output by the above candidate data augmentation strategy prediction model, a candidate data augmentation strategy corresponding to the above sample candidate augmentation strategy identifier is selected from N data augmentation strategies.

[0054] The above data augmentation strategy is used to process the above sample point cloud dataset to obtain a data-augmented sample candidate point cloud dataset.

[0055] Using the aforementioned candidate point cloud dataset and sample labels, a candidate object detection model matching the aforementioned candidate data augmentation strategy prediction model is trained, resulting in a new, trained candidate object detection model; and

[0056] Based on the evaluation metrics of the new candidate target object detection model and the difference information between the evaluation metrics of the sample target, the model parameters of the candidate data augmentation strategy prediction model are adjusted to obtain the trained data augmentation strategy prediction model.

[0057] The aforementioned data augmentation strategy prediction model is used in the aforementioned point cloud data processing method.

[0058] Another aspect of this disclosure provides an electronic device comprising:

[0059] One or more processors;

[0060] Memory, used to store one or more programs.

[0061] When the one or more programs are executed by the one or more processors, the one or more processors implement the method described above.

[0062] Another aspect of this disclosure provides a computer-readable storage medium storing computer-executable instructions that, when executed, are used to implement the method described above.

[0063] Another aspect of this disclosure provides a computer program product including computer-executable instructions that, when executed, are used to implement the method described above.

[0064] According to embodiments of this disclosure, by utilizing a data augmentation strategy prediction model that matches the target object detection model, a target data augmentation strategy suitable for processing the initial point cloud dataset is selected. Appropriate target data augmentation strategies are selectively chosen to augment the initial point cloud data, thereby improving the adaptability of the target point cloud data to the subsequent target object detection model. This at least partially overcomes the technical problem in related technologies where augmented point cloud data is difficult to adapt to the subsequent target object detection process, achieving the technical effect of improving the data augmentation quality of point cloud data. Attached Figure Description

[0065] The above and other objects, features and advantages of this disclosure will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0066] Figure 1This illustration schematically shows an exemplary system architecture to which point cloud data processing methods and apparatus can be applied according to embodiments of the present disclosure;

[0067] Figure 2 A flowchart illustrating a point cloud data processing method according to an embodiment of the present disclosure is shown schematically.

[0068] Figure 3 A flowchart illustrating a collaborative training method according to an embodiment of the present disclosure is shown schematically;

[0069] Figure 4 This diagram illustrates an application scenario of the point cloud data processing method according to an embodiment of the present disclosure.

[0070] Figure 5 A flowchart illustrating a training method for a data augmentation strategy prediction model according to an embodiment of the present disclosure is shown schematically.

[0071] Figure 6 The diagram illustrates an application scenario of a training method for a data augmentation strategy prediction model according to an embodiment of the present disclosure.

[0072] Figure 7 A block diagram of a point cloud data processing apparatus according to an embodiment of the present disclosure is shown schematically.

[0073] Figure 8 A block diagram illustrating a training apparatus for a data augmentation strategy prediction model according to embodiments of the present disclosure is shown schematically; and

[0074] Figure 9 The diagram illustrates an electronic device suitable for implementing the point cloud data processing method and the training method for the data augmentation strategy prediction model described above, according to embodiments of the present disclosure. Detailed Implementation

[0075] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0076] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0077] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0078] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0079] In the technical solution disclosed herein, the user's authorization or consent is obtained before acquiring or collecting the user's personal information.

[0080] In the technical solution disclosed herein, the collection, storage, use, processing, transmission, provision, disclosure, and application of user personal information comply with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and there is no violation of public order and good morals.

[0081] In applications such as autonomous driving assistance and automated vehicle inspection, point cloud data representing objects around the vehicle is typically acquired using target object detection devices such as LiDAR and millimeter-wave radar. Relevant target object detection models, such as the SMOKE (Single-Stage Monocular 3D Object Detection via Keypoint Estimation) model, are then used to process the point cloud data and detect objects in the space surrounding the vehicle. However, the performance of these target detection models or algorithms often depends on the quantity and quality of the sample point cloud data used in training. The quantity and quality of the point cloud data also affect the accuracy and precision of the target object detection results. To improve the quality and quantity of point cloud data, related technologies often involve acquiring large amounts of point cloud data. However, point cloud data acquisition is costly and easily introduces class imbalance.

[0082] Data augmentation techniques can significantly increase the quantity and quality of point cloud training sets. However, existing point cloud data augmentation methods typically involve selecting suitable data augmentation operations and parameters through several sets of comparative experiments. This approach is time-consuming, and manually selected methods rarely achieve optimal results. Furthermore, it usually requires highly experienced personnel, making it difficult to guarantee that the augmented point cloud dataset will meet practical application requirements.

[0083] Embodiments of this disclosure provide a point cloud data processing method, apparatus, device, storage medium, and program product. The point cloud data processing method includes:

[0084] An initial point cloud dataset representing the target object is input into a data augmentation strategy prediction model, which outputs a target strategy identifier. The data augmentation strategy prediction model matches the target object detection model used to detect the target object. A target data augmentation strategy corresponding to the target strategy identifier is selected from N data augmentation strategies. The target data augmentation strategy is used to augment the initial point cloud dataset, where N is a positive integer. The initial point cloud dataset is then processed based on the target data augmentation operation in the target data augmentation strategy and the target data augmentation parameters corresponding to the target data augmentation operation to obtain the target point cloud dataset.

[0085] According to embodiments of this disclosure, by utilizing a data augmentation strategy prediction model that matches the target object detection model, a target data augmentation strategy suitable for processing the initial point cloud dataset is selected. Appropriate target data augmentation strategies are selectively chosen to augment the initial point cloud data, thereby improving the adaptability of the target point cloud data to the subsequent target object detection model. This at least partially overcomes the technical problem in related technologies where augmented point cloud data is difficult to adapt to the subsequent target object detection process, achieving the technical effect of improving the data augmentation quality of point cloud data. This lays the foundation for subsequently improving the target object detection effect and enhancing the training effect of the intelligent algorithm model for identifying target objects.

[0086] Figure 1 This illustration schematically depicts an exemplary system architecture to which point cloud data processing methods and apparatus can be applied according to embodiments of this disclosure. It should be noted that... Figure 1 The examples shown are merely examples of system architectures that can be applied to the embodiments of this disclosure, in order to help those skilled in the art understand the technical content of this disclosure, but do not mean that the embodiments of this disclosure cannot be used in other devices, systems, environments or scenarios.

[0087] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, and 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the terminal devices 101, 102, and 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0088] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software, etc. (for example only).

[0089] Terminal devices 101, 102, and 103 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0090] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using terminal devices 101, 102, and 103 (for example only). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0091] It should be noted that the point cloud data processing method provided in this embodiment can generally be executed by server 105. Correspondingly, the point cloud data processing device provided in this embodiment can generally be located in server 105. The point cloud data processing method provided in this embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105. Correspondingly, the point cloud data processing device provided in this embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105. Alternatively, the point cloud data processing method provided in this embodiment can also be executed by terminal devices 101, 102, or 103, or by other terminal devices different from terminal devices 101, 102, or 103. Accordingly, the point cloud data processing device provided in this embodiment of the present disclosure may also be disposed in terminal device 101, 102 or 103, or disposed in other terminal devices different from terminal device 101, 102 or 103.

[0092] For example, the initial point cloud dataset may originally be stored in any one of terminal devices 101, 102, or 103 (e.g., terminal device 101, but not limited thereto), or it may be stored on an external storage device and imported into terminal device 101. Then, terminal device 101 may execute the point cloud data processing method provided in the embodiments of this disclosure locally, or send the initial point cloud dataset to other terminal devices, servers, or server clusters, and have the other terminal devices, servers, or server clusters that receive the initial point cloud dataset execute the point cloud data processing method provided in the embodiments of this disclosure.

[0093] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0094] Figure 2 A flowchart illustrating a point cloud data processing method according to an embodiment of the present disclosure is shown.

[0095] like Figure 2 As shown, the point cloud data processing method includes operations S210~S230.

[0096] In operation S210, the initial point cloud dataset representing the target object is input into the data augmentation strategy prediction model, and the target strategy identifier is output, wherein the data augmentation strategy prediction model is matched with the target object detection model used to detect the target object.

[0097] According to embodiments of this disclosure, the initial point cloud dataset may include data obtained after detecting a target object using a relevant target object detection device. The detection device may include lidar, millimeter-wave radar, binocular camera, etc. Embodiments of this disclosure do not limit the specific acquisition method and acquisition device for the initial point cloud data, and those skilled in the art can select according to actual needs.

[0098] In operation S220, a target data augmentation strategy corresponding to the target strategy identifier is selected from N data augmentation strategies. The target data augmentation strategy can be used to augment the initial point cloud dataset, and N is a positive integer.

[0099] In operation S230, the initial point cloud dataset is processed based on the target data augmentation operation in the target data augmentation strategy and the target data augmentation parameters corresponding to the target data augmentation operation to obtain the target point cloud dataset.

[0100] According to embodiments of this disclosure, the data augmentation strategy prediction model can be a model built based on relevant prediction algorithms. For example, it can be built based on time-series prediction algorithms, but it is not limited to this. It can also be built based on neural network algorithms. The embodiments of this disclosure do not limit the specific construction method of the data augmentation strategy prediction model, as long as it can predict the target data augmentation strategy used to process the initial point cloud dataset.

[0101] According to embodiments of this disclosure, the data augmentation strategy prediction model can combine the data processing characteristics of the target object detection model with the data characteristics of the initial point cloud dataset to select a target data augmentation strategy that is suitable for the initial point cloud dataset and adapted to the target object detection model.

[0102] It should be understood that the target data augmentation operation can include the operations used to augment the initial point cloud dataset within the target data augmentation strategy, such as flipping operations, moving operations, etc., in related technologies. The target data augmentation parameters can include the operation attribute parameters of the target data augmentation operation, such as flipping angle parameters, moving direction, moving distance, etc.

[0103] It should be noted that the embodiments of this disclosure do not limit the number of data augmentation strategies, and the number of sets N can be one or more. Similarly, the number of data augmentation operations in each data augmentation strategy can also be one or more. The embodiments of this disclosure do not limit the data augmentation strategies, nor the number of data augmentation operations contained in each data augmentation strategy, and those skilled in the art can select according to actual needs.

[0104] According to embodiments of this disclosure, by utilizing a data augmentation strategy prediction model that matches the target object detection model, a target data augmentation strategy suitable for processing the initial point cloud dataset is selected. Appropriate target data augmentation strategies are selectively chosen to augment the initial point cloud data, thereby improving the adaptability of the target point cloud data to the subsequent target object detection model. This at least partially overcomes the technical problem in related technologies where augmented point cloud data is difficult to adapt to the subsequent target object detection process, achieving the technical effect of improving the data augmentation quality of point cloud data. This lays the foundation for subsequently improving the target object detection effect and enhancing the training effect of the intelligent algorithm model for identifying target objects.

[0105] According to embodiments of this disclosure, the data augmentation strategy prediction model includes a model built based on a neural network algorithm.

[0106] According to embodiments of this disclosure, after processing the initial point cloud dataset, a data augmentation strategy prediction model constructed based on a neural network algorithm can output prediction results. These prediction results can be the probability values ​​corresponding to each of the N data augmentation strategies, thereby determining the corresponding target strategy identifier based on the prediction results.

[0107] According to embodiments of this disclosure, the model constructed based on a neural network algorithm includes at least one of the following:

[0108] Recurrent neural network model, long short-term memory model, gated recurrent neural network model, backpropagation network model, fully connected neural network model.

[0109] In one embodiment of this disclosure, a data augmentation strategy prediction model can be constructed based on a recurrent neural network model. This allows for the full extraction of temporal features from the initial point cloud dataset, and the prediction of target strategy identifiers based on these temporal features. This improves the accuracy of target strategy identifier prediction and enables the selection of a target data augmentation strategy suitable for augmenting the initial point cloud dataset. This enhances the data quality of the target point cloud dataset, enabling it to meet performance requirements such as training requirements and target object detection requirements.

[0110] According to embodiments of this disclosure, the target data augmentation operation includes at least one of the following:

[0111] Rotate, scale, move, flip, and mirror operations.

[0112] In one embodiment of this disclosure, among the N data augmentation strategies, each data augmentation strategy may include two data operations, and each data operation may correspond to data augmentation parameters. By performing various data operations on the initial point cloud dataset, multi-faceted data augmentation of the initial point cloud dataset can be achieved, thereby comprehensively improving the data quality of the target point cloud dataset from multiple data augmentation aspects.

[0113] According to embodiments of this disclosure, the data augmentation strategy prediction model is trained using a collaborative training method, which can be referred to as follows: Figure 3 The operation steps shown are implemented.

[0114] Figure 3 A flowchart illustrating a collaborative training method according to an embodiment of the present disclosure is shown schematically.

[0115] like Figure 3 As shown, the collaborative training method in this embodiment may include operations S310 to S340.

[0116] In operation S310, the sample point cloud dataset is input into the untrained candidate data augmentation strategy prediction model so that, based on the sample candidate augmentation strategy identifier output by the candidate data augmentation strategy prediction model, a candidate data augmentation strategy corresponding to the sample candidate augmentation strategy identifier is selected from N data augmentation strategies.

[0117] When operating S320, data augmentation strategies are used to process the sample point cloud dataset to obtain a data-augmented sample candidate point cloud dataset.

[0118] In operation S330, a candidate target object detection model matching the candidate data augmentation strategy prediction model is trained using the sample candidate point cloud dataset, resulting in a new candidate target object detection model after training.

[0119] In operation S340, based on the evaluation metrics of the new candidate target object detection model and the difference information between the evaluation metrics of the sample target, the model parameters of the candidate data augmentation strategy prediction model are adjusted to obtain the trained data augmentation strategy prediction model.

[0120] According to embodiments of this disclosure, the collaborative training method can be to coordinate the training process of the candidate data augmentation model and the training process of the candidate target object detection model. For example, it can be achieved through the aforementioned collaborative training method: first, a candidate data augmentation strategy is selected based on the candidate data augmentation strategy prediction model; then, the candidate target object detection model is trained on the sample target point cloud dataset after data augmentation using the candidate data augmentation strategy; and finally, the candidate data augmentation strategy prediction model is penalized based on the difference between the evaluation metric of the newly trained candidate target object detection model and the sample target evaluation metric, thereby adjusting the model parameters of the candidate data augmentation strategy prediction model. This collaborative training method can be iteratively executed using the sample point cloud dataset until the evaluation metric of the current candidate target object detection model meets the relevant requirements. Thus, a trained data augmentation strategy prediction model can be obtained during the collaborative training process. Correspondingly, the candidate target object detection model obtained with the trained data augmentation strategy prediction model can also be used to detect corresponding target objects, thereby obtaining a trained target object detection model. This achieves collaborative training of the two models, improving the robustness of each model and correspondingly increasing training efficiency.

[0121] According to embodiments of this disclosure, the evaluation metrics include at least one of the following:

[0122] Mean precision, accuracy, and recall.

[0123] According to embodiments of this disclosure, any one or more of the mean precision, accuracy, and recall can be selected as evaluation metrics to improve the robustness of the data augmentation strategy prediction model and the target object detection model, as well as to improve their respective precision and accuracy.

[0124] It should be noted that the evaluation metrics can be determined by using a portion of the sample point cloud dataset as validation data, thereby allowing the evaluation metrics of the candidate target object detection model to be determined based on the validation data.

[0125] According to embodiments of this disclosure, the sample point cloud dataset includes L subsets of sample point cloud data, where L is a positive integer. The L subsets of sample point cloud data are input into the candidate data augmentation strategy prediction model in batches, and the collaborative training method may further include the following operations:

[0126] Iteratively, the m-th sample point cloud data subset from the m-th batch of L sample point cloud data subsets is input into the current candidate data augmentation strategy prediction model to obtain the sample candidate augmentation strategy identifier output by the current candidate data augmentation strategy prediction model, where L≥M≥1; the m-th sample point cloud data subset is processed using the candidate data augmentation strategy corresponding to the current sample candidate augmentation strategy identifier from among N data augmentation strategies to obtain the data-augmented m-th sample candidate point cloud dataset; the current candidate target object detection model is trained based on the m-th sample candidate point cloud dataset to obtain a new candidate target object detection model after training; the model parameters of the current candidate data augmentation strategy prediction model are iteratively adjusted based on the difference information between the m-th evaluation index of the new candidate target object detection model and the evaluation index of the sample target to obtain a new candidate data prediction model; the candidate data prediction model corresponding to the convergence of the evaluation index difference information is determined as the data augmentation strategy prediction model.

[0127] According to embodiments of this disclosure, by inputting L subsets of sample point cloud data into the candidate data augmentation strategy prediction model in batches, the diversity of training data used to train the candidate data augmentation strategy prediction model and the candidate target object detection model can be improved while making full use of the sample point cloud data.

[0128] It should be noted that the sample point cloud data subset can be the point cloud data in one frame of probe image information, or it can be composed of point cloud data from multiple frames of probe image information. The embodiments of this disclosure do not limit the construction method of the sample point cloud data subset or the number of point cloud data. Those skilled in the art can design it according to actual needs.

[0129] According to embodiments of this disclosure, L initial point cloud data subsets can be input into the current candidate data augmentation strategy prediction model in batches based on their respective temporal characteristics to perform the above-described collaborative training method. This allows both the candidate data augmentation strategy prediction model and the candidate target object detection model to fully learn the temporal characteristics of the point cloud data. Alternatively, the initial point cloud data subsets can be input into the current candidate data augmentation strategy prediction model in batches based on other characteristics such as data quality and quantity. Embodiments of this disclosure do not limit the order in which the L sample point cloud data subsets are input into the candidate data augmentation strategy prediction model; those skilled in the art can design the model according to actual needs.

[0130] Figure 4 The diagram illustrates an application scenario of the point cloud data processing method according to an embodiment of the present disclosure.

[0131] like Figure 4 As shown, in this application scenario, the initial point cloud dataset 410 can be input into the data augmentation strategy prediction model 420. The data augmentation strategy prediction model 420 can output the target strategy identifier. Then, based on the target strategy identifier, the target data augmentation strategy can be determined as data augmentation strategy 431 from N data augmentation strategies 431, 432, ... to 43N.

[0132] The initial point cloud dataset 410 can be augmented using the target data augmentation operation and the target data augmentation parameters of the target data augmentation operation in the target data augmentation strategy 431, and then the augmented target point cloud dataset 440 can be obtained.

[0133] After the target point cloud dataset 440 is input into the target object detection model 450, the target object detection model 450 can output the detection result 460 for the target object. The detection result 460 can include the detection information such as the classification and location of the target detection box.

[0134] According to embodiments of this disclosure, the point cloud data processing method may further include the following operations:

[0135] Target detection is performed on the target object based on the target point cloud dataset to obtain the detection results for the target object.

[0136] According to embodiments of this disclosure, a target point cloud dataset can be input into a target object detection model to output the detection results of the target object, such as the location and classification of the target detection box. Alternatively, a target object detection model with the same or similar algorithmic attributes and detection characteristics as the target object detection model can be retrained based on the target point cloud dataset, and then the target object can be detected based on the newly trained target object detection model.

[0137] According to embodiments of this disclosure, the data quality and quantity of the target point cloud dataset can meet the relevant training and / or detection requirements, thereby achieving the technical effect of improving the accuracy and precision of the detection results for the target object.

[0138] Figure 5 A flowchart illustrating a training method for a data augmentation strategy prediction model according to an embodiment of the present disclosure is shown.

[0139] like Figure 5 As shown, the training method for this data augmentation strategy prediction model includes operations S510~S520.

[0140] In operation S510, training samples are obtained, including a sample point cloud dataset, sample labels corresponding to the sample point cloud dataset, and sample target evaluation metrics. The sample point cloud dataset is used to represent part or all of the sample target objects, and L is a positive integer.

[0141] In operation S520, a co-training method is used to train an untrained candidate data augmentation strategy prediction model, resulting in a trained data augmentation strategy prediction model. The co-training method includes the following operations:

[0142] The sample point cloud dataset is input into an untrained candidate data augmentation strategy prediction model. Based on the candidate augmentation strategy identifiers output by the model, a candidate data augmentation strategy corresponding to the identifier is selected from N data augmentation strategies. The sample point cloud dataset is then processed using the data augmentation strategy to obtain a data-augmented candidate point cloud dataset. Using the candidate point cloud dataset and sample labels, a candidate object detection model matching the candidate data augmentation strategy prediction model is trained, resulting in a new, trained candidate object detection model. Finally, based on the evaluation metrics of the new candidate object detection model and the difference between the new and sample object detection models, the model parameters of the candidate data augmentation strategy prediction model are adjusted, resulting in a trained data augmentation strategy prediction model. The data augmentation strategy prediction model is used in the aforementioned point cloud data processing method.

[0143] According to embodiments of this disclosure, the sample point cloud dataset can be point cloud data that requires data augmentation processing.

[0144] According to embodiments of this disclosure, the collaborative training method can be to coordinate the training process of the candidate data augmentation model and the training process of the candidate target object detection model. For example, it can be achieved through the aforementioned collaborative training method: first, a candidate data augmentation strategy is selected based on the candidate data augmentation strategy prediction model; then, the candidate target object detection model is trained on the sample target point cloud dataset after data augmentation using the candidate data augmentation strategy; and finally, the candidate data augmentation strategy prediction model is penalized based on the difference between the evaluation metric of the newly trained candidate target object detection model and the sample target evaluation metric, thereby adjusting the model parameters of the candidate data augmentation strategy prediction model. This collaborative training method can be iteratively executed using the sample point cloud dataset until the evaluation metric of the current candidate target object detection model meets the relevant requirements. Thus, a trained data augmentation strategy prediction model can be obtained during the collaborative training process. Correspondingly, the candidate target object detection model obtained with the trained data augmentation strategy prediction model can also be used to detect corresponding target objects, thereby obtaining a trained target object detection model. This achieves collaborative training of the two models, improving the robustness of each model and correspondingly increasing training efficiency.

[0145] According to embodiments of this disclosure, the evaluation metrics include at least one of the following:

[0146] Mean precision, accuracy, and recall.

[0147] According to embodiments of this disclosure, any one or more of the mean precision, accuracy, and recall can be selected as evaluation metrics to improve the robustness of the data augmentation strategy prediction model and the target object detection model, as well as to improve their respective precision and accuracy.

[0148] It should be noted that the evaluation metrics can be determined by using a portion of the sample point cloud dataset as validation data, thereby allowing the evaluation metrics of the candidate target object detection model to be determined based on the validation data.

[0149] In one embodiment of this disclosure, the evaluation metric may include the mAP metric (mean Average Precision).

[0150] The newly trained candidate object detection model needs to be evaluated using a validation point cloud dataset to obtain the mAP metric, which measures the performance of this new candidate object detection model. This involves processing the validation point cloud dataset with the trained new candidate object detection model to obtain the prediction results corresponding to the validation point cloud dataset. The prediction results for each frame of the validation point cloud dataset are then compared with the sample labels to calculate the mAP. For a well-trained data augmentation strategy prediction model, the selected data augmentation strategy is close to or even optimal. Correspondingly, the mAP metric of the newly trained candidate object detection model will be higher, thus converging the difference between the mAP metric of the new candidate object detection model and the sample object evaluation metric, resulting in a well-trained target object detection model.

[0151] In the initial training stage of the candidate data augmentation strategy prediction model, since the candidate data augmentation strategy prediction model is randomly initialized, the selected combination of data augmentation methods is also poor. As a result, the mAP corresponding to the candidate target object detection model in the initial training stage will also be low. Therefore, the robustness and prediction accuracy of the data augmentation prediction model can be improved by using a co-training method to train the candidate data augmentation strategy prediction model and the candidate target object detection model, thus laying the foundation for subsequent adaptive data augmentation of the initial point cloud dataset.

[0152] According to embodiments of this disclosure, the sample point cloud dataset includes L subsets of sample point cloud data, where L is a positive integer. The L subsets of sample point cloud data are input into the candidate data augmentation strategy prediction model in batches, and the collaborative training method further includes the following operations:

[0153] Iteratively, the m-th sample point cloud data subset from the m-th batch of L sample point cloud data subsets is input into the current candidate data augmentation strategy prediction model to obtain the sample candidate augmentation strategy identifier output by the current candidate data augmentation strategy prediction model, where L≥M≥1; the m-th sample point cloud data subset is processed using the candidate data augmentation strategy corresponding to the current sample candidate augmentation strategy identifier from among N data augmentation strategies to obtain the data-augmented m-th sample candidate point cloud dataset; the current candidate target object detection model is trained based on the m-th sample candidate point cloud dataset to obtain a new candidate target object detection model after training; the model parameters of the current candidate data augmentation strategy prediction model are iteratively adjusted based on the difference information between the m-th evaluation index of the new candidate target object detection model and the evaluation index of the sample target to obtain a new candidate data prediction model; the candidate data prediction model corresponding to the convergence of the evaluation index difference information is determined as the data augmentation strategy prediction model.

[0154] According to embodiments of this disclosure, by inputting L subsets of sample point cloud data into the candidate data augmentation strategy prediction model in batches, the diversity of training data used to train the candidate data augmentation strategy prediction model and the candidate target object detection model can be improved while making full use of the sample point cloud data.

[0155] According to embodiments of this disclosure, L initial point cloud data subsets can be input into the current candidate data augmentation strategy prediction model in batches based on their respective temporal characteristics to perform the above-described collaborative training method. This allows both the candidate data augmentation strategy prediction model and the candidate target object detection model to fully learn the temporal characteristics of the point cloud data. Alternatively, the initial point cloud data subsets can be input into the current candidate data augmentation strategy prediction model in batches based on other characteristics such as data quality and quantity. Embodiments of this disclosure do not limit the order in which the L sample point cloud data subsets are input into the candidate data augmentation strategy prediction model; those skilled in the art can design the model according to actual needs.

[0156] According to embodiments of this disclosure, iteratively adjusting the model parameters of the current candidate data augmentation strategy prediction model based on the difference information between the m-th evaluation index of the new candidate target object detection model and the evaluation index of the sample target may include the following operations:

[0157] The m-th evaluation index is processed according to the preset training algorithm, and the evaluation index difference information between it and the sample target evaluation index is used to iteratively adjust the model parameters of the current candidate data prediction model to obtain a new candidate data prediction model.

[0158] The preset training algorithm includes at least one of the following: Policy Gradient, Proximal Policy Optimization (PPO), and Trust Region Policy Optimization (TRPO).

[0159] According to embodiments of this disclosure, a candidate data augmentation strategy prediction model can be constructed based on a recurrent neural network (RNN) model, and the model parameters of the candidate data augmentation strategy prediction model can be iteratively optimized and adjusted based on a proximal policy optimization (PPO) algorithm.

[0160] The PPO algorithm is a novel policy gradient algorithm. Because policy gradient algorithms are highly sensitive to step size, but choosing an appropriate step size is difficult, large differences between the old and new policies during training can hinder the learning of data features from the point cloud dataset by the candidate data augmentation policy prediction model. PPO proposes a new objective function that allows for mini-batch updates through multiple training steps, solving the problem of determining the step size in policy gradient algorithms. Furthermore, the PPO algorithm is compatible with the training process of candidate data augmentation policy prediction models built on recurrent neural networks, improving the robustness of the resulting data augmentation policy prediction model.

[0161] In another embodiment of this disclosure, the TRPO algorithm can also be used for collaborative training, which has a faster computation speed during training and can save a certain amount of training time.

[0162] Figure 6 The diagram illustrates an application scenario of a training method for a data augmentation strategy prediction model according to an embodiment of the present disclosure.

[0163] like Figure 6 As shown, the application scenario of this embodiment may include L subsets of sample point cloud data, including a first subset 611, a second subset 612, ... up to the l-th subset 61L. N data augmentation prediction strategies may include candidate data augmentation strategies 631, 632, ... up to 63N. The L subsets of sample point cloud data can be used to train the candidate data augmentation strategy prediction model 620 and the candidate target object monitoring model 650 in batches.

[0164] For example, the first sample point cloud data subset 611 of the first batch can be input into the current candidate data augmentation strategy prediction model 620. The data augmentation strategy 631 corresponding to the candidate number of the sample candidate augmentation strategy identifier output by the current candidate data augmentation strategy prediction model 620 can process the first sample point cloud data subset 611 and output the data-augmented first candidate point cloud data subset 641. The first candidate point cloud data subset 641 and the corresponding sample labels can be used to train the current candidate target object detection model 650 to obtain a new candidate target object detection model 650 after training. The difference information between the first evaluation index 661 and the sample target evaluation index 671 of the new candidate target object detection model 650 is input into the preset training algorithm model 680, and the model parameters of the candidate data augmentation strategy prediction model 620 can be iteratively adjusted to obtain the next candidate data augmentation strategy prediction model 620.

[0165] Based on the same or similar collaborative training method, the second sample point cloud data subset 612 can be used as the second batch input into the candidate data augmentation strategy prediction model 620 after one training. Then, according to the candidate data augmentation strategy 632, the second sample point cloud data subset 612 is augmented. The second candidate point cloud data subset 642 obtained after data augmentation is used to train the candidate target object detection model 650 after one training, that is, the current candidate target object detection model 650 in the second batch training, and thus a new candidate target object detection model 650 and its corresponding second evaluation index 662 are obtained.

[0166] The second evaluation index 662 and the sample target evaluation index 671 are input into the preset training algorithm model, so that the current candidate data augmentation strategy prediction model 620 can be iteratively adjusted again.

[0167] By using the same or similar collaborative training methods, candidate data augmentation strategy prediction models can be trained in batches using L subsets of sample point cloud data until the difference information between the current m-th evaluation index and the sample target evaluation index 671 converges, thus obtaining the trained data augmentation strategy prediction model.

[0168] According to embodiments of this disclosure, the candidate data augmentation strategy prediction model can be constructed based on a recurrent neural network model. This candidate data augmentation strategy prediction model may include one or more recurrent neural network layers (i.e., hidden layers), and may also include an output layer constructed from activation functions (e.g., softmax functions). Each activation function can output a corresponding prediction probability, for example, 30 prediction probabilities. The candidate data augmentation strategy is determined using the policy identifier corresponding to each of the 30 prediction probabilities and a prediction probability threshold.

[0169] It should be noted that the training method of the data augmentation strategy prediction model in the embodiments of this disclosure can correspond to the point cloud data processing method in the above embodiments. The data augmentation strategy prediction model obtained after training in the embodiments of this disclosure can be used in the point cloud data processing method in the above embodiments.

[0170] Figure 7 A block diagram of a point cloud data processing apparatus according to an embodiment of the present disclosure is shown schematically.

[0171] like Figure 7 As shown, the point cloud data processing device 700 includes: a strategy identification prediction module 710, a strategy selection module 720, and a data augmentation module 730.

[0172] The strategy label prediction module 710 is used to input the initial point cloud dataset representing the target object into the data augmentation strategy prediction model and output the target strategy label, wherein the data augmentation strategy prediction model is matched with the target object detection model used to detect the target object.

[0173] The strategy selection module 720 is used to select the target data augmentation strategy corresponding to the target strategy identifier from N data augmentation strategies, wherein the target data augmentation strategy is used to augment the initial point cloud dataset, and N is a positive integer.

[0174] The data augmentation module 730 is used to process the initial point cloud dataset based on the target data augmentation operation in the target data augmentation strategy and the target data augmentation parameters corresponding to the target data augmentation operation, so as to obtain the target point cloud dataset.

[0175] According to embodiments of this disclosure, the data augmentation strategy prediction model is trained using a collaborative training method, which may include the following operations.

[0176] The sample point cloud dataset is input into an untrained candidate data augmentation strategy prediction model. Based on the sample candidate augmentation strategy identifier output by the candidate data augmentation strategy prediction model, a candidate data augmentation strategy corresponding to the sample candidate augmentation strategy identifier is selected from N data augmentation strategies. The sample point cloud dataset is then processed using the data augmentation strategy to obtain a data-augmented sample candidate point cloud dataset. A candidate target object detection model matching the candidate data augmentation strategy prediction model is trained using the sample candidate point cloud dataset to obtain a new trained candidate target object detection model. Finally, based on the evaluation metrics of the new candidate target object detection model and the difference information between the evaluation metrics of the sample target object detection model and the evaluation metrics of the sample target object detection model, the model parameters of the candidate data augmentation strategy prediction model are adjusted to obtain the trained data augmentation strategy prediction model.

[0177] According to embodiments of this disclosure, the sample point cloud dataset includes L sample point cloud data subsets, where L is a positive integer. The L sample point cloud data subsets are input into the candidate data augmentation strategy prediction model in batches, and the collaborative training method further includes the following operations.

[0178] Iteratively, the m-th sample point cloud data subset from the m-th batch of L sample point cloud data subsets is input into the current candidate data augmentation strategy prediction model to obtain the sample candidate augmentation strategy identifier output by the current candidate data augmentation strategy prediction model, where L≥M≥1; the m-th sample point cloud data subset is processed using the candidate data augmentation strategy corresponding to the current sample candidate augmentation strategy identifier from among N data augmentation strategies to obtain the data-augmented m-th sample candidate point cloud dataset; the current candidate target object detection model is trained based on the m-th sample candidate point cloud dataset to obtain a new candidate target object detection model after training; the model parameters of the current candidate data augmentation strategy prediction model are iteratively adjusted based on the difference information between the m-th evaluation index of the new candidate target object detection model and the evaluation index of the sample target to obtain a new candidate data prediction model; the candidate data prediction model corresponding to the convergence of the evaluation index difference information is determined as the data augmentation strategy prediction model.

[0179] According to embodiments of this disclosure, the target data augmentation operation includes at least one of the following:

[0180] Rotate, scale, move, flip, and mirror operations.

[0181] According to embodiments of this disclosure, the data augmentation strategy prediction model includes a model built based on a neural network algorithm.

[0182] According to embodiments of this disclosure, the model constructed based on a neural network algorithm includes at least one of the following:

[0183] Recurrent neural network model, long short-term memory model, gated recurrent neural network model, backpropagation network model, fully connected neural network model.

[0184] According to embodiments of this disclosure, the point cloud data processing apparatus further includes a target detection module.

[0185] The target detection module is used to perform target detection on target objects based on the target point cloud dataset, and obtain the detection results for the target objects.

[0186] Figure 8 A block diagram of a training apparatus for a data augmentation strategy prediction model according to an embodiment of the present disclosure is shown schematically.

[0187] like Figure 8 As shown, the training device 800 for the data augmentation strategy prediction model includes a sample acquisition module 810 and a training module 820.

[0188] The sample acquisition module 810 is used to acquire training samples, which include a sample point cloud dataset, sample labels corresponding to the sample point cloud dataset, and sample target evaluation indicators. The sample point cloud dataset is used to represent part or all of the sample target objects, and L is a positive integer.

[0189] The training module 820 is used to train the untrained candidate data augmentation strategy prediction model using a collaborative training method to obtain the trained data augmentation strategy prediction model. The collaborative training method is as follows.

[0190] The sample point cloud dataset is input into an untrained candidate data augmentation strategy prediction model. Based on the candidate augmentation strategy identifiers output by the model, a candidate data augmentation strategy corresponding to the identifier is selected from N data augmentation strategies. The sample point cloud dataset is then processed using the data augmentation strategy to obtain a data-augmented candidate point cloud dataset. Using the candidate point cloud dataset and sample labels, a candidate object detection model matching the candidate data augmentation strategy prediction model is trained, resulting in a new, trained candidate object detection model. Finally, based on the evaluation metrics of the new candidate object detection model and the difference between the new and sample object detection models, the model parameters of the candidate data augmentation strategy prediction model are adjusted, resulting in a trained data augmentation strategy prediction model. This data augmentation strategy prediction model is used in the aforementioned point cloud data processing method.

[0191] According to embodiments of this disclosure, the sample point cloud dataset includes L sample point cloud data subsets, where L is a positive integer. The L sample point cloud data subsets are input into the candidate data augmentation strategy prediction model in batches, and the collaborative training method further includes the following operations.

[0192] Iteratively, the m-th sample point cloud data subset from the m-th batch of L sample point cloud data subsets is input into the current candidate data augmentation strategy prediction model to obtain the sample candidate augmentation strategy identifier output by the current candidate data augmentation strategy prediction model, where L≥M≥1; the m-th sample point cloud data subset is processed using the candidate data augmentation strategy corresponding to the current sample candidate augmentation strategy identifier from among N data augmentation strategies to obtain the data-augmented m-th sample candidate point cloud dataset; the current candidate target object detection model is trained based on the m-th sample candidate point cloud dataset to obtain a new candidate target object detection model after training; the model parameters of the current candidate data augmentation strategy prediction model are iteratively adjusted based on the difference information between the m-th evaluation index of the new candidate target object detection model and the evaluation index of the sample target to obtain a new candidate data prediction model; the candidate data prediction model corresponding to the convergence of the evaluation index difference information is determined as the data augmentation strategy prediction model.

[0193] According to embodiments of this disclosure, iteratively adjusting the model parameters of the current candidate data augmentation strategy prediction model based on the m-th evaluation index of the new candidate target object detection model and the evaluation index difference information between the sample target evaluation index includes the following operations.

[0194] The m-th evaluation index is processed according to the preset training algorithm to obtain the evaluation index difference information between the m-th evaluation index and the sample target evaluation index, so as to iteratively adjust the model parameters of the current candidate data prediction model and obtain a new candidate data prediction model; wherein, the preset training algorithm includes at least one of the following: policy gradient algorithm, near-end policy optimization algorithm, and confidence region policy optimization algorithm.

[0195] According to embodiments of this disclosure, the evaluation metrics include at least one of the following:

[0196] Mean precision, accuracy, and recall.

[0197] Any one or more of the modules or units according to embodiments of this disclosure, or at least part of the functions of any one or more of them, can be implemented in one module. Any one or more of the modules or units according to embodiments of this disclosure can be implemented by dividing them into multiple modules. Any one or more of the modules or units according to embodiments of this disclosure can be at least partially implemented as hardware circuitry, such as a Field-Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a System-on-Chip, a System-on-Substrate, a System-on-Package, an Application-Specific Integrated Circuit (ASIC), or implemented by hardware or firmware in any other reasonable manner by integrating or packaging the circuitry, or implemented in any one of software, hardware, and firmware, or in a suitable combination of any of these. Alternatively, one or more of the modules or units according to embodiments of this disclosure can be at least partially implemented as computer program modules, which, when run, can perform corresponding functions.

[0198] For example, any plurality of the policy identification prediction module 710, policy selection module 720, and data enhancement module 730 may be combined into one module / unit, or any one of these modules / units may be split into multiple modules / units. Alternatively, at least part of the functionality of one or more of these modules / units may be combined with at least part of the functionality of other modules / units and implemented in one module / unit. According to embodiments of this disclosure, at least one of the policy identification prediction module 710, policy selection module 720, and data enhancement module 730 may be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the strategy identification prediction module 710, strategy selection module 720, and data augmentation module 730 may be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0199] It should be noted that the point cloud data processing device part in the embodiments of this disclosure corresponds to the point cloud data processing method part in the embodiments of this disclosure. For a detailed description of the point cloud data processing device part, please refer to the point cloud data processing method part, which will not be repeated here.

[0200] It should be noted that the training device part of the data augmentation strategy prediction model in the embodiments of this disclosure corresponds to the training method part of the data augmentation strategy prediction model in the embodiments of this disclosure. The specific description of the training device part of the data augmentation strategy prediction model can be found in the training method part of the data augmentation strategy prediction model, and will not be repeated here.

[0201] Figure 9 The diagram illustrates an electronic device suitable for implementing the point cloud data processing method and the training method for the data augmentation strategy prediction model described above, according to embodiments of the present disclosure. Figure 9 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0202] like Figure 9As shown, an electronic device 900 according to an embodiment of the present disclosure includes a processor 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage portion 908 into a random access memory (RAM) 903. The processor 901 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 901 may also include onboard memory for caching purposes. The processor 901 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0203] RAM 903 stores various programs and data required for the operation of electronic device 900. Processor 901, ROM 902, and RAM 903 are interconnected via bus 904. Processor 901 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 902 and / or RAM 903. It should be noted that the programs may also be stored in one or more memories other than ROM 902 and RAM 903. Processor 901 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0204] According to embodiments of this disclosure, the electronic device 900 may further include an input / output (I / O) interface 905, which is also connected to a bus 904. The system 900 may also include one or more of the following components connected to the I / O interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the I / O interface 905 as needed. A removable medium 911, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 910 as needed so that computer programs read from it can be installed into the storage section 908 as needed.

[0205] According to embodiments of this disclosure, the method flow according to embodiments of this disclosure can be implemented as a computer software program. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via communication section 909, and / or installed from removable medium 911. When the computer program is executed by processor 901, it performs the functions defined in the system of embodiments of this disclosure. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0206] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0207] According to embodiments of this disclosure, the computer-readable storage medium can be a non-volatile computer-readable storage medium. Examples include, but are not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0208] For example, according to embodiments of this disclosure, a computer-readable storage medium may include the ROM 902 and / or RAM 903 described above and / or one or more memories other than ROM 902 and RAM 903.

[0209] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods provided in the embodiments of this disclosure. When the computer program product is run on an electronic device, the program code is used to enable the electronic device to implement the point cloud data processing method and the training method for the data augmentation strategy prediction model provided in the embodiments of this disclosure.

[0210] When the computer program is executed by the processor 901, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0211] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via the communication section 909, and / or installed from a removable medium 911. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0212] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on a user's computing device, partially on a user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0213] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions. Those skilled in the art will understand that the features described in the various embodiments of the present disclosure can be combined and / or combined in various ways, even if such combinations are not explicitly described in the present disclosure. In particular, the features described in the various embodiments of this disclosure may be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0214] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A point cloud data processing method, comprising: An initial point cloud dataset representing the target object is input into a data augmentation strategy prediction model, and a target strategy identifier is output, wherein the data augmentation strategy prediction model is matched with a target object detection model used to detect the target object. Select a target data augmentation strategy corresponding to the target strategy identifier from N data augmentation strategies, wherein the target data augmentation strategy is used to augment the initial point cloud dataset, and N is a positive integer; and The initial point cloud dataset is processed based on the target data augmentation operation in the target data augmentation strategy and the target data augmentation parameters corresponding to the target data augmentation operation to obtain the target point cloud dataset; The data augmentation strategy prediction model is trained using a collaborative training method, which includes: The sample point cloud dataset is input into an untrained candidate data augmentation strategy prediction model so that, based on the sample candidate augmentation strategy identifier output by the candidate data augmentation strategy prediction model, a candidate data augmentation strategy corresponding to the sample candidate augmentation strategy identifier is selected from N data augmentation strategies. The data augmentation strategy is used to process the sample point cloud dataset to obtain a data-augmented sample candidate point cloud dataset. Using the aforementioned candidate point cloud dataset, a candidate object detection model matching the candidate data augmentation strategy prediction model is trained to obtain a new trained candidate object detection model; and Based on the evaluation metrics of the new candidate target object detection model and the difference information between the evaluation metrics of the sample target, the model parameters of the candidate data augmentation strategy prediction model are adjusted to obtain the trained data augmentation strategy prediction model.

2. The method according to claim 1, wherein, The sample point cloud dataset includes L subsets of sample point cloud data, where L is a positive integer. These L subsets are input into the candidate data augmentation strategy prediction model in batches. The collaborative training method further includes: Iteratively input the m-th sample point cloud data subset of the m-th batch into the current candidate data augmentation strategy prediction model from the L subsets of the sample point cloud data to obtain the sample candidate augmentation strategy identifier output by the current candidate data augmentation strategy prediction model, where L≥M≥1; The m-th sample point cloud data subset is processed using the candidate data augmentation strategy corresponding to the current sample candidate augmentation strategy identifier from among the N data augmentation strategies to obtain the data-augmented m-th sample candidate point cloud dataset. Train the current candidate target object detection model based on the m-th sample candidate point cloud dataset to obtain the trained new candidate target object detection model; Based on the difference information between the m-th evaluation index of the new candidate target object detection model and the evaluation index of the sample target, the model parameters of the current candidate data augmentation strategy prediction model are iteratively adjusted to obtain a new candidate data prediction model. The candidate data prediction model corresponding to the convergence of the evaluation index difference information is determined as the data augmentation strategy prediction model.

3. The method according to claim 1, wherein, The target data augmentation operation includes at least one of the following: Rotate, scale, move, flip, and mirror operations.

4. The method according to claim 1, wherein, The data augmentation strategy prediction model includes a model built based on a neural network algorithm.

5. The method according to claim 4, wherein, The model constructed based on the neural network algorithm includes at least one of the following: Recurrent neural network model, long short-term memory model, gated recurrent neural network model, backpropagation network model, fully connected neural network model.

6. The method according to any one of claims 1 to 5, further comprising: The target object is detected based on the target point cloud dataset to obtain the detection result for the target object.

7. A training method for a data augmentation strategy prediction model, comprising: Obtain training samples, wherein the training samples include a sample point cloud dataset, sample labels corresponding to the sample point cloud dataset, and sample target evaluation metrics, wherein the sample point cloud dataset is used to represent part or all of the sample target objects; and An untrained candidate data augmentation strategy prediction model is trained using a collaborative training method to obtain a trained data augmentation strategy prediction model. The collaborative training method includes: The sample point cloud dataset is input into an untrained candidate data augmentation strategy prediction model so that, based on the sample candidate augmentation strategy identifier output by the candidate data augmentation strategy prediction model, a candidate data augmentation strategy corresponding to the sample candidate augmentation strategy identifier is selected from N data augmentation strategies, where N is a positive integer. The data augmentation strategy is used to process the sample point cloud dataset to obtain a data-augmented sample candidate point cloud dataset. Using the candidate point cloud dataset and the sample labels, a candidate object detection model matching the candidate data augmentation strategy prediction model is trained, resulting in a new, trained candidate object detection model; and Based on the evaluation index of the new candidate target object detection model and the difference information between the evaluation index of the sample target, the model parameters of the candidate data augmentation strategy prediction model are adjusted to obtain the trained data augmentation strategy prediction model. The data augmentation strategy prediction model is used in the point cloud data processing method according to any one of claims 1 to 6.

8. The training method according to claim 7, wherein, The sample point cloud dataset includes L subsets of sample point cloud data, where L is a positive integer. These L subsets are input into the candidate data augmentation strategy prediction model in batches. The collaborative training method further includes: Iteratively input the m-th sample point cloud data subset of the m-th batch into the current candidate data augmentation strategy prediction model from the L subsets of the sample point cloud data to obtain the sample candidate augmentation strategy identifier output by the current candidate data augmentation strategy prediction model, where L≥M≥1; The m-th sample point cloud data subset is processed using the candidate data augmentation strategy corresponding to the current sample candidate augmentation strategy identifier from among the N data augmentation strategies to obtain the data-augmented m-th sample candidate point cloud dataset. Train the current candidate target object detection model based on the m-th sample candidate point cloud dataset to obtain the new candidate target object detection model after training. Based on the difference information between the m-th evaluation index of the new candidate target object detection model and the evaluation index of the sample target, the model parameters of the current candidate data augmentation strategy prediction model are iteratively adjusted to obtain a new candidate data prediction model. The candidate data prediction model corresponding to the convergence of the evaluation index difference information is determined as the data augmentation strategy prediction model.

9. The training method according to claim 8, wherein, Based on the difference information between the m-th evaluation index of the new candidate target object detection model and the evaluation index of the sample target, the model parameters of the current candidate data augmentation strategy prediction model are iteratively adjusted, including: The m-th evaluation index is processed according to a preset training algorithm to obtain the evaluation index difference information between the m-th evaluation index and the sample target evaluation index, so as to iteratively adjust the model parameters of the current candidate data prediction model and obtain a new candidate data prediction model. The preset training algorithm includes at least one of the following: policy gradient algorithm, near-end policy optimization algorithm, and confidence region policy optimization algorithm.

10. The training method according to any one of claims 7 to 9, wherein, The evaluation indicators include at least one of the following: Mean precision, accuracy, and recall.

11. A point cloud data processing device, comprising: The strategy identification prediction module is used to input the initial point cloud dataset representing the target object into the data augmentation strategy prediction model and output the target strategy identification, wherein the data augmentation strategy prediction model is matched with the target object detection model used to detect the target object. A strategy selection module is used to select a target data augmentation strategy corresponding to the target strategy identifier from N data augmentation strategies, wherein the target data augmentation strategy is used to augment the initial point cloud dataset, and N is a positive integer; and The data augmentation module is used to process the initial point cloud dataset based on the target data augmentation operation in the target data augmentation strategy and the target data augmentation parameters corresponding to the target data augmentation operation, to obtain the target point cloud dataset. The data augmentation strategy prediction model is trained using a collaborative training method, which includes: The sample point cloud dataset is input into an untrained candidate data augmentation strategy prediction model so that, based on the sample candidate augmentation strategy identifier output by the candidate data augmentation strategy prediction model, a candidate data augmentation strategy corresponding to the sample candidate augmentation strategy identifier is selected from N data augmentation strategies. The data augmentation strategy is used to process the sample point cloud dataset to obtain a data-augmented sample candidate point cloud dataset. Using the aforementioned candidate point cloud dataset, a candidate object detection model matching the candidate data augmentation strategy prediction model is trained to obtain a new trained candidate object detection model; and Based on the evaluation metrics of the new candidate target object detection model and the difference information between the evaluation metrics of the sample target, the model parameters of the candidate data augmentation strategy prediction model are adjusted to obtain the trained data augmentation strategy prediction model.

12. A training device for a data augmentation strategy prediction model, comprising: A sample acquisition module is used to acquire training samples, wherein the training samples include a sample point cloud dataset, sample labels corresponding to the sample point cloud dataset, and sample target evaluation metrics, wherein the sample point cloud dataset is used to represent part or all of the sample target objects, and L is a positive integer; and The training module is used to train an untrained candidate data augmentation strategy prediction model using a collaborative training method to obtain a trained data augmentation strategy prediction model. The collaborative training method includes: The sample point cloud dataset is input into an untrained candidate data augmentation strategy prediction model so that, based on the sample candidate augmentation strategy identifier output by the candidate data augmentation strategy prediction model, a candidate data augmentation strategy corresponding to the sample candidate augmentation strategy identifier is selected from N data augmentation strategies. The data augmentation strategy is used to process the sample point cloud dataset to obtain a data-augmented sample candidate point cloud dataset. Using the candidate point cloud dataset and the sample labels, a candidate object detection model matching the candidate data augmentation strategy prediction model is trained, resulting in a new, trained candidate object detection model; and Based on the evaluation index of the new candidate target object detection model and the difference information between the evaluation index of the sample target, the model parameters of the candidate data augmentation strategy prediction model are adjusted to obtain the trained data augmentation strategy prediction model. The data augmentation strategy prediction model is used in the point cloud data processing method according to any one of claims 1 to 6.

13. An electronic device, comprising: One or more processors; Memory, used to store one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1 to 10.

14. A computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the method of any one of claims 1 to 10.

15. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 10.

Images