Method, apparatus, device, storage medium and program product for processing point cloud data

Abstract

Embodiments of this disclosure relate to methods, apparatus, devices, storage media, and program products for processing point cloud data. The method includes: providing first point cloud data to a first point cloud segmentation model to generate a first set of labels corresponding to the first point cloud data; constructing a hybrid sample based on the first point cloud data and second point cloud data, the second point cloud data corresponding to a second set of labels, the hybrid sample including a first projection portion corresponding to the first point cloud data and a second projection portion corresponding to the second point cloud data; processing the hybrid sample using a second point cloud segmentation model to generate predicted labels; and training the second point cloud segmentation model based on a comparison of the predicted labels and reference labels, the reference labels being determined based on the first set of labels and the second set of labels. In this manner, embodiments of this disclosure can improve the efficiency of processing point cloud data.

Description

Technical Field

[0001] The exemplary embodiments disclosed herein generally relate to the field of computers, and particularly to methods, apparatus, devices, computer-readable storage media, and computer program products for processing point cloud data. Background Technology

[0002] With the development of computer technology, point cloud data acquired by radar sensors is widely used in target recognition scenarios, including autonomous driving and other semantic recognition applications. Therefore, how to segment point cloud data to improve the accuracy and efficiency of target recognition is a topic worthy of attention. Summary of the Invention

[0003] In a first aspect of this disclosure, a method for processing point cloud data is provided. The method includes: providing first point cloud data to a first point cloud segmentation model to generate a first set of labels corresponding to the first point cloud data; constructing a hybrid sample based on the first point cloud data and second point cloud data, the second point cloud data corresponding to a second set of labels, the hybrid sample including a first projection portion corresponding to the first point cloud data and a second projection portion corresponding to the second point cloud data; processing the hybrid sample using a second point cloud segmentation model to generate predicted labels; and training the second point cloud segmentation model based on a comparison of the predicted labels and reference labels, the reference labels being determined based on the first set of labels and the second set of labels.

[0004] In a second aspect of this disclosure, an apparatus for processing point cloud data is provided. The apparatus includes: a first generation module configured to provide first point cloud data to a first point cloud segmentation model to generate a first set of labels corresponding to the first point cloud data; a construction module configured to construct a hybrid sample based on the first point cloud data and second point cloud data, wherein the second point cloud data corresponds to a second set of labels, and the hybrid sample includes a first projection portion corresponding to the first point cloud data and a second projection portion corresponding to the second point cloud data; a second generation module configured to allow the second point cloud segmentation model to process the hybrid sample to generate predicted labels; and a training module configured to train the second point cloud segmentation model based on a comparison of the predicted labels and reference labels, wherein the reference labels are determined based on the first set of labels and the second set of labels.

[0005] In a third aspect of this disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the device to perform the method of the first aspect.

[0006] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program that can be executed by a processor to implement the method of the first aspect.

[0007] In a fifth aspect of this disclosure, a computer program product is provided. The computer program product includes computer-executable instructions that, when executed by a processor, implement the method of the first aspect.

[0008] It should be understood that the content described in this content section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0009] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0010] Figure 1 A schematic diagram is shown in which an example system in which embodiments of the present disclosure may be implemented;

[0011] Figure 2 A schematic diagram illustrating an example process for training a point cloud segmentation model according to some embodiments of the present disclosure is shown;

[0012] Figure 3 A schematic diagram illustrating an example process for processing point cloud data according to some embodiments of the present disclosure is shown;

[0013] Figure 4 A schematic structural block diagram of an example apparatus for processing point cloud data according to some embodiments of the present disclosure is shown; and

[0014] Figure 5 A block diagram of an electronic device capable of implementing several embodiments of the present disclosure is shown. Detailed Implementation

[0015] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0016] It should be noted that the headings of any section / subsection provided herein are not limiting. Various embodiments are described throughout this document, and embodiments of any type may be included under any section / subsection. Furthermore, embodiments described in any section / subsection may be combined in any way with any other embodiments described in the same section / subsection and / or different sections / subsections.

[0017] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below. The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0018] The embodiments of this disclosure may involve user data, data acquisition, and / or use. All of these aspects comply with applicable laws, regulations, and relevant provisions. In the embodiments of this disclosure, all data collection, acquisition, processing, manipulation, forwarding, and use are conducted with the user's knowledge and confirmation. Accordingly, in implementing the embodiments of this disclosure, the type, scope of use, and usage scenarios of any data or information that may be involved should be communicated to the user and their authorization obtained in accordance with relevant laws and regulations through appropriate means. The specific methods of notification and / or authorization may vary depending on the actual situation and application scenario, and the scope of this disclosure is not limited in this respect.

[0019] In this specification and the embodiments, any processing of personal information will be carried out only under the premise of legality (such as obtaining the consent of the personal information subject, or being necessary for the performance of a contract), and will only be carried out within the scope stipulated or agreed upon. A user's refusal to process personal information other than that necessary for basic functions will not affect the user's use of basic functions.

[0020] As mentioned above, with the development of computer technology, point cloud data acquired by radar sensors is widely used in target recognition scenarios, particularly in autonomous driving and other semantic recognition scenarios. Therefore, how to segment point cloud data to improve the accuracy and efficiency of target recognition is a crucial issue. However, traditional point cloud segmentation methods, on the one hand, are based on voxel-based segmentation, which cannot meet the needs of autonomous driving scenarios with limited chip resources. On the other hand, segmentation based on point cloud cylinders results in low accuracy. Neither approach satisfies the semantic recognition requirements of the autonomous driving field.

[0021] Embodiments of this disclosure propose a scheme for processing point cloud data. The scheme can provide first point cloud data to a first point cloud segmentation model to generate a first set of labels corresponding to the first point cloud data; construct a hybrid sample based on the first and second point cloud data, where the second point cloud data corresponds to a second set of labels, and the hybrid sample includes a first projection portion corresponding to the first point cloud data and a second projection portion corresponding to the second point cloud data; process the hybrid sample using a second point cloud segmentation model to generate predicted labels; and train the second point cloud segmentation model based on a comparison of the predicted labels and reference labels, where the reference labels are determined based on the first and second sets of labels.

[0022] In this manner, embodiments of this disclosure can process mixed samples constructed from the projections of first and second point cloud data based on a second segmentation model to obtain predicted labels. Furthermore, embodiments of this disclosure can train a second point cloud segmentation model based on a comparison between the predicted labels and reference labels determined from a first set of labels derived from the first point cloud data and the labels corresponding to the second point cloud data. Therefore, embodiments of this disclosure can train a point cloud segmentation model based on projections of both labeled and unlabeled point cloud data as samples. Thus, embodiments of this disclosure can effectively utilize unlabeled point cloud data to generate a point cloud segmentation model capable of processing projections of point cloud data, thereby improving both the accuracy and efficiency of point cloud data segmentation.

[0023] The following section provides a detailed description of various example implementations of this scheme, with reference to the accompanying drawings.

[0024] Example training point cloud segmentation model

[0025] Figure 1 A schematic diagram of an example system 100 in which embodiments of the present disclosure can be implemented is shown. For example... Figure 1 As shown, system 100 may include training device 120. Training device 120 can train a first point cloud segmentation model 110 (also known as a teacher model) and a second point cloud segmentation model 130 (also known as a student model).

[0026] The training device 120 can be implemented as an electronic device. The electronic device can be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, handheld computers, portable gaming terminals, VR / AR devices, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. In some embodiments, the training device 120 can also support any type of user-facing interface (such as "wearable" circuitry).

[0027] Training device 120 can be implemented as a service device. A service device can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and big data and artificial intelligence platforms. Service devices may include, for example, computing systems / servers such as mainframes, edge computing nodes, and computing devices in cloud environments, etc.

[0028] The first point cloud segmentation model 110 and the second point cloud segmentation model 130 can process point cloud data. For example, the first point cloud segmentation model 110 and the second point cloud segmentation model 130 can segment point cloud data to obtain the segmentation result (or classification result) corresponding to the point cloud data.

[0029] As an example, the first point cloud segmentation model 110 and the second point cloud segmentation model 130 can be two pillar-based point cloud segmentation models with identical structures but different weights (e.g., a backbone network (e.g., Backbone) and / or a neck network (e.g., Neck)). As an example, the parameters of the first point cloud segmentation model 110 are frozen during the training initialization phase and do not participate in gradient backpropagation updates.

[0030] The first point cloud segmentation model 110 and / or the second point cloud segmentation model 130 can be deployed on autonomous vehicles. Autonomous vehicles can be vehicles with autonomous driving capabilities (or driverless capabilities), also known as driverless cars, autonomous vehicles, etc. In some scenarios, autonomous vehicles can be assigned to provide travel services to users. For example, users can obtain travel services provided by autonomous vehicles through travel applications. In some scenarios, autonomous vehicles can also be called driverless taxis or robotaxis.

[0031] For example, a first point cloud segmentation model 110 and / or a second point cloud segmentation model 130 deployed on an autonomous vehicle can classify point cloud data collected by sensors mounted on the autonomous vehicle. The first point cloud segmentation model 110 and / or the second point cloud segmentation model 130 can acquire point cloud information (e.g., point cloud data) of the physical environment in which the vehicle is located based on radar (e.g., lidar, millimeter-wave radar, etc.). As an example, point cloud data is a data structure representing a set of points in three-dimensional space. Each point contains its coordinates in three-dimensional space (typically X, Y, Z coordinates), and sometimes additional information such as color, intensity, and normals. Point cloud data can be used to represent the surface shape of objects, terrain, buildings, and other entities in three-dimensional space.

[0032] The following will detail the specific implementation of the training process of the training device 120 for the first point cloud segmentation model 110 and / or the second point cloud segmentation model 130.

[0033] Figure 2 A flowchart illustrating an example process 200 for training a point cloud segmentation model according to some embodiments of the present disclosure is shown. Process 200 can be implemented in a training device 120. References are made below. Figure 1 Describe the process 200.

[0034] like Figure 2 As shown, the training device 120 can provide first point cloud data (e.g., unlabeled point cloud 205-2) to the first point cloud segmentation model 110 (also referred to as the teacher model). As an example, the first point cloud data can be voxelized point cloud data. As an example, by matching the three-dimensional coordinate information of the point cloud data with a set of preset three-dimensional meshes, a set of voxelized point cloud data corresponding to the preset three-dimensional meshes can be obtained. In some embodiments, the first point cloud data is unlabeled point cloud data (i.e., the first point cloud data does not contain labeling information).

[0035] Additionally, the first point cloud segmentation model 110 can generate a first set of labels corresponding to the first point cloud data based on the first point cloud data. As an example, the first set of labels can indicate a set of categories corresponding to the first point cloud data. As an example, during the initial training phase, the first set of labels generated by the first point cloud segmentation model 110 can be pseudo-labels (e.g., pseudo-label 215-2).

[0036] In some embodiments, the first point cloud segmentation model 110 can generate a label prediction result corresponding to the first point cloud data based on the first point cloud data. The training device 120 can add the label prediction result to the first set of labels based on the confidence level of the label prediction result (e.g., the probability that the label prediction result belongs to the corresponding category) being greater than a threshold (e.g., 0.95).

[0037] As an example, in pseudo-label 215-2, -1 can indicate that the confidence level of the point cloud data in the corresponding area is not greater than a threshold. 0 can indicate, for example, the first category, which indicates that the corresponding area does not contain point cloud data. 1 can indicate, for example, the second category, which indicates that the corresponding point cloud data is valid data. 2 can indicate, for example, the third category, which indicates that the point cloud data in the corresponding area is invalid data (or interference data).

[0038] In some embodiments, the training device 120 can construct a hybrid sample based on the first point cloud data and the second point cloud data. As an example, the second point cloud data can also be voxelized point cloud data. For instance, the second point cloud data corresponds to a second set of labels (e.g., label 215-1). The second set of labels can indicate ground truth information (i.e., true label information) corresponding to the second point cloud data.

[0039] As an example, the first point cloud data and the second point cloud data can correspond to a preset ratio (e.g., 1:1).

[0040] As an example, the mixed sample may include a first projection portion corresponding to the first point cloud data and a second projection portion corresponding to the second point cloud data. As an example, the first projection portion corresponds to the projection of the first point cloud data onto a preset plane (e.g., the first projection portion does not contain the height information of the first point cloud data). The second projection portion corresponds to the projection of the second point cloud data onto the preset plane (e.g., the second projection portion does not contain the height information of the second point cloud data). As an example, the preset plane may be the plane corresponding to a bird's-eye view.

[0041] In some embodiments, the training device 120 can determine a first set of point pillars corresponding to the first set of point cloud data and a second set of point pillars corresponding to the second set of point cloud data. As an example, the training device 120 can determine the first set of point pillars based on the projection of the first point cloud data onto a preset plane and based on a first set of preset grids in a preset region. As an example, the training device 120 can determine the second set of point pillars based on the projection of the second point cloud data onto a preset plane and based on a second set of preset grids in a preset region. As an example, the preset plane can be a plane perpendicular to the height direction, such as the plane corresponding to a bird's-eye view.

[0042] As an example, a pillar can preserve structured information. For instance, in a point cloud collected by a LiDAR system on an autonomous driving test vehicle traveling on a real-world road, points collected by rays at different tilt angles exhibit distinctly different category distributions. For example, points collected by rays at larger tilt angles (upper part) are highly associated with objects such as vegetation and buildings, and these objects are relatively far from the vehicle. Points collected by rays at intermediate tilt angles (middle part) are highly associated with dynamic objects such as vehicles, pedestrians, and bicycles, and these objects are at a moderate distance from the vehicle. Points collected by rays at smaller tilt angles (lower part) are highly associated with the road surface, and the road surface is relatively close to the vehicle. In a pillar representation, this inherent structured information exists in the form of the distance from the point to the vehicle. Therefore, this disclosure takes into account the actual applications in the industry. By incorporating the inherent structured information of point clouds in autonomous driving scenarios under Pillar representation into a semi-supervised point cloud segmentation algorithm, it utilizes a large amount of unlabeled point cloud data to optimize the performance of the point cloud segmentation model, thereby reducing the dependence of the point cloud segmentation model iteration on labeled data.

[0043] The training device 120 can fill a first set of grids in a target region using at least one point pillar from a first set of point pillars to determine a first projected portion. The training device 120 can also fill a second set of grids in the target region using at least one point pillar from a second set of point pillars to determine a second projected portion. Furthermore, the training device 120 can construct a hybrid sample corresponding to the target region based on the first and second projected portions. As an example, the number of at least one point pillar from the first set of point pillars filling the first set of grids in the target region is determined based on the number of grids in the first set of grids.

[0044] As an example, the first and second sets of grids are interwoven in the target region. For example, the mixed sample may include mixed point cloud 210-1 and mixed point cloud 210-2. Taking the number of regions of mixed point cloud 210-1 and the target region as 2, the target region may include a first region 211 (e.g., the inner circle region) and a second region 212 (e.g., the outer circle region). It is worth noting that the number of regions of the target region here is only an illustrative example, and the number of target regions can be randomly determined or set by relevant personnel as needed. As an example, the number of regions of the target region can be 2, 3, 4, 5, 6, etc. As an example, the interwoven distribution can be understood as the point pillars of adjacent regions in the target region corresponding to different point cloud data. For example, taking the number of regions of the target region as 3, the first set of grids in the first region (e.g., the inner circle region) corresponds to the first point cloud data, then the second set of grids in the second region (e.g., the region between the inner and outer circles) corresponds to the second point cloud data. And the third set of grids in the third region (e.g., the outer circle region) corresponds to the first point cloud data.

[0045] In some embodiments, the first set of grids may be distributed outside the second set of grids. For example, the first region 211 may correspond to the second set of grids (e.g., 4 grids). The second region 212 may correspond to the first set of grids (e.g., 10 grids).

[0046] Additionally, the training device 120 can use the second point cloud segmentation model 130 to process mixed samples to generate predicted labels.

[0047] As an example, training device 120 can determine reference labels based on the first set of labels and the second set of labels.

[0048] As an example, the training device 120 can determine a first label corresponding to the first projection portion based on a first set of labels. The training device 120 can determine a second label corresponding to the second projection portion based on a second set of labels. Further, the training device 120 can determine a reference label based on the first label and the second label. As an example, the training device 120 can construct a reference label based on a mixing operation using the first label and the second label. The mixing operation can refer to the aforementioned construction process of constructing mixed samples based on the first point cloud data and the second point cloud data, and will not be repeated here. As an example, the reference label may include, for example, reference label 220-1 and reference label 220-2.

[0049] Alternatively or additionally, training device 120 may train a second point cloud segmentation model 130 based on a comparison of predicted labels and reference labels.

[0050] As an example, training device 120 can determine a first loss based on a comparison of predicted labels and reference labels. The first loss can, for example, correspond to an unsupervised loss. As an example, training device 120 can process second point cloud data based on a second point cloud segmentation model 130 to determine a second loss. As an example, training device 120 can determine a second loss based on a comparison of the prediction results of the second segmentation model for the second point cloud data with a second set of labels. As an example, the second loss can correspond to a supervised loss. As an example, the first and second losses can be implemented as a cross-entropy loss function with class weights.

[0051] Furthermore, the training device 120 can train a second point cloud segmentation model based on the first loss and the second loss. For example, the training device 120 can adjust at least one parameter of the second point cloud segmentation model 130 through gradient backpropagation based on the first loss and the second loss.

[0052] In some embodiments, the training device 120 may also adjust the second set of parameters of the first point cloud segmentation model 110 based on the first set of parameters of the second point cloud segmentation model 130. As an example, the training device 120 may adjust the parameters of the first point cloud segmentation model 110 using the second point cloud segmentation model 130 based on an exponential moving average (EMA). As an example, the strength of the EMA may be set to a preset value (e.g., 0.99).

[0053] Alternatively or concurrently, the training device 120 may trigger the autonomous vehicle to deploy either the first point cloud segmentation model 110 or the second point cloud segmentation model 130. As an example, the training device 120 may determine which point cloud recognition model has a better training effect to deploy on the autonomous vehicle based on the training effects corresponding to the first point cloud segmentation model 110 and the second point cloud segmentation model 130. As an example, the training effect may be determined based on the accuracy of point cloud segmentation.

[0054] Based on the process described above, embodiments of this disclosure can process mixed samples constructed from the projections of first and second point cloud data using a second segmentation model to obtain predicted labels. Furthermore, embodiments of this disclosure can train a second point cloud segmentation model based on a comparison between the predicted labels and reference labels determined from a first set of labels derived from the first point cloud data and the labels corresponding to the second point cloud data. Therefore, embodiments of this disclosure can train a point cloud segmentation model using projections of both labeled and unlabeled point cloud data as samples. In this way, embodiments of this disclosure can effectively utilize unlabeled point cloud data to generate a point cloud segmentation model capable of processing projections of point cloud data, thereby improving both the accuracy and efficiency of point cloud data segmentation.

[0055] Example process

[0056] Figure 3 A flowchart of an example process 300 for processing point cloud data according to some embodiments of the present disclosure is shown. Process 300 can be implemented at training device 120. Reference is made below. Figure 1 To describe process 300.

[0057] like Figure 3 As shown in box 310, the training device 120 provides the first point cloud data to the first point cloud segmentation model to generate the first set of labels corresponding to the first point cloud data.

[0058] In box 320, training device 120 constructs a hybrid sample based on first point cloud data and second point cloud data. The second point cloud data corresponds to the second set of labels. The hybrid sample includes a first projection part corresponding to the first point cloud data and a second projection part corresponding to the second point cloud data.

[0059] In box 330, training device 120 uses a second point cloud segmentation model to process mixed samples to generate predicted labels.

[0060] In box 340, training device 120 trains a second point cloud segmentation model based on a comparison of predicted labels and reference labels, where the reference labels are determined based on the first set of labels and the second set of labels.

[0061] In some embodiments, training a second point cloud segmentation model based on a comparison of predicted labels and reference labels includes: determining a first loss based on the comparison of predicted labels and reference labels; processing second point cloud data using the second point cloud segmentation model to determine a second loss; and training the second point cloud segmentation model based on the first loss and the second loss.

[0062] In some embodiments, process 300 further includes: adjusting a second set of parameters of a first point cloud segmentation model based on a first set of parameters of a second point cloud segmentation model.

[0063] In some embodiments, process 300 further includes: triggering the deployment of a first point cloud segmentation model or a second point cloud segmentation model on the autonomous vehicle.

[0064] In some embodiments, constructing a hybrid sample based on first point cloud data and second point cloud data includes: determining a first set of point pillars corresponding to the first point cloud data and a second set of point pillars corresponding to the second point cloud data; filling a first set of grids in a target area with at least one point pillar from the first set of point pillars to determine a first projection portion; filling a second set of grids in the target area with at least one point pillar from the second set of point pillars to determine a second projection portion; and constructing a hybrid sample corresponding to the target area based on the first projection portion and the second projection portion.

[0065] In some embodiments, the first set of grids and the second set of grids are interwoven in the target area.

[0066] In some embodiments, the first set of grids is distributed outside the second set of grids.

[0067] In some embodiments, providing first point cloud data to a first point cloud segmentation model to generate a first set of labels corresponding to the first point cloud data includes: determining the label prediction results generated by the first point cloud segmentation model; and adding the label prediction results to the first set of labels in response to the confidence level of the label prediction results being greater than a threshold.

[0068] In some embodiments, process 300 further includes: determining a first label corresponding to a first projection portion based on a first set of labels; determining a second label corresponding to a second projection portion based on a second set of labels; and determining a reference label based on the first label and the second label.

[0069] Example devices and equipment

[0070] Figure 4 A schematic structural block diagram of an example apparatus 400 for processing point cloud data according to some embodiments of the present disclosure is shown. Apparatus 400 may be implemented as or included in training device 120. Various modules / components in apparatus 400 may be implemented by hardware, software, firmware, or any combination thereof.

[0071] like Figure 4 As shown, the device 400 includes a first generation module 410 configured to provide first point cloud data to a first point cloud segmentation model to generate a first set of labels corresponding to the first point cloud data; a construction module 420 configured to construct a hybrid sample based on the first point cloud data and the second point cloud data, wherein the second point cloud data corresponds to the second set of labels, and the hybrid sample includes a first projection portion corresponding to the first point cloud data and a second projection portion corresponding to the second point cloud data; a second generation module 430 configured to process the hybrid sample using the second point cloud segmentation model to generate predicted labels; and a training module 440 configured to train the second point cloud segmentation model based on a comparison of predicted labels and reference labels, wherein the reference labels are determined based on the first set of labels and the second set of labels.

[0072] In some embodiments, the training module 440 is further configured to: determine a first loss based on a comparison of the predicted label and the reference label; process the second point cloud data using the second point cloud segmentation model to determine a second loss; and train the second point cloud segmentation model based on the first loss and the second loss.

[0073] In some embodiments, the apparatus 400 further includes an adjustment module configured to adjust a second set of parameters of a first point cloud segmentation model based on a first set of parameters of the second point cloud segmentation model.

[0074] In some embodiments, the device 400 further includes a triggering module configured to trigger the deployment of a first point cloud segmentation model or a second point cloud segmentation model on an autonomous vehicle.

[0075] In some embodiments, the construction module 420 is further configured to: determine a first set of point pillars corresponding to the first point cloud data and a second set of point pillars corresponding to the second point cloud data; fill a first set of grids in the target area with at least one point pillar from the first set of point pillars to determine a first projection portion; fill a second set of grids in the target area with at least one point pillar from the second set of point pillars to determine a second projection portion; and construct a hybrid sample corresponding to the target area based on the first projection portion and the second projection portion.

[0076] In some embodiments, the first set of grids and the second set of grids are interwoven in the target area.

[0077] In some embodiments, the first set of grids is distributed outside the second set of grids.

[0078] In some embodiments, the first generation module 410 is further configured to: determine the label prediction result generated by the first point cloud segmentation model; and add the label prediction result to the first group of labels in response to the confidence level of the label prediction result being greater than a threshold.

[0079] In some embodiments, the apparatus 400 further includes a determining module configured to: determine a first label corresponding to a first projection portion based on a first set of labels; determine a second label corresponding to a second projection portion based on a second set of labels; and determine a reference label based on the first label and the second label.

[0080] The modules included in device 400 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units can be implemented using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to or as an alternative to machine-executable instructions, some or all of the modules in device 400 can be implemented at least partially by one or more hardware logic components. By way of example, and not limitation, exemplary types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific labeled circuits (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0081] Figure 5A block diagram of an electronic device 500 in which one or more embodiments of the present disclosure may be implemented is shown. It should be understood that... Figure 5 The electronic device 500 shown is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. The electronic device 500 can be used to implement... Figure 1 120 training devices.

[0082] like Figure 5 As shown, electronic device 500 is in the form of a general-purpose electronic device. Components of electronic device 500 may include, but are not limited to, one or more processors or processing units 510, memory 520, storage device 530, one or more communication units 540, one or more input devices 550, and one or more output devices 560. Processing unit 510 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 520. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 500.

[0083] Electronic device 500 typically includes multiple computer storage media. Such media can be any available media accessible to electronic device 500, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 520 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 530 can be a removable or non-removable medium and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data (e.g., training data for training) and can be accessed within electronic device 500.

[0084] Electronic device 500 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 5 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 520 may include computer program product 525 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0085] Communication unit 540 enables communication with other electronic devices via a communication medium. Additionally, the functionality of components of electronic device 500 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, electronic device 500 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0086] Input device 550 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 550 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 500 can also communicate with one or more external devices (not shown) via communication unit 540 as needed. External devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 500, or with any device that enables electronic device 500 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0087] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores one or more computer instructions, wherein one or more computer instructions are executed by a processor to implement the methods described above.

[0088] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0089] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0090] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0091] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0092] Various implementations of this disclosure have been described above. The foregoing description is exemplary and not exhaustive, nor is it limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the implementations disclosed herein.

Claims

1. A method for processing point cloud data, comprising: Provide the first point cloud data to the first point cloud segmentation model to generate the first set of labels corresponding to the first point cloud data; Based on the first point cloud data and the second point cloud data, a hybrid sample is constructed, wherein the second point cloud data corresponds to the second set of labels, and the hybrid sample includes a first projection part corresponding to the first point cloud data and a second projection part corresponding to the second point cloud data. The mixed samples are processed using a second point cloud segmentation model to generate predicted labels; as well as The second point cloud segmentation model is trained based on the comparison between the predicted labels and the reference labels, wherein the reference labels are determined based on the first set of labels and the second set of labels.

2. The method according to claim 1, wherein training the second point cloud segmentation model based on the comparison of the predicted label and the reference label comprises: Based on the comparison between the predicted label and the reference label, a first loss is determined; The second point cloud data is processed using the second point cloud segmentation model to determine the second loss; as well as The second point cloud segmentation model is trained based on the first loss and the second loss.

3. The method according to claim 1, further comprising: Based on the first set of parameters of the second point cloud segmentation model, adjust the second set of parameters of the first point cloud segmentation model.

4. The method according to claim 3, further comprising: Trigger the deployment of the first point cloud segmentation model or the second point cloud segmentation model on the autonomous vehicle.

5. The method according to claim 1, wherein constructing a hybrid sample based on the first point cloud data and the second point cloud data comprises: Determine the first set of point columns corresponding to the first point cloud data and the second set of point columns corresponding to the second point cloud data; The first set of grids in the target area is filled using at least one of the first set of point pillars to determine the first projected portion; The second set of grids in the target region is filled using at least one of the second set of point pillars to determine the second projected portion; as well as Based on the first projection portion and the second projection portion, the hybrid sample corresponding to the target region is constructed.

6. The method of claim 5, wherein the first set of grids and the second set of grids are interwoven in the target area.

7. The method of claim 5, wherein the first set of meshes is distributed outside the second set of meshes.

8. The method according to claim 1, wherein providing first point cloud data to a first point cloud segmentation model to generate a first set of labels corresponding to the first point cloud data comprises: Determine the label prediction results generated by the first point cloud segmentation model; as well as If the confidence level of the label prediction result is greater than a threshold, the label prediction result is added to the first group of labels.

9. The method according to claim 1, further comprising: Based on the first set of labels, determine the first label corresponding to the first projection portion; Based on the second set of labels, determine the second label corresponding to the second projection portion; as well as The reference label is determined based on the first label and the second label.

10. An apparatus for processing point cloud data, comprising: The first generation module is configured to provide first point cloud data to the first point cloud segmentation model in order to generate a first set of labels corresponding to the first point cloud data. The construction module is configured to construct a hybrid sample based on the first point cloud data and the second point cloud data, wherein the second point cloud data corresponds to a second set of labels, and the hybrid sample includes a first projection portion corresponding to the first point cloud data and a second projection portion corresponding to the second point cloud data. The second generation module is configured to process the mixed samples using a second point cloud segmentation model to generate predicted labels; as well as The training module is configured to train the second point cloud segmentation model based on a comparison of the predicted labels and the reference labels, wherein the reference labels are determined based on the first set of labels and the second set of labels.

11. An electronic device, comprising: At least one processing unit; as well as At least one memory, coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, which, when executed by the at least one processing unit, cause the electronic device to perform the method according to any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program stored thereon, the computer program being executable by a processor to implement the method according to any one of claims 1 to 9.

13. A computer program product comprising computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the method according to any one of claims 1 to 9.

Images