Methods for segmenting point cloud data, methods for generating segmentation models, devices and equipment

Abstract

This disclosure proposes a method, a method for generating a segmentation model, an apparatus, and a device for segmenting point cloud data, relating to the field of computer technology, particularly to artificial intelligence technologies such as computer vision, image processing, and large-scale modeling. The scheme is as follows: After receiving a first operation on a first mask, the first feature of the first object associated with the first operation is determined. Then, the first feature, the first point cloud feature corresponding to the first point cloud data, and the first mask feature are fused to obtain a first fused feature. This first fused feature is input into the segmentation model to obtain a second mask. After receiving a second operation on the second mask, based on the second feature of the second object associated with the second operation, the feature fusion operation is returned to be executed until the target mask corresponding to the first point cloud data is obtained. Thus, based on simple user operations on the generated mask, the point cloud data segmentation result is corrected, improving the efficiency of point cloud data segmentation and reducing the cost of point cloud data annotation.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to the fields of artificial intelligence technology such as computer vision, image processing, and large models. Specifically, it relates to a method for segmenting point cloud data, a method for generating segmentation models, an apparatus, and a device. Background Technology

[0002] Point cloud segmentation divides a point cloud based on spatial, geometric, and textural features, so that point clouds within the same segment have similar features.

[0003] In related technologies, deep learning models are typically used to segment large-scale point cloud data. However, training deep learning models requires significant manpower from labelers to correct the labels of each point or region in the initial segmented data, resulting in high labeling costs. Summary of the Invention

[0004] This disclosure aims to at least partially address one of the technical problems in the related art.

[0005] The first aspect of this disclosure proposes a method for segmenting point cloud data, including:

[0006] Upon receiving a first operation for a first mask, determine a first feature of the first object associated with the first operation;

[0007] The first feature, the first point cloud feature corresponding to the first point cloud data, and the first mask feature corresponding to the first mask are fused to obtain the first fused feature.

[0008] The first fused feature is input into the segmentation model to obtain the second mask;

[0009] Upon receiving a second operation for the second mask, based on the second feature of the second object associated with the second operation, the operation of performing the feature fusion is returned until the target mask corresponding to the first point cloud data is obtained.

[0010] A second aspect of this disclosure provides a method for generating a point cloud data segmentation model, comprising:

[0011] Obtain a training dataset, a first simulated object, and a first reference mask, wherein the training dataset includes second point cloud data and a corresponding labeled mask;

[0012] The second point cloud feature corresponding to the second point cloud data, the first simulation feature corresponding to the first simulation object, and the second mask feature corresponding to the first reference mask are fused to obtain the second fused feature.

[0013] The second fused feature is input into the initial segmentation model to obtain the first prediction mask;

[0014] Based on the difference between the first prediction mask and the labeled mask, the second simulation object and the loss value are determined;

[0015] The initial segmentation model is reverse-corrected based on the loss value to obtain the corrected segmentation model.

[0016] Based on the third simulated feature corresponding to the second simulated object and the third mask feature corresponding to the first predicted mask, the feature fusion operation is returned until the target segmentation model is obtained.

[0017] A third aspect of this disclosure provides a point cloud data segmentation apparatus, comprising:

[0018] The first determining module is configured to determine a first feature of a first object associated with the first operation upon receiving a first operation for the first mask.

[0019] The first fusion module is used to fuse the first feature, the first point cloud feature corresponding to the first point cloud data, and the first mask feature corresponding to the first mask to obtain the first fused feature.

[0020] The first input module is used to input the first fused feature into the segmentation model to obtain the second mask;

[0021] The first return module is used to, upon receiving a second operation for the second mask, return to perform the feature fusion operation based on the second feature of the second object associated with the second operation, until the target mask corresponding to the first point cloud data is obtained.

[0022] A fourth aspect of this disclosure provides an apparatus for generating a point cloud data segmentation model, comprising:

[0023] The acquisition module is used to acquire a training dataset, a first simulated object, and a first reference mask, wherein the training dataset includes second point cloud data and a corresponding labeled mask;

[0024] The second fusion module is used to fuse the second point cloud features corresponding to the second point cloud data, the first simulation features corresponding to the first simulation object, and the second mask features corresponding to the first reference mask to obtain the second fused features.

[0025] The second input module is used to input the second fused feature into the initial segmentation model to obtain the first prediction mask;

[0026] The second determining module is used to determine the second simulation object and the loss value based on the difference between the first prediction mask and the labeled mask;

[0027] The correction module is used to reverse correct the initial segmentation model based on the loss value to obtain the corrected segmentation model.

[0028] The second return module is used to return the operation of performing the feature fusion based on the third simulated feature corresponding to the second simulated object and the third mask feature corresponding to the first prediction mask, until the target segmentation model is obtained.

[0029] A fifth aspect of this disclosure provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the point cloud data segmentation method proposed in the first aspect of this disclosure and the point cloud data segmentation model generation method proposed in the second aspect of this disclosure.

[0030] A sixth aspect of this disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the point cloud data segmentation method proposed in the first aspect of this disclosure and the point cloud data segmentation model generation method proposed in the second aspect of this disclosure.

[0031] A seventh aspect embodiment of this disclosure provides a computer program product, including a computer program that, when executed by a processor, implements the point cloud data segmentation method proposed in the first aspect embodiment of this disclosure and the point cloud data segmentation model generation method proposed in the second aspect embodiment.

[0032] The point cloud data segmentation method, segmentation model generation method, apparatus, and equipment disclosed herein have the following beneficial effects:

[0033] In this embodiment of the disclosure, upon receiving a first operation on the first mask, the first feature of the first object associated with the first operation is first determined. Then, the first feature, the first point cloud feature corresponding to the first point cloud data, and the first mask feature corresponding to the first mask are fused to obtain a first fused feature. The first fused feature is then input into the segmentation model to obtain a second mask. Upon receiving a second operation on the second mask, based on the second feature of the second object associated with the second operation, the feature fusion operation is returned to be performed until the target mask corresponding to the first point cloud data is obtained. Thus, by simply operating the generated mask based on the user's operation, the point cloud data segmentation result can be corrected, thereby improving the efficiency of point cloud data segmentation and reducing the cost of point cloud data annotation.

[0034] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0035] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0036] Figure 1 This is a flowchart illustrating a point cloud data segmentation method provided in an embodiment of the present disclosure.

[0037] Figure 2 This is a flowchart illustrating a point cloud data segmentation method provided in an embodiment of the present disclosure.

[0038] Figure 3 This is a schematic diagram of the overall framework process for interactive 3D data segmentation provided in an embodiment of the present disclosure;

[0039] Figure 4 This is a flowchart illustrating a method for generating a point cloud data segmentation model according to an embodiment of the present disclosure.

[0040] Figure 5 This is a flowchart illustrating a method for generating a point cloud data segmentation model according to an embodiment of the present disclosure.

[0041] Figure 6 This is a flowchart illustrating a method for generating a point cloud data segmentation model according to an embodiment of the present disclosure.

[0042] Figure 7 A schematic diagram of a point cloud data segmentation device provided in an embodiment of this disclosure;

[0043] Figure 8 A schematic diagram of the structure of a point cloud data segmentation model generation device provided in an embodiment of this disclosure;

[0044] Figure 9 A block diagram of an exemplary computer device suitable for implementing embodiments of the present disclosure is shown. Detailed Implementation

[0045] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0046] This disclosure relates to the fields of artificial intelligence technology, such as computer vision, image processing, and large models.

[0047] Artificial Intelligence (AI) is a new technological science that studies and develops theories, methods, technologies, and application systems to simulate, extend, and expand human intelligence.

[0048] Computer vision refers to machine vision that uses cameras and computers to identify, track, and measure targets instead of human eyes, and further processes the images to make them more suitable for human observation or transmission to instruments for detection.

[0049] Image processing refers to the techniques of analyzing, processing, and manipulating images to meet visual, psychological, or other requirements.

[0050] Large models, also known as Foundation Models, are models that extract knowledge from hundreds of millions of corpora or images, learn, and then produce large models with hundreds of millions of parameters.

[0051] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0052] The following description, with reference to the accompanying drawings, outlines a method for segmenting point cloud data, a method for generating segmentation models, an apparatus, and a device for such data.

[0053] It should be noted that the execution entities of the point cloud data segmentation method and the segmentation model generation method in this embodiment are the point cloud data segmentation device and the segmentation model generation device, respectively. Both devices can be implemented by software and / or hardware, and can be configured in electronic devices. The electronic device can be any device with computing capabilities, such as a personal computer (PC), mobile terminal, server, etc. The mobile terminal can be, for example, a vehicle device, mobile phone, tablet computer, personal digital assistant, wearable device, smart speaker, etc., with various operating systems, touch screens and / or displays.

[0054] Figure 1 This is a flowchart illustrating a point cloud data segmentation method provided in an embodiment of the present disclosure.

[0055] like Figure 1 As shown, the point cloud data segmentation method may include the following steps:

[0056] Step 101: Upon receiving a first operation for the first mask, determine the first feature of the first object associated with the first operation.

[0057] The first mask is the mask output by the segmentation model in the previous step.

[0058] The first operation is an operation on the image data of the first mask, which can be an operation such as clicking, selecting, or text input, etc. This disclosure does not limit it.

[0059] It should be noted that, in this disclosure, the first operation on the first mask may refer to an operation only on the first mask; or it may refer to an operation on the data containing the first mask, such as the first operation being an operation on the data after the first mask and point cloud data are superimposed, etc. This disclosure does not limit it in this way.

[0060] The first object is the operation object determined based on user interaction.

[0061] In some possible implementations, the first object may include any of the following: point cloud data points, selection boxes, or text, so that users can modify the point cloud results through various operations.

[0062] The first feature can be a feature extracted by the encoder to identify the first object.

[0063] It should be noted that when the first operation is a click operation, its associated first object can be a point cloud data point; when the first operation is a selection operation, its associated first object can be a point cloud data selection box; and when the first operation is a text input operation, its associated first object can be text.

[0064] In some possible implementations, an encoder can be used to encode at least one of the content, location information, and label of the region where the first object is located in order to obtain the first feature of the first object.

[0065] The location information can be the coordinates of the first object. When the first object is a selection box, the location information can be the coordinates of the center point of the first object, or the coordinates of each vertex of the first object, etc., and this disclosure does not limit it.

[0066] The label of the region is used to identify whether the region where the first object is located is an error region or an omission region. It can be identified by taking a specific value, or it can be identified by other implementation methods. For example, the label of the region can take a specific value of "0" to identify the region where the first object is located as an error region, and the label of the region can take a specific value of "1" to identify the region where the first object is located as an omission region. This disclosure does not limit this.

[0067] In this disclosure, when the content of the first object is point cloud data points or a selection box, the encoder can encode the position information of the first object and the label of the area where it is located, thereby obtaining the first feature of the first object. When the content of the first object is text, the encoder can encode the content of the first object, thereby obtaining the first feature of the first object, so that the encoder can extract the first feature of the first object through different user operations.

[0068] Step 102: Perform feature fusion on the first feature, the first point cloud feature corresponding to the first point cloud data, and the first mask feature corresponding to the first mask to obtain the first fused feature.

[0069] The dimensions of the first feature, the first point cloud feature, and the first mask feature may be the same or different. In some embodiments, if the dimensions of the first feature, the first point cloud feature, and the first mask feature are different, the dimensions of the first feature, the first point cloud feature, and the first mask feature may be filled in according to the needs of feature fusion so that the three dimensions are the same. This disclosure does not limit this.

[0070] In some possible implementations, the first point cloud feature and the first mask feature can be fused first to obtain the first reference feature. Then, the first reference feature and the first feature are respectively input into the fusion model to obtain the first fused feature, so that when the user performs point cloud data segmentation, he / she can make corrections based on the previous segmentation results.

[0071] In this disclosure, the first point cloud features and the first mask features can be fused by adding them together, or they can be fused by concatenating them together; this disclosure does not limit the choice. However, since the feature fusion process is an iterative process, in order to reduce the computational load, the addition method can be chosen for fusion.

[0072] Step 103: Input the first fused feature into the segmentation model to obtain the second mask.

[0073] The second mask is the mask output by the segmentation model based on the first fusion feature.

[0074] Step 104: Upon receiving a second operation for the second mask, based on the second feature of the second object associated with the second operation, return to perform the feature fusion operation until the target mask corresponding to the first point cloud data is obtained.

[0075] The second operation is an image data operation on the second mask, which can be a click, a selection box, or text input. It can be the same as the first operation or different from the first operation. This disclosure does not limit it in this regard.

[0076] The second object is the operation object determined based on the user's interaction.

[0077] In some possible implementations, the second object may include any of the following: point cloud data points, selection boxes, or text.

[0078] The second feature is a feature extracted by the encoder and used to identify the second object.

[0079] The target mask is the final mask corresponding to the first point cloud data.

[0080] In this disclosure, upon receiving a second operation targeting the second mask, an error can be assumed to have occurred in the second mask. In this case, based on the second feature of the second object associated with the second operation, a feature fusion operation is performed until the obtained mask can be determined as the target mask corresponding to the first point cloud data. The correction of the point cloud data segmentation result can be completed through simple user interactions such as clicking or selecting boxes; the operation is simple and efficient.

[0081] In some possible implementations, if no operation is received on the newly generated mask within a preset time after its generation, it can be determined that the newly generated mask is an accurate mask, thus ending the mask update process.

[0082] The preset duration can be pre-set to represent the time limit for receiving operations on the newly generated mask; this disclosure does not limit its length.

[0083] In this embodiment of the disclosure, upon receiving a first operation on the first mask, the first feature of the first object associated with the first operation is first determined. Then, the first feature, the first point cloud feature corresponding to the first point cloud data, and the first mask feature corresponding to the first mask are fused to obtain a first fused feature. The first fused feature is then input into the segmentation model to obtain a second mask. Upon receiving a second operation on the second mask, based on the second feature of the second object associated with the second operation, the feature fusion operation is returned to be performed until the target mask corresponding to the first point cloud data is obtained. Thus, by simply manipulating the generated mask by the user, the point cloud data segmentation result can be corrected, thereby improving the efficiency of point cloud data segmentation and reducing the cost of point cloud data annotation.

[0084] Figure 2 This is a flowchart illustrating a point cloud data segmentation method provided in an embodiment of the present disclosure.

[0085] like Figure 2 As shown, the point cloud data segmentation method may include the following steps:

[0086] Step 201: Upon receiving a first operation for the first mask, determine the first feature of the first object associated with the first operation.

[0087] Step 202: Fuse the first point cloud features with the first mask features to obtain the first reference features.

[0088] The specific implementation of steps 201 and 202 can be found in the detailed descriptions of other embodiments in this disclosure, and will not be repeated here.

[0089] Step 203: Input the first feature into the self-attention processing network in the fusion model to obtain the first vector.

[0090] Here, the first vector is the vector generated after the first feature is processed by the self-attention network.

[0091] Step 204: Input the first reference vector and the first vector in the first reference feature into the first cross-attention network in the fusion model to obtain the first fusion vector.

[0092] The first reference vector is a vector in the first reference feature, which can be used to perform the first cross-attention calculation with the first vector.

[0093] The first fusion vector is the fusion vector generated after the first reference vector and the first vector are processed by cross-attention calculation.

[0094] Step 205: Input the first fusion vector into the perceptual network in the fusion model to obtain the second vector.

[0095] The specific implementation of step 205 can be found in the detailed descriptions of other embodiments in this disclosure, and will not be repeated here.

[0096] Step 206: Input the second vector and the second reference vector in the first reference feature into the second cross-attention network in the fusion model to obtain the first fusion feature.

[0097] The second reference vector is a vector in the first reference feature, which can be used to perform the second cross-attention calculation with the second vector.

[0098] The first fusion feature is the fusion feature obtained after the second vector and the second reference vector are processed by the second cross attention.

[0099] Step 207: Input the first fused feature into the segmentation model to obtain the second mask.

[0100] Step 208: Upon receiving a second operation for the second mask, based on the second feature of the second object associated with the second operation, return to perform the feature fusion operation until the target mask corresponding to the first point cloud data is obtained.

[0101] The specific implementation of steps 207 to 208 can be found in the detailed descriptions of other embodiments in this disclosure, and will not be repeated here.

[0102] The following is based on Figure 3 The point cloud data segmentation method provided in this disclosure will be described in detail using examples. Figure 3 This is a schematic diagram of the overall framework process for interactive 3D data segmentation provided in an embodiment of this disclosure, as shown below. Figure 3 As shown, kv is the first reference vector in the first reference feature, q is the second reference vector in the first reference feature, 3D Mask is the 3D mask, and Mask is the mask.

[0103] Taking 3D point cloud data input as an example, the point cloud encoder first extracts the point cloud features corresponding to the point cloud data. Then, the point cloud features are added to the features of the 3D mask output by the segmentation model in the previous step. Another branch takes 3D point cloud data points, selection boxes, or text as input, obtains the corresponding features through a prompt encoder, and then inputs the obtained features and point cloud features into a fusion model for fusion. Finally, the fused features are input into the 3D segmentation head to calculate the 3D mask output, thus achieving efficient interactive point cloud data segmentation.

[0104] In this embodiment of the disclosure, upon receiving a first operation on the first mask, the first feature of the first object associated with the first operation is first determined. Then, the first point cloud feature and the first mask feature are fused to obtain a first reference feature. The first feature is input into a self-attention processing network in the fusion model to obtain a first vector. Then, the first reference vector and the first vector in the first reference feature are respectively input into a first cross-attention network in the fusion model to obtain a first fusion vector. After that, the first fusion vector is input into a perceptual network in the fusion model to obtain a second vector. Then, the second vector and the second reference vector in the first reference feature are respectively input into a second cross-attention network in the fusion model to obtain a first fusion feature. After that, the first fusion feature is input into a segmentation model to obtain a second mask. Upon receiving a second operation on the second mask, based on the second feature of the second object associated with the second operation, the feature fusion operation is returned to be performed until the target mask corresponding to the first point cloud data is obtained. Thus, by utilizing the fusion model to fuse the features of the user-operated object with the reference features, the efficiency of the point cloud data segmentation and correction process is further improved.

[0105] Figure 4 This is a flowchart illustrating a method for generating a point cloud data segmentation model according to an embodiment of the present disclosure.

[0106] like Figure 4 As shown, the method for generating this point cloud data segmentation model may include the following steps:

[0107] Step 401: Obtain the training dataset, the first simulated object, and the first reference mask, wherein the training dataset includes the second point cloud data and the corresponding labeled mask.

[0108] The training dataset is a pre-trained dataset containing labeled second point cloud data and corresponding label masks.

[0109] The first simulated object is a virtual operation object determined to simulate the user interaction process.

[0110] In some possible implementations, the first simulated object can include any of the following: text, point cloud data points, or a selection box. By simulating and training with multiple manipulation objects, the user's interaction process can be made more convenient.

[0111] It should be noted that text objects may only be used once or a limited number of times during model training, while point or box objects can be used multiple times.

[0112] The first reference mask can be pre-set or randomly generated; this disclosure does not limit it.

[0113] The annotation mask is the mask generated after annotating the second point cloud data.

[0114] In this disclosure, before training the segmentation model, a training dataset, a first simulated object, and a first reference mask are first obtained.

[0115] Step 402: Perform feature fusion on the second point cloud feature corresponding to the second point cloud data, the first simulation feature corresponding to the first simulation object, and the second mask feature corresponding to the first reference mask to obtain the second fused feature.

[0116] In some possible implementations, an initial encoder can be used to encode at least one of the following: the content, location information, and label of the region containing the first simulated object, to obtain the first simulated feature of the first simulated object. Then, the second point cloud feature and the second mask feature are fused to obtain the second reference feature. Finally, the second reference feature and the first simulated feature are respectively input into the initial fusion model to obtain the second fused feature. By using the fusion model to fuse the features of the virtual operation object simulating user operations with the reference feature, the accuracy of the model training process is improved.

[0117] Step 403: Input the second fused feature into the initial segmentation model to obtain the first prediction mask.

[0118] The first prediction mask is the prediction mask output by the initial segmentation model, used to identify the prediction result of the second point cloud data.

[0119] Step 404: Based on the difference between the first prediction mask and the labeled mask, determine the second simulation object and the loss value.

[0120] The second simulated object is a virtual operation object determined based on the difference between the first predicted mask and the labeled mask.

[0121] In some possible implementations, the second simulation object may include any of the following: point cloud data points, selection boxes, etc., without limitation in this disclosure.

[0122] The loss value is a numerical value calculated based on the difference between the first predicted mask and the labeled mask.

[0123] In some possible implementations, the difference region between the first prediction mask and the labeled mask can be identified as the error region first, and then any point within the error region or a selection box containing any point can be identified as the second simulation object.

[0124] In this disclosure, when there is a difference between the first predicted mask and the labeled mask, it can be assumed that the initial segmentation model's segmentation prediction result for the second point cloud data has errors or omissions. In this case, the second simulated object and the loss value can be determined based on the difference between the first predicted mask and the labeled mask. When determining the second simulated object, the difference region can be first identified as the error region, and then any point within the error region or a selection box containing any point can be identified as the second simulated object, thereby improving the efficiency of training the model.

[0125] Step 405: Perform reverse correction on the initial segmentation model based on the loss value to obtain the corrected segmentation model.

[0126] In this disclosure, after obtaining the loss value between the first predicted mask and the labeled mask, the initial segmentation model can be reverse-corrected based on the loss value to obtain the corrected segmentation model, so that its output value is closer to the labeled mask.

[0127] Step 406: Based on the third simulated feature corresponding to the second simulated object and the third mask feature corresponding to the first predicted mask, return to perform feature fusion operation until the target segmentation model is obtained.

[0128] The target segmentation model is the final segmentation model obtained after the model training is completed.

[0129] It should be noted that returning to perform feature fusion means using the modified segmentation model to process the feature obtained by fusing the third simulated feature and the third mask feature corresponding to the first predicted mask.

[0130] In this disclosure, the condition for the end of model training may be that the number of corrections to the segmentation model has reached a preset number of corrections, or the loss value has reached a preset range. This disclosure does not limit this.

[0131] In this embodiment, a training dataset, a first simulated object, and a first reference mask are first acquired. The second point cloud features corresponding to the second point cloud data, the first simulated features corresponding to the first simulated object, and the second mask features corresponding to the first reference mask are fused to obtain a second fused feature. This second fused feature is then input into an initial segmentation model to obtain a first predicted mask. Based on the difference between the first predicted mask and the labeled mask, a second simulated object and a loss value are determined. The initial segmentation model is then reverse-corrected based on the loss value to obtain a corrected segmentation model. Finally, based on the third simulated features corresponding to the second simulated object and the third mask features corresponding to the first predicted mask, the feature fusion operation is returned until the target segmentation model is obtained. Therefore, during model training, by generating a second simulated object based on the difference between the predicted mask and the labeled mask to simulate user operations, the accuracy and reliability of the generated segmentation model are improved.

[0132] Figure 5 This is a flowchart illustrating a method for generating a point cloud data segmentation model according to an embodiment of the present disclosure.

[0133] like Figure 5 As shown, the method for generating this point cloud data segmentation model may include the following steps:

[0134] Step 501: Obtain the training dataset, the first simulated object, and the first reference mask, wherein the training dataset includes the second point cloud data and the corresponding labeled mask.

[0135] Step 502: Perform feature fusion on the second point cloud feature corresponding to the second point cloud data, the first simulation feature corresponding to the first simulation object, and the second mask feature corresponding to the first reference mask to obtain the second fused feature.

[0136] Step 503: Input the second fused feature into the initial segmentation model to obtain the first prediction mask.

[0137] The specific implementation of steps 501 to 503 can be found in the detailed descriptions of other embodiments in this disclosure, and will not be repeated here.

[0138] Step 504: The difference region between the first predicted mask and the labeled mask is identified as the error region.

[0139] In this disclosure, when there is a difference between the first prediction mask and the labeled mask, it can be considered that the prediction result of the first prediction mask for the second point cloud data is inaccurate in the region of difference, and therefore the region of difference can be identified as the error region.

[0140] Step 505: Determine the third distance between each point within the error region and the boundary line of the error region.

[0141] The third distance is the distance between a point within the error region and the boundary line of the error region.

[0142] It should be noted that an error region has multiple boundary lines. In this disclosure, the distance between each point in the error region and its nearest boundary line can be calculated and determined as the third distance.

[0143] Step 506: Determine the first threshold and / or the second threshold based on the third distance corresponding to each point.

[0144] Wherein, the first threshold is the critical distance between any point in the error region and the center point. It can be the average of each third distance, or the average of some third distance values, etc. This disclosure does not limit it.

[0145] The second threshold is the critical distance between the point contained in any selection box within the error area and the center point. It may be the same as the first threshold or it may be different from the first threshold. This disclosure does not limit this.

[0146] In this disclosure, a first threshold and / or a second threshold are determined based on the third distance corresponding to each point, thereby ensuring the accuracy of determining any point or any selection box as the second simulated object within the error area based on the first threshold and / or the second threshold.

[0147] In some possible implementations, the n third distances are first sorted from largest to smallest to obtain the top m third distances, where n is the amount of point cloud data contained in the error region and m is a value less than or equal to n / 2. Then, based on the top m third distances, a first threshold and / or a second threshold are determined.

[0148] It should be noted that when determining the first threshold and / or the second threshold based on the first m third distances, any distance value less than or equal to the average of the first m third distances can be determined as the first threshold and / or the second threshold, thereby improving the reliability of the first threshold and / or the second threshold values.

[0149] For example, if the average of the first m third distances is 2 mm, any distance value less than or equal to 2 mm can be determined as the first threshold and / or the second threshold. For example, the first threshold can be determined as 1.8 mm and the second threshold can be determined as 1.5 mm.

[0150] Step 507: Select any point within the error area or a selection box containing any point as the second simulation object.

[0151] In some possible implementations, the center point of the error region can be determined as the second simulated object.

[0152] In some possible implementations, any selection box located within the error region and containing a center point can also be designated as a second simulation object.

[0153] In some possible implementations, any point within the error region whose first distance from the center point is less than a first threshold can be identified as the second simulated object.

[0154] Wherein, the first distance is the distance between any point in the error region and the center point.

[0155] In some possible implementations, any selection box located within the error region and whose second distance between the included points and the center point is less than a second threshold can also be identified as the second simulated object.

[0156] The second distance is the distance between the point contained in any selection box within the error area and the center point. It may be the same as the first distance or it may be different from the first distance. This disclosure does not limit this.

[0157] In this embodiment, a training dataset, a first simulated object, and a first reference mask are first acquired. The second point cloud features corresponding to the second point cloud data, the first simulated features corresponding to the first simulated object, and the second mask features corresponding to the first reference mask are fused to obtain a second fused feature. This second fused feature is then input into an initial segmentation model to obtain a first predicted mask. The difference region between the first predicted mask and the labeled mask is identified as an error region. A third distance between each point within the error region and the boundary line of the error region is then determined. Based on the third distances corresponding to each point, a first threshold and / or a second threshold are determined. Finally, any point within the error region or any selection box containing any point is identified as a second simulated object. Thus, based on the error region between the first predicted mask and the labeled mask, any point or any selection box located within it is identified as a second simulated object, thereby enabling model training. This improves the efficiency of training the segmentation model while ensuring its accuracy.

[0158] Figure 6 This is a flowchart illustrating a method for generating a point cloud data segmentation model according to an embodiment of the present disclosure.

[0159] like Figure 6 As shown, the method for generating this point cloud data segmentation model may include the following steps:

[0160] Step 601: Obtain the training dataset, the first simulated object, and the first reference mask, wherein the training dataset includes the second point cloud data and the corresponding labeled mask.

[0161] Step 602: Encode at least one of the content, location information and label of the region of the first simulated object using the initial encoder to obtain the first simulated feature of the first simulated object.

[0162] Step 603: Fuse the second point cloud features with the second mask features to obtain the second reference features.

[0163] Step 604: Input the second reference feature and the first simulation feature into the initial fusion model to obtain the second fusion feature.

[0164] Step 605: Input the second fused feature into the initial segmentation model to obtain the first prediction mask.

[0165] Step 606: Based on the difference between the first prediction mask and the labeled mask, determine the second simulation object and the loss value.

[0166] The specific implementation of steps 601 to 606 can be found in the detailed descriptions of other embodiments in this disclosure, and will not be repeated here.

[0167] Step 607: Based on the loss value, reverse correction is performed on the initial encoder and the initial fusion model respectively to obtain the corrected encoder and fusion model.

[0168] In this disclosure, after obtaining the loss value between the first prediction mask and the labeled mask, the initial encoder and the initial fusion model can be reverse-corrected based on the loss value to obtain the corrected encoder and fusion model, thereby improving the accuracy of the encoder and fusion model.

[0169] Step 608: Perform reverse correction on the initial segmentation model based on the loss value to obtain the corrected segmentation model.

[0170] Steps 607 and 608 can be executed simultaneously.

[0171] It should be noted that when the initial encoder, initial fusion model, and initial segmentation model are simultaneously reverse-corrected based on the loss value, the correction gradient of each part can be different or the same.

[0172] Step 609: Based on the third simulated feature corresponding to the second simulated object and the third mask feature corresponding to the first predicted mask, return to perform feature fusion operation until the target segmentation model is obtained.

[0173] The specific implementation of steps 608 to 609 can be found in the detailed descriptions of other embodiments in this disclosure, and will not be repeated here.

[0174] In this embodiment, a training dataset, a first simulated object, and a first reference mask are first acquired. An initial encoder encodes at least one of the content, location information, and label of the region containing the first simulated object to obtain a first simulated feature. The second point cloud feature is then fused with the second mask feature to obtain a second reference feature. The second reference feature and the first simulated feature are then input into an initial fusion model to obtain a second fusion feature. The second fusion feature is then input into an initial segmentation model to obtain a first predicted mask. Based on the difference between the first predicted mask and the labeled mask, a second simulated object and a loss value are determined. Based on the loss value, the initial encoder and the initial fusion model are reverse-corrected to obtain a corrected encoder and fusion model. Simultaneously, the initial segmentation model is reverse-corrected based on the loss value to obtain a corrected segmentation model. Finally, based on the third simulated feature corresponding to the second simulated object and the third mask feature corresponding to the first predicted mask, the feature fusion operation is performed again until the target segmentation model is obtained. Thus, based on the difference between the predicted mask and the labeled mask, a second simulated object simulating user operations is generated, while the encoder, fusion model, and segmentation model are simultaneously reverse-corrected, thereby improving the accuracy of the encoder, fusion model, and segmentation model.

[0175] To achieve the above embodiments, this disclosure also proposes a point cloud data segmentation device.

[0176] Figure 7 This is a schematic diagram of the structure of a point cloud data segmentation device provided in an embodiment of the present disclosure.

[0177] like Figure 7 As shown, the point cloud data segmentation device 700 includes: a first determination module 701, a first fusion module 702, a first input module 703, and a first return module 704.

[0178] The first determining module 701 is configured to determine a first feature of a first object associated with the first operation upon receiving a first operation for a first mask.

[0179] The first fusion module 702 is used to fuse the first feature, the first point cloud feature corresponding to the first point cloud data, and the first mask feature corresponding to the first mask to obtain the first fused feature.

[0180] The first input module 703 is used to input the first fused feature into the segmentation model to obtain the second mask;

[0181] The first return module 704 is used to, upon receiving a second operation for the second mask, return an operation to perform feature fusion based on the second feature of the second object associated with the second operation, until the target mask corresponding to the first point cloud data is obtained.

[0182] In one possible implementation of this disclosure, the object includes any of the following: point cloud data points, selection boxes, and text.

[0183] In one possible implementation of this disclosure, the first determining module 701 is specifically used for:

[0184] The encoder is used to encode at least one of the content, location information and the label of the area where the first object is located, in order to obtain the first feature of the first object.

[0185] In one possible implementation of this disclosure, the first fusion module 702 is specifically used for:

[0186] The first point cloud feature and the first mask feature are fused together to obtain the first reference feature;

[0187] The first reference feature and the first feature are respectively input into the fusion model to obtain the first fusion feature.

[0188] In one possible implementation of this disclosure, the first fusion module 702 is further configured to:

[0189] The first feature is input into the self-attention processing network in the fusion model to obtain the first vector;

[0190] The first reference vector and the first vector in the first reference feature are respectively input into the first cross-attention network in the fusion model to obtain the first fusion vector;

[0191] The first fusion vector is input into the perceptual network in the fusion model to obtain the second vector;

[0192] The second vector and the second reference vector from the first reference feature are respectively input into the second cross-attention network in the fusion model to obtain the first fusion feature.

[0193] In one possible implementation of this disclosure, the first return module 704 is specifically used for:

[0194] If no operation is received for the newly generated mask within the preset time period, the newly generated mask is determined to be the target mask corresponding to the first point cloud data.

[0195] The functions and specific implementation principles of the modules described in this embodiment can be found in the above method embodiments, and will not be repeated here.

[0196] In this embodiment of the disclosure, upon receiving a first operation on the first mask, the first feature of the first object associated with the first operation is first determined. Then, the first feature, the first point cloud feature corresponding to the first point cloud data, and the first mask feature corresponding to the first mask are fused to obtain a first fused feature. The first fused feature is then input into the segmentation model to obtain a second mask. Upon receiving a second operation on the second mask, based on the second feature of the second object associated with the second operation, the feature fusion operation is returned to be performed until the target mask corresponding to the first point cloud data is obtained. Thus, by simply operating the generated mask based on the user's operation, the point cloud data segmentation result can be corrected, thereby improving the efficiency of point cloud data segmentation and reducing the cost of point cloud data annotation.

[0197] To implement the above embodiments, this disclosure also proposes an apparatus for generating point cloud data segmentation models.

[0198] Figure 8 This is a schematic diagram of the structure of a point cloud data segmentation model generation device provided in an embodiment of the present disclosure.

[0199] like Figure 8 As shown, the point cloud data segmentation model generation device 800 includes: an acquisition module 801, a second fusion module 802, a second input module 803, a second determination module 804, a correction module 805, and a second return module 806.

[0200] The acquisition module 801 is used to acquire the training dataset, the first simulated object, and the first reference mask, wherein the training dataset includes the second point cloud data and the corresponding labeled mask;

[0201] The second fusion module 802 is used to fuse the second point cloud features corresponding to the second point cloud data, the first simulation features corresponding to the first simulation object, and the second mask features corresponding to the first reference mask to obtain the second fused features.

[0202] The second input module 803 is used to input the second fused feature into the initial segmentation model to obtain the first prediction mask;

[0203] The second determination module 804 is used to determine the second simulation object and the loss value based on the difference between the first prediction mask and the labeled mask;

[0204] The correction module 805 is used to perform reverse correction on the initial segmentation model based on the loss value to obtain the corrected segmentation model.

[0205] The second return module 806 is used to return the operation of performing feature fusion based on the third simulated feature corresponding to the second simulated object and the third mask feature corresponding to the first prediction mask, until the target segmentation model is obtained.

[0206] In one possible implementation of this disclosure, the first simulated object includes any of the following: text, point cloud data points, and a selection box; the second simulated object includes any of the following: point cloud data points and a selection box.

[0207] In one possible implementation of this disclosure, the second determining module 804 is specifically used for:

[0208] The region of difference between the first predicted mask and the labeled mask is identified as the error region;

[0209] Select any point within the error area, or a selection box containing any point, as the second simulation object.

[0210] In one possible implementation of this disclosure, the second determining module 804 is specifically used for:

[0211] The center point of the error region is designated as the second simulation object; or...

[0212] Select any selection box located within the error area and containing the center point as the second simulation object; or,

[0213] Any point within the error region whose first distance from the center point is less than a first threshold is identified as the second simulation object; or...

[0214] Any selection box located within the error region and whose second distance between the included points and the center point is less than the second threshold is identified as the second simulation object.

[0215] In one possible implementation of this disclosure, the second determining module 804 is specifically used for:

[0216] Determine the third distance between each point within the error region and the boundary line of the error region;

[0217] Based on the third distance corresponding to each point, determine the first threshold and / or the second threshold.

[0218] In one possible implementation of this disclosure, the second determining module 804 is specifically used for:

[0219] Sort the n third distances in descending order to obtain the top m third distances, where n is the amount of point cloud data contained in the error region and m is a value less than or equal to n / 2.

[0220] Based on the first m third distances, determine the first threshold and / or the second threshold.

[0221] In one possible implementation of this disclosure, the second fusion module 802 is specifically used for:

[0222] The first simulated object's content, location information, and the label of its region are encoded using an initial encoder to obtain the first simulated feature of the first simulated object.

[0223] The second point cloud feature is fused with the second mask feature to obtain the second reference feature;

[0224] The second reference feature and the first simulation feature are respectively input into the initial fusion model to obtain the second fusion feature.

[0225] In one possible implementation of this disclosure, the second fusion module 802 is further configured to:

[0226] Based on the loss value, the initial encoder and the initial fusion model are reverse-corrected to obtain the corrected encoder and fusion model.

[0227] The functions and specific implementation principles of the modules described in this embodiment can be found in the above method embodiments, and will not be repeated here.

[0228] In this embodiment, a training dataset, a first simulated object, and a first reference mask are first acquired. The second point cloud features corresponding to the second point cloud data, the first simulated features corresponding to the first simulated object, and the second mask features corresponding to the first reference mask are fused to obtain a second fused feature. This second fused feature is then input into an initial segmentation model to obtain a first predicted mask. Based on the difference between the first predicted mask and the labeled mask, a second simulated object and a loss value are determined. The initial segmentation model is then reverse-corrected based on the loss value to obtain a corrected segmentation model. Finally, based on the third simulated features corresponding to the second simulated object and the third mask features corresponding to the first predicted mask, the feature fusion operation is returned until the target segmentation model is obtained. Therefore, during model training, by generating a second simulated object based on the difference between the predicted mask and the labeled mask to simulate user operations, the accuracy and reliability of the generated segmentation model are improved.

[0229] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0230] Figure 9A schematic block diagram of an example electronic device 900 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0231] like Figure 9 As shown, device 900 includes a computing unit 901, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 902 or a computer program loaded from storage unit 908 into random access memory (RAM) 903. RAM 903 may also store various programs and data required for the operation of device 900. The computing unit 901, ROM 902, and RAM 903 are interconnected via bus 904. Input / output (I / O) interface 905 is also connected to bus 904.

[0232] Multiple components in device 900 are connected to I / O interface 905, including: input unit 906, such as keyboard, mouse, etc.; output unit 907, such as various types of monitors, speakers, etc.; storage unit 908, such as disk, optical disk, etc.; and communication unit 909, such as network card, modem, wireless transceiver, etc. Communication unit 909 allows device 900 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0233] The computing unit 901 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 901 performs the various methods and processes described above, such as point cloud data segmentation methods. For example, in some embodiments, the point cloud data segmentation method can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 908. In some embodiments, part or all of the computer program can be loaded and / or installed on device 900 via ROM 902 and / or communication unit 909. When the computer program is loaded into RAM 903 and executed by the computing unit 901, one or more steps of the point cloud data segmentation method described above can be performed. Alternatively, in other embodiments, the computing unit 901 can be configured to perform point cloud data segmentation methods by any other suitable means (e.g., by means of firmware).

[0234] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0235] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0236] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0237] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0238] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), the Internet, and blockchain networks.

[0239] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service ecosystem, addressing the shortcomings of traditional physical hosts and VPS (Virtual Private Server, or simply "VPS") services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0240] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0241] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified. In the description of this disclosure, the words "if" and "suppose" as used may be interpreted as "when," "when," "in response to determination," or "in the circumstances."

[0242] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for segmenting point cloud data, comprising: Upon receiving a first operation for a first 3D mask, a first feature of a first object associated with the first operation is determined, wherein the first operation is an operation on image data of the first 3D mask, the first object is an operation object determined based on user interaction, and the first feature includes features extracted by an encoder to identify the first object. The first feature, the first point cloud feature corresponding to the first point cloud data, and the first mask feature corresponding to the first 3D mask are fused to obtain the first fused feature. The first fused feature is input into the 3D segmentation head of the segmentation model to obtain the second 3D mask; Upon receiving a second operation for the second 3D mask, based on the second feature of the second object associated with the second operation, the operation of feature fusion is returned to be performed until the target 3D mask corresponding to the first point cloud data is obtained. The second operation is an image data operation for the second 3D mask. The step of fusing the first feature, the first point cloud feature corresponding to the first point cloud data, and the first mask feature corresponding to the first 3D mask to obtain the first fused feature includes: The first point cloud feature is added to the first mask feature to obtain the first reference feature; The first feature is input into the self-attention processing network in the fusion model to obtain the first vector. The first feature is the feature of 3D point cloud data points and 3D selection box extracted based on the prompt encoder. The first reference vector and the first vector in the first reference feature are respectively input into the first cross-attention network in the fusion model to obtain the first fusion vector; The first fusion vector is input into the perceptual network in the fusion model to obtain the second vector; The second vector and the second reference vector from the first reference feature are respectively input into the second cross-attention network in the fusion model to obtain the first fusion feature.

2. The method as described in claim 1, wherein, The object includes any of the following: point cloud data points, selection boxes, and text.

3. The method as described in claim 2, wherein, The first feature of determining the first object associated with the first operation includes: The encoder is used to encode at least one of the content, location information and the label of the area where the first object is located, in order to obtain the first feature of the first object.

4. The method as described in any one of claims 1-3, wherein, The process of obtaining the target 3D mask corresponding to the first point cloud data includes: If no operation is received for the newly generated mask within a preset time period, the newly generated mask is determined to be the target 3D mask corresponding to the first point cloud data.

5. A method for generating a point cloud data segmentation model, comprising: Obtain a training dataset, a first simulated object, and a first reference mask, wherein the training dataset includes second point cloud data and a corresponding labeled mask, and the first simulated object is a virtual operation object determined to simulate the user interaction process; The second point cloud feature corresponding to the second point cloud data, the first simulation feature corresponding to the first simulation object, and the second mask feature corresponding to the first reference mask are fused to obtain the second fused feature. The first simulation feature includes the feature extracted by the encoder to identify the first simulation object. The second fused feature is input into the initial segmentation model to obtain the first prediction mask; Based on the difference between the first prediction mask and the labeled mask, the second simulation object and the loss value are determined; The initial segmentation model is reverse-corrected based on the loss value to obtain the corrected segmentation model. Based on the third simulated feature corresponding to the second simulated object and the third mask feature corresponding to the first predicted mask, return to perform the feature fusion operation until the target segmentation model is obtained; The step of determining the second simulated object based on the difference between the first predicted mask and the labeled mask includes: The region of difference between the first predicted mask and the labeled mask is identified as the error region; Any point within the error region or a selection box containing any point is designated as the second simulation object; The step of fusing the second point cloud features corresponding to the second point cloud data, the first simulation features corresponding to the first simulation object, and the second mask features corresponding to the first reference mask to obtain the second fused features includes: The second point cloud feature is fused with the second mask feature to obtain the second reference feature; The first simulated feature is input into the self-attention processing network in the fusion model to obtain the first vector; The first reference vector and the first vector in the second reference feature are respectively input into the first cross-attention network in the fusion model to obtain the first fusion vector; The first fusion vector is input into the perceptual network in the fusion model to obtain the second vector; The second vector and the second reference vector in the second reference feature are respectively input into the second cross-attention network in the fusion model to obtain the second fusion feature.

6. The method of claim 5, wherein, The first simulated object includes any one of the following: text, point cloud data points, and a selection box; the second simulated object includes any one of the following: point cloud data points and a selection box.

7. The method of claim 5, wherein, The step of determining any point within the error region or a selection box containing any point as the second simulation object includes: The center point of the error region is determined as the second simulated object; or... Select any selection box located within the error region and containing the center point as the second simulation object; or... Any point within the error region whose first distance from the center point is less than a first threshold is identified as the second simulated object; or... Any selection box located within the error region and whose second distance from the center point is less than a second threshold is identified as the second simulation object.

8. The method of claim 7, wherein, Before determining any point within the error region or a selection box containing any point as the second simulation object, the method further includes: Determine the third distance between each point within the error region and the boundary line of the error region; The first threshold and / or the second threshold are determined based on the third distance corresponding to each point.

9. The method of claim 8, wherein, The step of determining the first threshold and / or the second threshold based on the third distance corresponding to each point includes: Sort the n third distances from largest to smallest to obtain the top m third distances, where n is the amount of point cloud data contained in the error region, and m is a value less than or equal to n / 2; The first threshold and / or the second threshold are determined based on the first m third distances.

10. The method as described in any one of claims 5-9, wherein, The step of fusing the second point cloud features corresponding to the second point cloud data, the first simulation features corresponding to the first simulation object, and the second mask features corresponding to the first reference mask to obtain the second fused features includes: The first simulated object's content, location information, and the label of its region are encoded using an initial encoder to obtain the first simulated feature of the first simulated object. The second point cloud feature is fused with the second mask feature to obtain the second reference feature; The second reference feature and the first simulation feature are respectively input into the initial fusion model to obtain the second fusion feature.

11. The method of claim 10, wherein, After determining the second simulation object and the loss value, the process also includes: Based on the loss value, the initial encoder and the initial fusion model are respectively reverse-corrected to obtain the corrected encoder and fusion model.

12. A point cloud data segmentation device, comprising: The first determining module is configured to determine a first feature of a first object associated with the first operation upon receiving a first operation for a first 3D mask, wherein the first operation is an operation on image data of the first 3D mask, the first object is an operation object determined based on user interaction, and the first feature includes features extracted by an encoder to identify the first object. The first fusion module is used to fuse the first feature, the first point cloud feature corresponding to the first point cloud data, and the first mask feature corresponding to the first mask to obtain the first fused feature. The first input module is used to input the first fused feature into the 3D segmentation head of the segmentation model to obtain the second 3D mask; The first return module is used to, upon receiving a second operation for the second 3D mask, return to perform the feature fusion operation based on the second feature of the second object associated with the second operation, until the target 3D mask corresponding to the first point cloud data is obtained, wherein the second operation is an image data operation for the second mask; Specifically, the first fusion module is used to: add the first point cloud feature to the first mask feature to obtain a first reference feature; input the first feature into the self-attention processing network in the fusion model to obtain a first vector; the first feature is the feature of 3D point cloud data points and 3D selection boxes extracted based on the prompt encoder; input the first reference vector in the first reference feature and the first vector into the first cross-attention network in the fusion model to obtain a first fusion vector; input the first fusion vector into the perception network in the fusion model to obtain a second vector; and input the second vector and the second reference vector in the first reference feature into the second cross-attention network in the fusion model to obtain the first fusion feature.

13. The apparatus of claim 12, wherein, The object includes any of the following: point cloud data points, selection boxes, and text.

14. The apparatus of claim 13, wherein, The first determining module is specifically used for: The encoder is used to encode at least one of the content, location information and the label of the area where the first object is located, in order to obtain the first feature of the first object.

15. The apparatus according to any one of claims 12-14, wherein, The first return module is specifically used for: If no operation is received for the newly generated mask within a preset time period, the newly generated mask is determined to be the target mask corresponding to the first point cloud data.

16. An apparatus for generating a point cloud data segmentation model, comprising: The acquisition module is used to acquire a training dataset, a first simulated object, and a first reference mask. The training dataset includes second point cloud data and a corresponding labeled mask. The first simulated object is a virtual operation object determined to simulate the user interaction process. The second fusion module is used to fuse the second point cloud features corresponding to the second point cloud data, the first simulation features corresponding to the first simulation object, and the second mask features corresponding to the first reference mask to obtain the second fused features. The first simulation features include features extracted by the encoder to identify the first simulation object. The second input module is used to input the second fused feature into the initial segmentation model to obtain the first prediction mask; The second determining module is used to determine the second simulation object and the loss value based on the difference between the first prediction mask and the labeled mask; The correction module is used to perform reverse correction on the initial segmentation model based on the loss value to obtain the corrected segmentation model. The second return module is used to return the operation of performing the feature fusion based on the third simulated feature corresponding to the second simulated object and the third mask feature corresponding to the first prediction mask, until the target segmentation model is obtained; The step of determining the second simulated object based on the difference between the first predicted mask and the labeled mask includes: The region of difference between the first predicted mask and the labeled mask is identified as the error region; Any point within the error region or a selection box containing any point is designated as the second simulation object; The second fusion module is used for, package: The second point cloud feature is fused with the second mask feature to obtain the second reference feature; The first simulated feature is input into the self-attention processing network in the fusion model to obtain the first vector; The first reference vector and the first vector in the second reference feature are respectively input into the first cross-attention network in the fusion model to obtain the first fusion vector; The first fusion vector is input into the perceptual network in the fusion model to obtain the second vector; The second vector and the second reference vector in the second reference feature are respectively input into the second cross-attention network in the fusion model to obtain the second fusion feature.

17. The apparatus of claim 16, wherein, The first simulated object includes any one of the following: text, point cloud data points, and a selection box; the second simulated object includes any one of the following: point cloud data points and a selection box.

18. The apparatus of claim 16, wherein, The second determining module is specifically used for: The center point of the error region is determined as the second simulated object; or... Select any selection box located within the error region and containing the center point as the second simulation object; or... Any point within the error region whose first distance from the center point is less than a first threshold is identified as the second simulated object; or... Any selection box located within the error region and whose second distance from the center point is less than a second threshold is identified as the second simulation object.

19. The apparatus of claim 18, wherein, The second determining module is further configured to: Determine the third distance between each point within the error region and the boundary line of the error region; The first threshold and / or the second threshold are determined based on the third distance corresponding to each point.

20. The apparatus of claim 19, wherein, The second determining module is further configured to: Sort the n third distances from largest to smallest to obtain the top m third distances, where n is the amount of point cloud data contained in the error region, and m is a value less than or equal to n / 2; The first threshold and / or the second threshold are determined based on the first m third distances.

21. The apparatus according to any one of claims 16-20, wherein, The second fusion module is specifically used for: The first simulated object's content, location information, and the label of its region are encoded using an initial encoder to obtain the first simulated feature of the first simulated object. The second point cloud feature is fused with the second mask feature to obtain the second reference feature; The second reference feature and the first simulation feature are respectively input into the initial fusion model to obtain the second fusion feature.

22. The apparatus of claim 21, wherein, The second fusion module is also used for: Based on the loss value, the initial encoder and the initial fusion model are respectively reverse-corrected to obtain the corrected encoder and fusion model.

23. An electronic device, characterized in that, include: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores instructions that may be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method of any one of claims 1-4 or 5-11.

24. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-4 or 5-11.

25. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-4 or 5-11.

Images