Model training method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a model training method and device, electronic equipment and a storage medium, and the method comprises the steps: carrying out the selective segmentation of a region, where a target instance exists, in an original point cloud data set, and training a point cloud recognition model through employing point cloud image blocks obtained through segmentation, thereby reducing the size of a training sample as much as possible, reducing the model training time, and improving the training efficiency. Meanwhile, the target instance image features in the original point cloud image are fully utilized, and the performance of the point cloud recognition model is improved. The model training method comprises the following steps: receiving an original point cloud image; based on a target detection algorithm, marking a prediction area of a target instance in the original point cloud image; based on a prediction area, carrying out image segmentation on the original point cloud image to obtain a first point cloud image block containing a target instance; and training the initial neural network model by taking the first point cloud image block as a training sample to obtain the point cloud recognition model.

Description

[Technical field]

[0001] The embodiments of the present application relate to the field of machine learning technology, and in particular to a model training method, device, electronic device and storage medium. [Background technology]

[0002] In recent years, with the development of autonomous driving technology, the use of three-dimensional perception algorithms to identify target instances in the point cloud data output by automotive radars and extract the position, size, shape and category information of the target instances for the vehicle to perform subsequent operations such as road condition judgment and semantic map generation has become a more commonly used object recognition method.

[0003] In the prior art, supervised training is often used to train the point cloud recognition model in the 3D perception algorithm. This method requires manual or self-training annotation of the complete original point cloud image as a training sample. However, the original point cloud image contains both target instance information and a large amount of invalid noise information. Inputting the complete original point cloud image for training will cause the noise information to be read and recognized at the same time, resulting in a large waste of computing power, which greatly prolongs the training time and is not conducive to improving the performance of the point cloud recognition model. [Summary of the invention]

[0004] The embodiments of the present application provide a model training method, device, electronic device and storage medium, which selectively segments the area where the target instance exists in the original point cloud data set, and uses the segmented point cloud image blocks to train the point cloud recognition model, thereby minimizing the size of the training sample and reducing the model training time while retaining the complete image features of the target instance. At the same time, the target instance image features of the original point cloud image are fully utilized to improve the performance of the point cloud recognition model.

[0005] In a first aspect, an embodiment of the present application provides a model training method, which is applied to a terminal side, and the method includes:

[0006] Receiving an original point cloud image;

[0007] Based on the target detection algorithm, a predicted area of ​​the target instance is detected in the original point cloud image;

[0008] Based on the predicted area, performing image segmentation on the original point cloud image to obtain a first point cloud image block containing the target instance;

[0009] The initial neural network model is trained using the first point cloud image block as a training sample to obtain the point cloud recognition model.

[0010] In an embodiment of the present application, before training the point cloud neural network model, the prediction area in the target instance is marked based on the target detection algorithm, and the original point cloud image is cut based on the prediction area to obtain a first point cloud image block for training that contains the target instance. This ensures that the training sample contains valid target instance information and filters out a large amount of noise information in the original point cloud image. While fully retaining the target instance image features of the original point cloud image, it avoids the waste of computing power in the model training process, thereby shortening the training time of the point cloud recognition model and improving the model performance.

[0011] Optionally, the first point cloud image block is composed of a second point cloud image block and a third point cloud image block, and the performing image segmentation on the original point cloud image based on the predicted area to obtain the first point cloud image block containing the target instance includes:

[0012] Randomly segmenting the point cloud in the prediction area in the original point cloud image to obtain the second point cloud image block;

[0013] The original point cloud image is randomly segmented to include a single predicted area to obtain the third point cloud image block.

[0014] In an embodiment of the present application, by performing random segmentation based on different strategies on the point cloud and the original point cloud image within the prediction area, two different image blocks containing partial image features and complete image features of the target instance are obtained for training, thereby enriching the diversity of the training data set and enhancing the robustness and generalization ability of the trained point cloud recognition model.

[0015] Optionally, after randomly segmenting the original point cloud image to include a single prediction area to obtain the third point cloud image block, the method further includes:

[0016] Based on a filtering algorithm, filtering the second point cloud image block and the third point cloud image block;

[0017] Performing data enhancement processing on the filtered second point cloud image block and the filtered third point cloud image block to obtain an enhanced second point cloud image block and an enhanced third point cloud image block, wherein the data enhancement processing includes at least one of random displacement, random cropping, random rotation and flipping, random scaling, adding random noise, or performing random spatial projection;

[0018] The training of the initial neural network model using the first point cloud image block as a training sample to obtain the point cloud recognition model comprises:

[0019] The enhanced second point cloud image block and the enhanced third point cloud image block are used as training samples to train the initial neural network model to obtain the point cloud recognition model.

[0020] In an embodiment of the present application, by filtering and data enhancement processing on the second point cloud image block and the third point cloud image block, and using the enhanced second point cloud image block and the third point cloud image block for training, it is possible to improve the edge point cloud data quality of the training samples through filtering, and increase the complexity of the training samples through data enhancement processing, so that the point cloud recognition model can better adapt to the changes in point cloud images in real environments.

[0021] Optionally, after performing data enhancement processing on the second point cloud image block after point cloud filtering and the third point cloud image block after point cloud filtering to obtain the enhanced second point cloud image block and the enhanced third point cloud image block, the method further includes:

[0022] Inputting the enhanced third point cloud image block into a non-maximum suppression algorithm, extracting the third point cloud image block with an overlapping volume smaller than a set threshold or with the highest confidence to obtain a fourth point cloud image block;

[0023] The step of training the initial neural network model by using the enhanced second point cloud image block and the enhanced third point cloud image block as training samples to obtain the point cloud recognition model comprises:

[0024] The enhanced second point cloud image block and the fourth point cloud image block are used as training samples to train the initial neural network model to obtain the point cloud recognition model.

[0025] In an embodiment of the present application, the enhanced third point cloud image is input into a non-maximum suppression algorithm, and the effect of overlapping image blocks is found and eliminated with the help of the non-maximum suppression algorithm. The overlapping volume and confidence data are first calculated, and the third point cloud image blocks with lower overlapping volume or higher overlapping volume but the highest confidence are screened out to form a fourth point cloud image block, thereby removing point cloud image blocks with excessive overlap and inaccurate selection of target instances, thereby improving the overall data quality of the training samples.

[0026] Optionally, after inputting the enhanced third point cloud image block into a non-maximum suppression algorithm to calculate an intersection-and-union ratio, and confirming a fourth point cloud image block from the third point cloud image block according to the intersection-and-union ratio, the method further includes:

[0027] Based on the set volume value, a random position is selected in the unsegmented part of the original point cloud image to perform image segmentation to obtain a fifth point cloud image block, the number of the fifth point cloud image blocks is equal to the number of the first point cloud image blocks, and the set volume value is smaller than the volume of the original point cloud image;

[0028] The step of training the initial neural network model by using the enhanced second point cloud image block and the fourth point cloud image block as training samples to obtain the point cloud recognition model comprises:

[0029] The initial neural network model is trained using the enhanced second point cloud image block and the fourth point cloud image block as positive training samples and the fifth point cloud image block as negative training samples to obtain the point cloud recognition model.

[0030] In an embodiment of the present application, a fifth point cloud image block is obtained by cutting at a random position in the unsegmented part of the original point cloud image, and the second point cloud image block, the fourth point cloud image block and the fifth point cloud image block are input for training by simultaneously inputting positive and negative samples, thereby providing a more comprehensive training sample input for the training of the point cloud recognition model, learning both the features and contours of the target instance and the judgment basis for distinguishing the target from the background, thereby alleviating the overfitting problem of the model and enhancing the recognition and classification capabilities of the point cloud recognition model.

[0031] Optionally, the step of training the initial neural network model using the enhanced second point cloud image block and the fourth point cloud image block as positive training samples and the fifth point cloud image block as negative training samples to obtain the point cloud recognition model includes:

[0032] Obtaining a set number of training times for the initial neural network model;

[0033] Calculate the number of samples required to be input for each training batch according to the sum of the number of the positive training samples and the number of the negative training samples and the set number of training times;

[0034] Calculating a ratio between the number of the second point cloud image blocks after enhancement and the number of the fourth point cloud image blocks;

[0035] Calculate, according to the ratio and one half of the sample quantity, a first input amount occupied by the enhanced second point cloud image block and a second input amount occupied by the fourth point cloud image block in each of the training batches;

[0036] Using half of the number of samples as a third input quantity occupied by the fifth point cloud image block in each of the training batches;

[0037] Extracting the enhanced second point cloud image block of the first input amount, the fourth point cloud image block of the second input amount, and the fifth point cloud image block of the third input amount one by one, to generate a plurality of training sets required for a plurality of the training batches;

[0038] The initial neural network model is iteratively trained in sequence using the multiple groups of training sets to obtain the point cloud recognition model.

[0039] In an embodiment of the present application, by setting the number of training times, the total number of positive training samples and the total number of negative training samples, the first input, the second input and the third input required for a single training of the initial neural network model are calculated, so that while ensuring that the number of training times meets the requirements, the training samples generated by image segmentation are put into training as much as possible to maximize sample utilization; at the same time, the positive samples and negative samples are maintained to be input synchronously into the neural network model to reduce the risk of overfitting problems.

[0040] In a second aspect, an embodiment of the present application provides a model training device, the device comprising:

[0041] A receiving unit, used for receiving an original point cloud image;

[0042] An instance detection unit, configured to detect a predicted region of a target instance in the original point cloud image based on a target detection algorithm;

[0043] An image segmentation unit, configured to perform image segmentation on the original point cloud image based on the predicted area to obtain a first point cloud image block containing the target instance;

[0044] A model training unit is used to train an initial neural network model using the first point cloud image block as a training sample to obtain the point cloud recognition model.

[0045] Optionally, the first point cloud image block is composed of a second point cloud image block and a third point cloud image block, and the image segmentation unit is specifically used to:

[0046] Randomly segmenting the point cloud in the prediction area in the original point cloud image to obtain the second point cloud image block;

[0047] The original point cloud image is randomly segmented to include a single predicted area to obtain the third point cloud image block.

[0048] Optionally, the device further comprises:

[0049] A filtering unit, configured to filter the second point cloud image block and the third point cloud image block based on a filtering algorithm;

[0050] a data enhancement unit, configured to perform data enhancement processing on the second point cloud image block after filtering and the third point cloud image block after filtering to obtain an enhanced second point cloud image block and an enhanced third point cloud image block, wherein the data enhancement processing includes at least one of random displacement, random cropping, random rotation and flipping, random scaling, adding random noise or performing random spatial projection;

[0051] The model training unit is specifically used for:

[0052] The enhanced second point cloud image block and the enhanced third point cloud image block are used as training samples to train the initial neural network model to obtain the point cloud recognition model.

[0053] Optionally, the device further comprises:

[0054] an extraction unit, configured to input the enhanced third point cloud image block into a non-maximum suppression algorithm, and extract the third point cloud image block having an overlapping volume smaller than a set threshold or having the highest confidence level to obtain a fourth point cloud image block;

[0055] The model training unit is specifically used for:

[0056] The enhanced second point cloud image block and the fourth point cloud image block are used as training samples to train the initial neural network model to obtain the point cloud recognition model.

[0057] Optionally, the image segmentation unit is further used to select a random position in the unsegmented part of the original point cloud image to perform image segmentation based on a set volume value to obtain fifth point cloud image blocks, the number of the fifth point cloud image blocks is equal to the number of the first point cloud image blocks, and the set volume value is smaller than the volume of the original point cloud image;

[0058] The model training unit is specifically used for:

[0059] The initial neural network model is trained using the enhanced second point cloud image block and the fourth point cloud image block as positive training samples and the fifth point cloud image block as negative training samples to obtain the point cloud recognition model.

[0060] Optionally, the model training unit is specifically used for:

[0061] Obtaining a set number of training times for the initial neural network model;

[0062] Calculate the number of samples required to be input for each training batch according to the sum of the number of the positive training samples and the number of the negative training samples and the set number of training times;

[0063] Calculating a ratio between the number of the second point cloud image blocks after enhancement and the number of the fourth point cloud image blocks;

[0064] Calculate, according to the ratio and one half of the sample quantity, a first input amount occupied by the enhanced second point cloud image block and a second input amount occupied by the fourth point cloud image block in each of the training batches;

[0065] Using half of the number of samples as a third input quantity occupied by the fifth point cloud image block in each of the training batches;

[0066] Extracting the enhanced second point cloud image block of the first input amount, the fourth point cloud image block of the second input amount, and the fifth point cloud image block of the third input amount one by one, to generate a plurality of training sets required for a plurality of the training batches;

[0067] The initial neural network model is iteratively trained in sequence using the multiple groups of training sets to obtain the point cloud recognition model.

[0068] In a third aspect, an embodiment of the present application further provides an electronic device, comprising at least one processor and a memory connected to the at least one processor, wherein the at least one processor is used to implement the steps of the method as described in any one of the first aspects when executing a computer program stored in the memory.

[0069] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method as described in any one of the first aspects.

[0070] It should be understood that the second to fourth aspects of the embodiments of the present application are consistent with the technical solutions of the first aspect of the embodiments of the present application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation methods are similar and will not be repeated here.

Brief Description of the Drawings

[0071] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0072] Figure 1 A flowchart of a model training method provided in an embodiment of the present application;

[0073] Figure 2 A schematic diagram of a flow chart of a method for segmenting an original point cloud image provided in an embodiment of the present application;

[0074] Figure 3 A flowchart of a method for optimizing the data quality of a second point cloud image block and a third point cloud image block provided in an embodiment of the present application;

[0075] Figure 4 A schematic diagram of a flow chart of a screening method for a third point cloud image block provided in an embodiment of the present application;

[0076] Figure 5 A- Figure 5 D is a schematic diagram of the effect of using a non-maximum suppression algorithm to process a third point cloud image block provided by an embodiment of the present application;

[0077] Figure 6 A flowchart of a point cloud recognition model training method using both positive and negative training samples provided in an embodiment of the present application;

[0078] Figure 7 A flowchart of a method for inputting positive and negative training samples provided in an embodiment of the present application;

[0079] Figure 8 A schematic diagram of the structure of a model training device provided in an embodiment of the present application;

[0080] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. [Specific implementation method]

[0081] In order to better understand the technical solution of this specification, the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0082] It should be clear that the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.

[0083] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit this specification. The singular forms of "a", "said" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0084] In recent years, with the development of autonomous driving technology, the use of three-dimensional perception algorithms to identify target instances in the point cloud data output by automotive radars and extract the position, size, shape and category information of the target instances for the vehicle to perform subsequent operations such as road condition judgment and semantic map generation has become a more commonly used object recognition method.

[0085] The inventors of this application have found that in the prior art, supervised training is often used when training point cloud recognition models in three-dimensional perception algorithms. This method requires manual or self-training annotation of the complete original point cloud image as a training sample. However, the original point cloud image contains both target instance information and a large amount of invalid noise information. Inputting the complete original point cloud image for training will cause the noise information to be read and recognized at the same time, resulting in a large amount of computing power waste, which greatly prolongs the training time and is not conducive to improving the performance of the point cloud recognition model.

[0086] In view of this, an embodiment of the present application provides a model training method, which selectively segments the area where the target instance exists in the original point cloud dataset, and uses the segmented point cloud image blocks to train the point cloud recognition model, thereby minimizing the size of the training sample and reducing the model training time while retaining the complete image features of the target instance. At the same time, the target instance image features of the original point cloud image are fully utilized to improve the performance of the point cloud recognition model.

[0087] The technical solution provided by the embodiments of the present application is introduced below in conjunction with the accompanying drawings.

[0088] See also Figure 1 , is a model training method provided in an embodiment of the present application, which is applied to the terminal side, and the steps of the method are as follows:

[0089] Step 101: Receive an original point cloud image.

[0090] In the embodiment of the present application, the original point cloud image is used as the basic sample for the subsequent model training steps, and its source includes but is not limited to: point cloud image data synthesized and output by any device such as laser radar, millimeter wave radar, RGBD camera, three-dimensional scanner, ultrasonic scanner, etc. after scanning, or any public point cloud image data obtained by downloading, copying, etc. At the same time, its data storage format can be a format based on voxel structure storage, a format based on grid structure storage, or any three-dimensional point cloud data format that uses voxels and grids at the same time, or uses other data structures. In theory, as long as it is three-dimensional point cloud data composed of a large number of discrete point clouds, the solution provided in the embodiment of the present application can be applied.

[0091] After the original point cloud image is imported into the terminal that performs model training (such as a graphics processor server, cloud server, or high-performance computing cluster), preprocessing of the original point cloud image can be started to meet the training requirements of the point cloud recognition model.

[0092] Step 102: Based on the target detection algorithm, the predicted area of ​​the target instance is detected in the original point cloud image.

[0093] In the embodiment of the present application, in order to realize the point cloud annotation of the original point cloud image, it is a feasible method to detect and annotate the predicted area of ​​the target instance in the original point cloud image through the target detection algorithm. This annotation method usually detects, crops and classifies the point cloud image based on the anchor point or the region to find the object contained in the point cloud image. As the annotation result, the information output by the target detection algorithm at least includes the predicted area information of the target instance, that is, the three-dimensional coordinates, height, width, depth and other information corresponding to the predicted area of ​​the target instance. In addition, according to the specific training requirements, the annotation result can also be expanded to the form of "prediction area + semantic information", that is, the category information of the target instance is additionally attached on the basis of the information of the prediction area itself; further, the annotation result can also be expanded to "prediction area + instance information", that is, the attached target instance category information is replaced with individualized annotation information, and each target object is distinguished using a specific identifier to facilitate the training of a neural network model that performs specific functions such as: target tracking, behavior analysis, etc.

[0094] In addition to the above-mentioned target instance annotation method, the original point cloud image can also be annotated by the following methods in the embodiment of the present application:

[0095] (1) Labeling through annotation algorithms: Additional attributes are added to each point in the point cloud, such as the point’s color, normal vector, intensity, label, etc., to achieve the effect of labeling the target instance in the original point cloud image.

[0096] (2) Labeling through matching algorithms: Match the point cloud data with point cloud data from other sources or reference point clouds (for example, compare the point cloud data from a lidar with images taken by a camera), and compare the matched similar local areas with the feature descriptors of the template instances in the database, so as to find the target instances that meet the feature descriptors based on the similar local areas.

[0097] (3) Annotation through segmentation algorithm: The original point cloud image is divided into different parts using the segmentation algorithm. Different parts of the image carry different semantic labels, geometric attributes, materials and other information as annotation information.

[0098] It should be noted that since the target detection algorithm is used as a labeling tool before the selective cutting of the original point cloud image in the embodiment of the present application, it does not hinder the subsequent segmentation operation itself, but only provides the predicted area of ​​the target instance; while in the embodiment corresponding to method (3), the segmentation algorithm performs the cutting operation on the original point cloud image while labeling, which conflicts with the subsequent step 103 of the embodiment of the present application. A detailed solution to this conflict will be given later.

[0099] After the original point cloud image is annotated using the above method, subsequent steps such as image cutting and training based on the annotation can be performed on the annotated original point cloud image.

[0100] Step 103: Based on the predicted area, the original point cloud image is segmented to obtain a first point cloud image block containing the target instance.

[0101] In an embodiment of the present application, since the predicted area of ​​the target instance has been labeled in the aforementioned steps, the original point cloud image can be directly segmented based on the labeling results to obtain multiple first point cloud image blocks containing target instance point cloud data (which may further contain target instance related information).

[0102] Specifically, this operation is called "patch segmentation" of the original point cloud image, which means dividing the point cloud image into a series of smaller image blocks, so that the training of the point cloud recognition model can be performed in parallel based on multiple image blocks. Compared with directly inputting the annotated original point cloud image into the initial neural network model for training, patch segmentation of the original point cloud image can reduce the size of the training sample as much as possible while retaining the complete point cloud image features of the target instance, thereby reducing the terminal memory burden, while making full use of the target instance information of the original point cloud image to improve training efficiency and reduce the risk of model overfitting.

[0103] It should be noted that the patch segmentation of the original point cloud image in the embodiment of the present application is based on the annotation of the prediction area, rather than dividing the original point cloud image into multiple image blocks of fixed size and fixed shape. Although the latter method also has the advantages of reducing the size of a single training sample and reducing the single computing power consumption after dividing the original point cloud image into equal parts, the complete semantic information of the target instance in the image blocks obtained by equal division is likely to have been destroyed, resulting in poor training results; at the same time, since the image blocks are not segmented based on the prediction area, but are executed using an equal division strategy, the image blocks themselves still contain a lot of noise information, and the volume of the training samples cannot be reduced.

[0104] Therefore, in the embodiment of the present application, a method for segmenting the original point cloud image based on the predicted region annotation is adopted. The following is a possible implementation of the method:

[0105] Figure 2 103 is a flow chart of a method for segmenting an original point cloud image provided by an embodiment of the present application. Step 103 can be further implemented by executing sub-steps 1031 to 1032.

[0106] Step 1031: Randomly segment the point cloud in the prediction area of ​​the original point cloud image to obtain a second point cloud image block.

[0107] Step 1032: Randomly segment the original point cloud image to include a single prediction area to obtain a third point cloud image block.

[0108] In the embodiment of the present application, since the essence of the prediction area is a set of point clouds with the same point cloud data features, and this set is accompanied by labeled coordinate information, width, length, height and other parameters, in order to ensure the richness of the training samples, two segmentation strategies can be used to segment the original point cloud image:

[0109] (1) The point cloud contained in the prediction area is segmented into random volumes and random shapes. On the premise of ensuring that the segmented area does not exceed the range covered by the prediction area, part of the image features of the target instance are intercepted to obtain a second point cloud image block, and the second point cloud image block is used as a part of the first point cloud image block.

[0110] (2) The point cloud in the original point cloud image is segmented into random volumes and random shapes, and the segmented area must contain the complete prediction area, that is, the segmented image block must contain the complete target instance to obtain the third point cloud image block, and the third point cloud image block is used as another part of the first point cloud image block.

[0111] Among them, since the second point cloud image block only includes the point cloud of part of the target instance, using it as training can enable the initial neural network model to better learn the local features of the target instance. Since the third point cloud image block contains the complete target instance point cloud (including internal point cloud and edge point cloud), it is used as training, and the initial neural network model will be more inclined to learn the overall contour features of the target instance. By using the second point cloud image block and the third point cloud image block obtained by patch segmentation as training samples at the same time, compared with traditional whole-image training, the sample richness is enhanced on the basis of ensuring the accuracy of the training data, so that the obtained point cloud recognition model can have a stronger generalization ability.

[0112] It should be understood that for the second point cloud image block and the third point cloud image block used for training, their cutting shapes can also be flexibly adjusted according to the training requirements. For different training requirements, the shapes of the second point cloud image block and the third point cloud image block may include: cube, sphere, cylinder or other irregular polyhedron. Among them, the advantage of using a cube for cutting is that it is simple and easy to read; and compared with the cube, the sphere and cylinder have a shape that is closer to the real point cloud data, and can better capture the geometric and semantic features of the target; as for the irregular polyhedron image block, its cutting complexity can better adapt to the unstructured point cloud, but because the segmentation rules are more complex, it requires more adjustments in the algorithm implementation and hyperparameter customization before it can be realized. The specific adjustment methods are relatively diverse and will not be repeated here.

[0113] In addition, the previous article mentioned a method of using a segmentation algorithm to annotate the original point cloud image, and this method requires performing a segmentation operation on the image, which obviously conflicts with the patch segmentation in this link; therefore, the case of using a segmentation algorithm for annotation in the embodiment of the present application is discussed separately here.

[0114] Specifically, in order to avoid the conflict between patch segmentation and the segmentation algorithm, the relevant parameters of the segmentation algorithm can be adjusted, such as appropriate optimization in feature selection, classification method, neighborhood search, etc., and then based on the optimized segmentation algorithm, a more refined semantic segmentation (or instance segmentation) is performed on the original point cloud image, while retaining the target instance information in the segmented area and excluding invalid noise information as much as possible.

[0115] Furthermore, by integrating the segmented areas based on the contextual relationship between the segmented areas (which can also be understood as the positional relationship with the original point cloud image), we can obtain training samples that contain complete target instances and can be used for point cloud recognition model training.

[0116] The training samples obtained in this way can also be directly imported into the initial neural network model for training, because they retain the image features of the target instance while excluding most of the noise information during segmentation, which is equivalent to replacing the step of selectively segmenting the original point cloud image.

[0117] After the second point cloud image block and the third point cloud image block are obtained by patch segmentation, the second point cloud image block and the third point cloud image block can be used as training samples and imported into the initial neural network model for training to obtain the desired point cloud recognition model.

[0118] Step 104: Using the first point cloud image block as a training sample to train the initial neural network model to obtain a point cloud recognition model.

[0119] In the embodiment of the present application, the general process of training the point cloud recognition model through the initial neural network model is as follows:

[0120] (1) Input the training sample data set and the corresponding annotation content, and start iterative training.

[0121] (2) Map the input training samples to the various layers of the initial neural network model, and operate the mapped training samples together with the activation function layer by layer, and pass the output results layer by layer until the output layer is reached, and the initial neural network model outputs the prediction results including the occurrence position of the target instance in the training sample and the specific type of the instance.

[0122] (3) The prediction results are input into a normalized exponential function to calculate the exact prediction value (in the form of a set of probability value vectors associated with multiple target instance types).

[0123] (4) Substitute the calculated predicted value and the true value corresponding to the training sample data set (represented by one-hot vector encoding) into the loss function to calculate the loss value of the initial neural network model.

[0124] (5) Based on the loss value, use an optimization algorithm (such as the gradient descent algorithm) to backpropagate the neural network model, calculate the gradient and update the weight parameters of the model.

[0125] (6) Repeat steps (1) to (5) until the latest calculated loss value is less than the preset loss value, or the number of iterative training reaches the set number, stop training and export the point cloud recognition model.

[0126] In this process, since the first segmented point cloud image block is not a complete original point cloud image, while inputting the first point cloud image block to reduce the training sample volume and shorten the model training time, it is also necessary to integrate the recognition results after a single training is completed, so as to obtain the position and type of each target instance in the original point cloud image, which is convenient for the subsequent calculation of loss values ​​and adjustment of model parameters.

[0127] It should be understood that the initial neural network model used for training in the embodiments of the present application may include but is not limited to: convolutional neural network model, neural image processing network model, recurrent neural network model, residual neural network model, bidirectional recurrent neural network model, attention mechanism neural network model and transformer model (or Transformer Model) and other mainstream neural network models.

[0128] Among them, the transformer model, due to its strong long-distance dependency modeling ability and multi-layer self-attention mechanism, can better integrate and splice the recognition results of each partial image block after importing the first point cloud image block, thereby obtaining a more complete recognition result of the original point cloud image. In theory, it can achieve better training results in the training method provided in the embodiment of the present application. However, for other neural network models, the method provided in the embodiment of the present application can also be used to achieve the effects of shortening the training time and improving the recognition accuracy of the point cloud recognition model. Therefore, no restrictions are imposed on the type of initial neural network model used for training.

[0129] In addition to performing basic patch segmentation to shorten the model training time, the data quality of the training samples can be further enhanced before training begins, thereby enhancing the recognition accuracy of the point cloud recognition model after it is put into use.

[0130] Figure 31033 to 1034 may be further performed after step 1032.

[0131] Step 1033: filtering the second point cloud image block and the third point cloud image block based on a filtering algorithm;

[0132] Step 1034: Perform data enhancement processing on the filtered second point cloud image block and the filtered third point cloud image block to obtain an enhanced second point cloud image block and an enhanced third point cloud image block, wherein the data enhancement processing includes at least one of random displacement, random cropping, random rotation and flipping, random scaling, adding random noise or performing random spatial projection.

[0133] Meanwhile, step 104 may be implemented by further executing sub-step 1041 .

[0134] Step 1041: Use the enhanced second point cloud image block and the enhanced third point cloud image block as training samples to train the initial neural network model to obtain a point cloud recognition model.

[0135] In an embodiment of the present application, after randomly segmenting the original point cloud data to obtain the second point cloud image block and the third point cloud image block, some methods can be used to further enhance the data quality of these image blocks so that better model training effects can be obtained when used for training.

[0136] For example, the second point cloud image block and the third point cloud image block can be subjected to point cloud filtering based on a filtering algorithm to remove noise information and abnormal point cloud information therein to enhance the accuracy and reliability of the point cloud data. Among them, the filtering algorithm used for point cloud image block filtering may include but is not limited to: Gaussian filtering method, random sampling consistency filtering method, voxel grid filtering method and other commonly used filtering methods.

[0137] In addition, the second point cloud image block and the third point cloud image block can also be subjected to data enhancement processing. By introducing random displacement, random cropping, random rotation and flipping, random scaling, adding random noise or performing random spatial projection and other operations, the second point cloud image block and the third point cloud image block whose point cloud data remain unchanged can be expanded to generate more image blocks that simulate the point cloud acquisition situation in the real environment as training samples.

[0138] Since data enhancement processing will introduce random changes within a controllable range on the basis of the original second point cloud image block and the third point cloud image block to generate a new point cloud image block, and the newly generated point cloud image block and the original second and third point cloud image blocks are input into the initial neural network model for training, performing data enhancement processing will not destroy the original training samples, or cause the quality of sample data to decline due to excessive introduction of random changes, and can effectively increase the richness of training samples.

[0139] The filtering algorithm is mainly used to improve the edge point cloud data quality of the second and third point cloud image blocks, which are the basis for generating new point cloud image blocks; when used in conjunction with data enhancement processing, it will also improve the overall quality of the newly generated point cloud image blocks. Therefore, performing these two operations on the training samples in sequence will not only not produce a conflict of effects, but will also improve the overall data quality of the training samples while increasing the posture, size, and shape changes of the target instances in the training samples to improve the adaptability of the point cloud recognition model to complex real situations and enhance the robustness of the model.

[0140] It should be noted that in the process of optimizing the data quality of the second point cloud image block and the third point cloud image block, the two steps of using the filtering algorithm to filter the point cloud and performing data enhancement processing can be performed in no particular order, that is, the data enhancement processing can be performed after the point cloud filtering, and the filtering algorithm can also be started after the data enhancement processing, and there is no essential difference in the optimization effect.

[0141] At the same time, in some embodiments, before and after the point cloud filtering processing or data enhancement processing is performed on the second point cloud image block and the third point cloud image block, additional operations such as point cloud resampling (using a resampling algorithm to adjust the density of the point cloud to make its density more uniform), point cloud registration (using a point cloud registration algorithm to correct the registration error during data output), point cloud completion (automatically completing point cloud data with obvious data missing or incomplete), and data balancing (re-undersampling, oversampling, or generating synthetic samples for the imbalance in the total number of the second point cloud image block and the third point cloud image block) may be inserted. The insertion position and order of these operations are also not in particular order, and they can be selected and used according to specific application scenarios and needs to further optimize the quality of the point cloud data.

[0142] After optimizing the data quality of the second point cloud image block and the third point cloud image block, the enhanced second point cloud image block and the enhanced third point cloud image block can be used as training samples to input into the initial neural network model for training. The training steps are not significantly different from the training steps in the previous article, so they will not be repeated here.

[0143] In addition, after optimizing the data instructions of the training samples, multiple third point cloud image blocks with overlapping volumes may be further processed to improve the quality of the training samples.

[0144] Figure 4 1034 is a flowchart of a method for screening a third point cloud image block provided in an embodiment of the present application. As a possible implementation, after step 1034, step 1035 may be further performed.

[0145] Step 1035: input the enhanced third point cloud image block into a non-maximum suppression algorithm, extract the third point cloud image block with an overlapping volume smaller than a set threshold or with the highest confidence to obtain a fourth point cloud image block.

[0146] Meanwhile, step 1041 may be implemented by further executing sub-step 201 .

[0147] Step 201: Use the enhanced second point cloud image block and the fourth point cloud image block as training samples to train the initial neural network model to obtain a point cloud recognition model.

[0148] In the embodiment of the present application, since the target detection algorithm mainly relies on the sliding window method, point cloud projection and other methods to detect target instances in the entire original point cloud image, in this process, it is possible that repeated detection will occur in the point cloud of the same target instance or in multiple prediction areas generated around the point cloud, and then the repeated target instances will be included in the third point cloud image block; at the same time, when the target instances are densely packed in the point cloud image, since the randomly divided volumes and shapes are random, it is also easy for large areas of volume overlap to occur between the two third point cloud image blocks.

[0149] For the third point cloud image block with repeated target instances, an appropriate algorithm needs to be selected for deduplication. To this end, the embodiment of the present application introduces a non-maximum suppression algorithm in the solution to process the third point cloud image block.

[0150] Figure 5 A- Figure 5 D is a schematic diagram of the effect of using the non-maximum suppression algorithm to process the third point cloud image block in the embodiment of the present application. Figure 5 A. When there are at least two third point cloud images (labeled as ① and ②) that partially overlap each other, the non-maximum suppression algorithm will detect at least two third point cloud image blocks with overlapping volumes to determine whether their overlapping volumes (gray parts in the figure) are greater than the set threshold. The judgment is generally based on calculating the intersection-union ratio, that is, calculating the ratio between the overlapping volume (intersection) and the total volume (union) of the two third point cloud image blocks after overlapping.

[0151] If the intersection-and-union ratio is greater than the set threshold, then Figure 5 B. Determine the completeness and accuracy of the target instance segmentation of at least two third point cloud image blocks, and evaluate the confidence level based on the judgment results. The higher the confidence level, the better the segmentation effect of the third point cloud image block. For example, if the confidence level of image block ① (0.7) is less than the confidence level of image block ② (0.95), the segmentation completeness of image block ② for the target instance is better than that of image block ①. Then, Figure 5 D. Delete the third point cloud image block ① with a lower score, and only retain the third point cloud image block ② with the highest confidence, and then determine it as the fourth point cloud image block together with other third point cloud image blocks with no overlap or less overlap volume.

[0152] For at least two third point cloud image blocks whose overlapping volume is lower than the set threshold, such as Figure 5 C. The non-maximum suppression algorithm will treat image blocks ① and ② as image blocks containing different target instances, that is, it will not perform confidence evaluation and deletion on these third point cloud image blocks.

[0153] The threshold is generally set to 0.5 or 0.3 to set different judgment criteria for overlapping segmentation. If the user wishes to relax or enhance the de-overlapping effect of the third point cloud image block, the threshold can be appropriately adjusted according to training requirements.

[0154] It should be understood that the above-mentioned non-maximum suppression algorithms also include: Soft-NMS (soft non-maximum suppression algorithm), Weighted NMS (weighted non-maximum suppression algorithm), Soft-NMS++ (improved soft non-maximum suppression algorithm), Cluster-NMS (clustered non-maximum suppression algorithm), etc., which are algorithms that have made certain improvements to the non-maximum suppression algorithm, so as to perform more flexible and accurate screening of the third point cloud image blocks based on user needs.

[0155] In some embodiments, only deduplication is performed on the third point cloud image block, which cannot guarantee that there will be no repeated segmentation of the same area of ​​the target instance in the second point cloud image block. Therefore, when the target detection algorithm directly marks the prediction area for the original point cloud image block, the repeated detection problem can be optimized for the marked prediction area. This effectively reduces the overlapping samples in the second point cloud image block and the third point cloud image block.

[0156] After deleting the overlapping samples in the third point cloud image block and obtaining the fourth point cloud image block, the fourth point cloud image block and the enhanced second point cloud image block can be input into the model for training. The steps here are no different from the model training steps in the previous article, so they will not be repeated here.

[0157] In addition, in order to reduce the risk of overfitting and insufficient recognition accuracy in the point cloud recognition model, the "positive and negative sample synchronous input training" method can be additionally introduced into the previous point cloud recognition model training method to enhance the training effect.

[0158] Figure 6 1035 is a flowchart of a point cloud recognition model training method using positive and negative training samples at the same time provided in an embodiment of the present application. As a possible implementation, after step 1035, step 1036 may be further performed.

[0159] Step 1036: Based on the set volume value, select a random position in the unsegmented part of the original point cloud image for image segmentation to obtain a fifth point cloud image block. The number of the fifth point cloud image blocks is equal to the number of the first point cloud image blocks, and the set volume value is smaller than the volume of the original point cloud image.

[0160] Meanwhile, step 201 can be implemented by further executing sub-step 2011 .

[0161] Step 2011: The enhanced second point cloud image block and the fourth point cloud image block are used as positive training samples, and the fifth point cloud image block is used as a negative training sample to train the initial neural network model to obtain a point cloud recognition model.

[0162] In the embodiment of the present application, it is feasible to train only by introducing positive samples containing the image features of the target instance, but the point cloud recognition model obtained by this training method is lacking in generalization ability and cannot distinguish well between the foreground point cloud containing the target instance and the background point cloud not containing the target instance. Therefore, the method of introducing positive and negative training samples for training at the same time can better improve this problem.

[0163] Specifically, this training method requires the introduction of positive training samples containing target instance information and setting their label values ​​to 1, and the introduction of negative training samples containing only background information and setting their label values ​​to 0. In this way, during the training process, the point cloud recognition model will learn the difference in point cloud data between the target instance image and the background image of the non-target instance on the basis of learning the image contour and image features of the target instance. Therefore, when identifying the point cloud image containing the target instance, the selected prediction area can include all point clouds that meet the image features of the target instance, while avoiding the selection of point clouds that are more in line with the background point cloud image features as much as possible, so that the boundary recognition of the target instance is clearer, and the recognition accuracy is more significantly improved compared to the traditional training method.

[0164] Therefore, for the fifth point cloud image block used as a negative training sample, it can only be sampled in the unsegmented area that does not contain the target instance, and a simplest sampling strategy is to segment the random area based on a predetermined set volume value. The way to configure the set volume value can be to directly specify the specified shape and volume size of the segmentation; or the total volume size of the given segmentation, and the shape is randomly selected without exceeding the boundary of the original point cloud image.

[0165] In some embodiments, the fifth point cloud image block can also be obtained by performing a completely equal segmentation in the unsegmented part of the original point cloud image based on the shape and area of ​​the positive training sample. This segmentation method can enhance the contrast between the positive training samples and the negative training samples, slightly enhance the accuracy of the point cloud recognition model, and at the same time give the trained point cloud recognition model better interpretability.

[0166] As for the method of using positive and negative training samples for synchronous training, in order to ensure that the training effect meets expectations, it is also necessary to impose certain restrictions on the number and type of samples input for a single training session.

[0167] Figure 7 2011 is a flowchart of a method for inputting positive and negative training samples provided in an embodiment of the present application. As a possible implementation, step 2011 can be further implemented by executing sub-steps 301 to 307.

[0168] Step 301: Obtain the set number of training times of the initial neural network model.

[0169] Step 302: Calculate the number of samples required to be input for each training batch according to the sum of the number of positive training samples and the number of negative training samples and the set number of training times.

[0170] Step 303: Calculate the ratio between the number of enhanced second point cloud image blocks and the number of fourth point cloud image blocks.

[0171] Step 304: Calculate the first input amount occupied by the enhanced second point cloud image block and the second input amount occupied by the fourth point cloud image block in each training batch according to the ratio and half of the sample quantity.

[0172] Step 305: Use half of the number of samples as the third input quantity occupied by the fifth point cloud image block in each training batch.

[0173] Step 306: successively extract the enhanced second point cloud image block of the first input, the fourth point cloud image block of the second input, and the fifth point cloud image block of the third input to generate multiple training sets required for multiple training batches.

[0174] Step 307: Use multiple groups of training sets to iteratively train the initial neural network model in sequence to obtain a point cloud recognition model.

[0175] In an embodiment of the present application, in order to ensure that the previously generated point cloud image blocks are maximized, it is necessary to use a suitable strategy to match the input ratios of the enhanced second point cloud image block, the fourth point cloud image block, and the fifth point cloud image block to ensure that each input can achieve the best training effect.

[0176] Specifically, due to the way of using positive and negative training samples to train the point cloud recognition model at the same time, it is necessary to ensure that when the initial neural network model learns the image features of a positive training sample, it also learns the image features of a negative training sample synchronously to avoid overfitting of a certain type of training samples. Therefore, when a certain number of positive training samples are input, the same number of negative training samples must be input at the same time.

[0177] At the same time, since there is a set number of training times for the training of the neural network model, the specific value of this set number of times may be set for the purpose of converging the loss value to the optimal solution, avoiding model overfitting, balancing training time and computing resources, etc. After obtaining the set number of training times, the number of samples input for a single training batch can be determined based on the set number of training times and the sum of the number of positive training samples and negative training samples (since patch segmentation reduces the volume of training samples, the number of samples that can be input for a single training batch has also changed from the conventional one to multiple). Afterwards, the first input and the second input are calculated based on the ratio of the number of enhanced second point cloud image blocks and fourth point cloud image blocks, and half the number of samples, while the third input can be directly equal to half the number of samples.

[0178] The first input, second input and third input calculated by this method can maximize the average input ratio of the enhanced second point cloud image block, fourth point cloud image block and fifth point cloud image block in the training sample while ensuring that the number of training times meets the standard, thereby ensuring better training results.

[0179] In some embodiments, due to other factors such as learning rate, upper limit of the number of training batches, optimization algorithm, etc., the first input, the second input, and the third input may have to be subtracted from a fixed value at the same time, or reduced in proportion at the same time. This situation is likely to result in the generated training samples not being fully utilized. In this regard, when performing target detection and random segmentation, additional indicators such as the upper limit of the number of training batches and the calculation parameters of the optimization algorithm can be obtained first, and the optimal input amount of different training samples can be calculated based on these constraints and the set number of training times, positive training samples, and the sum of negative training samples. The redundant point cloud image blocks are discarded according to the calculated values, or the random segmentation is stopped in advance when the number of point cloud image blocks obtained by segmentation reaches the standard, to ensure that the generated training samples can be maximized.

[0180] In some embodiments, even if an upper limit is set for the number of training batches, the total amount of information carried by multiple point cloud image blocks may exceed the upper limit of the hardware memory. In this case, a more intelligent batch construction and input technology can be introduced to compress and normalize the training samples when preloading the data, or to automatically select multiple groups of training samples with the most appropriate data size of the point cloud image blocks to form a training batch, to ensure that the total amount of information in a single training batch does not exceed the upper limit of the hardware memory.

[0181] See also Figure 8 Based on the same inventive concept, the embodiment of the present application also provides a model training device, the structure of which is as follows:

[0182] A receiving unit 401 is used to receive an original point cloud image;

[0183] An instance detection unit 402 is used to mark the predicted area of ​​the target instance in the original point cloud image based on the target detection algorithm;

[0184] An image segmentation unit 403 is used to perform image segmentation on the original point cloud image based on the predicted area to obtain a first point cloud image block containing the target instance;

[0185] The model training unit 404 is used to train the initial neural network model using the first point cloud image block as a training sample to obtain a point cloud recognition model.

[0186] Optionally, the first point cloud image block is composed of the second point cloud image block and the third point cloud image block, and the image segmentation unit 403 is specifically used to:

[0187] Randomly segment the point cloud in the prediction area of ​​the original point cloud image to obtain a second point cloud image block;

[0188] The original point cloud image is randomly segmented to include a single prediction area to obtain a third point cloud image block.

[0189] Optionally, the device further comprises:

[0190] A filtering unit, configured to filter the second point cloud image block and the third point cloud image block based on a filtering algorithm;

[0191] A data enhancement unit is used to perform data enhancement processing on the filtered second point cloud image block and the filtered third point cloud image block to obtain an enhanced second point cloud image block and an enhanced third point cloud image block, wherein the data enhancement processing includes at least one of random displacement, random cropping, random rotation and flipping, random scaling, adding random noise or performing random spatial projection;

[0192] The model training unit 404 is specifically used for:

[0193] The enhanced second point cloud image block and the enhanced third point cloud image block are used as training samples to train the initial neural network model to obtain a point cloud recognition model.

[0194] Optionally, the device further comprises:

[0195] An extraction unit, used for inputting the enhanced third point cloud image block into a non-maximum suppression algorithm to calculate an intersection-and-union ratio, and determining a fourth point cloud image block from the third point cloud image block according to the intersection-and-union ratio;

[0196] The model training unit 404 is specifically used for:

[0197] The enhanced second point cloud image block and the fourth point cloud image block are used as training samples to train the initial neural network model to obtain a point cloud recognition model.

[0198] Optionally, the image segmentation unit 403 is further configured to select a random position in the unsegmented part of the original point cloud image for image segmentation based on the set volume value to obtain fifth point cloud image blocks, the number of the fifth point cloud image blocks is equal to the number of the first point cloud image blocks, and the set volume value is smaller than the volume of the original point cloud image;

[0199] The model training unit 404 is specifically used for:

[0200] The enhanced second point cloud image block and the fourth point cloud image block are used as positive training samples, and the fifth point cloud image block is used as a negative training sample to train the initial neural network model to obtain a point cloud recognition model.

[0201] Optionally, the model training unit 404 is specifically used for:

[0202] Get the set number of training times for the initial neural network model;

[0203] Calculate the number of samples required for each training batch based on the sum of the number of positive training samples and negative training samples and the set number of training times;

[0204] Calculating a ratio between the number of enhanced second point cloud image blocks and the number of fourth point cloud image blocks;

[0205] According to the ratio and half of the number of samples, calculate the first input amount occupied by the enhanced second point cloud image block and the second input amount occupied by the fourth point cloud image block in each training batch;

[0206] Use half of the number of samples as the third input quantity occupied by the fifth point cloud image block in each training batch;

[0207] Extracting the enhanced second point cloud image block of the first input, the fourth point cloud image block of the second input, and the fifth point cloud image block of the third input successively to generate multiple groups of training sets required for multiple training batches;

[0208] The initial neural network model is iteratively trained using multiple sets of training sets to obtain a point cloud recognition model.

[0209] See also Fig. 9 Based on the same inventive concept, the embodiment of the present application further provides an electronic device 500, which may include at least one processor, and the at least one processor is used to execute a computer program stored in a memory to implement the embodiment of the present application. Figure 1-Figure 4 , Figure 6 and Figure 7 The steps of the model training method are shown.

[0210] Optionally, the processor may specifically be a central processing unit, a specific ASIC, or one or more integrated circuits for controlling program execution.

[0211] Optionally, the electronic device may further include a memory 502 connected to the at least one processor 501, and the memory 502 may include a ROM, a RAM, and a disk memory. The memory 502 is used to store data required by the processor 501 when it is running, that is, it stores instructions that can be executed by the at least one processor 501. The at least one processor 501 executes the instructions stored in the memory 502 to execute the following. Figure 1-Figure 4 , Figure 6 and Figure 7 The method shown in the figure. The number of the memory 502 is one or more.

[0212] The physical devices corresponding to the receiving unit 401, the instance detection unit 402, the image segmentation unit 403 and the model training unit 404 can all be the aforementioned processor 501. The electronic device can be used to execute Figure 1-Figure 4 , Figure 6 and Figure 7 Therefore, for the functions that can be realized by each functional unit in the electronic device, please refer to Figure 1-Figure 4 , Figure 6 and Figure 7 The corresponding description in the illustrated embodiment will not be repeated here.

[0213] In addition, the embodiment of the present application also provides a computer storage medium, wherein the computer storage medium stores computer instructions, and when the computer instructions are executed on the computer, the computer executes Figure 1-Figure 4 , Figure 6 and Figure 7 The method described.

[0214] The above description is only a preferred embodiment of this specification and is not intended to limit this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of this specification should be included in the scope of protection of this specification.

Claims

1. A model training method, characterized in that: Applied to the terminal side, the method includes: Receiving an original point cloud image; Based on the target detection algorithm, a predicted area of ​​the target instance is detected in the original point cloud image; Based on the predicted area, performing image segmentation on the original point cloud image to obtain a first point cloud image block containing the target instance; The initial neural network model is trained using the first point cloud image block as a training sample to obtain the point cloud recognition model.

2. The method according to claim 1, characterized in that The first point cloud image block is composed of the second point cloud image block and the third point cloud image block, and the image segmentation is performed on the original point cloud image based on the predicted area to obtain the first point cloud image block containing the target instance, including: Randomly segmenting the point cloud in the prediction area in the original point cloud image to obtain the second point cloud image block; The original point cloud image is randomly segmented to include a single predicted area to obtain the third point cloud image block.

3. The method according to claim 2, characterized in that After randomly segmenting the original point cloud image to include a single prediction area to obtain the third point cloud image block, the method further includes: Based on a filtering algorithm, filtering the second point cloud image block and the third point cloud image block; Performing data enhancement processing on the filtered second point cloud image block and the filtered third point cloud image block to obtain an enhanced second point cloud image block and an enhanced third point cloud image block, wherein the data enhancement processing includes at least one of random displacement, random cropping, random rotation and flipping, random scaling, adding random noise, or performing random spatial projection; The training of the initial neural network model using the first point cloud image block as a training sample to obtain the point cloud recognition model comprises: The enhanced second point cloud image block and the enhanced third point cloud image block are used as training samples to train the initial neural network model to obtain the point cloud recognition model.

4. The method according to claim 3, characterized in that After performing data enhancement processing on the filtered second point cloud image block and the filtered third point cloud image block to obtain the enhanced second point cloud image block and the enhanced third point cloud image block, the method further includes: Inputting the enhanced third point cloud image block into a non-maximum suppression algorithm, extracting the third point cloud image block with an overlapping volume smaller than a set threshold or with the highest confidence to obtain a fourth point cloud image block; The step of training the initial neural network model by using the enhanced second point cloud image block and the enhanced third point cloud image block as training samples to obtain the point cloud recognition model comprises: The enhanced second point cloud image block and the fourth point cloud image block are used as training samples to train the initial neural network model to obtain the point cloud recognition model.

5. The method according to claim 4, characterized in that After inputting the enhanced third point cloud image block into a non-maximum suppression algorithm to calculate an intersection-and-union ratio, and identifying a fourth point cloud image block from the third point cloud image block according to the intersection-and-union ratio, the method further includes: Based on the set volume value, a random position is selected in the unsegmented part of the original point cloud image to perform image segmentation to obtain a fifth point cloud image block, the number of the fifth point cloud image blocks is equal to the number of the first point cloud image blocks, and the set volume value is smaller than the volume of the original point cloud image; The step of training the initial neural network model by using the enhanced second point cloud image block and the fourth point cloud image block as training samples to obtain the point cloud recognition model comprises: The initial neural network model is trained using the enhanced second point cloud image block and the fourth point cloud image block as positive training samples and the fifth point cloud image block as negative training samples to obtain the point cloud recognition model.

6. The method according to claim 5, characterized in that The step of training the initial neural network model by using the enhanced second point cloud image block and the fourth point cloud image block as positive training samples and the fifth point cloud image block as negative training samples to obtain the point cloud recognition model comprises: Obtaining a set number of training times for the initial neural network model; Calculate the number of samples required to be input for each training batch according to the sum of the number of the positive training samples and the number of the negative training samples and the set number of training times; Calculating a ratio between the number of the second point cloud image blocks after enhancement and the number of the fourth point cloud image blocks; Calculate, according to the ratio and one half of the sample quantity, a first input amount occupied by the enhanced second point cloud image block and a second input amount occupied by the fourth point cloud image block in each of the training batches; Using half of the number of samples as a third input quantity occupied by the fifth point cloud image block in each of the training batches; Extracting the enhanced second point cloud image block of the first input amount, the fourth point cloud image block of the second input amount, and the fifth point cloud image block of the third input amount one by one, to generate a plurality of training sets required for a plurality of the training batches; The initial neural network model is iteratively trained in sequence using the multiple groups of training sets to obtain the point cloud recognition model.

7. A model training device, characterized in that: The device comprises: A receiving unit, used for receiving an original point cloud image; An instance detection unit, configured to detect a predicted region of a target instance in the original point cloud image based on a target detection algorithm; An image segmentation unit, configured to perform image segmentation on the original point cloud image based on the predicted area to obtain a first point cloud image block containing the target instance; A model training unit is used to train an initial neural network model using the first point cloud image block as a training sample to obtain the point cloud recognition model.

8. The device according to claim 7, characterized in that The first point cloud image block is composed of the second point cloud image block and the third point cloud image block, and the image segmentation unit is specifically used for: Randomly segmenting the point cloud in the prediction area in the original point cloud image to obtain the second point cloud image block; The original point cloud image is randomly segmented to include a single predicted area to obtain the third point cloud image block.

9. An electronic device, characterized in that: The electronic device comprises at least one processor and a memory connected to the at least one processor, and the at least one processor is configured to implement the steps of the method according to any one of claims 1 to 6 when executing a computer program stored in the memory.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images