Training method, device, electronic device and medium for point cloud instance segmentation model
Through the knowledge distillation method, replacing the backbone network in the point cloud instance segmentation model, the problem of insufficient storage and computing resources when the model is run on edge devices is solved, and the model compression and accuracy balance is achieved.
Patent Information
- Application Number
- CN202110213343.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-02-25
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2041-02-25
AI Technical Summary
The existing point cloud instance segmentation model has been reduced due to changes in the way of handling point clouds, but its model scale is still large and difficult to run on edge devices. It lacks effective compression technology, which affects the user experience at the edge.
Using the knowledge distillation method, the training of small student models is guided by using a large teacher model, replacing the backbone network in the initial instance segmentation model, and using the output results of the teacher model as a label to train the student model, and then obtain the compressed model.
It effectively reduces the storage space and computing resources required for instance segmentation models, and at the same time reduces the accuracy loss during model compression, realizing the feasibility of point cloud instance segmentation models on edge devices.
Smart Images

Figure CN112950642B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of artificial intelligence, specifically to the field of deep neural network model compression, and more specifically to a training method for a point cloud instance segmentation model, a training device for a point cloud instance segmentation model, an electronic device, a readable storage medium, and a computer program product. Background Art
[0002] In computer vision, the instance segmentation task first classifies each pixel in the image to form a semantic segmentation result, and then classifies the pixels in each category according to the objects they belong to, in order to distinguish the instances to which each pixel belongs. Instance segmentation has a wide range of applications in real-world scenarios. For example, in the workpiece sorting scenario, instance segmentation is used to distinguish each object in the point cloud, and then through pose estimation, the robot arm is controlled to grasp and complete the grasping of the workpiece.
[0003] Instance segmentation of 3D vision is of great significance in real life. For example, in autonomous driving, in addition to detecting vehicles and pedestrians on the road, the distance between vehicles and pedestrians must be accurately controlled. If only two-dimensional images are used, the above goals are difficult to achieve due to the lack of crucial depth information. In 3D vision, point cloud is a common form of data, which can be directly collected by LiDAR. In the early development, point cloud processing was completed by converting point clouds into voxels. For example: voxel network, an end-to-end point cloud target detection model, voxel network achieves good detection effect, but because the storage space occupied by voxels is too large, the voxel network model for processing voxels also requires a large storage space and computing resources, which is not conducive to edge deployment.
[0004] In the process of implementing the concept of the present disclosure, it is found that although the point cloud instance segmentation model in the related art has gradually shrunk due to the change in the way of processing point clouds, the model is still large and insufficient to run on edge devices, so the model needs to be compressed again. So far, there is no effective technical solution to complete the compression of the point cloud instance segmentation model. Summary of the invention
[0005] In view of this, the present disclosure provides a training method for a point cloud instance segmentation model, a training device for a point cloud instance segmentation model, an electronic device, a readable storage medium, and a computer program product.
[0006] One aspect of the present disclosure provides a training method for a point cloud instance segmentation model, comprising: obtaining a teacher model and an initial student model, wherein the teacher model is obtained by training an initial instance segmentation model, the initial instance segmentation model comprises a first backbone network, a semantic segmentation network and an instance segmentation network connected in sequence, and the initial student model is obtained by replacing the first backbone network in the initial instance segmentation model with a second backbone network, wherein the network scale of the second backbone network is smaller than the network scale of the first backbone network; inputting original point cloud data into the teacher model to obtain a first feature matrix output by the backbone network of the teacher model, a first probability matrix output by the semantic segmentation network of the teacher model and a second feature matrix output by the instance segmentation network of the teacher model; and using the first feature matrix, the first probability matrix and the second feature matrix as labels for the original point cloud data, training the initial student model based on the original point cloud data to obtain a trained point cloud instance segmentation model.
[0007] According to an embodiment of the present disclosure, the above-mentioned first feature matrix, the above-mentioned first probability matrix and the above-mentioned second feature matrix are used as labels of the above-mentioned original point cloud data, and the above-mentioned initial student model is trained based on the above-mentioned original point cloud data to obtain a trained point cloud instance segmentation model, including: based on the above-mentioned original point cloud data, using the above-mentioned first feature matrix as a label to train the second backbone network of the above-mentioned initial student model to obtain a first network model; based on the above-mentioned original point cloud data, using the above-mentioned first probability matrix as a label to train the second network model to obtain a third network model, wherein the above-mentioned second network model includes the semantic segmentation network of the above-mentioned first network model and the above-mentioned initial student model; and based on the above-mentioned original point cloud data, using the above-mentioned second feature matrix as a label to train the fourth network model to obtain a trained point cloud instance segmentation model, wherein the above-mentioned fourth network model includes the above-mentioned third network model and the instance segmentation network of the above-mentioned initial student model.
[0008] According to an embodiment of the present disclosure, the second backbone network for training the initial student model based on the original point cloud data using the first feature matrix as a label includes: inputting the original point cloud data into the second backbone network, and outputting a first output matrix; processing the first output matrix using a normalization method so that the dimension of the first output matrix is equal to the dimension of the first feature matrix; calculating a first loss value based on the first feature matrix and the normalized first output matrix; and when the first loss value is greater than a preset value, adjusting the model parameters of the second backbone network based on the first loss value.
[0009] According to an embodiment of the present disclosure, the above-mentioned training of the second network model based on the above-mentioned original point cloud data using the above-mentioned first probability matrix as a label includes: inputting the above-mentioned original point cloud data into the above-mentioned second network model, and outputting a second output matrix; calculating a second loss value based on the above-mentioned second output matrix and the true label of the above-mentioned original point cloud data; calculating a third loss value based on the above-mentioned second output matrix and the above-mentioned first probability matrix; performing a weighted sum operation on the above-mentioned second loss value and the above-mentioned third loss value to obtain a fourth loss value; and when the above-mentioned fourth loss value is greater than a preset value, adjusting the model parameters of the above-mentioned second network model based on the above-mentioned fourth loss value.
[0010] According to an embodiment of the present disclosure, the above-mentioned training of the fourth network model based on the above-mentioned original point cloud data using the above-mentioned second feature matrix as a label includes: inputting the above-mentioned original point cloud data into the above-mentioned fourth network model, and outputting the obtained third output matrix; calculating the fifth loss value based on the above-mentioned second feature matrix and the above-mentioned third output matrix; and when the above-mentioned fifth loss value is greater than a preset value, adjusting the model parameters of the above-mentioned fourth network model based on the above-mentioned fifth loss value.
[0011] The method of the disclosed embodiment also includes: inputting the point cloud data to be segmented into the above-mentioned point cloud instance segmentation model to obtain a second probability matrix output by the semantic segmentation network of the above-mentioned point cloud instance segmentation model and a third feature matrix output by the instance segmentation network of the above-mentioned point cloud instance segmentation model; obtaining point cloud category information according to the above-mentioned second probability matrix, performing similarity measurement on the above-mentioned third feature matrix to obtain point cloud boundary information; and obtaining a point cloud instance segmentation result based on the above-mentioned point cloud category information and the above-mentioned point cloud boundary information.
[0012] Another aspect of the present disclosure provides a training device for a point cloud instance segmentation model, comprising a model acquisition module, a data storage module and a training module, wherein: the model acquisition module is used to acquire a teacher model and an initial student model, wherein the teacher model is obtained by training an initial instance segmentation model, the initial instance segmentation model comprises a first backbone network, a semantic segmentation network and an instance segmentation network connected in sequence, and the initial student model is obtained by replacing the first backbone network in the initial instance segmentation model with a second backbone network, wherein the network scale of the second backbone network is smaller than the network scale of the first backbone network; the data storage module is used to store the first feature matrix output by the backbone network of the teacher model, the first probability matrix output by the semantic segmentation network of the teacher model and the second feature matrix output by the instance segmentation network of the teacher model obtained by inputting the original point cloud data into the teacher model; and the training module is used to use the first feature matrix, the first probability matrix and the second feature matrix as labels of the original point cloud data, train the initial student model based on the original point cloud data, and obtain a trained point cloud instance segmentation model.
[0013] According to an embodiment of the present disclosure, the training module includes: a first training unit, which is used to train the second backbone network of the initial student model based on the original point cloud data and using the first feature matrix as a label to obtain a first network model; a second training unit, which is used to train the second network model based on the original point cloud data and using the first probability matrix as a label to obtain a third network model, wherein the second network model includes the first network model and the semantic segmentation network of the initial student model; and a third training unit, which is used to train the fourth network model based on the original point cloud data and using the second feature matrix as a label to obtain a trained point cloud instance segmentation model, wherein the fourth network model includes the third network model and the instance segmentation network of the initial student model.
[0014] According to an embodiment of the present disclosure, the above-mentioned first training unit includes: a first operator unit, used to input the above-mentioned original point cloud data into the above-mentioned second backbone network, and output a first output matrix; a second operator unit, used to process the above-mentioned first output matrix using a normalization method, so that the dimension of the above-mentioned first output matrix is equal to the dimension of the above-mentioned first feature matrix; a third operator unit, used to calculate the first loss value based on the above-mentioned first feature matrix and the first output matrix after normalization; and a fourth operator unit, used to adjust the model parameters of the above-mentioned second backbone network based on the above-mentioned first loss value when the above-mentioned first loss value is greater than a preset value.
[0015] According to an embodiment of the present disclosure, the above-mentioned second training unit includes: a fifth operator unit, which is used to input the above-mentioned original point cloud data into the above-mentioned second network model, and output a second output matrix; a sixth operator unit, which is used to calculate the second loss value based on the above-mentioned second output matrix and the true label of the above-mentioned original point cloud data; a seventh operator unit, which is used to calculate the third loss value based on the above-mentioned second output matrix and the above-mentioned first probability matrix; an eighth operator unit, which is used to perform a weighted sum operation on the above-mentioned second loss value and the above-mentioned third loss value to obtain a fourth loss value; and a ninth operator unit, which is used to adjust the model parameters of the above-mentioned second network model based on the above-mentioned fourth loss value when the above-mentioned fourth loss value is greater than a preset value.
[0016] According to an embodiment of the present disclosure, the above-mentioned third training unit includes: a tenth operator unit, used to input the above-mentioned original point cloud data into the above-mentioned fourth network model, and output the obtained third output matrix; an eleventh operator unit, used to calculate the fifth loss value based on the above-mentioned second feature matrix and the above-mentioned third output matrix; and a twelfth operator unit, used to adjust the model parameters of the above-mentioned fourth network model based on the above-mentioned fifth loss value when the above-mentioned fifth loss value is greater than a preset value.
[0017] The device of the disclosed embodiment also includes an execution module, wherein the execution module includes: an execution unit, used to input the point cloud data to be segmented into the point cloud instance segmentation model, and obtain a second probability matrix output by the semantic segmentation network of the point cloud instance segmentation model and a third feature matrix output by the instance segmentation network of the point cloud instance segmentation model; a first analysis unit, used to obtain point cloud category information according to the second probability matrix, and perform similarity measurement on the third feature matrix to obtain point cloud boundary information; and a second analysis unit, used to obtain a point cloud instance segmentation result based on the point cloud category information and the point cloud boundary information.
[0018] Another aspect of the present disclosure provides an electronic device, including one or more processors and a memory storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned method.
[0019] Another aspect of the present disclosure provides a readable storage medium storing executable instructions, which are used to implement the above method when executed.
[0020] Another aspect of the present disclosure provides a computer program product, which includes computer executable instructions, and when the instructions are executed, are used to implement the method as described above.
[0021] According to the embodiments of the present disclosure, a student model is obtained by replacing the backbone network in the initial instance segmentation model with a smaller backbone network, the initial instance segmentation model is trained to convergence to obtain a teacher model, and the original point cloud data is input into the teacher model to obtain the output results of different networks as labels, and the labels are used to train different networks in the student model to obtain a compressed model. Based on this method, the compression of the instance segmentation model for three-dimensional objects is solved, which effectively reduces the storage space and computing resources required for the instance segmentation model, and at the same time reduces the accuracy loss during the model compression process. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] The above and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:
[0023] Figure 1 An exemplary system architecture 100 to which a training method for a point cloud instance segmentation model can be applied according to an embodiment of the present disclosure is schematically shown;
[0024] Figure 2 A flowchart of a method 200 for training a point cloud instance segmentation model according to an embodiment of the present disclosure is schematically shown;
[0025] Figure 3 A schematic diagram schematically shows a training process of a point cloud instance segmentation model according to an embodiment of the present disclosure;
[0026] Figure 4a , Figure 4b and Figure 4c An application example of a point cloud instance segmentation model trained according to the method of an embodiment of the present disclosure is shown;
[0027] Figure 5 A schematic diagram of a training device 500 for a point cloud instance segmentation model according to an embodiment of the present disclosure is schematically shown;
[0028] Figure 6 A block diagram of an electronic device 600 suitable for implementing a training method for a point cloud instance segmentation model according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION
[0029] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.
[0030] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.
[0031] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification, and should not be interpreted in an idealized or overly rigid manner.
[0032] In the case of using expressions such as "at least one of A, B, and C, etc.", it should generally be interpreted in accordance with the meaning of the expression generally understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.). In the case of using expressions such as "at least one of A, B, or C, etc.", it should generally be interpreted in accordance with the meaning of the expression generally understood by those skilled in the art (for example, "a system having at least one of A, B, or C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).
[0033] In related technologies, the point cloud instance segmentation model has gradually shrunk due to changes in the way point clouds are processed, but its model scale is still large. At the same time, the edge end (such as mobile phones, embedded ends, etc.) is different from cloud computing centers. It has fewer computing resources. Large-scale models often cannot run or run too inefficiently when running on the edge end, thus affecting the user experience of the edge end. Therefore, the model needs to be compressed again to meet the needs of the point cloud instance segmentation model running on edge devices.
[0034] Based on the above problems, the knowledge distillation method can be used to solve them. Knowledge distillation is to use a large network (teacher model) to guide the training of a small network (student model), so that the student model can improve the model accuracy without increasing any computing and storage resources, thereby achieving the purpose of compressing the model. The knowledge distillation method has a relatively stable performance, a complete mathematical theoretical foundation, and is easy to deploy on the object side. It can be used to solve the point cloud model compression problem. After the point cloud model is compressed, it can be easily deployed on the edge, such as industrial sorting robots, mobile phones, embedded terminals, etc., to ensure that the point cloud instance segmentation work can be completed without large computing equipment. In related technologies, although there are many papers on model compression based on knowledge distillation, most of them focus on the compression of classification models, and there is no more exploration of the compression of point cloud instance segmentation models.
[0035] In the current model that directly performs instance segmentation on point clouds, such as the point cloud instance segmentation similarity group proposal network, the implementation process can be divided into four steps, namely, first extracting point cloud features (implemented by the backbone network), then classifying the point cloud according to the type of object based on the point cloud features (implemented by the semantic segmentation network), and then completing the distinction of each object (implemented by the instance segmentation network), and finally obtaining the instance segmentation result based on the object distinction and classification results. In the current instance segmentation work, the instance segmentation network results are presented as feature vectors, which have the following characteristics: the feature vectors of the same object are very close, and the feature vectors of different objects are very far away. For the learned feature vectors, after clustering, the instance segmentation result is obtained. It should be noted that the result of instance segmentation is only the division of objects and does not include the type of objects. The final instance segmentation result is obtained after confirming the classification of each object based on the semantic segmentation result.
[0036] The embodiments of the present disclosure provide a method for training a point cloud instance segmentation model, a training device for a point cloud instance segmentation model, an electronic device, a readable storage medium, and a computer program product. The method includes: obtaining a teacher model and an initial student model; inputting the original point cloud data into the teacher model to obtain a first feature matrix output by the backbone network of the teacher model, a first probability matrix output by the semantic segmentation network of the teacher model, and a second feature matrix output by the instance segmentation network of the teacher model; and using the first feature matrix, the first probability matrix, and the second feature matrix as labels for the original point cloud data, training the initial student model based on the original point cloud data, and obtaining a trained point cloud instance segmentation model.
[0037] Figure 1 An exemplary system architecture 100 to which a training method for a point cloud instance segmentation model can be applied according to an embodiment of the present disclosure is schematically shown. It should be noted that: Figure 1The system architecture 100 shown is merely an example to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.
[0038] like Figure 1 As shown, the system architecture 100 according to this embodiment may include edge devices 101, 102, 103, 104 and a cloud device 105. The edge devices 101, 102, 103, 104 and the terminal device 105 may exchange information via communication links including wired and / or wireless networks.
[0039] Edge devices 101, 102, 103, and 104 may be various electronic devices that require real-time processing of environmental images, including but not limited to smartphones, drones, smart driving cars, industrial robots, etc. Edge devices 101, 102, 103, and 104 have limited computing resources and storage capabilities. At the same time, when using these edge devices, users have high requirements for the speed of image information processing.
[0040] The cloud device 105 includes but is not limited to servers, server clusters, supercomputers, etc., which have sufficient computing resources and storage capabilities and can realize the training of deep networks.
[0041] In an embodiment of the present disclosure, the samples required for training the point cloud instance segmentation model can be obtained by edge devices 101, 102, 103, and 104. The cloud device 105 is used to train the teacher model, and use the teacher model to guide the training of the student model. The trained student model is sent to the edge devices 101, 102, 103, and 104 through a communication link. The edge devices 101, 102, 103, and 104 can implement instance segmentation of real-time captured image data based on the trained student model. It should be noted that the method provided in the embodiment of the present disclosure can also be executed by other devices that can establish communication with the cloud device 105. In this application scenario, the cloud device 105 is used to assist other devices in training the teacher model and the student model, or is only used to store training sample data or model data, or is only used to provide security for communication between other devices and edge devices 101, 102, 103, and 104.
[0042] It should be understood that Figure 1 The types and numbers of edge devices and cloud devices in the embodiment are only for illustration purposes. Any types or numbers of edge devices and cloud devices may be used as required.
[0043] Figure 2 A flowchart of a method 200 for training a point cloud instance segmentation model according to an embodiment of the present disclosure is schematically shown.
[0044] like Figure 2 As shown, the method 200 includes operations S210 to S230.
[0045] In operation S210, a teacher model and an initial student model are acquired.
[0046] According to an embodiment of the present disclosure, the initial point cloud instance segmentation model is decomposed according to the network structure to obtain a backbone network, a semantic segmentation network and an instance segmentation network, and the above three networks can be connected in series. The teacher model has the same network structure as the initial point cloud instance segmentation model. The teacher model in this embodiment can be obtained by training the initial point cloud instance segmentation model in the corresponding data until the model converges. In an embodiment of the present disclosure, the teacher model can be trained using the same point cloud data as when training the student model, or can use relevant data in other existing data sets.
[0047] According to the embodiments of the present disclosure, the backbone network in the point cloud instance segmentation model can be used to extract point cloud features. According to the relevant theories of neural networks, deep networks are conducive to the mining of deep features, thereby bringing better results. In order to improve the performance of the model, the backbone network has a much larger scale than the semantic segmentation network and the instance segmentation network, so the backbone network occupies the vast majority of storage space and computing power in the entire network results. Therefore, when constructing the student model, a smaller backbone network can be used to replace the backbone network in the original point cloud instance segmentation model.
[0048] For example, the teacher model is an association segmentation model of instances and semantics in point clouds, and its backbone network is a deep feature learning network of point sets in metric space. The backbone network of the teacher model is replaced with a deep learning network of point sets in three-dimensional classification and segmentation to obtain the student model. The size of the student model is only about 60% of the teacher model, and the computing resources required for the student model when in use are also reduced accordingly.
[0049] In operation S220, the original point cloud data is input into the teacher model to obtain a first feature matrix output by the backbone network of the teacher model, a first probability matrix output by the semantic segmentation network of the teacher model, and a second feature matrix output by the instance segmentation network of the teacher model.
[0050] In operation S230, the first feature matrix, the first probability matrix, and the second feature matrix are used as labels of the original point cloud data, and an initial student model is trained based on the original point cloud data to obtain a trained point cloud instance segmentation model.
[0051] According to an embodiment of the present disclosure, during the training of the initial student model, the output values of each network after the teacher model inputs the original point cloud data can be used as label supervised training. At the same time, depending on the selected loss function, the real labels of the original point cloud data can also be used to balance the training results.
[0052] According to an embodiment of the present disclosure, the trained point cloud instance segmentation model has a smaller model scale than the initial instance segmentation model, and consumes less storage space and computing resources of the device running the model.
[0053] According to the embodiments of the present disclosure, a student model is obtained by replacing the backbone network in the initial instance segmentation model with a smaller backbone network, the initial instance segmentation model is trained to convergence to obtain a teacher model, and the original point cloud data is input into the teacher model to obtain the output results of different networks as labels, and the labels are used to train different networks in the student model to obtain a compressed model. Based on this method, the compression of the instance segmentation model for three-dimensional objects is solved, which effectively reduces the storage space and computing resources required for the instance segmentation model, and at the same time reduces the accuracy loss during the model compression process.
[0054] Figure 3 A schematic diagram schematically illustrates a training process of a point cloud instance segmentation model according to an embodiment of the present disclosure.
[0055] like Figure 3 As shown, the process of the teacher model guiding the student model to train includes respectively training the backbone network, semantic segmentation network and instance segmentation network of the initial student model, which can be used to implement operation S230 in method 200. The training of the initial student model in the embodiment of the present disclosure includes the following steps:
[0056] Firstly, based on the original point cloud data, the second backbone network of the initial student model is trained using the first feature matrix as a label to obtain a first network model.
[0057] According to an embodiment of the present disclosure, training the second backbone network includes: inputting the original point cloud data into the second backbone network, and outputting a first output matrix; processing the first output matrix using a normalization method so that the dimension of the first output matrix is equal to the dimension of the first feature matrix; calculating a first loss value based on the first feature matrix and the normalized first output matrix; and when the first loss value is greater than a preset value, adjusting the model parameters of the second backbone network based on the first loss value.
[0058] In an embodiment of the present disclosure, the following loss function may be used to supervise the training of the backbone network of the initial student model:
[0059] L HT =||uh (x)-r(v g (x))|| 2 (1)
[0060] Where, L HT represents the first loss value; x represents the original point cloud data; u h (x) represents the first characteristic matrix; v g (x) represents the output matrix of the second backbone network; r(·) represents the normalization function used to convert v g The dimension of (x) is related to u h (x) dimensions are aligned.
[0061] Then, based on the original point cloud data, the second network model is trained using the first probability matrix as a label to obtain a third network model.
[0062] According to an embodiment of the present disclosure, the second network model includes a semantic segmentation network of the first network model and an initial student model.
[0063] According to an embodiment of the present disclosure, training the second network model includes: inputting the original point cloud data into the second network model, and outputting a second output matrix; calculating a second loss value based on the second output matrix and the true label of the original point cloud data; calculating a third loss value based on the second output matrix and the first probability matrix; performing a weighted sum operation on the second loss value and the third loss value to obtain a fourth loss value; and when the fourth loss value is greater than a preset value, adjusting the model parameters of the second network model based on the fourth loss value.
[0064] In an embodiment of the present disclosure, the following loss function is used to supervise the training of the second network model:
[0065] L cls =μL hard +(1-μ)L soft (P s -P t ) (2)
[0066] L soft =-∑P t logP s (3)
[0067] Among them, L cls represents the fourth loss value; μ represents the loss adjustment parameter, and its value range is [0, 1]; L hard represents the second loss value, which can be obtained through the cross entropy loss function; L soft represents the third loss value, which is obtained by the loss function shown in formula (3); t represents the first probability matrix; P sRepresents the output matrix of the second network model.
[0068] Finally, based on the original point cloud data, the fourth network model is trained using the second feature matrix as a label to obtain a trained point cloud instance segmentation model.
[0069] According to an embodiment of the present disclosure, the fourth network model includes the third network model and an instance segmentation network of the initial student model.
[0070] According to an embodiment of the present disclosure, training the fourth network model includes: inputting the original point cloud data into the fourth network model, outputting the third output matrix; calculating the fifth loss value based on the second feature matrix and the third output matrix; and when the fifth loss value is greater than a preset value, adjusting the model parameters of the fourth network model based on the fifth loss value.
[0071] In an embodiment of the present disclosure, the following loss function is used to supervise the training of the fourth network model:
[0072] L′ HT =||u′ h (x)-v′ g (x)|| 2 (4)
[0073] Among them, L HT represents the fifth loss value; u′ h (x) represents the second characteristic matrix; v′ g (x) represents the output matrix of the fourth network model.
[0074] According to an embodiment of the present disclosure, during the training of the second network model or the fourth network model, based on the gradient diffusion phenomenon, the gradient of the trained partial network in the student model during the retraining process is very small, that is, the derivative of the overall loss function relative to the weight of the trained partial network is very small, so the model parameters of the trained partial network are changed slowly, avoiding the overfitting phenomenon of the student model due to overtraining. In addition, in order to avoid overfitting, a lower learning rate can be set for the trained partial network, for example, the learning rate of the trained partial network can be set to 1 / 10 of the learning rate of the untrained partial network.
[0075] It should be noted that the loss function used in the training of each network of the initial student model in the embodiment of the present disclosure is only schematic. In practical applications, the loss function can be adjusted according to experimental results. For example, a square loss function, a logarithmic loss function, a Hinge loss function, etc. can be used.
[0076] According to an embodiment of the present disclosure, the teacher model is used to guide the training of each network in the student model, which can effectively reduce the accuracy loss in the model migration process.
[0077] Figure 4a , Figure 4b and Figure 4c An application example of a point cloud instance segmentation model trained according to the method of an embodiment of the present disclosure is shown.
[0078] According to the embodiments of the present disclosure, Figure 4a The image shown is converted into a pixel matrix, that is, converted into a data format that can be input into the point cloud instance segmentation model. The image data is input into the point cloud instance segmentation model to obtain the probability matrix output by the semantic segmentation network of the point cloud instance segmentation model and the feature matrix output by the instance segmentation network of the point cloud instance segmentation model. In this embodiment, since the pixel matrix obtained by image conversion is a 3-dimensional matrix, the output probability matrix and feature matrix are 4-dimensional matrices, and one or more pixel points in the pixel matrix correspond to a probability vector in the probability matrix and a feature vector in the feature matrix.
[0079] According to the embodiment of the present disclosure, for each probability vector in the probability matrix, the category corresponding to the maximum value in the probability vector, that is, the semantic category of one or more corresponding pixels, is processed to obtain the following: Figure 4b Schematic diagram of point cloud category information shown.
[0080] According to the embodiment of the present disclosure, for each eigenvector in the feature matrix, the different objects to which the pixel points belong can be classified using the similarity measurement method, that is, Figure 4c Schematic diagram of point cloud boundary information shown.
[0081] According to the embodiment of the present disclosure, based on the point cloud category information and the point cloud boundary information, it is possible to obtain Figure 4a Point cloud instance segmentation results for the image in .
[0082] For example, Figure 4a Contains a stapler, correction tape, and 3 pens, 5 items of each kind; Figure 4b In the above example, by processing the probability matrix, the five items are divided into three categories, including staplers, correction tapes, and pens; Figure 4c Through similarity measurement, five items are distinguished, namely items 1 to 5; comprehensive Figure 4b and Figure 4c The classification results can complete the instance segmentation of these five items.
[0083] Figure 5 A schematic diagram of a training device 500 for a point cloud instance segmentation model according to an embodiment of the present disclosure is schematically shown.
[0084] like Figure 5 As shown, the training device 500 for the point cloud instance segmentation model includes a model acquisition module 510, a data storage module 520 and a training module 530.
[0085] The model acquisition module 510 is used to acquire a teacher model and an initial student model, wherein the teacher model is obtained by training an initial instance segmentation model, the initial instance segmentation model includes a first backbone network, a semantic segmentation network and an instance segmentation network connected in sequence, and the initial student model is obtained by replacing the first backbone network in the initial instance segmentation model with a second backbone network, wherein the network scale of the second backbone network is smaller than the network scale of the first backbone network
[0086] A data storage module 520 is used to store the first feature matrix output by the backbone network of the teacher model, the first probability matrix output by the semantic segmentation network of the teacher model, and the second feature matrix output by the instance segmentation network of the teacher model obtained by inputting the original point cloud data into the teacher model; and
[0087] The training module 530 is used to use the first feature matrix, the first probability matrix and the second feature matrix as labels of the original point cloud data, train the initial student model based on the original point cloud data, and obtain a trained point cloud instance segmentation model.
[0088] According to an embodiment of the present disclosure, the training module 530 includes: a first training unit, which is used to train the second backbone network of the initial student model based on the original point cloud data using the first feature matrix as a label to obtain a first network model; a second training unit, which is used to train the second network model based on the original point cloud data using the first probability matrix as a label to obtain a third network model, wherein the second network model includes the first network model and a semantic segmentation network of the initial student model; and a third training unit, which is used to train the fourth network model based on the original point cloud data using the second feature matrix as a label to obtain a trained point cloud instance segmentation model, wherein the fourth network model includes the third network model and an instance segmentation network of the initial student model.
[0089] According to an embodiment of the present disclosure, the first training unit includes: a first operator unit, which is used to input the original point cloud data into the second backbone network and output a first output matrix; a second operator unit, which is used to process the first output matrix using a normalization method so that the dimension of the first output matrix is equal to the dimension of the first feature matrix; a third operator unit, which is used to calculate a first loss value based on the first feature matrix and the normalized first output matrix; and a fourth operator unit, which is used to adjust the model parameters of the second backbone network based on the first loss value when the first loss value is greater than a preset value.
[0090] According to an embodiment of the present disclosure, the second training unit includes: a fifth operator unit, which is used to input the original point cloud data into the second network model, and output a second output matrix; a sixth operator unit, which is used to calculate the second loss value based on the second output matrix and the true label of the original point cloud data; a seventh operator unit, which is used to calculate the third loss value based on the second output matrix and the first probability matrix; an eighth operator unit, which is used to perform a weighted sum operation on the second loss value and the third loss value to obtain a fourth loss value; and a ninth operator unit, which is used to adjust the model parameters of the second network model based on the fourth loss value when the fourth loss value is greater than a preset value.
[0091] According to an embodiment of the present disclosure, the third training unit includes: a tenth operator unit, used to input the original point cloud data into the fourth network model, and output the obtained third output matrix; an eleventh operator unit, used to calculate the fifth loss value based on the second feature matrix and the third output matrix; and a twelfth operator unit, used to adjust the model parameters of the fourth network model based on the fifth loss value when the fifth loss value is greater than a preset value.
[0092] The device of the disclosed embodiment also includes an execution module, wherein the execution module includes: an execution unit, used to input the point cloud data to be segmented into the point cloud instance segmentation model, and obtain a second probability matrix output by the semantic segmentation network of the point cloud instance segmentation model and a third feature matrix output by the instance segmentation network of the point cloud instance segmentation model; a first analysis unit, used to obtain point cloud category information according to the second probability matrix, and perform similarity measurement on the third feature matrix to obtain point cloud boundary information; and a second analysis unit, used to obtain a point cloud instance segmentation result based on the point cloud category information and the point cloud boundary information.
[0093] According to the embodiments of the present disclosure, a student model is obtained by replacing the backbone network in the initial instance segmentation model with a smaller backbone network, the initial instance segmentation model is trained to convergence to obtain a teacher model, and the original point cloud data is input into the teacher model to obtain the output results of different networks as labels, and the labels are used to train different networks in the student model to obtain a compressed model. Based on this method, the compression of the instance segmentation model for three-dimensional objects is solved, which effectively reduces the storage space and computing resources required for the instance segmentation model, and at the same time reduces the accuracy loss during the model compression process.
[0094] According to the modules and units of the embodiments of the present disclosure, any multiple or at least part of the functions of any multiple thereof can be implemented in one module. According to the modules and units of the embodiments of the present disclosure, any one or more can be split into multiple modules for implementation. According to the modules and units of the embodiments of the present disclosure, any one or more can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware of any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in any appropriate combination of any of them. Alternatively, according to the modules and units of the embodiments of the present disclosure, one or more can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be executed.
[0095] Figure 6 A block diagram of an electronic device 600 suitable for implementing a training method for a point cloud instance segmentation model according to an embodiment of the present disclosure is schematically shown. Figure 6 The electronic device 600 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.
[0096] like Figure 6 As shown, the electronic device 600 according to an embodiment of the present disclosure includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 into a random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include an onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.
[0097] In RAM 603, various programs and data required for the operation of electronic device 600 are stored. Processor 601, ROM 602 and RAM 603 are connected to each other via bus 604. Processor 601 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 602 and / or RAM 603. It should be noted that the program can also be stored in one or more memories other than ROM 602 and RAM 603. Processor 601 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more memories.
[0098] According to an embodiment of the present disclosure, the electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to the bus 604. The system 600 may further include one or more of the following components connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage section 608 as needed.
[0099] According to an embodiment of the present disclosure, the method flow according to an embodiment of the present disclosure can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the processor 601, the above-mentioned functions defined in the system of the embodiment of the present disclosure are executed. According to an embodiment of the present disclosure, the system, equipment, device, module, unit, etc. described above can be implemented by a computer program module.
[0100] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.
[0101] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium. For example, it may include, but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus, or device.
[0102] For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include the ROM 602 and / or the RAM 603 described above and / or one or more memories other than the ROM 602 and the RAM 603 .
[0103] An embodiment of the present disclosure also includes a computer program product, which includes a computer program, which contains program code for executing the method provided by the embodiment of the present disclosure. When the computer program product runs on an electronic device, the program code is used to enable the electronic device to implement the training method of the point cloud instance segmentation model provided by the embodiment of the present disclosure.
[0104] When the computer program is executed by the processor 601, the above functions defined in the system / device of the embodiment of the present disclosure are executed. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.
[0105] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 609, and / or installed from a removable medium 611. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.
[0106] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).
[0107] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram may represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box may also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions. It can be understood by those skilled in the art that the features recorded in the various embodiments and / or claims of the present disclosure can be combined and / or combined in a variety of ways, even if such a combination or combination is not explicitly recorded in the present disclosure. In particular, without departing from the spirit and teaching of the present disclosure, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways. All of these combinations and / or combinations fall within the scope of the present disclosure.
[0108] It should be noted that the training method and device of the point cloud instance segmentation model of the embodiments of the present disclosure can be used in the field of artificial intelligence. Specific applicable fields include the field of unmanned driving technology, the field of AR technology, the field of drone technology, etc.; in addition, the application field of the method and device of the embodiments of the present disclosure is not limited, and can also be used in any field except the field of artificial intelligence.
[0109] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. The scope of the present disclosure is defined by the attached claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.
Claims
1. A method for training a point cloud instance segmentation model, comprising: Acquire a teacher model and an initial student model, wherein the teacher model is obtained by training an initial instance segmentation model, the initial instance segmentation model includes a first backbone network, a semantic segmentation network, and an instance segmentation network connected in sequence, and the initial student model is obtained by replacing the first backbone network in the initial instance segmentation model with a second backbone network, wherein the network scale of the second backbone network is smaller than the network scale of the first backbone network; Inputting the original point cloud data into the teacher model, obtaining a first feature matrix output by the backbone network of the teacher model, a first probability matrix output by the semantic segmentation network of the teacher model, and a second feature matrix output by the instance segmentation network of the teacher model; Using the first feature matrix, the first probability matrix, and the second feature matrix as labels of the original point cloud data, and training the initial student model based on the original point cloud data to obtain a trained point cloud instance segmentation model; Inputting the point cloud data to be segmented into the point cloud instance segmentation model to obtain a second probability matrix output by a semantic segmentation network of the point cloud instance segmentation model and a third feature matrix output by an instance segmentation network of the point cloud instance segmentation model; Acquire point cloud category information according to the second probability matrix, and perform similarity measurement on the third feature matrix to acquire point cloud boundary information; and Based on the point cloud category information and the point cloud boundary information, a point cloud instance segmentation result is obtained.
2. The method according to claim 1, wherein: The step of using the first feature matrix, the first probability matrix, and the second feature matrix as labels of the original point cloud data, training the initial student model based on the original point cloud data, and obtaining a trained point cloud instance segmentation model comprises: Based on the original point cloud data, using the first feature matrix as a label to train a second backbone network of the initial student model to obtain a first network model; Based on the original point cloud data, using the first probability matrix as a label to train a second network model to obtain a third network model, wherein the second network model includes a semantic segmentation network of the first network model and the initial student model; and Based on the original point cloud data, the fourth network model is trained using the second feature matrix as a label to obtain a trained point cloud instance segmentation model, wherein the fourth network model includes the instance segmentation network of the third network model and the initial student model.
3. The method according to claim 2, wherein: The second backbone network of the initial student model trained based on the original point cloud data using the first feature matrix as a label comprises: Inputting the original point cloud data into the second backbone network, and outputting a first output matrix; Processing the first output matrix using a normalization method so that the dimension of the first output matrix is equal to the dimension of the first feature matrix; Calculating a first loss value based on the first feature matrix and the normalized first output matrix; and When the first loss value is greater than a preset value, the model parameters of the second backbone network are adjusted based on the first loss value.
4. The method according to claim 2, wherein: The step of training a second network model based on the original point cloud data and using the first probability matrix as a label comprises: Inputting the original point cloud data into the second network model, and outputting a second output matrix; A second loss value is calculated based on the second output matrix and the true label of the original point cloud data; Calculate a third loss value based on the second output matrix and the first probability matrix; performing a weighted sum operation on the second loss value and the third loss value to obtain a fourth loss value; and When the fourth loss value is greater than a preset value, a model parameter of the second network model is adjusted based on the fourth loss value.
5. The method according to claim 2, wherein: The training of the fourth network model based on the original point cloud data and using the second feature matrix as a label comprises: Inputting the original point cloud data into the fourth network model, and outputting the obtained third output matrix; Calculate a fifth loss value based on the second feature matrix and the third output matrix; and When the fifth loss value is greater than a preset value, a model parameter of the fourth network model is adjusted based on the fifth loss value.
6. A training device for a point cloud instance segmentation model, comprising: A model acquisition module, used to acquire a teacher model and an initial student model, wherein the teacher model is obtained by training an initial instance segmentation model, the initial instance segmentation model includes a first backbone network, a semantic segmentation network and an instance segmentation network connected in sequence, and the initial student model is obtained by replacing the first backbone network in the initial instance segmentation model with a second backbone network, wherein the network scale of the second backbone network is smaller than the network scale of the first backbone network; A data storage module is used to store the first feature matrix output by the backbone network of the teacher model, the first probability matrix output by the semantic segmentation network of the teacher model, and the second feature matrix output by the instance segmentation network of the teacher model obtained by inputting the original point cloud data into the teacher model; a training module, configured to use the first feature matrix, the first probability matrix, and the second feature matrix as labels of the original point cloud data, and train the initial student model based on the original point cloud data to obtain a trained point cloud instance segmentation model; and The application module is used to input the point cloud data to be segmented into the point cloud instance segmentation model to obtain a second probability matrix output by the semantic segmentation network of the point cloud instance segmentation model and a third feature matrix output by the instance segmentation network of the point cloud instance segmentation model; obtain point cloud category information according to the second probability matrix, perform similarity measurement on the third feature matrix to obtain point cloud boundary information; and obtain a point cloud instance segmentation result based on the point cloud category information and the point cloud boundary information.
7. An electronic device comprising: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 5.
8. A readable storage medium having executable instructions stored thereon, which, when executed by a processor, enables the processor to implement the method according to any one of claims 1 to 5.
9. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 5.
