Semi-supervised bimodal semantic segmentation method and system based on parallel dual-flow network

Abstract

The application provides a kind of semi-supervised bimodal semantic segmentation method and system based on parallel double-flow network, method includes: obtaining to be segmented object, to be segmented object includes to be segmented point cloud data and to be segmented point cloud data corresponding to to be segmented image, to be segmented point cloud data and to be segmented image are not carried label;To be segmented object is input to target network, the feature of to be segmented point cloud data and the feature of to be segmented image are obtained, the target network is used to carry out semantic segmentation to the to be segmented object, the feature of the to be segmented object after segmentation is output.The system executes the method.The application only utilizes a small amount of labeled point cloud data, a large amount of unlabeled point cloud data and unlabeled image data for training, which is of great significance in alleviating the labeling difficulty problem and fully utilizing bimodal data, and has a wide practical application prospect.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of semantic segmentation, and particularly relates to a semi-supervised bimodal semantic segmentation method and system based on a parallel double-flow network. BACKGROUND

[0002] The semantic segmentation task is an important research topic in the field of computer vision, aiming to assign a semantic class label to each point or each pixel. In recent years, with the development of deep learning, three-dimensional point cloud semantic segmentation methods and two-dimensional image semantic segmentation methods based on neural networks have made major breakthroughs. Three-dimensional point cloud data has rich geometric structure, and two-dimensional image data has dense texture information. In the field of autonomous driving and robotics, three-dimensional point cloud and two-dimensional image data can be easily obtained by sensors. And the correspondence between three-dimensional points and image pixels can be calculated through the internal and external parameters of the camera.

[0003] In recent years, many researchers have tried to integrate three-dimensional point cloud and two-dimensional image semantic segmentation tasks to promote the segmentation accuracy of neural networks under different modal data. Although these methods can complement the advantages of three-dimensional point cloud and two-dimensional image, most of them are for fully supervised semantic segmentation tasks. Because the amount of point cloud and image data is very large in complex scenes, the cost of manually labeling semantic labels in fully supervised semantic segmentation tasks is high. In order to solve the problem of high cost of manual labeling of three-dimensional point cloud, some researchers have proposed methods to solve the problem of three-dimensional point cloud semi-supervised semantic segmentation. In the problem of three-dimensional point cloud semi-supervised semantic segmentation, the training set only has a small amount of labeled point cloud scenes and a large amount of unlabeled point cloud scenes. Although these methods can introduce additional constraints to extract semantic features of unlabeled point cloud, they do not make use of the existing image data in the scene, and there is a problem of difficulty in labeling three-dimensional point cloud data and two-dimensional image. SUMMARY

[0004] The semi-supervised bimodal semantic segmentation method and system based on a parallel double-flow network provided by the present application are used to solve the problems of difficulty and high cost in labeling three-dimensional point cloud data and two-dimensional image in the prior art.

[0005] The semi-supervised bimodal semantic segmentation method based on a parallel double-flow network provided by the present application comprises:

[0006] Obtaining a to-be-segmented object, the to-be-segmented object comprising to-be-segmented point cloud data and to-be-segmented image corresponding to the to-be-segmented point cloud data, the to-be-segmented point cloud data and the to-be-segmented image not carrying labels;

[0007] The object to be segmented is input into a target network to obtain features of the point cloud data to be segmented and features of the image to be segmented, the target network being used for semantic segmentation of the object to be segmented and outputting features of the segmented object to be segmented.

[0008] According to the parallel double-flow network-based semi-supervised bimodal semantic segmentation method provided in the application, the target network is obtained in the following manner:

[0009] The first data set is input into the original branch module of the preset network for training, and the parameters of the original branch module are updated based on the labels of the first data and the labels of the first image, the first data set comprising first data and first images corresponding to the first data, the first data being point cloud data carrying labels;

[0010] The original branch module is updated according to the updated parameters of the original branch module;

[0011] The pseudo-label prediction branch module of the preset network is initialized based on the updated parameters of the original branch module;

[0012] The second data set is input into the initialized pseudo-label prediction branch module for training to obtain labels of second data and labels of second images, the second data set comprising second data and second images corresponding to the second data, the second data being point cloud data not carrying labels;

[0013] The labels of the second data and the labels of the second images are optimized to obtain optimized labels of the second data and optimized labels of the second images;

[0014] The third data set is input into the updated original branch module for training to obtain the target network, the third data set comprising third data, third images corresponding to the third data and the second data set, the third data being point cloud data carrying first labels, the third images carrying second labels, the first labels being obtained by optimizing the labels of the third data, and the second labels being obtained by optimizing the labels of the third images.

[0015] According to the parallel double-flow network-based semi-supervised bimodal semantic segmentation method provided in the application, the labels of the first images are obtained in the following manner:

[0016] Based on the parameters of the sensor, the correspondence between the first data and the pixels of the first image is determined;

[0017] According to the correspondence, the labels of the first data are mapped into the first image to obtain the labels of the first image.

[0018] The application provides a semi-supervised bimodal semantic segmentation method based on a parallel double-flow network, wherein in the case that a value of a first loss function of the original branch module is less than or equal to a first preset threshold, the training of the original branch module is stopped, the first loss function is determined according to a cross-entropy loss function of a coder-decoder of the original branch module and a consistency constraint function, and the consistency constraint function is determined based on features of first data output by an encoder in the coder-decoder and features of a first image.

[0019] The application provides a semi-supervised bimodal semantic segmentation method based on a parallel double-flow network, wherein the labels of the second data and the labels of the second image are optimized to obtain optimized labels of the second data and optimized labels of the second image, and the optimization comprises:

[0020] In the case that a first preset condition is met, the label of the second data is taken as the optimized label of the second data, the first preset condition comprises that the label of the second data is consistent with a third label, or a feature of a first dimension of the second data output by the pseudo-label prediction branch module is greater than a second preset threshold, the third label is a projected label of the second data obtained by projecting the label of the second image to the second data, and the first dimension is determined according to a value of the label of the second data;

[0021] In the case that a second preset condition is met, the label of the second image is taken as the optimized label of the second image, the second preset condition comprises that the label of the second image is consistent with a fourth label, or a feature of a second dimension of the second image output by the pseudo-label prediction branch module is greater than the second preset threshold, the fourth label is a projected label of the second image obtained by projecting the label of the second data to the second image and performing dense processing through average pooling, and the second dimension is determined according to a value of the label of the second image.

[0022] The application provides a semi-supervised bimodal semantic segmentation method based on a parallel double-flow network, wherein in the case that a value of a second loss function of the updated original branch module is less than or equal to a third preset threshold or a training frequency of the updated original branch module reaches a fourth preset threshold, the training of the updated original branch module is stopped, and the second loss function is determined according to the first loss function and a cross-entropy loss function of a coder-decoder of the updated original branch module.

[0023] The application further provides a semi-supervised bimodal semantic segmentation system based on a parallel double-flow network, comprising an acquisition module and a segmentation module.

[0024] The acquisition module is configured to acquire an object to be segmented, wherein the object to be segmented comprises point cloud data to be segmented and an image to be segmented corresponding to the point cloud data to be segmented, and the point cloud data to be segmented and the image to be segmented are both unlabeled.

[0025] The segmentation module is configured to input the object to be segmented into a target network, acquire features of the point cloud data to be segmented and features of the image to be segmented, and output features of the object to be segmented after segmentation, wherein the target network is configured to perform semantic segmentation on the object to be segmented.

[0026] The application further provides an electronic device comprising a processor and a memory storing a computer program, wherein the processor implements the parallel double-flow network-based semi-supervised bimodal semantic segmentation method according to any one of the above when executing the program.

[0027] The application further provides a non-transitory computer-readable storage medium storing a computer program, wherein the computer program implements the parallel double-flow network-based semi-supervised bimodal semantic segmentation method according to any one of the above when executed by a processor.

[0028] The application further provides a computer program product comprising a computer program, wherein the computer program implements the parallel double-flow network-based semi-supervised bimodal semantic segmentation method according to any one of the above when executed by a processor.

[0029] The parallel double-flow network-based semi-supervised bimodal semantic segmentation method and system provided by the application can acquire features of unlabeled point cloud data to be segmented and unlabeled images to be segmented based on a target network, and can realize high-quality semantic labeling of point cloud data and images by using the features, thereby solving the problems of high cost and difficulty in labeling and acquiring point cloud data and images in the prior art. BRIEF DESCRIPTION OF DRAWINGS

[0030] In order to more clearly illustrate the technical solutions of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0031] Figure 1 is a flowchart of the parallel double-flow network-based semi-supervised bimodal semantic segmentation method provided by the application;

[0032] Figure 2 is a structural diagram of the target network provided by the application;

[0033] Figure 3A structure schematic diagram of a semi-supervised bimodal semantic segmentation system based on a parallel double-flow network provided by the present application is shown in the figure.

[0034] Figure 4 A physical structure schematic diagram of an electronic device provided by the present application is shown in the figure. DETAILED DESCRIPTION

[0035] To make the objectives, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below with reference to the accompanying drawings in the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the present application.

[0036] The semi-supervised bimodal semantic segmentation method based on the parallel double-flow network provided by the present application can solve the problem that the existing point cloud data and image segmentation method needs high-quality semantic labels with high cost and difficulty. The target network provided by the present application includes two parallel double-flow branch modules: an original branch module and a pseudo-label prediction branch module. The pseudo-label prediction branch module is used to predict the pseudo-labels of the unlabeled point cloud data and the corresponding images, and the predicted pseudo-labels are sent to the original branch module for self-training. The original branch module and the pseudo-label prediction branch module both include 3D and 2D encoder-decoder branches and multiple bimodal fusion modules. The 3D and 2D encoder-decoder branches are used to extract the features of the point cloud data and the images respectively, and the bimodal fusion modules are used to fuse the point cloud and image features. In addition, the present application also includes a pseudo-label optimization module, which is used to refine the pseudo-labels output by the pseudo-label prediction branch module to generate high-quality pseudo-labels, so as to ensure the effectiveness of the unlabeled data training. The specific implementation is as follows:

[0037] Figure 1 A flowchart of the semi-supervised bimodal semantic segmentation method based on the parallel double-flow network provided by the present application is shown in the figure. Figure 1 As shown in the figure, the method includes:

[0038] In step 110, an object to be segmented is obtained, which includes point cloud data to be segmented and an image corresponding to the point cloud data to be segmented. The point cloud data to be segmented and the image to be segmented do not carry labels.

[0039] In step 120, the object to be segmented is input into a target network to obtain the features of the point cloud data to be segmented and the features of the image to be segmented. The target network is used for semantic segmentation of the object to be segmented, and outputs the features of the segmented object to be segmented.

[0040] It should be noted that the execution subject of the above method can be a computer device.

[0041] Optionally, the to-be-segmented object can specifically include to-be-segmented point cloud data not carrying a label and to-be-segmented images not carrying a label corresponding to the to-be-segmented point cloud data, the to-be-segmented point cloud data being three-dimensional point cloud data, and the to-be-segmented images being two-dimensional images.

[0042] The to-be-segmented object can be specifically obtained through a sensor that collects to-be-segmented point cloud data and to-be-segmented images.

[0043] Optionally, the target network can be specifically used for semantic segmentation of the to-be-segmented object and output of features of the segmented to-be-segmented object, the features can be used as labels of the to-be-segmented object, and the to-be-segmented object is semantically annotated, wherein the target network is obtained through training.

[0044] The obtained to-be-segmented object is input into the target network, and features of the to-be-segmented point cloud data and features of the to-be-segmented images are obtained according to an output of the target network.

[0045] The parallel double-flow network-based semi-supervised bimodal semantic segmentation method provided by the application obtains features of to-be-segmented point cloud data not carrying a label and to-be-segmented images not carrying a label based on a target network, and high-quality semantic annotation of the point cloud data and the images can be realized by using the features, thereby solving the problems of high cost and difficulty in annotation acquisition of the point cloud data and the images in the prior art.

[0046] Further, in one embodiment, the target network can be obtained in the following manner:

[0047] The first data set is input into the original branch module of the preset network for training, and parameters of the original branch module are updated based on labels of the first data and labels of the first images, the first data set includes the first data and first images corresponding to the first data, and the first data is point cloud data carrying a label;

[0048] The original branch module is updated according to the updated parameters of the original branch module;

[0049] The pseudo-label prediction branch module of the preset network is initialized based on the updated parameters of the original branch module;

[0050] The second data set is input into the initialized pseudo-label prediction branch module for training to obtain labels of the second data and labels of the second images, the second data set includes the second data and second images corresponding to the second data, and the second data is point cloud data not carrying a label;

[0051] optimizing the label of the second data and the label of the second image to obtain an optimized label of the second data and an optimized label of the second image;

[0052] inputting a third data set into the updated original branch module for training to obtain the target network, the third data set comprising third data, a third image corresponding to the third data and the second data set, the third data being point cloud data carrying a first label, the third image carrying a second label, the first label being obtained by optimizing the label of the third data, and the second label being obtained by optimizing the label of the third image.

[0053] Optionally, Figure 2 is a structural schematic diagram of the target network provided by the application, as Figure 2 indicated, the target network can specifically be a semi-supervised dual-modal semantic segmentation network based on a parallel dual-flow network, which can specifically include an original branch module and a pseudo label prediction branch module, and is obtained by training a preset network having the same structure as the target network.

[0054] The first data set can specifically include first data and an image corresponding to the first data, i.e., a first image, and it should be noted that the first data can specifically be point cloud data carrying a label, and the first image does not carry a label.

[0055] The first data is input into the original branch module in the preset network for training in a supervised manner. The point cloud data carrying a label and the first image corresponding thereto not carrying a label are taken as inputs of the original branch module, and the correspondence between the point cloud data and the first image is established by obtaining the parameters of the sensor of the first data set to map the label of the point cloud data to the first image to obtain a pixel-level semantic label, i.e., the label of the first image. The point cloud data carrying a label and the first image are respectively subjected to 3D and 2D codec (including an encoder and a decoder) structures and a plurality of dual-modal fusion modules in the original branch to obtain the features (i.e., 3D semantic features) of the point cloud data and the features (i.e., 2D semantic features) of the first image, and the output of the decoder is supervised using the label of the point cloud data and the label of the first image obtained by mapping, the parameters in the two codec structures are updated by back propagation, and then the parameters of the original branch module are updated.

[0056] The parameters of the original branch module are updated using the updated parameters of the original branch module, and the pseudo label prediction branch module in the preset network is initialized based on the updated parameters of the original branch module.

[0057] Optionally, the second data set can specifically include point cloud data (i.e., second data) without carrying a label and its corresponding image, i.e., a second image, which does not carry a label.

[0058] The pseudo-label prediction branch module is trained using the point cloud data without carrying a label and its corresponding second image to obtain pseudo-labels of the second data in the second data set and the second image, i.e., labels of the second data and the second image, and the updated original branch module is trained using the third data set to obtain the target network. The pseudo-label prediction branch module is initialized using the updated original branch parameters, the second data set is input into the pseudo-label prediction branch module to obtain the labels of the second data and the second image, and the labels of the second data and the second image are input into the pseudo-label optimization module for further optimization.

[0059] Optionally, the third data set can specifically include third data, an image corresponding to the third data (i.e., a third image), and the second data set, the third data being point cloud data carrying a first label, and the third image carrying a second label.

[0060] The first label can be specifically obtained by optimizing the label of the third data using the pseudo-label optimization module, and the second label can be specifically obtained by optimizing the label of the third image using the pseudo-label optimization module.

[0061] The updated labels of the second data and the second image are used as supervision information of the label-free data (i.e., the second data set), and the updated original branch module is trained using the third data set to obtain the target network.

[0062] Further, in an embodiment, the optimizing the labels of the second data and the second image to obtain the updated labels of the second data and the updated labels of the second image can specifically include:

[0063] In a case where a first preset condition is met, the label of the second data is used as the updated label of the second data, and the first preset condition includes that the label of the second data is consistent with a third label, or a first dimension of the second data output by the pseudo-label prediction branch module is greater than a second preset threshold, the third label being a projected second data label obtained by projecting the label of the second image to the second data, and the first dimension being determined according to a value of the label of the second data.

[0064] In a case of meeting a second preset condition, taking the label of the second image as the label of the optimized second image, the second preset condition comprises that the label of the second image and a fourth label are consistent, or a second dimension feature of the second image output by the pseudo label prediction branch module is greater than the second preset threshold, the fourth label is a projected second image label obtained by projecting the label of the second data to the second image and densifying through average pooling, and the second dimension is determined according to a value of the label of the second image.

[0065] Optionally, the application optimizes the label of the second data and the label of the second image through a pseudo label optimization module, and specifically:

[0066] The pseudo label prediction branch module also comprises 3D and 2D codec structures and a plurality of bimodal fusion modules, and is used for outputting pseudo labels of the second data and the corresponding second image. The application optimizes the output pseudo labels through the design of the pseudo label optimization module, so as to improve the quality of the pseudo labels.

[0067] Since the pseudo label optimization module is only used for point cloud data without carrying a label and an image without carrying a label, the point cloud data-image pixel matching pair without a label is represented as where N u is the number of point cloud data-image pixel matching pairs without a label. The pseudo label optimization module takes 3D coarse pseudo labels (i.e., the label of the second data) output by the pseudo label prediction branch and 2D coarse pseudo labels (i.e., the label of the second image) as inputs, and outputs optimized 3D fine pseudo labels (i.e., the optimized label of the second data) and 2D fine pseudo labels (i.e., the optimized label of the second image)

[0068] The calculation process of the optimized label of the second data, i.e., the 3D fine pseudo label may be represented as:

[0069]

[0070] wherein, is a projected second data label 3D pseudo label (i.e., a third label) obtained by projecting the label of the second image, i.e., the 2D coarse pseudo label to the second data, is a first dimension feature of the second data output by the pseudo label prediction branch module, t conf is a preset threshold, i.e., a second preset threshold, and the first dimension can be specifically determined according to a value of the label of the second data .

[0071] In the case of meeting the first preset condition, i.e., if the 3D coarse pseudo label and the 2D coarse pseudo label meet or , the 3D coarse pseudo label is retained as the label of the second data after optimization, and if the condition is not met, the coarse pseudo label is deleted.

[0072] Similarly, the calculation process of the label of the second image after optimization, i.e., the 2D fine pseudo label , can be represented as:

[0073]

[0074] wherein, is the projected 2D pseudo label (i.e., the fourth label) obtained by projecting the label of the second data, the 3D coarse pseudo label , onto the second image and performing dense processing through average pooling, is the feature of the second dimension of the 2D feature of the second image output by the pseudo label prediction branch module, wherein the second dimension is determined according to the value of the label of the second image .

[0075] In the case of meeting the second preset condition, i.e., if or is met, the 2D coarse pseudo label is retained as the label of the second image after optimization, and if the condition is not met, the coarse pseudo label is deleted.

[0076] After obtaining the labels of the second data and the corresponding second image after optimization and , they can be used as the supervision information of the unlabeled data, and at the same time, the third data set is put into the updated original branch module for training.

[0077] The application provides a semi-supervised bimodal semantic segmentation method based on a parallel double-flow network, which predicts pseudo labels of point cloud data and corresponding images without carrying labels through two parallel double-flow branches in a target network, i.e., an original branch module and a pseudo label prediction branch module, the predicted pseudo labels are sent to the original branch module for self-training, the original branch and the pseudo label prediction branch both contain 3D and 2D codec branches and multiple bimodal fusion modules, wherein the 3D and 2D codec branches are used for extracting point cloud and image features respectively, the bimodal fusion modules are used for fusing point cloud and image features, and a pseudo label optimization module is used for refining the pseudo labels to generate high-quality labels, so as to guarantee the effectiveness of the label-free data training, only part of the labels of the point cloud data need to be provided during training, and the labels of the images do not need to be provided, thereby effectively solving the problems of difficult and high-cost labeling of point cloud data and images in the prior art.

[0078] Further, in one embodiment, the manner of obtaining the label of the first image can specifically include:

[0079] determining a correspondence between the first data and pixels of the first image based on parameters of the sensor;

[0080] mapping the label of the first data to the first image according to the correspondence to obtain the label of the first image.

[0081] Optionally, the correspondence between the first data and pixels of the first image is obtained according to parameters of the sensor for obtaining the first data and the first image, and the label of the first data is mapped to the first image according to the correspondence to obtain the pixel-level semantic label, i.e., the label of the first image.

[0082] and the N matched point cloud data-first image pixel pairs are represented as: where p i is a three-dimensional coordinate of a point, and x i is a two-dimensional coordinate of a pixel.

[0083] The bimodal fusion module takes the matched 3D feature (i.e., the feature of the point cloud data) f(p i ) and the 2D feature (i.e., the feature of the image) f(x i ) in a certain layer of the decoder as input, outputs the fused 3D feature g(p i ) and the 2D feature g(x i ), and sends them to the next layer of the decoder.

[0084] The bimodal fusion module uses a multi-head attention mechanism to fuse the features of the two modalities. Specifically, Minkowski convolution M and ordinary convolution C are used to process the input 3D feature f(p i ) and the 2D feature f(xi ) is calculated to obtain the key feature K(p i ), K(x i ), the query feature Q(p i ), Q(x i ) and the value feature V(p i ), V(x i ).

[0085] The fused 3D feature g(p i ) can be calculated by the following formula:

[0086]

[0087] wherein A(p i ) = Softmax(∑(K(p i )⊙Q(x i ))), M represents the Minkowski convolution operation, ⊙ represents the dot product operation, represents the concatenation operation.

[0088] Similarly, the fused 2D feature g(x i ) can be calculated by the following formula:

[0089]

[0090] wherein A(x i ) = Softmax(∑(K(x i )⊙Q(p i ))), C represents the normal convolution operation.

[0091] Further, in an embodiment, when the value of the first loss function of the original branch module is less than or equal to a first preset threshold, the training of the original branch module is stopped, the first loss function is determined according to the cross-entropy loss function of the encoder-decoder of the original branch module and a consistency constraint function, and the consistency constraint function is determined based on the features of the first data output by the encoder in the encoder-decoder and the features of the first image.

[0092] Optionally, in terms of loss function, the application proposes a consistency constraint function based on mean square error, which can make the 3D feature and the 2D feature also consistent in the output feature space. The dual-modal fusion module and the consistency constraint function make the neural network fuse the 3D and 2D semantic features in the feature space at different levels.

[0093] The application uses multiple bimodal fusion modules to fuse the bimodal features in the encoder at multiple scales. In order to ensure the consistency of the first data features of the encoder output of the original branch module, i.e., the 3D features y(p i ) and the features of the first image, i.e., the 2D features y(x i ), the application also proposes a consistency constraint function based on L2 norm

[0094]

[0095] The bimodal fusion module and the consistency constraint function enable the neural network to fuse the 3D and 2D features in the feature space at different levels.

[0096] When training the original branch module, the input point cloud data is labeled point cloud data, and the label of the first image is obtained by mapping the label of the point cloud data, so the loss function (i.e., the first loss function) of the original branch module in this step is

[0097]

[0098] wherein, and are common cross-entropy loss functions, representing the cross-entropy loss function of the 3D encoder-decoder and the cross-entropy loss function of the 2D encoder-decoder, respectively, and the loss term uses the provided label to constrain the output semantic features.

[0099] In the case where the value of the first loss function of the original branch module is less than or equal to a first preset threshold, the training of the original branch module is stopped, and the first preset threshold can be set according to actual conditions.

[0100] Further, in an embodiment, in the case where the value of the second loss function of the updated original branch module is less than or equal to a third preset threshold or the number of training times of the updated original branch module reaches a fourth preset threshold, the training of the updated original branch module is stopped, and the second loss function is determined according to the first loss function and the cross-entropy loss function of the encoder-decoder of the updated original branch module.

[0101] Optionally, the loss function (i.e., the second loss function) of the updated original branch module can be specifically obtained according to the first loss function of the original branch module and the cross-entropy loss function of the updated encoder-decoder, and is specifically as follows:

[0102]

[0103] wherein, represent a second loss function, and represent cross-entropy loss functions of 3D codecs and 2D codecs in the updated original branch module respectively, That is, the common cross-entropy loss, which uses the provided label to constrain the output semantic feature.

[0104] In the case that the value of the second loss function of the updated original branch module is less than or equal to a third preset threshold, or the number of training times of the updated original branch module reaches a fourth preset threshold, the training of the updated original branch module is stopped, and the third preset threshold and the fourth preset threshold can be set according to actual conditions.

[0105] The application provides a semi-supervised bimodal semantic segmentation method based on a parallel double-flow network, in order to fuse three-dimensional point cloud data and two-dimensional image data to solve the problem of semi-supervised bimodal semantic segmentation, the application proposes a semi-supervised bimodal semantic segmentation network based on a parallel double-flow network, the application trains a target network by using only a small amount of labeled point cloud data and a large amount of unlabeled point cloud data and unlabeled image data, and based on the target network, the semantic annotation of the unlabeled point cloud data and the unlabeled image data can effectively alleviate the problem of difficulty in labeling point cloud data and image data.

[0106] The semi-supervised bimodal semantic segmentation system based on the parallel double-flow network provided by the application is described below, and the semi-supervised bimodal semantic segmentation system based on the parallel double-flow network described below can be mutually corresponding to the semi-supervised bimodal semantic segmentation method based on the parallel double-flow network described above.

[0107] Figure 3 is a structural schematic diagram of the semi-supervised bimodal semantic segmentation system based on the parallel double-flow network provided by the application, as Figure 3 shown, comprising:

[0108] an acquisition module 310 and a segmentation module 311;

[0109] The acquisition module 310 is configured to acquire a to-be-segmented object, the to-be-segmented object comprising to-be-segmented point cloud data and to-be-segmented image corresponding to the to-be-segmented point cloud data, and the to-be-segmented point cloud data and the to-be-segmented image both do not carry labels.

[0110] The segmentation module 311 is configured to input the to-be-segmented object into a target network, acquire features of the to-be-segmented point cloud data and features of the to-be-segmented image, the target network being configured to perform semantic segmentation on the to-be-segmented object, and output features of the segmented to-be-segmented object.

[0111] The application provides a semi-supervised bimodal semantic segmentation system based on a parallel double-flow network, which obtains features of unlabeled to-be-segmented point cloud data and to-be-segmented images based on a target network, and high-quality semantic labeling of the point cloud data and the images can be realized by using the features, thereby solving the problems of high cost and high difficulty in labeling and obtaining the point cloud data and the images in the prior art.

[0112] Figure 4 is a schematic diagram of an entity structure of an electronic device provided by the application, as shown in Figure 4 The electronic device can include a processor 410, a communication interface 411, a memory 412 and a bus 413, wherein the processor 410, the communication interface 411 and the memory 412 can communicate with each other through the bus 413. The processor 410 can call logical instructions in the memory 412 to execute the following method:

[0113] Obtaining a to-be-segmented object, the to-be-segmented object including to-be-segmented point cloud data and to-be-segmented images corresponding to the to-be-segmented point cloud data, wherein the to-be-segmented point cloud data and the to-be-segmented images are both unlabeled;

[0114] Inputting the to-be-segmented object into a target network to obtain features of the to-be-segmented point cloud data and features of the to-be-segmented images, wherein the target network is used for semantic segmentation of the to-be-segmented object, and outputs features of the segmented to-be-segmented object.

[0115] In addition, the logical instructions in the memory described above can be implemented in the form of a software functional unit and sold or used as an independent product, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions for causing a computer power supply screen (which can be a personal computer, a server, or a network power supply screen) to execute all or part of the steps of the method described in the various embodiments of the application. The foregoing storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM, Read-only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk and various program code storage media.

[0116] Further, the present application discloses a computer program product, which comprises a computer program stored on a non-transitory computer-readable storage medium, and the computer program comprises program instructions, and when the program instructions are executed by a computer, the computer can execute the parallel double-flow network based semi-supervised bimodal semantic segmentation method provided by the above-mentioned method embodiments, for example, comprising:

[0117] An object to be segmented is acquired, the object to be segmented comprising point cloud data to be segmented and an image to be segmented corresponding to the point cloud data to be segmented, and the point cloud data to be segmented and the image to be segmented both do not carry labels;

[0118] The object to be segmented is input into a target network to acquire features of the point cloud data to be segmented and features of the image to be segmented, the target network being used for semantic segmentation of the object to be segmented, and features of the segmented object to be segmented are output.

[0119] In another aspect, the present application further provides a non-transitory computer-readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the parallel double-flow network based semi-supervised bimodal semantic segmentation method provided by the above-mentioned embodiments, for example, comprising:

[0120] An object to be segmented is acquired, the object to be segmented comprising point cloud data to be segmented and an image to be segmented corresponding to the point cloud data to be segmented, and the point cloud data to be segmented and the image to be segmented both do not carry labels;

[0121] The object to be segmented is input into a target network to acquire features of the point cloud data to be segmented and features of the image to be segmented, the target network being used for semantic segmentation of the object to be segmented, and features of the segmented object to be segmented are output.

[0122] The system embodiments described above are only illustrative, wherein the units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. According to actual needs, part or all of the modules can be selected to achieve the purpose of the present embodiment. Those skilled in the art can understand and implement without creative labor.

[0123] Those skilled in the art can clearly understand the technical solutions of the various embodiments from the above description of the embodiments, and the various embodiments can be implemented by means of software with the necessary general hardware platforms, and of course, can also be implemented by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part of the prior art that makes a contribution, can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, and the like, and includes a number of instructions for causing a computer power supply screen (which can be a personal computer, a server, or a network power supply screen, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0124] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of the present application.

Claims

1. A semi-supervised bimodal semantic segmentation method based on parallel two-stream networks, characterized in that, include: Obtain the object to be segmented, which includes point cloud data to be segmented and the corresponding image to be segmented, wherein neither the point cloud data to be segmented nor the image to be segmented carries a label; The object to be segmented is input into the target network to obtain the features of the point cloud data to be segmented and the features of the image to be segmented. The target network is used to perform semantic segmentation on the object to be segmented and output the features of the segmented object to be segmented. The methods for obtaining the target network include: The first dataset is input into the original branch module of the preset network for training, and the parameters of the original branch module are updated based on the labels of the first data and the labels of the first image. The first dataset includes the first data and the first image corresponding to the first data. The first data is point cloud data with labels. Update the original branch module according to the updated parameters of the original branch module; Based on the updated parameters of the original branch module, the pseudo-label prediction branch module of the preset network is initialized; The second dataset is input into the initialized pseudo-label prediction branch module for training to obtain the label of the second data and the label of the second image. The second dataset includes the second data and the second image corresponding to the second data. The second data is point cloud data without labels. The labels of the second data and the second image are optimized to obtain the optimized labels of the second data and the optimized labels of the second image; The third dataset is input into the updated original branch module for training to obtain the target network. The third dataset includes third data, the third image corresponding to the third data, and the second dataset. The third data is point cloud data carrying a first label, and the third image carries a second label. The first label is obtained by optimizing the label of the third data, and the second label is obtained by optimizing the label of the third image. The step of optimizing the labels of the second data and the second image to obtain optimized labels for the second data and optimized labels for the second image includes: Under the condition of satisfying the first preset condition, the label of the second data is used as the label of the optimized second data. The first preset condition includes that the label of the second data and the third label are the same, or the feature of the first dimension of the second data output by the pseudo-label prediction branch module is greater than the second preset threshold. The third label is the label of the projected second data obtained by projecting the label of the second image onto the second data. The first dimension is determined according to the value of the label of the second data. Under the condition of satisfying the second preset condition, the label of the second image is used as the label of the optimized second image. The second preset condition includes that the label of the second image and the fourth label are consistent, or the feature of the second dimension of the second image output by the pseudo label prediction branch module is greater than the second preset threshold. The fourth label is the label of the projected second image obtained by projecting the label of the second data onto the second image and then densening it through average pooling. The second dimension is determined according to the value of the label of the second image.

2. The semi-supervised bimodal semantic segmentation method based on parallel two-stream networks according to claim 1, characterized in that, The method for obtaining the label of the first image includes: Based on the parameters of the sensor, the correspondence between the first data and the first image pixels is determined; Based on the correspondence, the labels of the first data are mapped to the first image to obtain the labels of the first image.

3. The semi-supervised bimodal semantic segmentation method based on parallel two-stream networks according to claim 1, characterized in that, If the value of the first loss function of the original branch module is less than or equal to a first preset threshold, training of the original branch module is stopped. The first loss function is determined based on the cross-entropy loss function and consistency constraint function of the encoder and decoder of the original branch module. The consistency constraint function is determined based on the features of the first data output by the encoder in the encoder and the features of the first image.

4. The semi-supervised bimodal semantic segmentation method based on parallel two-stream networks according to claim 1, characterized in that, If the value of the second loss function of the updated original branch module is less than or equal to the third preset threshold, or if the number of training iterations of the updated original branch module reaches the fourth preset threshold, training of the updated original branch module is stopped. The second loss function is determined based on the first loss function and the cross-entropy loss function of the encoder and decoder of the updated original branch module.

5. A semi-supervised bimodal semantic segmentation system based on parallel two-stream networks, applying the semi-supervised bimodal semantic segmentation method based on parallel two-stream networks as described in any one of claims 1 to 4, characterized in that, include: Acquisition module and segmentation module; The acquisition module is used to acquire the object to be segmented. The object to be segmented includes point cloud data to be segmented and the image to be segmented corresponding to the point cloud data. Neither the point cloud data to be segmented nor the image to be segmented carries a label. The segmentation module is used to input the object to be segmented into a target network, obtain the features of the point cloud data to be segmented and the features of the image to be segmented, and the target network is used to perform semantic segmentation on the object to be segmented and output the features of the segmented object.

6. An electronic device comprising a processor and a memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the semi-supervised bimodal semantic segmentation method based on parallel two-stream networks as described in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the semi-supervised bimodal semantic segmentation method based on a parallel two-stream network as described in any one of claims 1 to 4.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the semi-supervised bimodal semantic segmentation method based on a parallel two-stream network as described in any one of claims 1 to 4.

Images