Method and device for determining target semantic segmentation network model and processor

Abstract

The invention relates to the field of neural networks, and discloses a method and device for determining a target semantic segmentation network model and a processor. The method comprises the steps that semantic segmentation results are input into a plurality of discriminators to obtain a plurality of classification results, the classification results belong to any one of a 2D source domain, a 3D source domain, a 2D target domain and a 3D target domain, and the discriminators comprise at least one of a first discriminator, a second discriminator, a third discriminator and a fourth discriminator; the first discriminator is used for judging whether the semantic segmentation result belongs to a 2D source domain or a 2D target domain, the second discriminator is used for judging whether the semantic segmentation result belongs to a 3D source domain or a 3D target domain, the third discriminator is used for judging whether the semantic segmentation result belongs to the 2D source domain or the 3D target domain, and the fourth discriminator is used for judging whether the semantic segmentation result belongs to the 3D source domain or the 2D target domain; obtaining a plurality of semantic segmentation network models to be tested after adversarial training; and determining a target semantic segmentation network model according to the test segmentation result. And the semantic segmentation accuracy of the target domain is improved.

Description

Technical Field

[0001] The present invention relates to the field of neural networks, and in particular to a method, a device and a processor for determining a target semantic segmentation network model. Background Art

[0002] Taking the field of industrial intelligence as an example, semantic segmentation plays a vital role in the field of industrial intelligence. For example, in projects such as helmet detection and seat belt detection, it is necessary to identify and segment different parts more accurately to make the identification more accurate. At the same time, in the field of chemical production, semantic segmentation is also needed to achieve more accurate monitoring. For inspection robots, effective semantic segmentation of scenes is the basis for realizing subsequent functions. The task of semantic segmentation is to assign a label to each pixel in the image to achieve pixel-level classification of the image content. However, collecting expert-labeled data sets, especially pixel-level annotations, is a process that requires a lot of manpower and material resources. The current common solution is to perform domain adaptation on the source domain and the target domain, so that the model trained on the labeled source domain can be migrated to the unlabeled target domain and achieve acceptable segmentation performance. As mentioned above, the current research on domain adaptation of semantic segmentation is mainly aimed at single-modality (image) domain adaptation, and the segmentation effect of the semantic segmentation network model is poor. Summary of the invention

[0003] In order to overcome the deficiencies in the prior art, an embodiment of the present invention provides a method, an apparatus, and a processor for determining a target semantic segmentation network model.

[0004] In order to achieve the above object, the first aspect of the present invention provides a method for determining a target semantic segmentation network model, comprising:

[0005] Inputting a first training data set and a second training data set into a semantic segmentation network model to obtain semantic segmentation results of the first training data set and the second training data set, wherein the first training data set includes a source domain 2D image, a source domain 3D point cloud data, a pixel-level label of the source domain 2D image, and a point-by-point label of the source domain 3D point cloud data, and the second training data set includes a target domain 2D image and a target domain 3D point cloud data;

[0006] Inputting the semantic segmentation results into a plurality of discriminators respectively, and obtaining a plurality of classification results of the semantic segmentation results, wherein the classification results include any one of a 2D source domain, a 3D source domain, a 2D target domain, and a 3D target domain, and the discriminator includes at least one of a first discriminator, a second discriminator, a third discriminator, and a fourth discriminator, wherein the first discriminator is used to determine whether the semantic segmentation result belongs to the 2D source domain or the 2D target domain, the second discriminator is used to determine whether the semantic segmentation result belongs to the 3D source domain or the 3D target domain, the third discriminator is used to determine whether the semantic segmentation result belongs to the 2D source domain or the 3D target domain, and the fourth discriminator is used to determine whether the semantic segmentation result belongs to the 3D source domain or the 2D target domain;

[0007] According to the semantic segmentation results and classification results, the semantic segmentation network model is used as a generator to perform adversarial training with multiple discriminators to obtain multiple semantic segmentation network models to be tested;

[0008] Inputting a test data set into multiple semantic segmentation network models to be tested to obtain test segmentation results of each of the multiple semantic segmentation network models to be tested, wherein the test data set includes a target domain 2D image, a target domain 3D point cloud data, a pixel-level label of the target domain 2D image, and a point-by-point label of the target domain 3D point cloud data;

[0009] A target semantic segmentation network model among multiple semantic segmentation network models to be tested is determined according to the respective test segmentation results.

[0010] In an embodiment of the present invention, the semantic segmentation network model includes a 2D semantic segmentation branch and a 3D semantic segmentation branch, the 2D semantic segmentation branch includes a U-Net network, a feature extraction network and a true 2D segmentation head, and the 3D semantic segmentation branch includes a spatial sparse convolutional neural network and a true 3D segmentation head;

[0011] Inputting the first training data set and the second training data set into the semantic segmentation network model, obtaining the semantic segmentation results of the first training data set and the second training data set includes:

[0012] Inputting the 2D images in the first training data set and the second training data set into the U-Net network to obtain 2D images in dense pixel form, where the 2D images include source domain 2D images and target domain 2D images;

[0013] Input the 2D image in dense pixel form into the feature extraction network to obtain a 2D feature map;

[0014] Input the 2D feature map into the real 2D segmentation head to obtain the semantic segmentation result of the 2D image;

[0015] Inputting the first training data set and the second training data set into the semantic segmentation network model to obtain the semantic segmentation results of the first training data set and the second training data set also includes:

[0016] Inputting the 3D point cloud data in the first training data set and the second training data set into a spatial sparse convolutional neural network to obtain 3D point cloud data in sparse voxel form, wherein the 3D point cloud data includes source domain 3D point cloud data and target domain 3D point cloud data;

[0017] The 3D point cloud data in sparse voxel form is input into the real 3D segmentation head to obtain the semantic segmentation result of the 3D point cloud data.

[0018] In an embodiment of the present invention, the method further includes:

[0019] Project the 3D point cloud data to obtain a projection image;

[0020] The 2D feature map is cropped and feature sampled according to the size of the projected image to obtain a cropped 2D feature map;

[0021] The 2D feature map is input into the real 2D segmentation head, and the semantic segmentation results of the 2D image are obtained, including:

[0022] The cropped 2D feature map is input into the real 2D segmentation head to obtain the semantic segmentation result of the 2D image.

[0023] In an embodiment of the present invention, the 2D semantic segmentation branch includes imitating a 3D segmentation head, the 3D semantic segmentation branch includes imitating a 2D segmentation head, and the method further includes:

[0024] The cropped 2D feature map is input into the imitation 3D segmentation head to obtain the 3D segmentation prediction result;

[0025] Input the 3D point cloud data in the form of sparse voxels into the simulated 2D segmentation head to obtain the 2D segmentation prediction result;

[0026] The imitated 3D segmentation head, the imitated 2D segmentation head, the real 3D segmentation head and the real 2D segmentation head are retrained according to the first segmentation loss of the 3D segmentation prediction result and the real 3D segmentation result of the real 3D segmentation head, and the second segmentation loss of the 2D segmentation prediction result and the real 2D segmentation result of the real 2D segmentation head.

[0027] In the embodiment of the present invention, the sizes of output data of the simulated 3D segmentation head and the real 3D segmentation head are the same, and the sizes of output data of the simulated 2D segmentation head and the real 2D segmentation head are the same.

[0028] In the embodiment of the present invention, the number of the multiple discriminators and the number of the multiple semantic segmentation network models to be tested are both 15.

[0029] A second aspect of the present invention provides a device for determining a target semantic segmentation network model, the device comprising:

[0030] A first input module is used to input the first training data set and the second training data set into the semantic segmentation network model to obtain semantic segmentation results of the first training data set and the second training data set, wherein the first training data set includes a source domain 2D image, a source domain 3D point cloud data, a pixel-level label of the source domain 2D image, and a point-by-point label of the source domain 3D point cloud data, and the second training data set includes a target domain 2D image and a target domain 3D point cloud data;

[0031] a second input module, for inputting the semantic segmentation results into a plurality of discriminators respectively, to obtain a plurality of classification results of the semantic segmentation results, wherein the classification results include any one of a 2D source domain, a 3D source domain, a 2D target domain and a 3D target domain, and the discriminator includes at least one of a first discriminator, a second discriminator, a third discriminator and a fourth discriminator, wherein the first discriminator is used to determine whether the semantic segmentation result belongs to a 2D source domain or a 2D target domain, the second discriminator is used to determine whether the semantic segmentation result belongs to a 3D source domain or a 3D target domain, the third discriminator is used to determine whether the semantic segmentation result belongs to a 2D source domain or a 3D target domain, and the fourth discriminator is used to determine whether the semantic segmentation result belongs to a 3D source domain or a 2D target domain;

[0032] A training module is used to perform adversarial training on the semantic segmentation network model as a generator and multiple discriminators according to the semantic segmentation results and classification results, so as to obtain multiple semantic segmentation network models to be tested;

[0033] A third input module is used to input a test data set into multiple semantic segmentation network models to be tested to obtain test segmentation results of each of the multiple semantic segmentation network models to be tested, wherein the test data set includes a target domain 2D image, a target domain 3D point cloud data, a pixel-level label of the target domain 2D image, and a point-by-point label of the target domain 3D point cloud data;

[0034] The determination module is used to determine a target semantic segmentation network model among multiple semantic segmentation network models to be tested according to their respective test segmentation results.

[0035] In an embodiment of the present invention, the semantic segmentation network model includes a 2D semantic segmentation branch and a 3D semantic segmentation branch, the 2D semantic segmentation branch includes a U-Net network, a feature extraction network and a true 2D segmentation head, and the 3D semantic segmentation branch includes a spatial sparse convolutional neural network and a true 3D segmentation head;

[0036] The first input module includes:

[0037] A first input unit is used to input the 2D images in the first training data set and the second training data set into the U-Net network to obtain 2D images in dense pixel form, where the 2D images include source domain 2D images and target domain 2D images;

[0038] A second input unit is used to input the 2D image in dense pixel form into the feature extraction network to obtain a 2D feature map;

[0039] The third input unit is used to input the 2D feature map into the real 2D segmentation head to obtain the semantic segmentation result of the 2D image;

[0040] The first input module also includes:

[0041] a fourth input unit, configured to input the 3D point cloud data in the first training data set and the second training data set into the spatial sparse convolutional neural network to obtain 3D point cloud data in a sparse voxel form, wherein the 3D point cloud data includes source domain 3D point cloud data and target domain 3D point cloud data;

[0042] The fifth input unit is used to input the 3D point cloud data in the form of sparse voxels into the real 3D segmentation head to obtain the semantic segmentation result of the 3D point cloud data.

[0043] A third aspect of the present invention provides a processor configured to execute the above-mentioned method for determining a target semantic segmentation network model.

[0044] A fourth aspect of the present invention provides a machine-readable storage medium having instructions stored thereon, the instructions being used to enable a machine to execute the above-mentioned method for determining a target semantic segmentation network model.

[0045] In the embodiment of the present invention, the technical means of domain adaptation are adopted, and there is no need to collect a large number of labeled training data sets in the target domain (that is, there is no need for a large number of pixel-level labels and point-level labels corresponding to the target domain 2D images and the target domain 3D point cloud data). The existing labeled training data sets in the source domain are used (that is, the source domain 2D images, the source domain 3D point cloud data, the pixel-level labels of the source domain 2D images, and the point-by-point labels of the source domain 3D point cloud data), thereby avoiding the problem of insufficient training data sets and avoiding the manual re-labeling of a large number of training data sets, thereby reducing the burden of manual labor. In addition, the first training data set and the second training data set both include 2D images and 3D point cloud data, that is, they involve multimodal and cross-modal model training, which solves the problem of lack of depth information in the single modality during the domain adaptation process, and thus can improve the semantic segmentation effect of the target domain.

[0046] In an embodiment of the present invention, the first discriminator is used to determine whether the semantic segmentation result belongs to the 2D source domain or the 2D target domain, the second discriminator is used to determine whether the semantic segmentation result belongs to the 3D source domain or the 3D target domain, the third discriminator is used to determine whether the semantic segmentation result belongs to the 2D source domain or the 3D target domain, and the fourth discriminator is used to determine whether the semantic segmentation result belongs to the 3D source domain or the 2D target domain. One or more of the first discriminator, the second discriminator, the third discriminator and the fourth discriminator can be used, so that a total of 15 different types of discriminator forms are involved for selection, including both single-domain discriminators and cross-domain discriminators. The discriminator is used to determine the classification result (2D source domain, 3D source domain, 2D target domain or 3D target domain) to which the semantic segmentation result belongs, and the generator is used to generate the semantic segmentation result and increase the discriminant difficulty of the discriminator. The discriminator and the generator jointly improve their respective performance in the process of adversarial learning, and ultimately improve the accuracy and segmentation effect of the semantic segmentation of the target domain. When the discriminator has difficulty in distinguishing the classification result to which the semantic segmentation result belongs and the test effect of the semantic segmentation meets expectations, it means that the network model has been well adapted from the source domain to the target domain, and a network model that can be well applied in the target domain has been trained.

[0047] Specifically, the test data set is input into multiple (15) semantic segmentation network models to be tested, and the test segmentation results (15) of each of the multiple (15) semantic segmentation network models to be tested are obtained, wherein the test data set can be a small amount of labeled data of the target domain, that is, the test data set includes the target domain 2D image, the target domain 3D point cloud data, the pixel-level label of the target domain 2D image, and the point-by-point label of the target domain 3D point cloud data; the target semantic segmentation network model among the multiple semantic segmentation network models to be tested is determined according to the respective test segmentation results. In this way, the target semantic segmentation network model finally determined has the best accuracy and segmentation effect for the semantic segmentation of the target domain, and can also better meet user needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the embodiments of the present invention, but do not constitute a limitation on the embodiments of the present invention. In the accompanying drawings:

[0049] Figure 1 Schematically shows a flow chart of a method for determining a target semantic segmentation network model according to an embodiment of the present invention;

[0050] Figure 2 A network architecture diagram schematically shows a target semantic segmentation network model according to an embodiment of the present invention in which the discriminator is a first discriminator and a second discriminator. DETAILED DESCRIPTION

[0051] The specific implementation of the embodiment of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the embodiment of the present invention, and is not used to limit the embodiment of the present invention.

[0052] It should be noted that if the implementation methods of the present application involve directional indications (such as up, down, left, right, front, back...), such directional indications are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0053] In addition, if there are descriptions involving "first", "second", etc. in the implementation methods of this application, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various implementation methods can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0054] Figure 1 The flowchart of the method for determining the target semantic segmentation network model according to an embodiment of the present invention is schematically shown. Figure 1 As shown, in one embodiment of the present invention, a method for determining a target semantic segmentation network model is provided, comprising the following steps:

[0055] Step 101, inputting a first training data set and a second training data set into a semantic segmentation network model to obtain semantic segmentation results of the first training data set and the second training data set, wherein the first training data set includes a source domain 2D image, a source domain 3D point cloud data, a pixel-level label of the source domain 2D image, and a point-by-point label of the source domain 3D point cloud data, and the second training data set includes a target domain 2D image and a target domain 3D point cloud data;

[0056] Step 102, inputting the semantic segmentation results into a plurality of discriminators respectively, and obtaining a plurality of classification results of the semantic segmentation results, wherein the classification results include belonging to any one of a 2D source domain, a 3D source domain, a 2D target domain and a 3D target domain, and the discriminator includes at least one of a first discriminator, a second discriminator, a third discriminator and a fourth discriminator, wherein the first discriminator is used to determine whether the semantic segmentation result belongs to a 2D source domain or a 2D target domain, the second discriminator is used to determine whether the semantic segmentation result belongs to a 3D source domain or a 3D target domain, the third discriminator is used to determine whether the semantic segmentation result belongs to a 2D source domain or a 3D target domain, and the fourth discriminator is used to determine whether the semantic segmentation result belongs to a 3D source domain or a 2D target domain;

[0057] Step 103, according to the semantic segmentation results and the classification results, the semantic segmentation network model is used as a generator to perform adversarial training with multiple discriminators to obtain multiple semantic segmentation network models to be tested;

[0058] Step 104, inputting the test data set into multiple semantic segmentation network models to be tested, and obtaining test segmentation results of each of the multiple semantic segmentation network models to be tested, wherein the test data set includes a target domain 2D image, a target domain 3D point cloud data, a pixel-level label of the target domain 2D image, and a point-by-point label of the target domain 3D point cloud data;

[0059] Step 105 , determining a target semantic segmentation network model among multiple semantic segmentation network models to be tested according to their respective test segmentation results.

[0060] In step 101, the first training data set is a labeled data set in the source domain, namely, source domain 2D images, source domain 3D point cloud data, pixel-level labels of source domain 2D images, and point-by-point labels of source domain 3D point cloud data. The second training data set is an unlabeled data set in the target domain, namely, target domain 2D images and target domain 3D point cloud data. In the embodiment of the present invention, domain adaptation technology is used, and there is no need to collect a large number of labeled training data sets in the target domain (i.e., there is no need to collect a large number of pixel-level labels and point-by-point labels corresponding to target domain 2D images and target domain 3D point cloud data, respectively). The existing labeled training data sets in the source domain (i.e., source domain 2D images, source domain 3D point cloud data, pixel-level labels of source domain 2D images, and point-by-point labels of source domain 3D point cloud data) are used to avoid the problem of insufficient training data sets, avoid manually re-labeling a large number of training data sets, and thus reduce the burden of manual labor. In addition, the first training data set and the second training data set both include 2D images and 3D point cloud data, that is, they involve multimodal and cross-modal model training, which solves the problem of lack of depth information in the domain adaptation process of a single modality; thus, the semantic segmentation effect of the target domain can be improved.

[0061] It should be noted that, in one embodiment, the pixel-level label of the source domain 2D image can be understood as the real semantic result of artificially annotating the source domain 2D image; the point-by-point label of the source domain 3D point cloud data can be understood as the real semantic result of artificially annotating the source domain 3D point cloud data.

[0062] The number of multiple discriminators and multiple semantic segmentation network models to be tested is 15. Specifically, the multiple discriminators include: (1) the first discriminator; (2) the second discriminator; (3) the third discriminator; (4) the fourth discriminator; (5) the first discriminator and the second discriminator; (6) the first discriminator and the third discriminator; (7) the first discriminator and the fourth discriminator; (8) the second discriminator and the third discriminator; (9) the second discriminator and the fourth discriminator; (10) the third discriminator and the fourth discriminator; (11) the first discriminator, the second discriminator and the third discriminator; (12) the first discriminator, the third discriminator and the fourth discriminator; (13) the second discriminator, the third discriminator and the fourth discriminator; (14) the first discriminator, the second discriminator, the fourth discriminator; (15) the first discriminator, the second discriminator, the third discriminator and the fourth discriminator. The semantic segmentation network model is used as a generator to perform adversarial training with the 15 different forms of discriminators mentioned above, and 15 semantic segmentation network models to be tested will be obtained. That is, it can be understood that a group of different domain adaptation network models for semantic segmentation based on cross-modal network architecture are designed.

[0063] The first discriminator is used to determine whether the semantic segmentation result belongs to the 2D source domain or the 2D target domain, and the second discriminator is used to determine whether the semantic segmentation result belongs to the 3D source domain or the 3D target domain. Therefore, the first discriminator and the second discriminator are both single-domain discriminators, which can also be called single-modal discriminators, that is, the target domain and the source domain are from the same modality of data. The third discriminator is used to determine whether the semantic segmentation result belongs to the 2D source domain or the 3D target domain, and the fourth discriminator is used to determine whether the semantic segmentation result belongs to the 3D source domain or the 2D target domain. Therefore, the third discriminator and the fourth discriminator are both cross-domain discriminators, which can also be called cross-modal discriminators, that is, the target domain and the source domain are from data of different modalities.

[0064] The discriminator is used to distinguish the classification result (2D source domain, 3D source domain, 2D target domain or 3D target domain) to which the semantic segmentation result belongs. The generator is used to generate the semantic segmentation result and increase the difficulty of the discriminator. The discriminator and the generator jointly improve their respective performance in the process of adversarial learning, and ultimately improve the accuracy and segmentation effect of the semantic segmentation of the target domain. When the discriminator has difficulty distinguishing the classification result to which the semantic segmentation result belongs and the test effect of the semantic segmentation meets expectations, it means that the network model has been well adapted from the source domain to the target domain, and a network model that can be well applied in the target domain has been trained.

[0065] Specifically, the test data set is input into multiple (15) semantic segmentation network models to be tested, and the test segmentation results (15) of each of the multiple (15) semantic segmentation network models to be tested are obtained, wherein the test data set can be a small amount of labeled data of the target domain, that is, the test data set includes the target domain 2D image, the target domain 3D point cloud data, the pixel-level label of the target domain 2D image, and the point-by-point label of the target domain 3D point cloud data; the target semantic segmentation network model among the multiple semantic segmentation network models to be tested is determined according to the respective test segmentation results. In this way, the target semantic segmentation network model finally determined has the best accuracy and segmentation effect for the semantic segmentation of the target domain, and can also better meet user needs.

[0066] In the process of selecting the target semantic segmentation network model from the 15 semantic segmentation network models to be tested, the model with the best overall semantic segmentation effect for the target domain can be selected from the 15 semantic segmentation network models to be tested, or a personalized selection can be made according to user needs. For example, in the field of autonomous driving, the focus is on the semantic segmentation effect for people. Then, the model with the best semantic segmentation effect for people in the target domain can be selected from the 15 semantic segmentation network models to be tested and determined as the target semantic segmentation network model. At this time, the overall semantic segmentation effect can be appropriately discarded.

[0067] It should be noted that the target semantic segmentation network model can be used alone as a 2D semantic segmentation network model to perform semantic segmentation on 2D images only; it can also be used alone as a 3D semantic segmentation network model to perform semantic segmentation on 3D point cloud data only; it can also be used as a combination of a 2D semantic segmentation network model and a 3D semantic segmentation network model to perform semantic segmentation on both 2D images and 3D point cloud data.

[0068] Figure 2 The network architecture diagram in which the discriminator is the first discriminator and the second discriminator in the target semantic segmentation network model according to an embodiment of the present invention is schematically shown. The number of multiple discriminators and multiple semantic segmentation network models to be tested is 15. Now we take the discriminator as the first discriminator and the second discriminator for example (i.e. Figure 2 The discriminator used to distinguish between 2D source and 2D target and the discriminator used to distinguish between 3D source and 3D target) can be understood similarly for other types of discriminators and the semantic segmentation network models to be tested. Figure 2 , Figure 2 The upper part is the 2D semantic segmentation branch. Figure 2 The lower part of is the 3D semantic segmentation branch. Figure 2In the present invention, the 2D camera image is understood as the 2D image in the first training data set and the second training data set, and the 3D point cloud file is understood as the 3D point cloud data in the first training data set and the second training data set.

[0069] See also Figure 2 In one embodiment, the semantic segmentation network model includes a 2D semantic segmentation branch and a 3D semantic segmentation branch, the 2D semantic segmentation branch includes a U-Net network, a feature extraction network and a true 2D segmentation head, and the 3D semantic segmentation branch includes a spatial sparse convolutional neural network and a true 3D segmentation head;

[0070] See also Figure 2 , inputting the first training data set and the second training data set into the semantic segmentation network model, and obtaining the semantic segmentation results of the first training data set and the second training data set includes: inputting the 2D images in the first training data set and the second training data set into the U-Net network to obtain 2D images in dense pixel form, wherein the 2D images include source domain 2D images and target domain 2D images; inputting the 2D images in dense pixel form into the feature extraction network to obtain a 2D feature map; inputting the 2D feature map into the real 2D segmentation head to obtain the semantic segmentation result of the 2D image;

[0071] See also Figure 2 , inputting the first training data set and the second training data set into the semantic segmentation network model, and obtaining the semantic segmentation results of the first training data set and the second training data set also includes: inputting the 3D point cloud data in the first training data set and the second training data set into the spatial sparse convolutional neural network to obtain 3D point cloud data in sparse voxel form, the 3D point cloud data includes source domain 3D point cloud data and target domain 3D point cloud data; inputting the 3D point cloud data in sparse voxel form into the real 3D segmentation head to obtain the semantic segmentation result of the 3D point cloud data.

[0072] In one embodiment, the method further includes: projecting the 3D point cloud data to obtain a projection image; cropping and feature sampling the 2D feature map according to the size of the projection image to obtain a cropped 2D feature map;

[0073] Inputting the 2D feature map into a real 2D segmentation head to obtain a semantic segmentation result of the 2D image includes: inputting the cropped 2D feature map into the real 2D segmentation head to obtain a semantic segmentation result of the 2D image.

[0074] In one embodiment, the 2D semantic segmentation branch includes an imitating 3D segmentation head, the 3D semantic segmentation branch includes an imitating 2D segmentation head, and the method further includes: inputting the cropped 2D feature map into the imitating 3D segmentation head to obtain a 3D segmentation prediction result; inputting 3D point cloud data in the form of sparse voxels into the imitating 2D segmentation head to obtain a 2D segmentation prediction result; and retraining the imitating 3D segmentation head, the imitating 2D segmentation head, the real 3D segmentation head, and the real 2D segmentation head according to the 3D segmentation prediction result and the first segmentation loss of the real 3D segmentation result of the real 3D segmentation head, the 2D segmentation prediction result and the real 2D segmentation result of the real 2D segmentation head.

[0075] In order to enable the 2D semantic segmentation branch and the 3D semantic segmentation branch to assist each other in training and supplement dimensional information, it is necessary to allow the two branches to exchange information. The basis for this is that the two branches need to have data of the same form and size, and the two branches produce outputs of the same size for processing. To achieve this goal, in an embodiment of the present invention, the three-dimensional point cloud is projected to two dimensions, and then used to crop and sample features of the two-dimensional image so that the two-dimensional data and the three-dimensional data can correspond. After achieving this foundation, two imitation segmentation heads (i.e., imitation 3D segmentation head and imitation 2D segmentation head) and cross-modal loss are introduced to achieve information exchange between the two branches. The imitation segmentation head is a segmentation network in which one dimensional branch is used to imitate the segmentation effect of another dimensional branch. The cross-modal loss achieves the effect of improving the overall network by comparing the difference between the imitation segmentation head and the real segmentation head; this effectively utilizes information of different dimensions and makes the information of the two dimensions complement each other.

[0076] It can be understood that in order to allow the 2D semantic segmentation branch and the 3D semantic segmentation branch to exchange information and assist each other in training, a new imitation segmentation head is added to each branch. On the 2D semantic segmentation branch, the 3D segmentation head is imitated to give a 3D segmentation prediction result, that is, the segmentation effect of the 3D point cloud is predicted based on the 2D data. On the 3D semantic segmentation branch, the 2D segmentation head is imitated to give a 2D segmentation prediction result, that is, the segmentation effect of the 2D image is predicted based on the 3D data. Then, a newly added cross-modal loss (i.e. Figure 2 The first segmentation loss of the 3D segmentation prediction result and the real 3D segmentation result of the real 3D segmentation head, the second segmentation loss of the 2D segmentation prediction result and the real 2D segmentation result of the real 2D segmentation head are calculated, and then the performance of the two new imitation segmentation heads (i.e., the imitation 3D segmentation head and the imitation 2D segmentation head) is improved; and the first segmentation loss and the second segmentation loss generated are fed back to the segmentation network to improve the accurate segmentation ability of the two segmentation networks (i.e., the real 3D segmentation head and the real 2D segmentation head).

[0077] In one embodiment, the sizes of output data of the simulated 3D segmentation head and the real 3D segmentation head are the same, and the sizes of output data of the simulated 2D segmentation head and the real 2D segmentation head are the same.

[0078] In one embodiment, in the design of the discriminator, a simple design architecture of connecting multiple linear layers to a pooling layer can be selected. In practical applications, the corresponding discriminator can be designed according to differences in scene characteristics, actual application effects, etc. In order to maintain such flexibility, a direct and easily adjustable connection method is required to connect the newly added part of adversarial learning to the main network. Therefore, a relatively novel method is adopted, that is, the semantic segmentation result is processed into the self-information space using some mathematical methods, and the semantic segmentation result is directly sent to the discriminator, thus completing a relatively simple connection method and maintaining the training effect and overall performance of the discriminator. In one embodiment, the neural network is developed using the pytorch framework of python, and the model file formed is saved in the format of a checkpoint (.ckpt) file.

[0079] In the embodiment of the present invention, adversarial learning is used to enhance the domain adaptation effect. Specifically, the segmentation results generated by each segmentation head are directly used and input into the discriminator in the self-information space for adversarial learning. The training goal is to enable the segmentation network to more accurately integrate 2D and 3D extracted features for segmentation, so that the discriminator can distinguish data from the original domain and the target domain. With the assistance of adversarial learning, the main network can better adapt from the source domain to the target domain, and train a network model that can be well applied in the target domain.

[0080] In an embodiment of the present invention, in addition to adversarial learning, a cross-modal network architecture is also constructed so that multi-modal information can be jointly trained, thereby increasing data dimensions and supplementing data information, achieving effects that cannot be achieved by single-modal semantic segmentation. In the cross-modal network architecture, there are two independent branches: a 2D semantic segmentation branch and a 3D semantic segmentation branch, and these two branches can be trained separately. Multi-modal joint training further improves the effect of semantic segmentation in specified scenarios based on existing methods.

[0081] Taking the chemical industry as an example, semantic segmentation is an important part of computer vision and the basis for realizing a series of intelligent functions. For example, the application fields can be chemical automation, intelligent chemical production safety, and industrial Internet+. The system application process mainly includes three parts: (1) data collection and model training; (2) construction of intelligent terminals; and (3) realization of intelligent terminal functions.

[0082] (1) Regarding data collection and model training, data is collected through terminal devices to train the network and generate a semantic segmentation model suitable for the scenario where the terminal is located. Terminal device: a device with multi-camera and lidar (laser radar) collection equipment that can be integrated to achieve certain functions. For example, an intelligent robot that performs automatic cruising in a chemical production scenario, and an intelligent camera that detects whether employees are using safety equipment through intelligent analysis.

[0083] The terminal device uses a camera to collect 2D images, while the lidar device collects 3D point cloud data. The collected data information is transmitted to the computer for preprocessing and integration, and fed into the above-mentioned training network as input for model training. Through training, users can obtain a set of semantic segmentation models for the scene where the terminal is located. After the system is automatically trained, a model file is generated, and the user can choose different ways to embed it into the terminal device.

[0084] Data can be transmitted in two ways: 1) Communication; this method can transmit data to the host remotely and continuously, but the efficiency is relatively low, and it requires the construction of a communication module and consumes certain resources; 2) Mobile hard disk transmission; this method is difficult to achieve the function of "transmit anytime, anywhere", but the transmission speed is fast and does not occupy a large amount of computing resources. According to the actual needs of the system, in the data collection of the model training stage, since a large amount of data with high memory usage is required, the mobile hard disk transmission method is selected to achieve the purpose of quickly collecting sufficient data and completing training.

[0085] (2) Regarding the construction of smart terminals, users embed the trained network model back into the terminal device, so that the terminal device can call the semantic segmentation module at any time to achieve more complex functions. When the terminal device is embedded with various automation functions, it can be called a smart terminal. In addition, regarding the selection of model parameters, the model parameters are built into the system and do not require users to change them. This is more convenient and more suitable for everyone to use, increasing the system's usage scenarios and scope of application.

[0086] (3) Regarding the realization of intelligent terminal functions, when the intelligent terminal is built, some automated functions can be realized by calling the semantic segmentation module. For example, during the cruising process of an automatic cruising robot, the semantic segmentation module can be used to analyze the composition of the scene in front, and based on this, commands such as steering and speed change can be issued to control the cruising car to reach the specified location via the specified route; for another example, the system can be deployed on the construction site monitoring equipment, so that the equipment can perform intelligent analysis of the scene to monitor whether there are any violations of safety regulations, whether there are any facilities in dangerous conditions, etc.

[0087] The terminal device plays the role of the client during the model training process, and the model trainer plays the role of the server. During the function execution process, the terminal device is the server, and the user can use the client for real-time monitoring and control. The client can be a remote application or an application embedded in the terminal device.

[0088] Taking the automatic safety cruise robot as an example, the specific execution mode of the system when the function is executed is explained. When the automatic safety cruise robot is working, the robot collects data through the camera equipment and radar collection equipment on the body, and automatically performs fast preprocessing. The trained segmentation network performs segmentation based on the preprocessed data to obtain the semantic segmentation result. It is worth noting that if the terminal device does not have the conditions to embed a camera or radar device, a single-modal model can also be used for segmentation.

[0089] The semantic segmentation results of the model can be used by other modules in the terminal device. During the cruising process of the automatic safety cruise robot, the semantic segmentation results will first be fed into the object recognition module. The object recognition module identifies key elements in the cruising process, such as roads, markings, human bodies, tank areas, etc., based on the content of semantic segmentation. Then, the data is transmitted to the following two basic modules: (1) Automatic cruise module; This module calculates the robot control parameters, such as speed, offset direction and angle, based on the road conditions and the target cruise route; (2) Safety monitoring and alarm module; This module can be composed of various sub-modules. For example, the tank area information will be used by the tank area safety alarm module to monitor whether there are abnormal phenomena and violations in the tank area; the alarm content can be fed back to the remote client, and the alarm is sent in real time and the information is stored.

[0090] In the embodiment of the present invention, the performance can be continuously and automatically improved. When the system performs model training, it is not necessary to annotate the second data set of the target domain. Therefore, as the data collected by the terminal device increases, the system can train and update the model to automatically improve the accuracy of the model. This function can be implemented in a scheduled manner, for example, setting the system to automatically update once every six months, continuously enhancing the semantic segmentation module's targeted segmentation performance for specific scenarios and improving the accuracy of the network model.

[0091] The system takes model training and model use as a whole, and endows chemical scene intelligent equipment with automatic model training capabilities, accurate semantic segmentation capabilities, multi-module coordination and linkage capabilities, and continuous maintenance and improvement capabilities, which is very practical. During the implementation process, the terminal equipment needs to be equipped with hardware facilities such as lidar collection equipment and modules, multi-camera collection equipment and modules, and synchronous command issuance modules.

[0092] In an embodiment of the present invention, (1) a domain adaptation algorithm is used to reduce the impact of insufficient annotated data sets for chemical scenes on semantic segmentation training. (2) Based on the 2D images of chemical scenes, 3D point cloud data collected from the scenes are used for joint training to enhance the accuracy, reliability, and robustness of semantic segmentation of chemical scenes. (3) The huge differences in 3D point cloud data caused by different collection device settings are reduced to improve the performance of the semantic segmentation model on different data sets. (4) The trained network model is embedded in the device to build a practical and scalable intelligent system based on semantic segmentation.

[0093] In the embodiments of the present invention, (1) when encountering the problem of lack of data sets, previous methods often use manual annotation to construct a large number of data sets to avoid insufficient training, while the present invention uses domain adaptation to better solve the problem of lack of data sets in chemical scenes; (2) a variety of means are used in combination for domain adaptation, thereby improving the accuracy of semantic segmentation and the effect of domain adaptation: when performing semantic segmentation of chemical scenes, not only 2D images of chemical production scenes are used, but also 3D point cloud data collected by lidar or radar for chemical scenes are used, and a novel cross-modal loss is introduced to enable 2D and 3D semantic segmentation branches to generate information exchange; a cross-modal network architecture is used to enable 2D and 3D semantic segmentation branches Collaborative training is used to improve the semantic segmentation effects of the 2D and 3D semantic segmentation branches, and the information of the two is integrated to make better use of depth information in the semantic segmentation of chemical scenes and produce better segmentation effects; adversarial learning is introduced to assist training, which improves the effect of domain adaptation and thus improves the final semantic segmentation effect; ③ Using the camera and lidar collection equipment on the intelligent robot or other intelligent terminal devices to collect data in real time, and then using the model trained by the innovative algorithm in the present invention to perform semantic segmentation, and using the feedback semantic segmentation results as a guide to realize various functions of the terminal device, such as automatic cruising, helmet recognition, and seat belt detection, and corresponding solutions can be taken to form a practical system. (4) When training the model, instead of using a single network for one-time training, a group of novel networks designed by themselves are trained, and a series of algorithm networks are designed. The optimal training model is taken as the final target semantic segmentation network model to improve the overall performance.

[0094] An embodiment of the present invention provides a device for determining a target semantic segmentation network model, the device comprising:

[0095] A first input module is used to input the first training data set and the second training data set into the semantic segmentation network model to obtain semantic segmentation results of the first training data set and the second training data set, wherein the first training data set includes a source domain 2D image, a source domain 3D point cloud data, a pixel-level label of the source domain 2D image, and a point-by-point label of the source domain 3D point cloud data, and the second training data set includes a target domain 2D image and a target domain 3D point cloud data;

[0096] a second input module, for inputting the semantic segmentation results into a plurality of discriminators respectively, to obtain a plurality of classification results of the semantic segmentation results, wherein the classification results include any one of a 2D source domain, a 3D source domain, a 2D target domain and a 3D target domain, and the discriminator includes at least one of a first discriminator, a second discriminator, a third discriminator and a fourth discriminator, wherein the first discriminator is used to determine whether the semantic segmentation result belongs to a 2D source domain or a 2D target domain, the second discriminator is used to determine whether the semantic segmentation result belongs to a 3D source domain or a 3D target domain, the third discriminator is used to determine whether the semantic segmentation result belongs to a 2D source domain or a 3D target domain, and the fourth discriminator is used to determine whether the semantic segmentation result belongs to a 3D source domain or a 2D target domain;

[0097] A training module is used to perform adversarial training on the semantic segmentation network model as a generator and multiple discriminators according to the semantic segmentation results and classification results, so as to obtain multiple semantic segmentation network models to be tested;

[0098] A third input module is used to input a test data set into multiple semantic segmentation network models to be tested to obtain test segmentation results of each of the multiple semantic segmentation network models to be tested, wherein the test data set includes a target domain 2D image, a target domain 3D point cloud data, a pixel-level label of the target domain 2D image, and a point-by-point label of the target domain 3D point cloud data;

[0099] The determination module is used to determine a target semantic segmentation network model among multiple semantic segmentation network models to be tested according to their respective test segmentation results.

[0100] In an embodiment of the present invention, the semantic segmentation network model includes a 2D semantic segmentation branch and a 3D semantic segmentation branch, the 2D semantic segmentation branch includes a U-Net network, a feature extraction network and a true 2D segmentation head, and the 3D semantic segmentation branch includes a spatial sparse convolutional neural network and a true 3D segmentation head;

[0101] The first input module includes:

[0102] A first input unit is used to input the 2D images in the first training data set and the second training data set into the U-Net network to obtain 2D images in dense pixel form, where the 2D images include source domain 2D images and target domain 2D images;

[0103] A second input unit is used to input the 2D image in dense pixel form into the feature extraction network to obtain a 2D feature map;

[0104] The third input unit is used to input the 2D feature map into the real 2D segmentation head to obtain the semantic segmentation result of the 2D image;

[0105] The first input module also includes:

[0106] a fourth input unit, configured to input the 3D point cloud data in the first training data set and the second training data set into the spatial sparse convolutional neural network to obtain 3D point cloud data in a sparse voxel form, wherein the 3D point cloud data includes source domain 3D point cloud data and target domain 3D point cloud data;

[0107] The fifth input unit is used to input the 3D point cloud data in the form of sparse voxels into the real 3D segmentation head to obtain the semantic segmentation result of the 3D point cloud data.

[0108] An embodiment of the present invention provides a processor configured to execute any one of the methods for determining a target semantic segmentation network model in the above embodiments.

[0109] Specifically, the processor may be configured to:

[0110] Inputting a first training data set and a second training data set into a semantic segmentation network model to obtain semantic segmentation results of the first training data set and the second training data set, wherein the first training data set includes a source domain 2D image, a source domain 3D point cloud data, a pixel-level label of the source domain 2D image, and a point-by-point label of the source domain 3D point cloud data, and the second training data set includes a target domain 2D image and a target domain 3D point cloud data;

[0111] Inputting the semantic segmentation results into a plurality of discriminators respectively, and obtaining a plurality of classification results of the semantic segmentation results, wherein the classification results include any one of a 2D source domain, a 3D source domain, a 2D target domain, and a 3D target domain, and the discriminator includes at least one of a first discriminator, a second discriminator, a third discriminator, and a fourth discriminator, wherein the first discriminator is used to determine whether the semantic segmentation result belongs to the 2D source domain or the 2D target domain, the second discriminator is used to determine whether the semantic segmentation result belongs to the 3D source domain or the 3D target domain, the third discriminator is used to determine whether the semantic segmentation result belongs to the 2D source domain or the 3D target domain, and the fourth discriminator is used to determine whether the semantic segmentation result belongs to the 3D source domain or the 2D target domain;

[0112] According to the semantic segmentation results and classification results, the semantic segmentation network model is used as a generator to perform adversarial training with multiple discriminators to obtain multiple semantic segmentation network models to be tested;

[0113] Inputting a test data set into multiple semantic segmentation network models to be tested to obtain test segmentation results of each of the multiple semantic segmentation network models to be tested, wherein the test data set includes a target domain 2D image, a target domain 3D point cloud data, a pixel-level label of the target domain 2D image, and a point-by-point label of the target domain 3D point cloud data;

[0114] A target semantic segmentation network model among multiple semantic segmentation network models to be tested is determined according to the respective test segmentation results.

[0115] In an embodiment of the present invention, the semantic segmentation network model includes a 2D semantic segmentation branch and a 3D semantic segmentation branch, the 2D semantic segmentation branch includes a U-Net network, a feature extraction network and a true 2D segmentation head, and the 3D semantic segmentation branch includes a spatial sparse convolutional neural network and a true 3D segmentation head;

[0116] The processor is configured to:

[0117] Inputting the first training data set and the second training data set into the semantic segmentation network model, obtaining the semantic segmentation results of the first training data set and the second training data set includes:

[0118] Inputting the 2D images in the first training data set and the second training data set into the U-Net network to obtain 2D images in dense pixel form, where the 2D images include source domain 2D images and target domain 2D images;

[0119] Input the 2D image in dense pixel form into the feature extraction network to obtain a 2D feature map;

[0120] Input the 2D feature map into the real 2D segmentation head to obtain the semantic segmentation result of the 2D image;

[0121] The processor is configured to:

[0122] Inputting the first training data set and the second training data set into the semantic segmentation network model to obtain the semantic segmentation results of the first training data set and the second training data set also includes:

[0123] Inputting the 3D point cloud data in the first training data set and the second training data set into a spatial sparse convolutional neural network to obtain 3D point cloud data in sparse voxel form, wherein the 3D point cloud data includes source domain 3D point cloud data and target domain 3D point cloud data;

[0124] The 3D point cloud data in sparse voxel form is input into the real 3D segmentation head to obtain the semantic segmentation result of the 3D point cloud data.

[0125] In this embodiment of the present invention, the processor is further configured to:

[0126] Project the 3D point cloud data to obtain a projection image;

[0127] The 2D feature map is cropped and feature sampled according to the size of the projected image to obtain a cropped 2D feature map;

[0128] The 2D feature map is input into the real 2D segmentation head, and the semantic segmentation results of the 2D image are obtained, including:

[0129] The cropped 2D feature map is input into the real 2D segmentation head to obtain the semantic segmentation result of the 2D image.

[0130] In an embodiment of the present invention, the 2D semantic segmentation branch includes imitating a 3D segmentation head, the 3D semantic segmentation branch includes imitating a 2D segmentation head, and the processor is further configured to:

[0131] The cropped 2D feature map is input into the imitation 3D segmentation head to obtain the 3D segmentation prediction result;

[0132] Input the 3D point cloud data in the form of sparse voxels into the simulated 2D segmentation head to obtain the 2D segmentation prediction result;

[0133] The imitated 3D segmentation head, the imitated 2D segmentation head, the real 3D segmentation head and the real 2D segmentation head are retrained according to the first segmentation loss of the 3D segmentation prediction result and the real 3D segmentation result of the real 3D segmentation head, and the second segmentation loss of the 2D segmentation prediction result and the real 2D segmentation result of the real 2D segmentation head.

[0134] In an embodiment of the present invention, the processor is configured to:

[0135] The sizes of output data of the simulated 3D segmentation head and the real 3D segmentation head are the same, and the sizes of output data of the simulated 2D segmentation head and the real 2D segmentation head are the same.

[0136] In an embodiment of the present invention, the processor is configured to:

[0137] The number of multiple discriminators and multiple semantic segmentation network models to be tested is 15 each.

[0138] An embodiment of the present invention provides a machine-readable storage medium having instructions stored thereon, the instructions being used to enable a machine to execute any one of the methods for determining a target semantic segmentation network model in the above-mentioned embodiments.

[0139] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0140] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0141] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0142] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0143] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0144] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0145] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0146] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0147] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A method for determining a target semantic segmentation network model, characterized in that: The method comprises: Inputting a first training data set and a second training data set into a semantic segmentation network model to obtain semantic segmentation results of the first training data set and the second training data set, wherein the first training data set includes a source domain 2D image, a source domain 3D point cloud data, a pixel-level label of the source domain 2D image, and a point-by-point label of the source domain 3D point cloud data, and the second training data set includes a target domain 2D image and a target domain 3D point cloud data; Inputting the semantic segmentation results into a plurality of discriminators respectively to obtain a plurality of classification results of the semantic segmentation results, wherein the classification results include any one of a 2D source domain, a 3D source domain, a 2D target domain and a 3D target domain, and the discriminator includes at least one of a first discriminator, a second discriminator, a third discriminator and a fourth discriminator, the first discriminator is used to determine whether the semantic segmentation result belongs to a 2D source domain or a 2D target domain, the second discriminator is used to determine whether the semantic segmentation result belongs to a 3D source domain or a 3D target domain, the third discriminator is used to determine whether the semantic segmentation result belongs to a 2D source domain or a 3D target domain, and the fourth discriminator is used to determine whether the semantic segmentation result belongs to a 3D source domain or a 2D target domain; According to the semantic segmentation result and the classification result, the semantic segmentation network model is used as a generator to perform adversarial training with the multiple discriminators respectively to obtain multiple semantic segmentation network models to be tested; Inputting the test data sets into the multiple semantic segmentation network models to be tested respectively, and obtaining the test segmentation results of the multiple semantic segmentation network models to be tested respectively, wherein the test data sets include the target domain 2D image, the target domain 3D point cloud data, the pixel-level label of the target domain 2D image, and the point-by-point label of the target domain 3D point cloud data; A target semantic segmentation network model among the multiple semantic segmentation network models to be tested is determined according to the respective test segmentation results.

2. The method according to claim 1, characterized in that: The semantic segmentation network model includes a 2D semantic segmentation branch and a 3D semantic segmentation branch, the 2D semantic segmentation branch includes a U-Net network, a feature extraction network and a real 2D segmentation head, and the 3D semantic segmentation branch includes a spatial sparse convolutional neural network and a real 3D segmentation head; The step of inputting the first training data set and the second training data set into the semantic segmentation network model to obtain the semantic segmentation results of the first training data set and the second training data set includes: Inputting the 2D images in the first training data set and the second training data set into the U-Net network to obtain 2D images in dense pixel form, wherein the 2D images include source domain 2D images and target domain 2D images; Inputting the 2D image in dense pixel form into the feature extraction network to obtain a 2D feature map; Inputting the 2D feature map into the real 2D segmentation head to obtain a semantic segmentation result of the 2D image; The step of inputting the first training data set and the second training data set into the semantic segmentation network model to obtain semantic segmentation results of the first training data set and the second training data set further comprises: Inputting the 3D point cloud data in the first training data set and the second training data set into the spatial sparse convolutional neural network to obtain 3D point cloud data in sparse voxel form, wherein the 3D point cloud data includes source domain 3D point cloud data and target domain 3D point cloud data; The 3D point cloud data in sparse voxel form is input into the real 3D segmentation head to obtain a semantic segmentation result of the 3D point cloud data.

3. The method according to claim 2, characterized in that The method further comprises: Projecting the 3D point cloud data to obtain a projection image; Cropping and feature sampling the 2D feature map according to the size of the projection image to obtain a cropped 2D feature map; Inputting the 2D feature map into the real 2D segmentation head to obtain the semantic segmentation result of the 2D image includes: The cropped 2D feature map is input into the real 2D segmentation head to obtain a semantic segmentation result of the 2D image.

4. The method according to claim 3, characterized in that The 2D semantic segmentation branch includes imitating a 3D segmentation head, the 3D semantic segmentation branch includes imitating a 2D segmentation head, and the method further includes: Inputting the cropped 2D feature map into the imitation 3D segmentation head to obtain a 3D segmentation prediction result; Inputting the 3D point cloud data in the form of sparse voxels into the simulated 2D segmentation head to obtain a 2D segmentation prediction result; The simulated 3D segmentation head, the simulated 2D segmentation head, the real 3D segmentation head and the real 2D segmentation head are retrained according to the first segmentation loss of the 3D segmentation prediction result and the real 3D segmentation result of the real 3D segmentation head, and the second segmentation loss of the 2D segmentation prediction result and the real 2D segmentation result of the real 2D segmentation head.

5. The method according to claim 4, characterized in that The sizes of output data of the simulated 3D segmentation head and the real 3D segmentation head are the same, and the sizes of output data of the simulated 2D segmentation head and the real 2D segmentation head are the same.

6. The method according to claim 1, characterized in that The number of the multiple discriminators and the number of the multiple semantic segmentation network models to be tested are both 15.

7. A device for determining a target semantic segmentation network model, characterized in that: The device comprises: A first input module is used to input a first training data set and a second training data set into a semantic segmentation network model to obtain semantic segmentation results of the first training data set and the second training data set, wherein the first training data set includes a source domain 2D image, a source domain 3D point cloud data, a pixel-level label of the source domain 2D image, and a point-by-point label of the source domain 3D point cloud data, and the second training data set includes a target domain 2D image and a target domain 3D point cloud data; A second input module is used to input the semantic segmentation results into multiple discriminators respectively to obtain multiple classification results of the semantic segmentation results, wherein the classification results include belonging to any one of a 2D source domain, a 3D source domain, a 2D target domain and a 3D target domain, and the discriminator includes at least one of a first discriminator, a second discriminator, a third discriminator and a fourth discriminator, the first discriminator is used to determine whether the semantic segmentation result belongs to a 2D source domain or a 2D target domain, the second discriminator is used to determine whether the semantic segmentation result belongs to a 3D source domain or a 3D target domain, the third discriminator is used to determine whether the semantic segmentation result belongs to a 2D source domain or a 3D target domain, and the fourth discriminator is used to determine whether the semantic segmentation result belongs to a 3D source domain or a 2D target domain; A training module, used for performing adversarial training on the semantic segmentation network model as a generator and the multiple discriminators according to the semantic segmentation result and the classification result, so as to obtain multiple semantic segmentation network models to be tested; A third input module is used to input the test data set into the multiple semantic segmentation network models to be tested respectively, and obtain the test segmentation results of each of the multiple semantic segmentation network models to be tested, wherein the test data set includes a target domain 2D image, a target domain 3D point cloud data, a pixel-level label of the target domain 2D image, and a point-by-point label of the target domain 3D point cloud data; A determination module is used to determine a target semantic segmentation network model among the multiple semantic segmentation network models to be tested according to the respective test segmentation results.

8. The device according to claim 7, characterized in that The semantic segmentation network model includes a 2D semantic segmentation branch and a 3D semantic segmentation branch, the 2D semantic segmentation branch includes a U-Net network, a feature extraction network and a real 2D segmentation head, and the 3D semantic segmentation branch includes a spatial sparse convolutional neural network and a real 3D segmentation head; The first input module comprises: A first input unit is used to input the 2D images in the first training data set and the second training data set into the U-Net network to obtain 2D images in dense pixel form, wherein the 2D images include source domain 2D images and target domain 2D images; A second input unit is used to input the 2D image in dense pixel form into the feature extraction network to obtain a 2D feature map; A third input unit is used to input the 2D feature map into the real 2D segmentation head to obtain a semantic segmentation result of the 2D image; The first input module also includes: a fourth input unit, configured to input the 3D point cloud data in the first training data set and the second training data set into the spatial sparse convolutional neural network to obtain 3D point cloud data in sparse voxel form, wherein the 3D point cloud data includes source domain 3D point cloud data and target domain 3D point cloud data; The fifth input unit is used to input the 3D point cloud data in the form of sparse voxels into the real 3D segmentation head to obtain a semantic segmentation result of the 3D point cloud data.

9. A processor, characterized in that: The method is configured to execute the method for determining a target semantic segmentation network model according to any one of claims 1 to 6.

10. A machine-readable storage medium having instructions stored thereon, characterized in that: The instruction is used to enable the machine to execute the method for determining a target semantic segmentation network model according to any one of claims 1 to 6.

Images