Machine learning model training method and device, and semantic segmentation method and device

Abstract

The present disclosure provides a machine learning model training method and device, a semantic segmentation method and device, and relates to the field of artificial intelligence. The machine learning model training method comprises: alternately training a first machine learning model by using source domain images and target domain images; alternately training a second machine learning model by using source domain three-dimensional data and target domain three-dimensional data; training the first machine learning model by using the source domain three-dimensional data, and training the second machine learning model by using the source domain images; and training the first machine learning model by using the target domain three-dimensional data, and training the second machine learning model by using the target domain images. The present disclosure effectively improves the robustness of the machine learning model in different environments by fusing the characteristics of different modalities.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of artificial intelligence, and in particular, to a machine learning model training method and device, and a semantic segmentation method and device. BACKGROUND

[0002] At present, three-dimensional semantic segmentation has been widely applied in scenarios such as robots, autonomous driving and virtual reality. In order to improve the semantic segmentation effect, two-dimensional images and three-dimensional point cloud data are usually used to train a semantic segmentation model. SUMMARY

[0003] The inventors have noticed that in the training process of the related art, the semantic segmentation model is trained by using two-dimensional images and three-dimensional point cloud data respectively, so that the trained semantic segmentation model cannot fuse the characteristics of different modalities, thereby leading to the under-robustness of the semantic segmentation model in different environments.

[0004] Accordingly, the present disclosure provides a machine learning model training scheme, which effectively improves the robustness of the machine learning model in different environments by fusing the characteristics of different modalities.

[0005] According to a first aspect of an embodiment of the present disclosure, a machine learning model training method is provided, comprising: training a first machine learning model by alternately using source domain images and target domain images; training a second machine learning model by alternately using source domain three-dimensional data and target domain three-dimensional data; training the first machine learning model by using the source domain three-dimensional data, and training the second machine learning model by using the source domain images; training the first machine learning model by using the target domain three-dimensional data, and training the second machine learning model by using the target domain images.

[0006] In some embodiments, the training of the first machine learning model by using the source domain three-dimensional data comprises: obtaining first depth information and first height information of each pixel point in the source domain image by using the source domain three-dimensional data; converting a first feature vector corresponding to the source domain image output by the first machine learning model from a pixel feature space to a geometric space to obtain second depth information and second height information of each pixel point; determining a first loss function by using the first depth information, the first height information, the second depth information and the second height information; and training the first machine learning model by using the first loss function.

[0007] In some embodiments, training the second machine learning model using the source domain image includes: obtaining a first gray value of each pixel in the source domain image and a first gray-level transformation amount of the local neighborhood of each pixel; predicting a second gray value of each pixel and a second gray-level transformation amount of the local neighborhood of each pixel based on a second feature vector output by the second machine learning model corresponding to the three-dimensional data of the source domain; determining a second loss function using the first gray value, the first gray-level transformation amount, the second gray value, and the second gray-level transformation amount; and training the second machine learning model using the second loss function.

[0008] In some embodiments, training the first machine learning model using the target domain 3D data includes: using the target domain 3D data to obtain third depth information and third height information of each pixel in the target domain image; converting the third feature vector output by the first machine learning model corresponding to the target domain image from the pixel feature space to the geometric space to obtain fourth depth information and fourth height information of each pixel; determining a third loss function using the third depth information, the third height information, the fourth depth information, and the fourth height information; and training the first machine learning model using the third loss function.

[0009] In some embodiments, training the second machine learning model using the target domain image includes: obtaining the third gray value of each pixel in the target domain image and the third gray-level transformation amount of the local neighborhood of each pixel; predicting the fourth gray value of each pixel and the fourth gray-level transformation amount of the local neighborhood of each pixel based on the fourth feature vector corresponding to the three-dimensional data of the target domain output by the second machine learning model; determining a fourth loss function using the third gray value, the third gray-level transformation amount, the fourth gray value, and the fourth gray-level transformation amount; and training the second machine learning model using the fourth loss function.

[0010] In some embodiments, training a first machine learning model using a source domain image includes: inputting the source domain image into the first machine learning model to generate a first feature vector; performing classification processing on the first feature vector to generate a first semantic segmentation prediction result; determining a fifth loss function value using the labels included in the source domain image and the first semantic segmentation prediction result; and training the first machine learning model using the fifth loss function value.

[0011] In some embodiments, training a second machine learning model using source domain 3D data includes: inputting the source domain 3D data into the second machine learning model to generate a second feature vector; classifying the second feature vector to generate a second semantic segmentation prediction result; determining a sixth loss function value using the labels included in the source domain 3D data and the second semantic segmentation prediction result; and training the second machine learning model using the sixth loss function value.

[0012] In some embodiments, a first joint loss function value is determined using the first semantic segmentation prediction result and the second semantic segmentation prediction result; the first machine learning model and the second machine learning model are trained using the first joint loss function value.

[0013] In some embodiments, the first joint loss function value is the relative entropy of the first semantic segmentation prediction result and the second semantic segmentation prediction result.

[0014] In some embodiments, training a first machine learning model using a target domain image includes: inputting the target domain image into the first machine learning model to generate a third feature vector; classifying the third feature vector to generate a third semantic segmentation prediction result; determining a seventh loss function value using a first pseudo-label and the third semantic segmentation prediction result; and training the first machine learning model using the seventh loss function value.

[0015] In some embodiments, a first pseudo-label is assigned to the target domain image based on the feature similarity of each pixel in the source domain image and the target domain image.

[0016] In some embodiments, training the second machine learning model using the target domain 3D data includes: inputting the target domain 3D data into the second machine learning model to generate a fourth feature vector; classifying the fourth feature vector to generate a fourth semantic segmentation prediction result; determining an eighth loss function value using the second pseudo-label and the fourth semantic segmentation prediction result; and training the second machine learning model using the eighth loss function value.

[0017] In some embodiments, a second pseudo-label is assigned to the target domain 3D data based on the feature similarity of each data point in the source domain 3D data and the target domain 3D data.

[0018] In some embodiments, the second joint loss function value is determined using the third semantic segmentation prediction result and the fourth semantic segmentation prediction result; the first machine learning model and the second machine learning model are trained using the second joint loss function value, respectively.

[0019] In some embodiments, the second joint loss function value is the relative entropy of the third semantic segmentation prediction result and the fourth semantic segmentation prediction result.

[0020] In some embodiments, a first modality category prototype is determined using a first feature vector output by the first machine learning model that corresponds to the source domain image; a second modality category prototype is determined using a third feature vector output by the first machine learning model that is associated with the target domain image; a ninth loss function value is determined based on the consistency between the first modality category prototype and the second modality category prototype; and the first machine learning model is trained using the ninth loss function value.

[0021] In some embodiments, a third modality category prototype is determined using a second feature vector output by the second machine learning model that corresponds to the source domain 3D data; a fourth modality category prototype is determined using a fourth feature vector output by the second machine learning model that corresponds to the target domain 3D data; a tenth loss function value is determined based on the consistency between the third modality category prototype and the fourth modality category prototype; and the second machine learning model is trained using the tenth loss function value.

[0022] According to a second aspect of the present disclosure, a machine learning model training apparatus is provided, comprising: a first training processing module configured to alternately train a first machine learning model using source domain images and target domain images, and alternately train a second machine learning model using source domain 3D data and target domain 3D data; and a second training processing module configured to train the first machine learning model using the source domain 3D data, and train the second machine learning model using the source domain images, and train the first machine learning model using the target domain 3D data, and train the second machine learning model using the target domain images.

[0023] According to a third aspect of the present disclosure, a machine learning model training apparatus is provided, comprising: a memory configured to store instructions; and a processor coupled to the memory, the processor being configured to execute instructions stored in the memory to implement the machine learning model training method as described in any of the foregoing embodiments.

[0024] According to a fourth aspect of the present disclosure, a semantic segmentation method is provided, comprising: inputting a target domain image into a first machine learning model so that the first machine learning model outputs a first semantic feature, wherein the first machine learning model is trained using the training method described in any of the above embodiments; performing classification processing on the first semantic feature to generate a first semantic segmentation result; inputting three-dimensional data of the target domain into a second machine learning model so that the second machine learning model outputs a second semantic feature, wherein the second machine learning model is trained using the training method described in any of the above embodiments; and performing classification processing on the second semantic feature to generate a second semantic segmentation result.

[0025] According to a fifth aspect of the present disclosure, a semantic segmentation apparatus is provided, comprising: a first segmentation processing module configured to input a target domain image into a first machine learning model, such that the first machine learning model outputs a first feature vector, wherein the first machine learning model is trained using the training method described in any of the above embodiments, and the first feature vector is classified to generate a first semantic segmentation result; and a second segmentation processing module configured to input target domain three-dimensional data into a second machine learning model, such that the second machine learning model outputs a second feature vector, wherein the second machine learning model is trained using the training method described in any of the above embodiments, and the second feature vector is classified to generate a second semantic segmentation result.

[0026] According to a sixth aspect of the present disclosure, a semantic segmentation apparatus is provided, comprising: a memory configured to store instructions; and a processor coupled to the memory, the processor being configured to execute instructions stored in the memory to implement the semantic segmentation method described in any of the above embodiments.

[0027] According to a seventh aspect of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method as described in any of the above embodiments.

[0028] Other features and advantages of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1This is a flowchart illustrating a machine learning model training method according to an embodiment of the present disclosure;

[0031] Figure 2 This is a flowchart illustrating a machine learning model training method according to another embodiment of the present disclosure;

[0032] Figure 3 This is a flowchart illustrating a machine learning model training method according to yet another embodiment of this disclosure.

[0033] Figure 4 This is a flowchart illustrating a machine learning model training method according to yet another embodiment of this disclosure.

[0034] Figure 5 This is a flowchart illustrating a machine learning model training method according to yet another embodiment of this disclosure.

[0035] Figure 6 This is a schematic diagram of a machine learning model training framework according to an embodiment of the present disclosure;

[0036] Figure 7 This is a flowchart illustrating a machine learning model training method according to yet another embodiment of this disclosure.

[0037] Figure 8 This is a schematic diagram of a machine learning model training framework according to another embodiment of the present disclosure;

[0038] Figure 9 This is a schematic diagram of the structure of a machine learning model training apparatus according to an embodiment of the present disclosure;

[0039] Figure 10 This is a schematic diagram of the structure of a machine learning model training apparatus according to another embodiment of the present disclosure;

[0040] Figure 11 This is a schematic diagram of the structure of a machine learning model training apparatus according to yet another embodiment of the present disclosure;

[0041] Figure 12 This is a flowchart illustrating a semantic segmentation method according to an embodiment of the present disclosure;

[0042] Figure 13 This is a schematic diagram of the structure of a semantic segmentation apparatus according to an embodiment of the present disclosure;

[0043] Figure 14 This is a schematic diagram of the structure of a semantic segmentation apparatus according to another embodiment of the present disclosure. Detailed Implementation

[0044] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0045] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0046] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0047] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0048] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0049] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0050] Figure 1 This is a schematic flowchart illustrating a machine learning model training method according to an embodiment of the present disclosure. In some embodiments, the following machine learning model training method is performed by a machine learning model training apparatus.

[0051] In step 101, the first machine learning model is trained alternately using source domain images and target domain images.

[0052] In some embodiments, the first machine learning model is a fully convolutional network (U-net) model.

[0053] In some embodiments, during the training of a first machine learning model using a source domain image, the source domain image is input into the first machine learning model to generate a first feature vector. The first feature vector is then classified to generate a first semantic segmentation prediction result.

[0054] For example, a classifier is used to classify the first feature vector to generate a first semantic segmentation prediction result.

[0055] Next, the loss function value is determined using the labels included in the source domain image and the first semantic segmentation prediction result, and the determined loss function value is used to train the first machine learning model.

[0056] In some embodiments, during the training of a first machine learning model using a target domain image, the target domain image is input into the first machine learning model to generate a third feature vector. The third feature vector is then classified to generate a third semantic segmentation prediction result. A loss function value is determined using a first pseudo-label and the third semantic segmentation prediction result, and this loss function value is used to train the first machine learning model.

[0057] It should be noted that since the target domain image does not have a label, a pre-set first pseudo-label is required for training.

[0058] In some embodiments, a first pseudo-label is assigned to the target domain image based on the feature similarity of each pixel in the source domain image and the target domain image.

[0059] In step 102, the second machine learning model is trained alternately using source domain 3D data and target domain 3D data.

[0060] In some embodiments, the source domain 3D data is source domain 3D point cloud data, and the target domain 3D data is target domain 3D point cloud data.

[0061] In some embodiments, the second machine learning model is a fully convolutional network model.

[0062] In some embodiments, during the training of a second machine learning model using source domain 3D data, the source domain 3D data is input into the second machine learning model to generate a second feature vector. The second feature vector is then classified to generate a second semantic segmentation prediction result.

[0063] For example, a classifier is used to classify the second feature vector to generate a second semantic segmentation prediction result.

[0064] Next, the loss function value is determined using the labels included in the source domain 3D data and the second semantic segmentation prediction results, and the second machine learning model is trained using this loss function value.

[0065] In some embodiments, during the training of the second machine learning model using target domain 3D data, the target domain 3D data is input into the second machine learning model to generate a fourth feature vector. The fourth feature vector is then classified to generate a fourth semantic segmentation prediction result. A loss function value is determined using the second pseudo-label and the fourth semantic segmentation prediction result, and this loss function value is used to train the second machine learning model.

[0066] It should be noted that since the target domain 3D data does not have labels, a pre-set second pseudo-label is required for training.

[0067] In some embodiments, a second pseudo-label is assigned to the target domain 3D data based on the feature similarity of each data point in the source domain 3D data and the target domain 3D data.

[0068] In some embodiments, consistency constraints are applied to the semantic segmentation prediction results of the first machine learning model and the second machine learning model for the same domain (source domain or target domain) so that the first machine learning model and the second machine learning model can produce consistent prediction results for the same domain.

[0069] For example, a first machine learning model generates a first feature vector based on a source domain image. A classifier classifies the first feature vector to generate a first semantic segmentation prediction result. A second machine learning model generates a second feature vector based on 3D data from the source domain. A classifier classifies the second feature vector to generate a second semantic segmentation prediction result. The first and second semantic segmentation prediction results are used to determine a first joint loss function value. The first and second machine learning models are then trained using the first joint loss function value, respectively.

[0070] In some embodiments, the first joint loss function value is the relative entropy (KL divergence) between the first semantic segmentation prediction result and the second semantic segmentation prediction result.

[0071] For example, if the first semantic segmentation prediction result is The second semantic segmentation prediction result is: Then the first joint loss function value L xM (x) is shown in formula (1).

[0072]

[0073] Among them, D KL This is the relative entropy constraint function.

[0074] For example, the first machine learning model generates a third feature vector based on the target domain image. A classifier classifies this third feature vector to generate a third semantic segmentation prediction result. The second machine learning model generates a fourth feature vector based on the target domain 3D data. A classifier classifies this fourth feature vector to generate a fourth semantic segmentation prediction result. The values ​​of the third and fourth semantic segmentation prediction results are used to determine the value of the second joint loss function. The first and second machine learning models are then trained using the value of the second joint loss function.

[0075] In some embodiments, the second joint loss function value is the relative entropy (KL divergence) of the third semantic segmentation prediction result and the fourth semantic segmentation prediction result.

[0076] For example, the value of the second joint loss function can be calculated using the above formula (1).

[0077] In step 103, a first machine learning model is trained using source domain 3D data, and a second machine learning model is trained using source domain images to achieve bidirectional self-supervision.

[0078] In some embodiments, the steps of training a first machine learning model using source domain 3D data are as follows: Figure 2 As shown.

[0079] In step 201, the first depth information and the first height information of each pixel in the source domain image are obtained using the source domain 3D data.

[0080] Since there is a correspondence between the source domain image and the source domain 3D data, the first depth information and the first height information of each pixel in the source domain image can be obtained by using the 3D coordinates of each data point in the source domain 3D data.

[0081] For example, using the maximum-minimum normalization method, the 3D coordinates of each data point in the source domain 3D data are normalized to the range [0, 1]. Next, the first depth information and the first height information of the corresponding pixel in the source domain image are obtained.

[0082] For example, the height information z of a pixel i For the corresponding data point, z is the z-coordinate value, and d is the depth information. i for Where x i y i These are the x and y coordinates of the corresponding data points.

[0083] In step 202, the first feature vector output by the first machine learning model corresponding to the source domain image is transformed from the pixel feature space to the geometric space to obtain the second depth information and the second height information of each pixel.

[0084] In some embodiments, an MLP (Multilayer Perceptron) is used to transform the first feature vector from the pixel feature space to the geometric space.

[0085] For example, the geometric space is g i =M pd (e i ,θ),∈R 2 Obtain the second depth information of the pixel from the geometric space. Second height information

[0086] In step 203, the loss function is determined using the first depth information, the first height information, the second depth information, and the second height information.

[0087] For example, the loss function is shown in Equation (2), where N is the total number of samples.

[0088]

[0089] In step 204, the first machine learning model is trained using a loss function.

[0090] In some embodiments, the steps of training a second machine learning model using source domain images are as follows: Figure 3 As shown.

[0091] In step 301, the first gray value of each pixel in the source domain image and the first gray value transformation amount of the local neighborhood of each pixel are obtained.

[0092] For example, p i Given the grayscale value of a pixel, the local neighborhood N(p) of that pixel is... i The grayscale value p) j p j ∈N(p i The grayscale transformation amount is p. j -p i .

[0093] In step 302, the second gray value of each pixel and the second gray transformation amount of the local neighborhood of each pixel are predicted based on the second feature vector corresponding to the source domain 3D data output by the second machine learning model.

[0094] In some embodiments, MLP is used to predict the second gray value of each pixel and the second gray transformation amount of the local neighborhood of each pixel.

[0095] In step 303, the loss function is determined using the first gray value, the first gray transformation amount, the second gray value, and the second gray transformation amount.

[0096] For example, the loss function is shown in Equation (3).

[0097]

[0098] Where N is the total number of samples, and K is the color space parameter. i p′ is the first grayscale value. i p is the second grayscale value. ij p′ is the first grayscale transformation value. ij This is the second grayscale transformation value.

[0099] In step 304, the second machine learning model is trained using the loss function.

[0100] return Figure 1 In step 104, a first machine learning model is trained using the 3D data of the target domain, and a second machine learning model is trained using the image of the target domain to achieve bidirectional self-supervision.

[0101] In some embodiments, the steps of training a first machine learning model using target domain 3D data are as follows: Figure 4 As shown.

[0102] In step 401, the third depth information and third height information of each pixel in the target domain image are obtained using the target domain three-dimensional data.

[0103] In step 402, the third feature vector output by the first machine learning model corresponding to the target domain image is transformed from the pixel feature space to the geometric space to obtain the fourth depth information and the fourth height information of each pixel.

[0104] In some embodiments, an MLP is used to transform the third feature vector from the pixel feature space to the geometric space.

[0105] In step 403, the loss function is determined using the third depth information, the third height information, the fourth depth information, and the fourth height information.

[0106] For example, the loss function can be calculated using the formula (2) above.

[0107] In step 404, the first machine learning model is trained using a loss function.

[0108] In some embodiments, the steps of training a second machine learning model using a target domain image are as follows: Figure 5 As shown.

[0109] In step 501, the third gray value of each pixel in the target domain image and the third gray transformation amount of the local neighborhood of each pixel are obtained.

[0110] In step 502, the fourth gray value of each pixel and the fourth gray transformation amount of the local neighborhood of each pixel are predicted based on the fourth feature vector corresponding to the three-dimensional data of the target domain output by the second machine learning model.

[0111] In some embodiments, MLP is used to predict the fourth gray value of each pixel and the fourth gray transformation amount of the local neighborhood of each pixel.

[0112] In step 503, the loss function is determined using the third gray value, the third gray transformation amount, the fourth gray value, and the fourth gray transformation amount.

[0113] In some embodiments, the loss function is calculated using equation (3) above.

[0114] In step 504, the second machine learning model is trained using the loss function.

[0115] Figure 6 This is a schematic diagram of a machine learning model training framework according to an embodiment of the present disclosure.

[0116] like Figure 6 As shown, a first machine learning model is trained alternately using source domain images and target domain images, and a second machine learning model is trained alternately using source domain 3D data and target domain 3D data. The classifier then outputs the corresponding classification results. During training, a first joint loss function value is determined using the first semantic segmentation prediction result corresponding to the source domain image and the second semantic segmentation prediction result corresponding to the source domain 3D data. This first joint loss function value is then used to train both the first and second machine learning models. Similarly, a second joint loss function value is determined using the third and fourth semantic segmentation prediction results corresponding to the target domain, and this second joint loss function value is then used to train both the first and second machine learning models.

[0117] Furthermore, in the bidirectional self-supervised process, a first machine learning model is trained using source domain 3D data, and a second machine learning model is trained using source domain images. Similarly, a first machine learning model is trained using target domain 3D data, and a second machine learning model is trained using target domain images. Figure 6 In this process, the first MLP is used to transform the corresponding feature vector from the pixel feature space to the geometric space, and the second MLP is used to predict the gray value of the pixel and the gray transformation amount of the local neighborhood of the pixel using the corresponding feature vector.

[0118] In the machine learning model training method provided in the above embodiments of this disclosure, the robustness of the machine learning model in different environments is effectively improved by integrating the characteristics of different modalities.

[0119] Figure 7 This is a schematic flowchart illustrating a machine learning model training method according to yet another embodiment of this disclosure. In some embodiments, the following machine learning model training method is performed by a machine learning model training apparatus.

[0120] In step 701, the first machine learning model is trained alternately using the source domain image and the target domain image.

[0121] In step 702, the second machine learning model is trained alternately using source domain 3D data and target domain 3D data.

[0122] In step 703, a first machine learning model is trained using the source domain 3D data, and a second machine learning model is trained using the source domain image.

[0123] In step 704, a first machine learning model is trained using the target domain 3D data, and a second machine learning model is trained using the target domain image.

[0124] In step 705, the first feature vector output by the first machine learning model corresponding to the source domain image is used to determine the corresponding first modality category prototype, and the third feature vector output by the first machine learning model associated with the target domain image is used to determine the corresponding second modality category prototype.

[0125] For example, the prototype of the first modality category is shown in formula (4).

[0126]

[0127] Where C is the total number of categories, N is the total number of samples, and y (i) Let e ​​be the label of the i-th sample. (i) Let be the feature of the i-th sample, and ∈ be the system parameter.

[0128] During training, the EMA (Exponential Moving Average) method can be used to update the class prototype.

[0129] For example, the prototype of the second modality category is shown in Equation (5).

[0130]

[0131] in, Let be the pseudo-label of the i-th sample.

[0132] In step 706, a loss function value is determined based on the consistency between the first modality category prototype and the second modality category prototype, and the first machine learning model is trained using the loss function value.

[0133] For example, the loss function value is shown in formula (6).

[0134]

[0135] The above-mentioned category-aware adversarial learning can effectively improve the consistency of the first machine learning model with the category prototypes of the source domain image and the target domain image.

[0136] In step 707, the corresponding third modality category prototype is determined by using the second feature vector output by the second machine learning model that corresponds to the source domain 3D data, and the corresponding fourth modality category prototype is determined by using the fourth feature vector output by the second machine learning model that corresponds to the target domain 3D data.

[0137] In some embodiments, the prototype of the third modality category is calculated using the above formula (4), and the prototype of the fourth modality category is calculated using the above formula (5).

[0138] In step 708, the loss function value is determined based on the consistency between the third modality category prototype and the fourth modality category prototype, and the second machine learning model is trained using the loss function value.

[0139] In some embodiments, the loss function value is as shown in formula (6) above.

[0140] Through the aforementioned category-aware adversarial learning, the consistency of the second machine learning model with the category prototypes of the source domain 3D data and the target domain 3D data can be effectively improved.

[0141] Figure 8 This is a schematic diagram of a machine learning model training framework according to another embodiment of the present disclosure. Figure 8 and Figure 6 The difference is that, in Figure 8 The embodiment also includes a first adversarial learning module and a second adversarial learning module.

[0142] The first adversarial learning module uses the first feature vector output by the first machine learning model corresponding to the source domain image to determine the corresponding first modality category prototype, uses the third feature vector output by the first machine learning model associated with the target domain image to determine the corresponding second modality category prototype, determines the loss function value based on the consistency between the first modality category prototype and the second modality category prototype, and uses the loss function value to train the first machine learning model.

[0143] The second adversarial learning module uses the second feature vector output by the second machine learning model, which corresponds to the source domain 3D data, to determine the corresponding third modality category prototype. It also uses the fourth feature vector output by the second machine learning model, which corresponds to the target domain 3D data, to determine the corresponding fourth modality category prototype. The module then determines the loss function value based on the consistency between the third and fourth modality category prototypes and uses this loss function value to train the second machine learning model.

[0144] Therefore, this disclosure, on the one hand, effectively improves the robustness of machine learning models in different environments by integrating the characteristics of different modalities. On the other hand, by constraining the consistency of class prototypes between the source and target domains, it can effectively improve the domain invariance of machine learning models.

[0145] Figure 9 This is a schematic diagram of the structure of a machine learning model training apparatus according to an embodiment of this disclosure. Figure 9 As shown, the machine learning model training device includes a first training processing module 91 and a second training processing device 92.

[0146] The first training processing module 91 is configured to alternately train a first machine learning model using source domain images and target domain images, and alternately train a second machine learning model using source domain 3D data and target domain 3D data.

[0147] In some embodiments, the first machine learning model and the second machine learning model are fully convolutional network models.

[0148] In some embodiments, the source domain 3D data is source domain 3D point cloud data, and the target domain 3D data is target domain 3D point cloud data.

[0149] In some embodiments, during the training of a first machine learning model using a source domain image, a first training processing module 91 inputs the source domain image into the first machine learning model to generate a first feature vector. The first feature vector is then classified to generate a first semantic segmentation prediction result. Next, a loss function value is determined using the labels included in the source domain image and the first semantic segmentation prediction result, and the determined loss function value is used to train the first machine learning model.

[0150] In some embodiments, during the training of a first machine learning model using a target domain image, the first training processing module 91 inputs the target domain image into the first machine learning model to generate a third feature vector. The third feature vector is then classified to generate a third semantic segmentation prediction result. A loss function value is determined using the first pseudo-label and the third semantic segmentation prediction result, and this loss function value is used to train the first machine learning model.

[0151] It should be noted that since the target domain image does not have a label, a pre-set first pseudo-label is required for training.

[0152] In some embodiments, a first pseudo-label is assigned to the target domain image based on the feature similarity of each pixel in the source domain image and the target domain image.

[0153] In some embodiments, during the training of a second machine learning model using source domain 3D data, the first training processing module 91 inputs the source domain 3D data into the second machine learning model to generate a second feature vector. The second feature vector is then classified to generate a second semantic segmentation prediction result. Next, a loss function value is determined using the labels included in the source domain 3D data and the second semantic segmentation prediction result, and this loss function value is used to train the second machine learning model.

[0154] In some embodiments, during the training of the second machine learning model using target domain 3D data, the first training processing module 91 inputs the target domain 3D data into the second machine learning model to generate a fourth feature vector. The fourth feature vector is then classified to generate a fourth semantic segmentation prediction result. A loss function value is determined using the second pseudo-label and the fourth semantic segmentation prediction result, and this loss function value is used to train the second machine learning model.

[0155] It should be noted that since the target domain 3D data does not have labels, a pre-set second pseudo-label is required for training.

[0156] In some embodiments, a second pseudo-label is assigned to the target domain 3D data based on the feature similarity of each data point in the source domain 3D data and the target domain 3D data.

[0157] In some embodiments, the first training processing module 91 determines a first joint loss function value using the first semantic segmentation prediction result and the second semantic segmentation prediction result. Then, the first machine learning model and the second machine learning model are trained using the first joint loss function value, respectively.

[0158] In some embodiments, the first joint loss function value is the relative entropy (KL divergence) between the first semantic segmentation prediction result and the second semantic segmentation prediction result. For example, the first joint loss function value can be calculated using the above formula (1).

[0159] In some embodiments, the first training processing module 91 determines the value of the second joint loss function using the third semantic segmentation prediction result and the fourth semantic segmentation prediction result. Then, the first machine learning model and the second machine learning model are trained using the value of the second joint loss function. In some embodiments, the value of the second joint loss function is the relative entropy (KL divergence) between the third and fourth semantic segmentation prediction results. For example, the value of the second joint loss function is calculated using the above formula (1).

[0160] The second training processing module 92 is configured to train a first machine learning model using source domain 3D data and a second machine learning model using source domain images, and to train the first machine learning model using target domain 3D data and the second machine learning model using target domain images.

[0161] In some embodiments, during the training of the first machine learning model using source domain 3D data, the second training processing module 92 uses the source domain 3D data to obtain the first depth information and the first height information of each pixel in the source domain image, transforms the first feature vector output by the first machine learning model corresponding to the source domain image from the pixel feature space to the geometric space to obtain the second depth information and the second height information of each pixel, uses the first depth information, the first height information, the second depth information and the second height information to determine the loss function, and uses the loss function to train the first machine learning model.

[0162] In some embodiments, during the process of training a second machine learning model using a source domain image, the second training processing module 92 obtains the first gray value of each pixel in the source domain image and the first gray value transformation of the local neighborhood of each pixel, predicts the second gray value of each pixel and the second gray value transformation of the local neighborhood of each pixel based on the second feature vector output by the second machine learning model corresponding to the source domain three-dimensional data, determines the loss function using the first gray value, the first gray value transformation, the second gray value, and the second gray value transformation, and trains the second machine learning model using the loss function.

[0163] In some embodiments, during the training of the first machine learning model using the target domain 3D data, the second training processing module 92 uses the target domain 3D data to obtain the third depth information and third height information of each pixel in the target domain image, transforms the third feature vector output by the first machine learning model corresponding to the target domain image from the pixel feature space to the geometric space to obtain the fourth depth information and fourth height information of each pixel, uses the third depth information, third height information, fourth depth information and fourth height information to determine the third loss function, and uses the third loss function to train the first machine learning model.

[0164] In some embodiments, during the training of the second machine learning model using the target domain image, the second training processing module 92 obtains the third gray value of each pixel in the target domain image and the third gray transformation amount of the local neighborhood of each pixel, predicts the fourth gray value of each pixel and the fourth gray transformation amount of the local neighborhood of each pixel based on the fourth feature vector corresponding to the three-dimensional data of the target domain output by the second machine learning model, determines the loss function using the third gray value, the third gray transformation amount, the fourth gray value, and the fourth gray transformation amount, and trains the second machine learning model using the fourth loss function.

[0165] Figure 10 This is a schematic diagram of the structure of a machine learning model training apparatus according to another embodiment of the present disclosure. Figure 10 and Figure 9 The difference is that, in Figure 10 The machine learning model training device shown also includes a third training processing module 93.

[0166] The third training processing module 93 uses the first feature vector output by the first machine learning model corresponding to the source domain image to determine the corresponding first modality category prototype, uses the third feature vector output by the first machine learning model associated with the target domain image to determine the corresponding second modality category prototype, determines the loss function value based on the consistency between the first modality category prototype and the second modality category prototype, and uses the loss function value to train the first machine learning model.

[0167] The above processing can effectively improve the consistency of the first machine learning model with the category prototypes of the source domain image and the target domain image.

[0168] In addition, the third training processing module 93 uses the second feature vector output by the second machine learning model corresponding to the source domain three-dimensional data to determine the corresponding third modality category prototype, uses the fourth feature vector output by the second machine learning model corresponding to the target domain three-dimensional data to determine the corresponding fourth modality category prototype, determines the loss function value based on the consistency between the third modality category prototype and the fourth modality category prototype, and uses the loss function value to train the second machine learning model.

[0169] The above processing can effectively improve the consistency of the category prototypes of the second machine learning model with respect to the source domain 3D data and the target domain 3D data.

[0170] Figure 11 This is a schematic diagram of the structure of a machine learning model training apparatus according to another embodiment of this disclosure. Figure 11 As shown, the machine learning model training device includes a memory 111 and a processor 112.

[0171] Memory 111 is used to store instructions, and processor 112 is coupled to memory 111. Processor 112 is configured to execute instructions based on the memory storage, as shown in the example below. Figures 1-5 The method involved in any of the embodiments in 7.

[0172] like Figure 11 As shown, the machine learning model training device also includes a communication interface 113 for exchanging information with other devices. Additionally, the machine learning model training device includes a bus 114, through which the processor 112, communication interface 113, and memory 111 communicate with each other.

[0173] The memory 111 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk storage device. The memory 111 may also be a memory array. The memory 111 may also be divided into blocks, and the blocks may be combined into virtual volumes according to certain rules.

[0174] Furthermore, processor 112 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present disclosure.

[0175] This disclosure also relates to a computer-readable storage medium storing computer instructions that, when executed by a processor, implement... Figures 1-5 The method involved in any of the embodiments in 7.

[0176] Figure 12 This is a schematic flowchart illustrating a semantic segmentation method according to an embodiment of the present disclosure. In some embodiments, the following semantic segmentation method is performed by a semantic segmentation apparatus.

[0177] In step 1201, the target domain image is input into the first machine learning model so that the first machine learning model outputs the first semantic feature. The first machine learning model utilizes... Figures 1-5 The training is obtained by any of the training methods involved in any of the embodiments in 7.

[0178] In step 1202, the first semantic features are classified to generate the first semantic segmentation result.

[0179] In step 1203, the target domain 3D data is input into the second machine learning model so that the second machine learning model outputs the second semantic feature. The second machine learning model utilizes... Figures 1-5 The training is obtained by any of the training methods involved in any of the embodiments in 7.

[0180] In step 1204, the second semantic features are classified to generate the second semantic segmentation result.

[0181] Figure 13 This is a schematic diagram of the structure of a semantic segmentation apparatus according to an embodiment of this disclosure. Figure 13 As shown, the semantic segmentation device includes a first segmentation processing module 131 and a second segmentation processing module 132.

[0182] The first segmentation processing module 131 is configured to input the target domain image into a first machine learning model, so that the first machine learning model outputs a first feature vector, wherein the first machine learning model utilizes... Figures 1-5The training method involved in any of the embodiments in 7 is used to train the first feature vector and perform classification processing to generate the first semantic segmentation result;

[0183] The second segmentation processing module is configured to input the target domain 3D data into a second machine learning model, so that the second machine learning model outputs a second feature vector, wherein the second machine learning model utilizes... Figures 1-5 The training method involved in any of the embodiments in 7 is used to train the second feature vector, and the second feature vector is classified to generate the second semantic segmentation result.

[0184] Figure 14 This is a schematic diagram of the structure of a semantic segmentation apparatus according to another embodiment of this disclosure. Figure 14 As shown, the semantic segmentation device includes a memory 141, a processor 142, a communication interface 143, and a bus 144. Figure 14 and Figure 11 The difference is that, in Figure 14 In the illustrated embodiment, processor 142 is configured to implement memory-based instruction execution as follows: Figure 12 The method involved in any of the embodiments.

[0185] This disclosure also relates to a computer-readable storage medium storing computer instructions that, when executed by a processor, implement... Figure 12 The method involved in any of the embodiments.

[0186] In some embodiments, the functional unit modules described above may be implemented as general-purpose processors, programmable logic controllers (PLCs), digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or any suitable combination thereof for performing the functions described herein.

[0187] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0188] The description in this disclosure is provided for illustrative and descriptive purposes only and is not intended to be exhaustive or to limit the disclosure to its forms. Many modifications and variations will be apparent to those skilled in the art. The embodiments were chosen and described in order to better illustrate the principles and practical application of this disclosure and to enable those skilled in the art to understand this disclosure and to design various embodiments with various modifications suitable for a particular purpose.

Claims

1. A method for training a machine learning model, comprising: The first machine learning model is trained by alternating between source domain images and target domain images; A second machine learning model is trained alternately using source domain 3D data and target domain 3D data; The first machine learning model is trained using the source domain 3D data, and the second machine learning model is trained using the source domain image; The first machine learning model is trained using the target domain 3D data, and the second machine learning model is trained using the target domain image; The first feature vector output by the first machine learning model, corresponding to the source domain image, is used to determine the corresponding first modality category prototype. The corresponding second modality category prototype is determined by using the third feature vector associated with the target domain image output by the first machine learning model; The ninth loss function value is determined based on the consistency between the first modality category prototype and the second modality category prototype, and the first machine learning model is trained using the ninth loss function value.

2. The method according to claim 1, wherein, The step of training the first machine learning model using the source domain 3D data includes: Using the source domain 3D data, obtain the first depth information and the first height information of each pixel in the source domain image; The first feature vector output by the first machine learning model, corresponding to the source domain image, is transformed from the pixel feature space to the geometric space to obtain the second depth information and the second height information of each pixel. A first loss function is determined using the first depth information, the first height information, the second depth information, and the second height information; The first machine learning model is trained using the first loss function.

3. The method according to claim 1, wherein, Training the second machine learning model using the source domain image includes: Obtain the first gray value of each pixel in the source image and the first gray value transformation amount of the local neighborhood of each pixel; Based on the second feature vector output by the second machine learning model corresponding to the source domain three-dimensional data, predict the second gray value of each pixel and the second gray transformation amount of the local neighborhood of each pixel. The second loss function is determined using the first gray value, the first gray value transformation amount, the second gray value, and the second gray value transformation amount. The second machine learning model is trained using the second loss function.

4. The method according to claim 1, wherein, The step of training the first machine learning model using the target domain 3D data includes: Using the target domain 3D data, obtain the third depth information and third height information of each pixel in the target domain image; The third feature vector output by the first machine learning model, corresponding to the target domain image, is transformed from the pixel feature space to the geometric space to obtain the fourth depth information and the fourth height information of each pixel. The third loss function is determined using the third depth information, the third height information, the fourth depth information, and the fourth height information; The first machine learning model is trained using the third loss function.

5. The method according to claim 1, wherein, Training the second machine learning model using the target domain image includes: Obtain the third gray value of each pixel in the target domain image and the third gray transformation amount of the local neighborhood of each pixel; Based on the fourth feature vector output by the second machine learning model corresponding to the three-dimensional data of the target domain, predict the fourth gray value of each pixel and the fourth gray transformation amount of the local neighborhood of each pixel. The fourth loss function is determined using the third gray value, the third gray transformation amount, the fourth gray value, and the fourth gray transformation amount. The second machine learning model is trained using the fourth loss function.

6. The method according to claim 1, wherein, The step of training the first machine learning model using source domain images includes: The source domain image is input into the first machine learning model to generate a first feature vector; The first feature vector is classified to generate a first semantic segmentation prediction result; The fifth loss function value is determined using the labels included in the source domain image and the first semantic segmentation prediction result, and the first machine learning model is trained using the fifth loss function value.

7. The method according to claim 6, wherein, The method of training the second machine learning model using source domain 3D data includes: The source domain 3D data is input into the second machine learning model to generate a second feature vector; The second feature vector is classified to generate a second semantic segmentation prediction result; The sixth loss function value is determined using the labels included in the source domain 3D data and the second semantic segmentation prediction results, and the second machine learning model is trained using the sixth loss function value.

8. The method according to claim 7, further comprising: The value of the first joint loss function is determined using the first semantic segmentation prediction result and the second semantic segmentation prediction result; The first machine learning model and the second machine learning model are trained using the first joint loss function value, respectively.

9. The method according to claim 8, wherein, The first joint loss function value is the relative entropy of the first semantic segmentation prediction result and the second semantic segmentation prediction result.

10. The method according to claim 1, wherein, The step of training the first machine learning model using the target domain image includes: The target domain image is input into the first machine learning model to generate a third feature vector; The third feature vector is classified to generate a third semantic segmentation prediction result; The first pseudo-label and the third semantic segmentation prediction result are used to determine the value of the seventh loss function, and the first machine learning model is trained using the value of the seventh loss function.

11. The method of claim 10, further comprising: A first pseudo-label is assigned to the target domain image based on the feature similarity of each pixel in the source domain image and the target domain image.

12. The method according to claim 10, wherein, The step of training the second machine learning model using target domain 3D data includes: The target domain 3D data is input into the second machine learning model to generate a fourth feature vector; The fourth feature vector is classified to generate a fourth semantic segmentation prediction result; The value of the eighth loss function is determined using the second pseudo-label and the fourth semantic segmentation prediction result, and the second machine learning model is trained using the value of the eighth loss function.

13. The method of claim 12, further comprising: A second pseudo-label is assigned to the target domain 3D data based on the feature similarity of each data point in the source domain 3D data and the target domain 3D data.

14. The method of claim 12, further comprising: The value of the second joint loss function is determined using the third semantic segmentation prediction result and the fourth semantic segmentation prediction result; The first machine learning model and the second machine learning model are trained using the second joint loss function value, respectively.

15. The method according to claim 14, wherein, The second joint loss function value is the relative entropy of the third semantic segmentation prediction result and the fourth semantic segmentation prediction result.

16. The method according to any one of claims 1-15, further comprising: The corresponding third modality category prototype is determined by using the second feature vector output by the second machine learning model that corresponds to the source domain three-dimensional data; The corresponding fourth modality category prototype is determined by using the fourth feature vector output by the second machine learning model that corresponds to the three-dimensional data of the target domain; The tenth loss function value is determined based on the consistency between the third modality category prototype and the fourth modality category prototype, and the second machine learning model is trained using the tenth loss function value.

17. A machine learning model training device, comprising: The first training processing module is configured to alternately train a first machine learning model using source domain images and target domain images, and alternately train a second machine learning model using source domain 3D data and target domain 3D data. The second training processing module is configured to train the first machine learning model using the source domain 3D data and train the second machine learning model using the source domain image, and to train the first machine learning model using the target domain 3D data and train the second machine learning model using the target domain image. The third training processing module is configured to determine the corresponding first modality category prototype using the first feature vector output by the first machine learning model corresponding to the source domain image, determine the corresponding second modality category prototype using the third feature vector output by the first machine learning model associated with the target domain image, determine the ninth loss function value based on the consistency between the first modality category prototype and the second modality category prototype, and train the first machine learning model using the ninth loss function value.

18. A machine learning model training device, comprising: The memory is configured to store instructions; A processor, coupled to a memory, configured to implement the method as described in any one of claims 1-16 based on memory-stored instruction execution.

19. A semantic segmentation method, comprising: The target domain image is input into the first machine learning model so that the first machine learning model outputs a first feature vector, wherein the first machine learning model is trained using the training method of any one of claims 1-16; The first feature vector is classified to generate a first semantic segmentation result; The target domain three-dimensional data is input into the second machine learning model so that the second machine learning model outputs a second feature vector, wherein the second machine learning model is trained using the training method of any one of claims 1-16; The second feature vector is classified to generate a second semantic segmentation result.

20. A semantic segmentation apparatus, comprising: The first segmentation processing module is configured to input a target domain image into a first machine learning model so that the first machine learning model outputs a first semantic feature, wherein the first machine learning model is trained using the training method of any one of claims 1-16, and performs classification processing on the first semantic feature to generate a first semantic segmentation result. The second segmentation processing module is configured to input the target domain three-dimensional data into a second machine learning model so that the second machine learning model outputs a second semantic feature, wherein the second machine learning model is trained using the training method of any one of claims 1-16, and performs classification processing on the second semantic feature to generate a second semantic segmentation result.

21. A semantic segmentation apparatus, comprising: The memory is configured to store instructions; A processor, coupled to memory, is configured to implement the method of claim 19 based on memory-stored instruction execution.

22. A computer-readable storage medium, wherein, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method as described in any one of claims 1-16 and 19.

Images