Semantic segmentation model deployment method, semantic recognition method, device and storage medium

Abstract

Disclosed in the present application are a semantic segmentation model deployment method, a semantic recognition method, a device and a storage medium. Said deployment method comprises: acquiring a training image and a first label corresponding to the training image; inputting the training image into a teacher model and two student models for processing, acquiring a first segmentation output and a first feature output of the teacher model, and acquiring second segmentation outputs and second feature outputs of the two student models; on the basis of the first feature output and the two second feature outputs, determining a target distillation loss; on the basis of the two second segmentation outputs, determining a consistency loss; on the basis of the first label and the two second segmentation outputs, determining a target supervised loss; and on the basis of the target distillation loss, the consistency loss and the target supervised loss, determining trained student models, and deploying the trained student models on an edge device. The method can ensure the feature extraction capability, the anti-interference capability, the model recognition accuracy and other model performance of trained student models.

Description

Semantic segmentation model deployment method, semantic recognition method, device and storage medium

[0001] Cross-reference to related applications

[0002] This application claims priority to the application filed on June 27, 2024, in the China Patent Office, with the application number 202410851297.6, and the patent name: Semantic segmentation model deployment method, semantic recognition method, device and storage medium, the entire content of which is incorporated herein by reference. TECHNICAL FIELD

[0003] The present application relates to the field of artificial intelligence, in particular to a semantic segmentation model deployment method, a semantic recognition method, a device and a storage medium. BACKGROUND

[0004] Semantic segmentation is a task in the field of computer vision, aiming to classify each pixel point in an image into a specific semantic category, achieving pixel-level understanding and analysis. In the field of autonomous driving, semantic segmentation can help autonomous driving systems to recognize and understand different objects on the road surface, such as lane lines, drivable areas, pedestrians, vehicles, traffic signs, etc., by accurately perceiving and understanding the surrounding environment, making intelligent decisions and planning the driving path.

[0005] Existing semantic segmentation models generally need to be deployed in edge devices to intelligently recognize the semantic categories based on real-time collected images, while the edge devices have limited computing power, making the semantic segmentation models that can be deployed on edge devices generally small computing power models. The feature expression ability of small computing power models is generally poor, and the recognition accuracy is poor, which cannot meet the accuracy requirements in actual scenarios. TECHNICAL SOLUTION

[0006] The embodiments of the present application provide a semantic segmentation model deployment method, a semantic recognition method, a device and a storage medium to solve the problem of low recognition accuracy of semantic segmentation models that can be deployed on edge devices.

[0007] A semantic segmentation model deployment method, comprising:

[0008] Obtaining first training data, the first training data comprising training images and first labels corresponding to the training images;

[0009] Inputting the training images into a pre-trained teacher model and two student models for processing, obtaining first segmentation outputs and first feature outputs of the teacher model, and obtaining second segmentation outputs and second feature outputs of the two student models;

[0010] Determining a target distillation loss based on the first feature outputs and the two second feature outputs;

[0011] determine a consistency loss based on the two second segmentation outputs;

[0012] determine a target supervision loss based on the first label and the two second segmentation outputs;

[0013] determine a trained student model based on the target distillation loss, the consistency loss and the target supervision loss, and deploy the trained student model on an edge device.

[0014] A semantic recognition method, comprising:

[0015] obtain an image to be recognized;

[0016] recognize the image to be recognized by using the trained student model deployed by the semantic segmentation model deployment method, and obtain a semantic segmentation output corresponding to the image to be recognized.

[0017] A computer device, comprising a memory, a processor, and computer instructions stored in the memory and executable on the processor, wherein the processor executes the computer instructions to implement the semantic segmentation model deployment method or the semantic recognition method.

[0018] A computer readable storage medium, storing computer instructions, wherein the computer instructions are executed by a processor to implement the semantic segmentation model deployment method or the semantic recognition method.

[0019] The semantic segmentation model deployment method, the semantic recognition method, the device and the storage medium have the following advantages. The teacher model with large computing power is used to guide the student model with small computing power to perform knowledge distillation. Since the teacher model does not need to be deployed in the edge device, the model computing power can be ignored, and more reliable information can be provided to the student model to guide the training of the student model, thereby helping to ensure the model performance of the student model. The target distillation loss determined based on the first feature output and the two second feature outputs can reflect the loss of the knowledge distillation process, thereby helping to ensure the feature extraction capability of the student model. The consistency learning is performed through the two second segmentation outputs of the two student models, thereby helping to improve the anti-interference capability of the student model. The target supervision loss determined based on the first label and the two second segmentation outputs can help to ensure the accuracy of the recognition result of the student model. The model whether converges or not is evaluated based on the target distillation loss, the consistency loss and the target supervision loss, so that whether the model performance of the student model meets the standard can be evaluated from multiple dimensions, thereby ensuring the model performance of the student model. The student model is trained in the training device with large computing power by using the teacher model, so that the training process of the student model is not restricted by the computing power, thereby helping to ensure the model performance of the trained student model. The trained student model is deployed in the edge device, thereby ensuring the accuracy of the semantic segmentation and recognition of the image by the learning model deployed on the edge device. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort based on these drawings.

[0021] FIG. 1 is a flowchart of a semantic segmentation model deployment method according to an embodiment of the present application;

[0022] FIG. 2 is a flowchart of step S102 in FIG. 1;

[0023] FIG. 3 is another flowchart of step S201 in FIG. 2;

[0024] FIG. 4 is a flowchart before step S101 in FIG. 1;

[0025] FIG. 5 is a flowchart after step S402 in FIG. 4;

[0026] FIG. 6 is a flowchart of step S503 in FIG. 5;

[0027] FIG. 7 is a flowchart of step S603 in FIG. 6;

[0028] FIG. 8 is a schematic diagram of a semantic segmentation model architecture according to an embodiment of the present application;

[0029] FIG. 9 is a flowchart of a semantic recognition method according to an embodiment of the present application.

[0030] Embodiments of the present application

[0031] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0032] The semantic segmentation model deployment method provided by the embodiments of the present application can be applied to a training device for model training, and is specifically applicable to a semantic segmentation model architecture built on the training device. The semantic segmentation model architecture includes a teacher model and two student models. The teacher model generally does not need to be deployed in an edge device, and can not consider model computing power. The teacher model can be a model with strong feature extraction capability and capable of fully obtaining data information, i.e., the teacher model can be a large-computing-power model. The student models generally need to be deployed in an edge device and are constrained by the computing power of the edge device, and are generally small-computing-power models. In the present example, the teacher model and the student models are heterogeneous models, and the two student models are homogeneous models, i.e., the two student models have the same neural network structure but different initialization parameters. Since the teacher model does not need to be deployed in an edge device and can not consider model computing power, the teacher model can provide more reliable pseudo labels or output features to supervise the training of the small-computing-power student models, so as to guarantee the feature extraction capability and recognition result accuracy of the student models. Consistency learning is performed by the two homogeneous student models to improve the anti-interference capability of the models, and thus guarantee the model indicators of the trained student models.

[0033] In an embodiment, as shown in FIGS. 1 and 8, a semantic segmentation model deployment method is provided. Taking the training device in FIG. 1 as an example, the method includes the following steps:

[0034] S101: Obtain first training data, the first training data including training images and first labels corresponding to the training images;

[0035] S102: Input the training images into a pre-trained teacher model and two student models for processing, to obtain first segmentation outputs and first feature outputs of the teacher model, and second segmentation outputs and second feature outputs of the two student models;

[0036] S103: Determine a target distillation loss based on the first feature outputs and the two second feature outputs;

[0037] S104: Determine a consistency loss based on the two second segmentation outputs;

[0038] S105: Determine the target supervision loss based on the first label and the two second segmentation outputs;

[0039] S106: Based on the target distillation loss, consistency loss, and target supervision loss, determine the trained student model and deploy the trained student model to the edge device.

[0040] As an example, in step S101, the training device may obtain first training data for model training from the training dataset. The first training data includes training images and first labels corresponding to the training images. Here, the first training data can be understood as the training data used by the teacher model to guide the student model in the knowledge distillation process. The training image is an image from the training images, and the first label is the label of the first training data, which is used to identify the semantic category of any pixel in the training image.

[0041] For example, the first training data can be labeled data, i.e., from the labeled dataset D. L In the process, labeled data is obtained, including training images X. L and its corresponding real label Y L This allows for supervised learning based on labeled data. Alternatively, the first training data may include not only labeled data but also pseudo-labeled data and / or unlabeled data, i.e., it may include data from the pseudo-labeled dataset D. PL Obtain pseudo-labeled data and / or unlabeled dataset D U By acquiring unlabeled data and combining it with labeled data, along with pseudo-labeled and / or unlabeled data, semi-supervised learning can be performed. This allows for improved model performance with limited labeled data, utilizing a large amount of pseudo-labeled and / or unlabeled data. The pseudo-labeled data includes training images. and its corresponding first pseudo-tag and reliable pixel templates Reliable pixel template Used to distinguish between reliable and unreliable pixels, for example, a reliable pixel template. In the dataset, the values ​​of different pixels (i,j) are either 1 or 0, representing the first reliable pixel and the unreliable pixel, respectively. The unlabeled data contains only training images.

[0042] The pre-trained teacher model refers to a teacher model whose model parameters have already been trained; it is used to guide the student model in knowledge distillation. The student model is an untrained student model, meaning it needs to update its model parameters under the guidance of the teacher model.

[0043] The first segmentation output and the second segmentation output are respectively the segmentation outputs of the teacher model and the student model for processing the training image in the first training data. The segmentation output is used to reflect the semantic segmentation recognition output. The segmentation output can be represented by prob. The first segmentation output is specifically represented by prob1, and the second segmentation output is specifically represented by prob2.

[0044] The first feature output and the second feature output are respectively the feature outputs of the teacher model and the student model for processing the training image in the first training data. The feature output is used to reflect the pixel feature in the training image. The feature output can be understood as the feature extracted from the training image by the model. The feature output can be represented by feat. The first feature output is specifically represented by feat1, and the second feature output is specifically represented by feat2.

[0045] As an example, in step S102, the training device inputs the training image X in the first training data into the teacher model and the two student models for processing. Since the teacher model and the two student models both contain the semantic segmentation network classifier for performing semantic segmentation and the feature extraction network projector for performing feature extraction in the distillation learning process, the teacher model can output the first segmentation output prob1 and the first feature output feat1 when processing the training image X, and the two student models can output the two second segmentation outputs prob2 and the two second feature outputs feat2 when processing the training image X. That is, the semantic segmentation model architecture formed by the teacher model and the two student models can obtain one first segmentation output prob1 and two second segmentation outputs prob2, and one first feature output feat1 and two second feature outputs feat2 when processing the training image X.

[0046] As an example, in step S103, since the teacher model is suitable for all semantic categories when performing feature distillation on the student model, that is, the feature distillation process is irrelevant to the image label, that is, irrelevant to the semantic segmentation output, the training device only needs to process the feature outputs of the three models, that is, process the first feature output feat1 and the two second feature outputs feat2 to determine the target distillation loss. The target distillation loss is used to reflect the loss of the teacher model performing feature distillation on the two student models, and can be represented by Ldistill. The target distillation loss is used to reflect the feature extraction capability of the student model. In this example, the loss function such as cross-entropy and KL divergence can be used to calculate the target distillation loss of the entire semantic segmentation model architecture, but is not limited thereto.

[0047] As an example, in step S104, the consistency learning of the two student models is applicable to all semantic categories, and the consistency learning of the two student models exists in two aspects. On the one hand, the two student models have the same neural network structure, but their initialization parameters are different. The difference in the initialization parameters should not cause the two student models to produce differences in the segmentation output. On the other hand, the input data of the two student models are different. For example, different random data perturbations (including but not limited to color jittering, image blurring, and grayscale scaling, etc.) are applied to the same training image X. These perturbations should not cause the two student models to produce differences in the segmentation output. Therefore, the training device can perform consistency loss calculation on the two second segmentation outputs prob2. Specifically, but not limited to, loss functions such as KL divergence, JS divergence, and MSE loss, etc. can be used to perform consistency loss calculation on the two second segmentation outputs prob2 to determine the consistency loss between the two student models. L consistency consistency loss is used to reflect the consistency of the semantic segmentation learning of the two student models.

[0048] As an example, in step S105, since the first training data includes a training image and a first label corresponding to the training image, the first label is used to identify the semantic category of each pixel point in the training image. The two second segmentation outputs prob2 are used to reflect the predicted category of the training image after the network processing of the student model. By comparing the semantic category in the first label and the predicted category of the second segmentation output prob2, the supervision loss of the first training data in each student model is determined, so as to determine the target supervision loss L supervision .

[0049] As an example, in step S106, during the teacher model guided learning model training process, the student model needs to reacquire the target distillation loss L distill , the consistency loss L consistency , and the target supervision loss L supervision after each iteration of the student model, and based on the above losses, it is evaluated whether the entire semantic segmentation model architecture converges. If it has converged, it is determined that the teacher model guides the student model to complete knowledge distillation, that is, it is determined that the model performance of the student model at this time meets the preset requirements. Therefore, the student model can be determined as a trained student model, so as to deploy the trained student model in the edge device. Specifically, the semantic segmentation network in the trained student model is arranged in the edge device, and the feature extraction network projector is not deployed in the edge device to adapt to the small computing power constraint of the edge device.

[0050] In this embodiment, the teacher model with high computing power is used to guide the student model with low computing power to perform knowledge distillation. Since the teacher model does not need to be deployed in the edge device, the model computing power can be ignored, and more reliable information can be provided to the student model to guide the training of the student model, which helps to ensure the model performance of the trained student model. The target distillation loss determined based on the first feature output and the two second feature outputs can reflect the loss of the knowledge distillation process, which helps to ensure the feature extraction capability of the student model. The consistency learning is performed through the two second segmentation outputs of the two student models, which helps to improve the anti-interference capability of the student model. The target supervision loss determined based on the first label and the two second segmentation outputs helps to ensure the accuracy of the recognition result of the student model. The model convergence is evaluated based on the target distillation loss, the consistency loss, and the target supervision loss, which can evaluate whether the model performance of the student model meets the standard from multiple dimensions, so as to ensure the model performance of the student model. The teacher model is used to train the student model in the training device with high computing power, so that the training process of the student model is not restricted by the computing power, which helps to ensure the model performance of the trained student model. The trained student model is deployed in the edge device to ensure the accuracy of the semantic segmentation and recognition of the image by the learning model deployed on the edge device.

[0051] In an embodiment, as shown in FIGS. 2 and 8, in step S102, the training image is input into the pre-trained teacher model and two student models for processing to obtain the first segmentation output and the first feature output of the teacher model, and to obtain the second segmentation output and the second feature output of the two student models, including:

[0052] S201: performing enhancement processing on the training image to obtain a first enhanced image and two second enhanced images;

[0053] S202: inputting the first enhanced image into the teacher model for processing to obtain the first segmentation output and the first feature output of the teacher model;

[0054] S203: inputting the two second enhanced images into the two student models respectively for processing to obtain the second segmentation output and the second feature output of the two student models respectively.

[0055] As an example, in step S201, the training device can perform different degrees of enhancement processing on the training image X to obtain a first enhanced image and two second enhanced images. Here, the first enhanced image and the second enhanced image can be understood as images processed by different enhancement methods. The two second enhanced images are images processed by the same enhancement method but with different enhancement parameters. In this example, the training device can perform weak enhancement processing on the training image X to obtain the first enhanced image X Wand two independent strong enhancement processes are performed to obtain a second enhanced image X corresponding to the training image X S For example, the second enhanced images obtained by the two strong enhancement processes are X S1 and X S2 , so that the processed enhanced image has more interference factors compared with the training image X, and the model is trained based on the enhanced image to ensure the anti-interference ability of the trained model.

[0056] As an example, after obtaining the first enhanced image X W , the training device can input the first enhanced image X W to the teacher model for processing to obtain the first segmentation output prob1 and the first feature output feat1 output by the teacher model. Since the first enhanced image X W is an image obtained by weakly enhancing the training image X, it has less difference from the training image X, so that the first segmentation output prob1 and the first feature output feat1 output by the first enhanced image X W can be used to guide the training of the student model with large difference.

[0057] As an example, after obtaining the two second enhanced images X S1 and X S2 , the training device can input the two second enhanced images X S1 and X S2 to the two student models, respectively, for example, input the second enhanced image X S1 to the first student model to obtain the second segmentation output prob2 and the second feature output feat2 output by the first student model, and input the second enhanced image X S2 to the second student model to obtain the second segmentation output prob2 and the second feature output feat2 output by the second student model.

[0058] In this example, after obtaining the first segmentation output prob1 and the first feature output feat1 output by the teacher model and the second segmentation output prob2 and the second feature output feat2 output by the two student models, the training device can perform model training based on all the segmentation outputs and feature outputs. After the model training is completed, only the segmentation output corresponding to the semantic segmentation network classifier of the student model needs to be deployed on the edge device, so that the input image can be subjected to semantic segmentation and recognition, and the process requires less computing power and can adapt to the deployment requirements of the edge device. The feature output of the model is mainly used in the model training stage and is mainly used for feature distillation of the small-power student model by the large-power teacher model to ensure that the student model learned from the teacher model can learn the feature extraction ability of the teacher model, which helps to ensure the accuracy of the recognition result.

[0059] In an embodiment, as shown in FIG. 3 and FIG. 8, the step S201 of performing the enhancement processing on the training image to obtain the first enhanced image and the two second enhanced images comprises:

[0060] S301: performing weak enhancement on the training image by using a first enhancement rule to obtain the first enhanced image;

[0061] S302: performing strong enhancement on the first enhanced image by using a second enhancement rule to obtain the two second enhanced images respectively.

[0062] The first enhancement rule and the second enhancement rule are two independent enhancement rules. For example, the first enhancement rule can be a rule for implementing weak enhancement, so that the enhanced image has a small difference with the image before enhancement. The second enhancement rule is a rule for implementing strong enhancement, so that the enhanced image has a large difference with the image before enhancement. In this example, the enhancement rules used in the weak enhancement and the strong enhancement are different, but the enhancement rules used in the two strong enhancement processes are the same, and the specific control parameters used in the enhancement process are different, so that the two second enhanced images after the two strong enhancements are different.

[0063] As an example, in the step S301, the training device first performs weak enhancement processing on the training image X by using the first enhancement rule to obtain the first enhanced image. For example, the first enhancement rule can be a rule for performing weak enhancement processing, specifically including but not limited to a mapping function f() of weak enhancement processing, i.e., the first enhanced image X W = f(X).

[0064] As an example, in the step S302, after obtaining the first enhanced image, the training device performs two independent strong enhancement processes on the first enhanced image X W by using the second enhancement rule. For example, the second enhancement rule can be a rule for performing strong enhancement processing, specifically including but not limited to a mapping function g() of strong enhancement processing, then the two second enhanced images obtained are X S1 = g(X W ) and X S2 = g(X W ).

[0065] In this example, two enhancement rules are used for enhancement processing, so that the enhanced image after enhancement processing has more interference factors compared with the training image X, so that subsequent model training based on the enhanced image can guarantee the anti-interference ability of the trained model. Moreover, the second enhanced image has relevance with the first enhanced image by using the second enhancement rule to perform re-enhancement on the first enhanced image determined by the first enhancement rule, so that the two second enhanced images have a basis for consistent learning.

[0066] In an embodiment, the first enhancement rule comprises at least one of a random flip rule, an image zooming rule, and a random shift rule.

[0067] The second enhancement rule comprises at least one of a color flip rule, a contrast adjustment rule, and an image blurring rule.

[0068] As an example, the training device can adopt at least one of a random flip rule and a random shift rule to perform weak enhancement processing on the training image X, so that the first enhanced image X W Compared with the training image X, the first enhanced image X

[0069] As an example, the training device can adopt, but is not limited to, at least one of a color flip rule, a contrast adjustment rule, and an image blurring rule to perform twice independent strong enhancement processing on the first enhanced image X W The second enhancement rule adopted by the two strong enhancement processes is the same, but the specific control parameters are different, which can guarantee that the two second enhanced images X S1 and X S2 Compared with the training image X, the two second enhanced images X S1 and X S2 Based on the same second enhancement rule, the two second enhanced images X

[0070] In an embodiment, the target distillation loss is determined based on first distillation losses corresponding to the two student models;

[0071] The first distillation loss corresponding to each student model is determined based on the first feature output and the second feature output corresponding to the student model.

[0072] As an example, the training device can adopt, but is not limited to, loss functions such as cross-entropy and KL divergence to process the first feature output feat1 output by the teacher model processing the training image and the second feature output feat2 output by any student model processing the training image, to determine the first distillation loss corresponding to the student model. The first distillation loss herein is used to reflect the loss of the teacher model performing feature distillation on each student model, and can also reflect the feature extraction capability of the student model.

[0073] As an example, after obtaining the first distillation loss corresponding to the two student models, the training device can perform fusion processing on the two first distillation losses by using a preset fusion algorithm, and determine the processing result as the target distillation loss of the entire semantic segmentation model architecture. The fusion processing here can be understood as a processing process for processing multiple input data so that the output data is related to the multiple input data. In this example, the target distillation loss can reflect the overall loss of the teacher model performing knowledge distillation on the two student models with different initialization parameters and / or different input data.

[0074] In an embodiment, the target distillation loss is an arithmetic mean of the first distillation loss corresponding to the two student models.

[0075] As an example, after obtaining the first distillation loss corresponding to the two student models, the training device can perform mean processing on the first distillation loss corresponding to the two student models by using an arithmetic mean algorithm, so as to determine the arithmetic mean of the two as the target distillation loss of the entire semantic segmentation model architecture. The calculation formula of the target distillation loss is as follows: L distill is the target distillation loss, and are the first distillation loss corresponding to the two student models. In this example, the target distillation loss can reflect the overall loss of the teacher model performing knowledge distillation on the two student models with different initialization parameters and / or different input data.

[0076] In an embodiment, the target supervision loss is determined based on the first supervision loss corresponding to the two student models.

[0077] The first supervision loss corresponding to each student model is determined based on the first label and the second segmentation output corresponding to the student model.

[0078] As an example, the training device can use, but is not limited to, a cross-entropy loss function to process the first label corresponding to the training image and the second segmentation output prob2 of each student model processed on the training image, to determine the first supervision loss corresponding to each student model. The first supervision loss here is used to reflect the loss of each student model performing supervised learning or semi-supervised learning.

[0079] As an example, after obtaining the first supervision loss corresponding to the two student models, the training device can perform fusion processing on the first supervision loss corresponding to the two student models by using a preset fusion algorithm, and determine the processing result as the target supervision loss of the entire semantic segmentation model architecture. The fusion processing here can be understood as a processing process for processing multiple input data so that the output data is related to the multiple input data. In this example, the target supervision loss can reflect the overall loss of the two student models with different initialization parameters and / or different input data for supervised learning or semi-supervised learning on the first training data, for reflecting the accuracy of the prediction results.

[0080] In an embodiment, the target supervision loss is an arithmetic mean of the first supervision loss corresponding to the two student models.

[0081] As an example, after obtaining the first supervision loss corresponding to the two student models, the training device can perform mean processing on the first supervision loss corresponding to the two student models by using an arithmetic mean algorithm, so as to determine the arithmetic mean of the two as the target supervision loss of the entire semantic segmentation model architecture, that is, the calculation formula of the target supervision loss is as follows: Lsupervision is the target distillation loss, and are the first supervision loss corresponding to the two student models. In this example, the target supervision loss can reflect the overall loss of the two student models with different initialization parameters and / or different input data for supervised learning or semi-supervised learning on the first training data, for reflecting the accuracy of the prediction results.

[0082] In an embodiment, the first training data includes labeled data, and the first label includes a real label in the labeled data.

[0083] The first supervision loss corresponding to each student model is determined based on the labeled data loss corresponding to the student model.

[0084] The labeled data loss corresponding to each student model is determined based on the real label and the second segmentation output corresponding to the student model.

[0085] As an example, the first training data can include labeled data, and the labeled data includes a training image X L and its corresponding real label Y L The real label in the labeled data is the first label, which is used to reflect the training image X La real class corresponding to each pixel point. The training device can supervise learning of the second segmentation output prob2 output by each student model based on the labeled data, determine a labeled data loss corresponding to each student model, and the labeled data loss can use L S represent. That is, each training image X L corresponding to the real label Y L , the training image X L corresponds to the second augmented image X S The training device supervises learning of the second segmentation output prob2 of the student model, and the process can use, but is not limited to, cross-entropy loss calculation to calculate the labeled data loss L S reflects the difference between the second segmentation output prob2 of the student model and the real label Y L , which can reflect the accuracy of the semantic segmentation recognition result.

[0086] In the present example, when the first training data only includes labeled data, the real label Y L of the labeled data and the second segmentation output prob2 output by each student model can be used to determine the labeled data loss L S corresponding to each student model; then, the labeled data loss L S corresponding to each student model is determined as the first supervision loss corresponding to the student model.

[0087] In an embodiment, the labeled data loss corresponding to each student model is further determined based on the target weight corresponding to the student model.

[0088] The target weight corresponding to each student model is determined based on the second segmentation output corresponding to the student model.

[0089] As an example, after obtaining the second segmentation output prob2 output by the student model, the training device can determine the prediction probability of each pixel point (i, j) for each class c. Then, the prediction probability is analyzed to determine the reliability of the prediction probability , and the target weight corresponding to the prediction probability is determined according to the reliability. Generally, the higher the reliability of the prediction probability , the higher the target weight corresponding to it; conversely, the lower the reliability of the prediction probability , the lower the target weight corresponding to it.

[0090] As an example, after determining the target weight corresponding to each pixel point, the training device can use, but is not limited to, cross-entropy loss algorithm to determine the real label Y LThe second segmentation output prob2 of each student model is used to calculate the pixel supervision loss corresponding to each pixel. Then, the pixel supervision loss corresponding to each pixel and its corresponding target weight are weighted to determine the labeled data loss L corresponding to each student model. S This results in a loss of L of labeled data. S It can reflect the reliability of the student model in predicting categories, thereby strengthening the learning power of the student model.

[0091] In one embodiment, the target weights corresponding to each student model are:

[0092] Among them, w i,j The target weight for each pixel (i,j); H is the predicted probability of each pixel (i,j) for category c determined based on the second segmentation output. i,j The probability entropy for each pixel (i,j); For the labeled dataset D L The number of labeled data in the dataset; s is the number of labeled datasets D. L Any of the labeled data in the data; This represents the true category of each pixel (i,j) determined based on the true labels in the labeled data.

[0093] As an example, after obtaining the second segmentation output prob2 from the student model, the training device can process the second segmentation output prob2 using, but is not limited to, the softmax function, to determine the predicted probability of each pixel (i,j) for each class c. Then, using, but not limited to, cross-entropy, to predict probabilities. The process is performed to determine the probability entropy H of each pixel (i,j). i,j The probability entropy H i,j Determine the target weight w for each pixel. i,j ,Right now Generally speaking, probability entropy can reflect the uncertainty of the student model's class at a certain pixel. The higher the probability entropy, the higher the uncertainty, that is, the lower the reliability. This indicates that the student model's feature learning for that pixel is not enough, and the penalty for that pixel should be increased to strengthen its learning efforts.

[0094] Accordingly, the training device acquires the target weight w for each pixel. i,j Then, based on the target weight w i,j Given labeled data and a second split output, determine the labeled data loss L corresponding to the student model. S, and the specific calculation process can adopt but is not limited to the following formula:

[0095] wherein, is the number of labeled data in the labeled data set D L ; s is any labeled data in the labeled data set D L ; w i,j is the target weight; is the real class of each pixel point (i, j) for each class c, and is determined according to the real label Y L in the labeled data; is the prediction probability of each pixel point (i, j) for each class c, and is determined according to the second segmentation output prob2 of each student model.

[0096] In this example, the labeled data loss is determined based on the target weight w i,j , the real label Y L and the second segmentation output prob2, and the target weight w i,j is determined by the second segmentation output prob2, so that the labeled data loss L S can reflect the uncertainty of the student model in predicting the class, thereby enhancing the learning ability of the student model.

[0097] In an embodiment, the first training data further includes pseudo-label data, and the first label further includes a first pseudo-label in the pseudo-label data;

[0098] The first supervision loss corresponding to each student model is further determined based on a pseudo-label data loss corresponding to the student model;

[0099] The pseudo-label data loss corresponding to each student model is determined based on the first pseudo-label and the second segmentation output corresponding to the student model.

[0100] As an example, the first training data includes not only the labeled data but also the pseudo-label data, the labeled data includes a training image X L and a corresponding real label Y L of the training image X L ; the pseudo-label data includes a training image and a corresponding first pseudo-label of the training image . That is, the first label includes the real label Y L and the first pseudo-label . The training device can supervise the learning of the second segmentation output prob2 output by each student model based on the real label Y L of the labeled data, and determine the labeled data loss corresponding to each student model, which can adopt L SIt can also be represented by the first pseudo-label based on the pseudo-label data. Supervised learning is performed on the second segmentation output prob2 of each student model to determine the pseudo-label data loss for each student model. This pseudo-label data loss can be achieved using... This is represented by the labeled data loss L corresponding to each student model. S and pseudo-label data loss The first supervised loss for this student model is determined to be L, specifically the labeled data loss corresponding to the student model. S and pseudo-label data loss The sum of these is determined to be its corresponding first supervisory loss, i.e. Here, Si refers to the i-th student model.

[0101] In this example, it is based on labeled data loss L S and pseudo-label data loss By determining the corresponding first supervised loss, it is possible to improve the model performance of the trained student model by utilizing a large amount of pseudo-labeled data, even with limited labeled data.

[0102] In one embodiment, the pseudo-tag data further includes a reliable pixel template, which is used to determine the first reliable pixel.

[0103] The pseudo-label data loss for each student model is determined based on the first pseudo-label corresponding to the first reliable pixel and the second segmentation output corresponding to the student model. The pseudo-label data loss is as follows:

[0104] in, For the loss of pseudo-label data, For pseudo-label dataset D PL The number of pseudo-labeled data in the dataset, s, is the number of pseudo-labeled datasets D. PL Any pseudo-label data in the Mask; i,j This refers to the values ​​in the reliable pixel template, Mask. i,j =1 refers to the first reliable pixel.

[0105] As an example, since the pseudo-label data includes training images and its corresponding first pseudo-tag and reliable pixel templates Reliable pixel template Used to distinguish between reliable and unreliable pixels, for example, a reliable pixel template. The values ​​of different pixels (i,j) are either 1 or 0, which are used to represent the first reliable pixel or the unreliable pixel, respectively, and are used to reflect the first pseudo-label corresponding to them. The reliability of the middle pixel points. In this example, the first pseudo label corresponding to the first reliable pixel point can be used to determine the corresponding pseudo label data loss The second segmentation output prob2 corresponding to each student model to determine its corresponding pseudo label data loss, and the specific calculation formula is as follows:

[0106] wherein, is the pseudo label data loss, is the number of pseudo label data in the pseudo label data set D PL s is any pseudo label data in the pseudo label data set D PL Mask i,j is the reliable value corresponding to the pixel point (i, j) in the reliable pixel template, and Mask i,j = 1 means the first reliable pixel point.

[0107] In this example, when performing semi-supervised learning based on labeled data and pseudo label data, the first supervision loss corresponding to each student model can be determined based on the labeled data loss L S and the pseudo label data loss corresponding to all first reliable pixel points, and the specific first supervision loss is the sum of the labeled data loss L S and the pseudo label data loss corresponding to all first reliable pixel points. This can utilize the pseudo label data of the first reliable pixel points in a large amount of pseudo label data to improve the model performance of the student model trained under the condition of limited labeled data.

[0108] In an embodiment, the reliable pixel template is also used to determine the unreliable pixel points;

[0109] The first supervision loss corresponding to each student model is also determined based on the contrastive learning loss corresponding to the student model.

[0110] The contrastive learning loss corresponding to each student model is determined based on the first positive sample feature, the second positive sample feature and the negative sample feature. The first positive sample feature is the feature corresponding to the reliable pixel point in the first feature output, the second positive sample feature is the feature corresponding to the reliable pixel point in the second feature output corresponding to the student model, and the negative sample feature is the feature corresponding to the unreliable pixel point in the second feature output corresponding to the student model.

[0111] As an example, the reliable pixel template is used to distinguish the first reliable pixel point and the unreliable pixel point, for example, the reliable pixel template The values of different pixel points (i, j) are 1 or 0, which are used to represent the first reliable pixel point or the unreliable pixel point. When calculating the first supervision loss corresponding to each student model, the training device also needs to determine the reliable pixel template The reliable pixel points and the unreliable pixel points are specified for feature comparison. Specifically, the first positive sample feature corresponding to each reliable pixel point is extracted from the first feature output feat1, and the second positive sample feature corresponding to each reliable pixel point and the negative sample feature corresponding to the unreliable pixel point are extracted from the second feature output feat2. Then, the first positive sample feature, the second positive sample feature and the negative sample feature are compared to determine the comparison student loss L contra The positive sample feature here is the feature corresponding to the reliable pixel point, and the negative sample feature is the feature corresponding to the unreliable pixel point.

[0112] In this example, when semi-supervised learning is performed based on labeled data and pseudo-label data, the first supervision loss of each student model can be determined based on the labeled data loss L S , the pseudo-label data loss of all first reliable pixel points , and the comparison learning loss L contra of the unreliable pixel points. Specifically, the first supervision loss of each student model is determined by summing the labeled data loss L S , the pseudo-label data loss , and the comparison learning loss L contra . In this way, based on the first reliable pixel points and the unreliable pixel points in the pseudo-label data, loss calculation can be performed respectively, so that a large amount of pseudo-label data can be utilized to improve the model performance of the student model obtained by training.

[0113] In an embodiment, the comparison learning loss of each student model is as follows:

[0114] wherein L contra is the comparison student loss, M is the number of positive samples of each class, N is the number of negative samples corresponding to each positive sample point, C is the total number of semantic segmentation classes, r is a constant, is the mean of the plurality of first positive sample features corresponding to the class c determined based on the first feature output feat1 of the teacher model, Z c,i is the mean of the plurality of second positive sample features corresponding to the class c determined based on the second feature output feat2 of the student model, is the mean of the plurality of negative sample features corresponding to the class c determined based on the second feature output feat2 of the student model.

[0115] As an example, the reliable pixel template is also used to determine unreliable pixel points, which are pixel points with a value of 0 For all pixel points in the unreliable pixel points, a contrastive learning loss can be determined based on the corresponding first feature output feat1 and second feature output feat2, and specifically for each class c, a plurality of positive sample points pos corresponding to the first reliable pixel points are sampled c,i,j , and a plurality of negative sample points are sampled from the unreliable pixel points c,i,j The contrastive feature penalty is then used to make the features of the positive sample points pos closer to the class feature center, while the features of the negative sample points farther away from the class feature center.

[0116] In this example, the feature mean of the plurality of positive sample points corresponding to class c can be obtained based on the first feature output feat1 of the teacher model (which can be represented as ), i.e. is the mean of the first positive sample features corresponding to the reliable pixel points in the first feature output feat1, and the feature mean of the plurality of positive sample points corresponding to class c (which can be represented as c,i ) and the feature mean of the plurality of negative sample points corresponding to class c (which can be represented as ) are obtained based on the second feature output feat2 of the student model, i.e. c,i is the mean of the second positive sample features corresponding to the reliable pixel points in the second feature output feat2, is the mean of the negative sample features corresponding to the unreliable pixel points in the second feature output feat2, where the negative sample points refer to sample points corresponding to other classes except class c, M is the number of positive sample points for each class, N is the number of negative samples corresponding to each positive sample point, C is the total number of semantic segmentation classes, and r is a constant. Then the contrastive learning loss L contra determined by it is as follows:

[0117] In an embodiment, the first training data further includes unlabeled data.

[0118] The first supervision loss corresponding to each student model is further determined based on the unlabeled data loss corresponding to the student model.

[0119] The unlabeled data loss corresponding to each student model is determined based on the second pseudo-label and the second segmentation output corresponding to the student model, and the second pseudo-label is determined based on the first segmentation output.

[0120] As an example, since the unlabelled data only includes training images During the feature loss calculation process on the student model, the first segmentation output prob1 of the teacher model can be learned, and the corresponding second pseudo label is determined according to the first segmentation output prob1 Then, the second segmentation output prob2 output by the student model can be processed by, but not limited to, a softmax network to determine the prediction probability of each pixel point (i, j) for each class c Finally, the second pseudo label and the second segmentation output prob2 of the student model are calculated by loss to determine the unlabelled data loss, which can be represented by .

[0121] As an example, semi-supervised learning can be performed based on the labelled data and the unlabelled data, and specifically, the sum of the labelled data loss L S and the unlabelled data loss can be determined to correspond to the first supervised loss, so as to assist in training with a large amount of unlabelled data in the case of limited labelled data, so as to improve the model performance of the student model obtained by training.

[0122] As another example, semi-supervised learning can also be performed based on the labelled data, the pseudo label data and the unlabelled data, and specifically, the sum of the labelled data loss L S , the pseudo label data loss and the unlabelled data loss can be determined to correspond to the first supervised loss, that is so as to assist in training with a large amount of pseudo label data and unlabelled data in the case of limited labelled data, so as to improve the model performance of the student model obtained by training.

[0123] As yet another example, semi-supervised learning can also be performed based on the labelled data, the pseudo label data and the unlabelled data, and specifically, the sum of the labelled data loss L S , the pseudo label data loss contrastive learning loss L contra and the unlabelled data loss can be determined to correspond to the first supervised loss, that is so as to assist in training with a large amount of pseudo label data and unlabelled data in the case of limited labelled data, so as to improve the model performance of the student model obtained by training.

[0124] In an embodiment, the unlabeled data loss corresponding to each student model is determined based on the second pseudo label corresponding to the second reliable pixel point and the second segmentation output corresponding to the student model.

[0125] The second reliable pixel point is a pixel point whose predicted probability determined based on the first segmentation output is greater than a preset threshold.

[0126] As an example, after obtaining the first segmentation output prob1 corresponding to the teacher model, the training device can process the first segmentation output prob1 using, but not limited to, a softmax function to determine the predicted probability of each pixel point (i, j) for each class c Then, the predicted probability is compared with a preset threshold T, if the predicted probability is greater than the preset threshold T, the pixel point is determined as a second reliable pixel point; if the predicted probability is not greater than the preset threshold T, the pixel point is directly discarded, which helps to save computing power and avoid the interference of unreliable pixel points, and helps to ensure the accuracy of the result.

[0127] As an example, after determining the second reliable pixel point based on the first segmentation output prob1, the training device can use, but not limited to, a cross-entropy loss function to calculate the loss of the second pseudo label corresponding to the second reliable pixel point and the second segmentation output prob2 corresponding to the student model, to determine the unlabeled data loss corresponding to the student model In this example, the unlabeled data loss corresponding to the student model is determined based on the second reliable pixel point determined based on the first segmentation output prob1, which helps to ensure the accuracy of the result and improve the processing efficiency.

[0128] In an embodiment, the unlabeled data loss corresponding to each student model is as follows:

[0129] wherein, is the unlabeled data loss, is the number of unlabeled data in the unlabeled data set D U s is any unlabeled data in the unlabeled data set D U Validij is a second reliable pixel point, and T is a preset threshold; is a second pseudo label determined based on the first segmentation output prob1; is the predicted probability of each pixel point (i, j) for each class c.

[0130] As an example, after obtaining the first segmentation output prob1 corresponding to the teacher model, the training device can process the first segmentation output prob1 using, but not limited to, a softmax function to determine the prediction probability of each pixel point (i, j) for each class c Then, the prediction probability can be processed using, but not limited to, an argmax function to obtain the second pseudo label corresponding to the first segmentation output prob1 And the training device selects the pixel point whose prediction probability is greater than the preset threshold T to determine the second reliable pixel point Validij, Understandably, the pixel point whose prediction probability is not greater than the preset threshold T can be directly discarded without further processing.

[0131] As an example, after determining the second pseudo label and the second reliable pixel point Valid ij based on the first segmentation output prob1, the training device can determine the unlabeled data loss corresponding to the student model based on the second pseudo label and the second reliable pixel point Valid ij ,in combination with the second segmentation output prob2 corresponding to the student model which can guarantee the accuracy of the unlabeled data loss The specific calculation process can use, but not limited to, the following formula:

[0132] wherein, is the number of unlabeled data in the unlabeled data set D U ; s is any unlabeled data in the unlabeled data set D U ; Valid ij is the second reliable pixel point; is the second pseudo label determined based on the first segmentation output prob1; is the prediction probability of each pixel point (i, j) for each class c, which is determined according to the second segmentation output prob2 of each student model.

[0133] In an embodiment, step S106, i.e., determining the trained student model based on the target distillation loss, the consistency loss and the target supervision loss, comprises:

[0134] determining a first model loss based on the target distillation loss, the consistency loss and the target supervision loss;

[0135] In response to the first model loss satisfying a first convergence condition, the student model is determined as a trained student model.

[0136] As an example, the training device obtains, after each model parameter iteration update of the student model, a target distillation loss L distill , a consistency loss L consistency , and a target supervision loss L supervision , and determines a sum of the target distillation loss L distill , the consistency loss L consistency , and the target supervision loss L supervision as a first model loss L model1 , i.e., L model1 = L distill + L consistency + L supervision . Then, the first model loss L model1 may be compared with a first loss threshold preset, and if the first model loss L model1 is less than the first loss threshold, it is determined that the student model in the semantic segmentation model architecture has converged, i.e., it is determined that the model performance of the student model at this time meets the preset requirements, and therefore the student model can be determined as a trained student model. The first loss threshold is a loss threshold preset for evaluating whether the student model converges.

[0137] In an embodiment, as shown in FIG. 4, before step S101, i.e., before the first training data is obtained, the semantic segmentation model deployment method further includes:

[0138] S401: obtaining second training data, the second training data including training images and second labels corresponding to the training images;

[0139] S402: inputting the training images into the teacher model for processing to obtain third segmentation outputs and third feature outputs corresponding to the teacher model;

[0140] S403: determining a classification loss based on the second labels and the third segmentation outputs;

[0141] S404: determining a discrimination loss based on the third feature outputs;

[0142] S405: determining a trained teacher model based on the classification loss and the discrimination loss.

[0143] As an example, in step S401, the training device can obtain second training data for model training from the training data set, and the second training data includes training images and second labels corresponding to the training images. The second training data here can be understood as training data for training the teacher model, the training images are images in the second training data, and the second labels are labels in the second training data. In this example, the second training data includes labeled data and unlabeled data, the labeled data carries the second label, and the unlabeled data does not carry the training label.

[0144] The teacher model is an untrained teacher model, that is, a teacher model that needs to be trained using training data to update its model parameters.

[0145] As an example, in step S402, after obtaining the second training data, the training device can input the training images into the teacher model for processing. Specifically, the training images are first input into the encoder for encoding to determine the encoding features corresponding to each training image, and then the semantic segmentation network classifier and the feature extraction network projector are used to process the encoding features to obtain the third segmentation output prob3 and the third feature output feat3, so as to evaluate whether the teacher model meets the model convergence condition based on the third segmentation output prob3 and the third feature output feat3.

[0146] As an example, in step S403, since the second label in the labeled data can determine the real label of any pixel point in the training image belonging to a certain category, and the third segmentation output prob3 can determine the predicted label of any pixel point in the training image belonging to a certain category, the training device can use, but not limited to, cross-entropy to calculate the loss of the second label corresponding to the training image and the third segmentation output prob3 to determine the classification loss.

[0147] As an example, in step S404, after obtaining the third feature output feat3 corresponding to all training images, the training device can input the third feature output feat3 into the discriminator network to determine whether the third feature output feat3 comes from the labeled data or the unlabeled data through the discriminator network, and obtain the discrimination output corresponding to the discriminator network; and then compare and learn the discrimination output of the discriminator network with the second label corresponding to the training image to determine the discrimination loss, so as to further improve the feature extraction capability of the trained teacher model. In this example, the discrimination loss is based on the loss of all training images in the second training data, which not only includes the loss of the labeled data, but also includes the loss of the unlabeled data, so that the model performance can be improved by using a large amount of unlabeled data under the condition of limited labeled data.

[0148] As an example, in step S405, the training device obtains the classification loss and the discrimination loss after each model iteration, and determines whether the teacher model converges based on the classification loss and the discrimination loss. If the teacher model has converged, the teacher model is determined as a trained teacher model, so as to guide the student model for knowledge distillation subsequently.

[0149] In this embodiment, in the teacher model training process, not only the second label of the labeled data is used to determine the classification loss, but also a discriminator network is additionally added to determine the corresponding discrimination loss through the feature output of the discriminator network, so as to guarantee the model performance of the trained teacher model through the feature extraction capability of the discriminator network.

[0150] In an embodiment, step S404, that is, determining the discrimination loss based on the third feature output, includes:

[0151] The third feature output is processed by the discriminator network to obtain a discrimination output;

[0152] The discrimination loss is determined based on the second label and the discrimination output.

[0153] As an example, after obtaining the third feature output feat3 of all the second training data, the training device can input the third feature output feat3 into the discriminator network to determine whether the third feature output feat3 comes from the labeled data or the unlabeled data, and obtain the discrimination output of the discriminator network. For example, if the discrimination output of the discriminator network is 1, it indicates that the third feature output feat3 corresponds to the labeled data; if the discrimination output of the discriminator network is 0, it indicates that the third feature output feat3 corresponds to the unlabeled data.

[0154] As an example, the cross-entropy loss function and the mean square error loss function are used for loss calculation of the discrimination output and the corresponding second label to determine the discrimination loss, so as to further improve the feature extraction capability of the trained teacher model. In this example, the discrimination loss is based on the loss of the training image in all the second training data, which not only includes the loss of the labeled data, but also includes the loss of the unlabeled data. Therefore, the model performance can be improved by using a large amount of unlabeled data in the case of limited labeled data.

[0155] In an embodiment, step S405, that is, determining the teacher model based on the classification loss and the discrimination loss, includes:

[0156] The second model loss is determined based on the classification loss and the discrimination loss.

[0157] In response to the second model loss satisfying a second convergence condition, the teacher model is determined as a trained teacher model.

[0158] As an example, after each model parameter iteration update of the teacher model, the training device can obtain the corresponding classification loss and discrimination loss, and then determine the sum of the classification loss and the discrimination loss as the second model loss corresponding to the teacher model. Then, the second model loss is compared with a second loss threshold set in advance. If the second model loss is less than the second loss threshold, it is determined that the model performance of the teacher model at this time meets the preset requirement, and therefore the teacher model can be determined as a trained teacher model. The second loss threshold is a loss threshold set in advance for evaluating whether the teacher model converges, which can be the same as or different from the first loss threshold.

[0159] In an embodiment, as shown in FIG. 5, after step S402, after the training image is input to the teacher model for processing to obtain the third segmentation output and the third feature output corresponding to the teacher model, the semantic segmentation model deployment method further includes:

[0160] S501: determining a plurality of third pseudo labels based on the third segmentation output corresponding to the training image;

[0161] S502: performing similarity calculation on the plurality of third pseudo labels to determine a label similarity;

[0162] S503: in response to the label similarity being greater than a preset similarity, determining a first pseudo label and a reliable pixel template based on the plurality of third pseudo labels;

[0163] S504: determining pseudo label data based on the training image, the first pseudo label and the reliable pixel template.

[0164] As an example, in step S501, during the training of the teacher model, the teacher model processes the input training image (which can be a label-free image) to determine the third segmentation output prob3 and the third feature output feat3. The softmax function can be used to process the third segmentation output prob3 to determine the prediction probability of each pixel point (i, j) for each class c The third pseudo label corresponding to the training image can be determined based on all prediction probabilities ​The pseudo label class corresponding to each pixel point (i, j). Since the teacher model needs to be updated multiple times, the model parameters are saved after each update, and the third pseudo label corresponding to the training image after this update is obtained. In this example, if the teacher model needs to be updated N times, the teacher model can generate a third pseudo label at least once in N times during the training and updating process, for example, after the second model loss Loss of the teacher model reaches a certain condition and appears to be relatively stable, M times are selected for the generation of the third pseudo label. The third pseudo label can be generated during the training of the teacher model, or the model parameters at different stages can be saved, and after the training is completed, M parameters at different stages are used to infer M times to generate the third pseudo label.

[0165] As an example, in step S502, after determining the N third pseudo labels, the training device can use a similarity algorithm to calculate the similarity of the plurality of third pseudo labels to determine the label similarity, for example, but not limited to, the IoU function can be used to calculate the similarity of the plurality of third pseudo labels to determine the label similarity. IoU is the intersection over union function.

[0166] As an example, in step S503, after calculating the label similarity, the training device can compare the label similarity with the preset similarity, and in response to the label similarity being greater than the preset similarity, it is determined that the training image meets the preset condition for evaluating whether to generate a pseudo label. At this time, based on the N third pseudo labels, the first pseudo label used for model training can be generated and reliable pixel templates On the contrary, if the label similarity is not greater than the preset similarity, the training image can be determined as a no-label image in the no-label data.

[0167] As an example, in step S504, after determining the first pseudo label and reliable pixel templates , the training device can determine the training image, the first pseudo label and reliable pixel templates as pseudo label data, so as to train the student model based on the pseudo label data, that is, execute steps S101-S106 to train the student model.

[0168] In this example, after determining the pseudo-label data, semi-supervised learning can be performed based on the labeled data and the pseudo-label data to update the model parameters of the teacher model, and the update process is shown as S401-S405, so as to improve the model performance of the teacher model. Alternatively, semi-supervised learning can be performed based on the labeled data, the pseudo-label data, and the unlabeled data to update the model parameters of the student model, and the update process is shown as S101-S106, so as to improve the model performance of the student model. In this example, the pseudo-label data can be determined according to the model parameters saved during the training of the teacher model, and the automatic generation of the pseudo-label data can be realized, so as to update the model parameters by using the pseudo-label data subsequently, which is helpful to improve the model training efficiency and the model performance.

[0169] In an embodiment, as shown in FIG. 6, step S503, i.e., determining the first pseudo-label and the reliable pixel template based on the plurality of third pseudo-labels, includes:

[0170] S601: determining, based on each third pseudo-label, a predicted class corresponding to each pixel point in the third pseudo-label;

[0171] S602: determining the first pseudo-label based on the pseudo-label classes of all pixel points, the pseudo-label class being the predicted class with the largest number in the plurality of predicted classes corresponding to the same pixel point;

[0172] S603: performing reliability analysis on the predicted classes corresponding to each pixel point in the plurality of third pseudo-labels to determine the reliable pixel template.

[0173] As an example, in step S601, after obtaining the N third pseudo-labels corresponding to the same training image, the training device can determine the predicted class c corresponding to each pixel point in the training image based on each third pseudo-label.

[0174] As an example, in step S602, the training device counts the number of the plurality of predicted classes corresponding to the same pixel point from the obtained N third pseudo-labels to determine the number of each predicted class corresponding to the same pixel point, and determines the predicted class with the largest number as the pseudo-label class corresponding to the pixel point; and determines the first pseudo-label corresponding to the training image based on the pseudo-label classes corresponding to all pixel points, i.e., is the pseudo-label class corresponding to the pixel point (i, j) in the first pseudo-label, and The prediction categories corresponding to the pixel point (i, j) in the first, second, and Nth third pseudo labels, respectively, and maxvote is a function of selecting the most number. In this example, among the multiple prediction categories of the same pixel point in the N third pseudo labels, the prediction category with the most number is determined as the pseudo label category, which helps to ensure the accuracy of the first pseudo label determined based on the pseudo label category, and helps to ensure the accuracy of the model obtained by subsequent training.

[0175] As an example, in step S603, the training device determines the prediction category c corresponding to each pixel point in the training image when determining each third pseudo label. The reliability of the prediction category c of a certain pixel point can be evaluated by analyzing whether the prediction categories c corresponding to the same pixel point in the N third pseudo labels are the same, and then determining whether the pixel point is a first reliable pixel point or an unreliable pixel point. Based on all the first reliable pixel points and the unreliable pixel points, a reliable pixel template is determined. So as to determine the reliable pixel template based on the reliable pixel template Reflect the reliability of the pixel point, and further ensure the model performance of the trained model.

[0176] In an embodiment, as shown in FIG. 7, step S603, i.e., based on the prediction categories corresponding to each pixel point in the plurality of third pseudo labels, performing reliability analysis to determine a reliable pixel template, includes:

[0177] S701: Based on the plurality of prediction categories, determine a plurality of category templates corresponding to each third pseudo label, each category template corresponding to a prediction category;

[0178] S702: Taking the intersection of the same category template in the plurality of third pseudo labels, determine the first reliable pixel point corresponding to the category template;

[0179] S703: Taking the union of the first reliable pixel points corresponding to the plurality of category templates, determine the reliable pixel template.

[0180] As an example, in step S701, the training device divides the N third pseudo labels obtained into C category templates according to the different prediction categories, and the value of the pixel point belonging to the category in each category template is 1, and the value of the pixel point not belonging to the category is 0, so as to form C category templates corresponding to the same third pseudo label. That is, each third pseudo label of the N third pseudo labels will form C category templates, and a total of N*C category templates are formed. The category here can be a certain category identified in the semantic segmentation process, such as a car, a road, and other categories.

[0181] As an example, in step S702, the training device can take the intersection of the same category template based on the same pixel point from the N*C category templates, take the intersection of the N category templates corresponding to any category based on the pixel point, if the value of the same pixel point in the N category templates is 1, it is determined that the pixel point is the first reliable pixel point of the corresponding category, otherwise, if the value of the same pixel point in the N category templates is not all 1, it is determined that it is an unreliable pixel point.

[0182] As an example, in step S703, after determining the first reliable pixel point corresponding to any category, the training device can take the union of all first reliable pixel points corresponding to C categories to determine all first reliable pixel points corresponding to all predicted categories of the training image, and the remaining pixel points are determined as unreliable pixel points, thereby obtaining the reliable pixel template corresponding to the training image, that is The category template of the category c generated for the pth third pseudo label.

[0183] In an embodiment, a semantic recognition method is provided, as shown in FIG. 9. This method is applied to an edge device that deploys a student model trained by the above-mentioned embodiments. The steps include:

[0184] S901: Obtain an image to be recognized;

[0185] S902: Use the trained student model deployed by the above-mentioned semantic segmentation model deployment method to recognize the image to be recognized, and obtain the semantic segmentation output corresponding to the image to be recognized.

[0186] Wherein, the image to be recognized is an image that needs to be recognized.

[0187] As an example, the edge device has deployed the student model trained by the above-mentioned embodiments. The edge device can obtain the image to be recognized through a pre-set interface or communication method. Then, the above-mentioned student model trained by the above-mentioned embodiments is used to perform semantic segmentation recognition on the image to be recognized, and determine the semantic segmentation output corresponding to the image to be recognized.

[0188] In this example, the trained student model deployed on the edge device is trained by the teacher model in a large computing power training device, so that the training process of the student model is not constrained by computing power, which helps to ensure the model performance of the trained student model, and also ensures the accuracy of the semantic segmentation recognition performed by the trained student model in the edge device.

[0189] It should be understood that the size of the serial number of each step in the above-mentioned embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0190] In an embodiment, a computer device is provided, which can be a training device, comprising a memory, a processor, and computer instructions stored on the memory and executable on the processor, the processor implementing the semantic segmentation model deployment method in the above embodiments, such as S101-S106 shown in FIG. 1, or shown in FIGS. 2-7, which will not be repeated here.

[0191] In an embodiment, a computer device is provided, which can be an edge device, comprising a memory, a processor, and computer instructions stored on the memory and executable on the processor, the processor implementing the semantic recognition method in the above embodiments, such as S901-S902 shown in FIG. 9, which will not be repeated here.

[0192] In an embodiment, a computer readable storage medium is provided, which stores computer instructions, the computer instructions being executed by a processor to implement the semantic segmentation model deployment method in the above embodiments, such as S101-S106 shown in FIG. 1, or shown in FIGS. 2-7, which will not be repeated here, or the computer instructions being executed by a processor to implement the semantic recognition method in the above embodiments, such as S901-S902 shown in FIG. 9, which will not be repeated here.

[0193] A person of ordinary skill in the art can understand that all or part of the processes in the above embodiments can be completed by computer instructions instructing related hardware, and the computer instructions can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above embodiments. Any reference to memory, storage, database or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0194] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional component, device is exemplified, and in actual application, the above-mentioned functional distribution can be completed by different functional devices or devices according to needs, that is, the internal structure of the device is divided into different functional devices or devices to complete all or part of the functions described above.

[0195] The above examples are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that the technical solutions recorded in the foregoing examples can still be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A semantic segmentation model deployment method, wherein, The method comprises: obtaining first training data, the first training data comprising training images and first labels corresponding to the training images; inputting the training images into a pre-trained teacher model and two student models to obtain first segmentation outputs and first feature outputs of the teacher model, and second segmentation outputs and second feature outputs of the two student models; determining a target distillation loss based on the first feature outputs and the two second feature outputs; determining a consistency loss based on the two second segmentation outputs; determining a target supervision loss based on the first labels and the two second segmentation outputs; and determining a trained student model based on the target distillation loss, the consistency loss and the target supervision loss, and deploying the trained student model on an edge device. The inputting the training images into the pre-trained teacher model and the two student models to obtain the first segmentation outputs and the first feature outputs of the teacher model, and the second segmentation outputs and the second feature outputs of the two student models comprises: 2.The semantic segmentation model deployment method of claim 1, wherein, performing enhancement processing on the training images to obtain a first enhanced image and two second enhanced images; inputting the first enhanced image into the teacher model to obtain the first segmentation outputs and the first feature outputs of the teacher model; and inputting the two second enhanced images into the two student models respectively to obtain the second segmentation outputs and the second feature outputs of the two student models respectively. The performing enhancement processing on the training images to obtain the first enhanced image and the two second enhanced images comprises: 3.The semantic segmentation model deployment method of claim 2, wherein, performing weak enhancement on the training images using a first enhancement rule to obtain the first enhanced image; performing strong enhancement on the first enhanced image using a second enhancement rule to obtain the two second enhanced images respectively. The first enhancement rule comprises at least one of a random flip rule, an image scaling rule and a random shift rule; 4. The semantic segmentation model deployment method of claim 3, wherein, The second enhancement rule comprises at least one of a color flip rule, a contrast adjustment rule and an image blurring rule. The target distillation loss is determined based on first distillation losses corresponding to the two student models. 5.The semantic segmentation model deployment method of any of claims 1-4, wherein, The first distillation loss corresponding to each student model is determined based on the first feature outputs and the second feature outputs corresponding to the student model. The target distillation loss is an arithmetic mean of the first distillation losses corresponding to the two student models. 6.The semantic segmentation model deployment method of claim 5, wherein, The target supervision loss is determined based on first supervision losses corresponding to the two student models.

7. The semantic segmentation model deployment method of any one of claims 1 to 6, wherein, The first supervision loss corresponding to each student model is determined based on the first labels and the second segmentation outputs corresponding to the student model.

8. The semantic segmentation model deployment method according to claim 7, wherein The target supervision loss is an arithmetic mean of the first supervision losses corresponding to the two student models. The first training data comprises labeled data, and the first labels comprise true labels in the labeled data. 9.The semantic segmentation model deployment method of claim 7, wherein, The first supervision loss corresponding to each student model is determined based on a labeled data loss corresponding to the student model. ​ The labeled data loss corresponding to each student model is determined based on the real label and the second segmentation output corresponding to the student model.

10. The semantic segmentation model deployment method of claim 9, wherein, The labeled data loss corresponding to each student model is further determined based on a target weight corresponding to the student model. The target weight corresponding to each student model is determined based on the second segmentation output corresponding to the student model.

11. The semantic segmentation model deployment method of claim 9, wherein, The target weight corresponding to each of the student models is The labeled data loss corresponding to each of the student models is where w i,j is the target weight for each pixel point (i,j); H is the prediction probability for each pixel point (i,j) being of class c determined from the second segmentation output i,j H is the probability entropy for each pixel point (i,j); Let D be a labeled dataset L s is the number of labeled data in the labeled dataset D L s is the number of labeled data in the labeled dataset D The real class of each pixel point (i, j) is determined according to the real label in the labeled data. 12.The semantic segmentation model deployment method of claim 9, wherein, The first training data further comprises pseudo-label data, and the first label further comprises a first pseudo-label in the pseudo-label data; The first supervision loss corresponding to each student model is further determined based on a pseudo-label data loss corresponding to the student model; The pseudo-label data loss corresponding to each student model is determined based on the first pseudo-label and the second segmentation output corresponding to the student model. 13.The semantic segmentation model deployment method of claim 12, wherein, The pseudo-label data further comprises a reliable pixel template, and the reliable pixel template is used to determine a first reliable pixel point; The pseudo label data loss corresponding to each student model is determined based on the first pseudo label corresponding to the first reliable pixel point and the second segmentation output corresponding to the student model, and the pseudo label data loss is as follows: wherein for pseudo-label data loss, For the number of pseudo-label data in the pseudo-label data set D PL s is any pseudo-label data in the pseudo-label data set D PL Mask i,j refers to the value in the reliable pixel template, Mask i,j = 1 refers to the first reliable pixel point. 14.The semantic segmentation model deployment method of claim 13, wherein, The reliable pixel template is further used to determine an unreliable pixel point; The first supervision loss corresponding to each student model is further determined based on a contrastive learning loss corresponding to the student model; The contrastive learning loss corresponding to each student model is determined based on a first positive sample feature, a second positive sample feature and a negative sample feature, the first positive sample feature being a feature corresponding to the reliable pixel point in the first feature output, the second positive sample feature being a feature corresponding to the reliable pixel point in the second feature output corresponding to the student model, and the negative sample feature being a feature corresponding to the unreliable pixel point in the second feature output corresponding to the student model. 15.The semantic segmentation model deployment method of claim 14, wherein, The contrastive learning loss corresponding to each of the student models is as follows: wherein L contra is the contrastive student loss, M is the number of positive samples for each class, N is the number of negative samples corresponding to each positive sample point, C is the total number of classes of semantic segmentation, and r is a constant, a mean of a plurality of first positive sample features corresponding to the class c determined based on the first feature output feat1 of the teacher model, c,i a mean of a plurality of second positive sample features corresponding to the class c determined based on the second feature output feat2 of the student model, The mean of a plurality of negative sample features corresponding to a class c determined based on the second feature output feat2 of the student model.

16. The semantic segmentation model deployment method of any one of claims 9-15, wherein, The first training data further comprises unlabeled data; The first supervision loss corresponding to each student model is further determined based on an unlabeled data loss corresponding to the student model; The unlabeled data loss corresponding to each student model is determined based on a second pseudo-label and the second segmentation output corresponding to the student model, the second pseudo-label being determined based on the first segmentation output.

17. The semantic segmentation model deployment method of claim 16, wherein, The unlabeled data loss corresponding to each student model is determined based on a second pseudo-label corresponding to a second reliable pixel point and the second segmentation output corresponding to the student model; The second reliable pixel point is a pixel point whose predicted probability determined based on the first segmentation output is greater than a preset threshold.

18. The semantic segmentation model deployment method of claim 17, wherein, The loss of unlabeled data corresponding to each student model is as follows: wherein for no-tag data loss, wherein the number of unlabeled data in the unlabeled dataset D U s is any unlabeled data in the unlabeled dataset D U Valid ij is a second reliable pixel point, and T is a preset threshold value. a second pseudo label determined based on the first segmentation output prob1; The predicted probability of each pixel point (i, j) for each class c.

19. The semantic segmentation model deployment method of any one of claims 1 to 18, wherein, Based on the target distillation loss, the consistency loss and the target supervision loss, a trained student model is determined, comprising: A first model loss is determined based on the target distillation loss, the consistency loss and the target supervision loss; In response to the first model loss satisfying a first convergence condition, the student model is determined as the trained student model.

20. The semantic segmentation model deployment method of any one of claims 1 to 19, wherein, Before the first training data is acquired, the semantic segmentation model deployment method further comprises: acquiring second training data, the second training data comprising training images and second labels corresponding to the training images; inputting the training images into a teacher model for processing to acquire third segmentation outputs and third feature outputs corresponding to the teacher model; determining a classification loss based on the second labels and the third segmentation outputs; determining a discrimination loss based on the third feature outputs; and determining the teacher model based on the classification loss and the discrimination loss.

21. The semantic segmentation model deployment method of claim 20, wherein, The determining of the discrimination loss based on the third feature outputs comprises: processing the third feature outputs using a discriminator network to acquire discrimination outputs; performing loss calculation based on the second labels and the discrimination outputs to determine the discrimination loss.

22. The semantic segmentation model deployment method of claim 20, wherein, The determining of the teacher model based on the classification loss and the discrimination loss comprises: determining a second model loss based on the classification loss and the discrimination loss; in response to the second model loss satisfying a second convergence condition, determining the teacher model as a trained teacher model.

23. The semantic segmentation model deployment method of claim 20, wherein, After the inputting of the training images into the teacher model for processing to acquire the third segmentation outputs and the third feature outputs corresponding to the teacher model, the semantic segmentation model deployment method further comprises: determining a plurality of third pseudo labels based on the third segmentation outputs corresponding to the training images; performing similarity calculation on the plurality of third pseudo labels to determine label similarity; in response to the label similarity being greater than a preset similarity, determining a first pseudo label and a reliable pixel template based on the plurality of third pseudo labels; and determining pseudo label data based on the training images, the first pseudo label, and the reliable pixel template. The determining of the first pseudo label and the reliable pixel template based on the plurality of third pseudo labels comprises:

24. The semantic segmentation model deployment method of claim 23, wherein, determining, based on each third pseudo label, a predicted class corresponding to each pixel point in the third pseudo label; determining the first pseudo label based on pseudo label classes of all the pixel points, the pseudo label class being a predicted class with the largest quantity among a plurality of predicted classes corresponding to a same pixel point; and determining a reliable pixel template based on reliability analysis of the predicted classes corresponding to each pixel point in the plurality of third pseudo labels. The determining of the reliable pixel template based on the reliability analysis of the predicted classes corresponding to each pixel point in the plurality of third pseudo labels comprises:

25. The semantic segmentation model deployment method of claim 24, wherein, determining, based on the plurality of predicted classes, a plurality of class templates corresponding to each third pseudo label, each class template corresponding to a predicted class; determining first reliable pixel points corresponding to a same class template by taking an intersection of the same class template in the plurality of third pseudo labels; and determining the reliable pixel template by taking a union of the first reliable pixel points corresponding to the plurality of class templates. The method comprises:

26. A method of semantic recognition, wherein, acquiring a to-be-identified image; identifying the to-be-identified image using the trained student model deployed by the semantic segmentation model deployment method according to any one of claims 1 to 25 to acquire a semantic segmentation output corresponding to the to-be-identified image. ​ 27. A computer device comprising a memory, a processor, and computer instructions stored in the memory and executable on the processor, wherein, The computer instructions are executed by the processor to implement the semantic segmentation model deployment method according to any one of claims 1-25, or the semantic recognition method according to claim 26. 28.A computer readable storage medium applied to the semantic segmentation model deployment method of claim 1 or the semantic recognition method of claim 26, the computer readable storage medium storing computer instructions, wherein, The computer instructions are executed by the processor to implement the semantic segmentation model deployment method according to any one of claims 1-25, or the semantic recognition method according to claim 26.

Images