Semantic segmentation model training method, device, electronic device and storage medium

Through the image training set and segmentation loss function based on sparse point annotation, the semantic segmentation model is trained to generate dense pixel-level pseudo-notations, and the use of contrast learning to constrain training on the features of the intermediate layer is solved, and the problem of high labeling cost in semantic segmentation model training is solved. , achieving better performance of weak supervision model.

CN113569852BActive Publication Date: 2024-11-12INST OF AUTOMATION CHINESE ACAD OF SCI
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110643779.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-06-09
Publication Date
2024-11-12
Estimated Expiration
2041-06-09

AI Technical Summary

Technical Problem

The existing semantic segmentation model training relies on a large amount of manually labeled pixel-level label data, which leads to high costs. Weak supervised labeling methods such as image-level category labeling and target box labeling are expensive, making it difficult to effectively train the semantic segmentation model.

Method used

Using an image training set based on sparse point annotation, dense pixel-level pseudo-labels are generated through multiple updates of the semantic segmentation model. Constrained training is applied on the intermediate layer features by using contrast learning to make up for the shortcomings of less supervision information of sparse point annotation, and generate Dense pixel-level pseudo-notations are used to supervise model training.

Benefits of technology

At the smallest annotation cost, we can obtain relatively good weak supervised model performance, balance the annotation cost and model performance, and fully explore and utilize the implicit information in sparse point annotation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113569852B_ABST
    Figure CN113569852B_ABST
Patent Text Reader

Abstract

The embodiment of the present application discloses a training method, device, electronic device and storage medium for a semantic segmentation model, the method comprising: at the sparse point annotation position in the semantic segmentation result, based on the first loss function of the semantic segmentation model, updating the model parameters of the semantic segmentation model for the first time; based on the semantic segmentation result and the sparse point label corresponding to the image, obtaining the dense pixel-level pseudo-label corresponding to the image, and based on the dense pixel-level pseudo-label and the second loss function of the semantic segmentation model, updating the model parameters of the semantic segmentation model for the second time; based on the feature data of the image, the dense pixel-level pseudo-label and the third loss function of the semantic segmentation model, updating the model parameters of the semantic segmentation model for the third time. The embodiment of the present application makes full use of the information implicit in the sparse point annotation to train the deep model, so as to obtain better weak supervision model performance at the lowest possible annotation cost.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and specifically to a training method, device, electronic device and storage medium for a semantic segmentation model. Background Art

[0002] Semantic segmentation is to divide the input image into multiple regions with semantic information to achieve perceptual understanding of the visual scene. It is a very valuable and difficult computer vision task.

[0003] Traditional semantic segmentation model training relies on a large amount of manually annotated pixel-level label data. In order to alleviate the high cost of pixel-level label data, weakly supervised semantic segmentation models attempt to train semantic segmentation models using only coarse weakly supervised annotations. Common forms of weakly supervised annotations include image-level category annotations, object box annotations, etc.

[0004] However, the existing image-level category labeling method requires more target spatial position information, which is not conducive to the training of semantic segmentation models. The target box labeling method requires multiple position labeling on targets of the same category, which has a high labeling cost. Summary of the invention

[0005] Since the existing methods have the above-mentioned problems, the embodiments of the present application provide a training method, device, electronic device and storage medium for a semantic segmentation model.

[0006] Specifically, the embodiments of the present application provide the following technical solutions:

[0007] In a first aspect, an embodiment of the present application provides a method for training a semantic segmentation model, comprising:

[0008] Acquire an image training set based on sparse point annotation, wherein the image training set includes images and their corresponding sparse point labels, and the sparse point labels corresponding to any image include two-dimensional coordinates of the sparse points and categories of the sparse points;

[0009] Inputting the images in the image training set into the semantic segmentation model to obtain the semantic segmentation results and feature data of the images;

[0010] At the sparse point marking position in the semantic segmentation result, based on the first loss function of the semantic segmentation model, updating the model parameters of the semantic segmentation model for the first time;

[0011] Based on the semantic segmentation result and the sparse point label corresponding to the image, a dense pixel-level pseudo label corresponding to the image is obtained, and based on the dense pixel-level pseudo label and a second loss function of the semantic segmentation model, a model parameter of the semantic segmentation model is updated for a second time;

[0012] Based on the feature data of the image, the dense pixel-level pseudo-label and a third loss function of the semantic segmentation model, the model parameters of the semantic segmentation model are updated for a third time.

[0013] Optionally, the semantic segmentation model includes a basic network model f=F(I; θ f ), used to map the input image I to the intermediate layer feature f, where θ f is the learnable model parameter of the basic network model, F is the basic network model; the basic network model is connected to a semantic segmentation branch model s=S(f; θ s ), used to map the intermediate layer feature f to the semantic segmentation result s, where θ s is the learnable model parameter of the semantic segmentation branch model, S is the semantic segmentation branch model; the basic network model is also connected to a feature embedding branch model e=E(f; θ e ), used to map the intermediate layer feature f to the embedded feature space for intermediate layer feature contrast learning, θ e are the learnable model parameters of the feature embedding branch model, and E is the feature embedding branch model.

[0014] Optionally, at the sparse point marking position in the semantic segmentation result, based on the first loss function of the semantic segmentation model, updating the model parameters of the semantic segmentation model for the first time includes:

[0015] The sparse point marking position in the semantic segmentation result is trained based on the first loss function of the semantic segmentation model, and the model parameter θ of the semantic segmentation model is updated. f and θ s ;

[0016] Among them, the first loss function L pnt (θ s ,θ f ; 1) is:

[0017]

[0018] Among them, Y is the sparse point label |Y| is the number of sparse point labels, (h k , w k ) is the two-dimensional coordinate of the sparse point, L k is the category of sparse points, The semantic segmentation model predicts the image at sparse points (h k , w k ) The category of the position is L k The probability of k refers to the kth sparse point label.

[0019] Optionally, updating the model parameters of the semantic segmentation model for the second time based on the dense pixel-level pseudo-label and the second loss function of the semantic segmentation model includes:

[0020] Training is performed based on the dense pixel-level pseudo-label and the second loss function of the semantic segmentation model, and the model parameters θ of the semantic segmentation model are updated for the second time. f and θ s ;

[0021] Among them, the second loss function L dse (θ s ,θ f ; 1) is:

[0022]

[0023] Among them, Y′ is a dense pixel-level pseudo label, |Y′| is the number of dense pixel-level pseudo labels, (h k , w k ) is the two-dimensional coordinate of the pixel point, L k is the category of the pixel, The semantic segmentation model predicts the image at the pixel point (h k , w k ) The category of the position is L k The probability of k is k, and k refers to the kth dense pixel-level pseudo label.

[0024] Optionally, updating the model parameters of the semantic segmentation model for the third time based on the feature data of the image, the dense pixel-level pseudo-label, and a third loss function of the semantic segmentation model includes:

[0025] Training is performed based on the feature data of the image, the dense pixel-level pseudo-labels and the third loss function of the semantic segmentation model, and the model parameter θ of the semantic segmentation model is updated for the third time. f and θ e ;

[0026] Among them, the third loss function L cl (θ e ,θ f ; 1) is:

[0027]

[0028] Where H and W are the height and width of the semantic segmentation result image output by the model, respectively, and e is the feature embedding expression of the semantic segmentation model. is the positive sample pair of features at each dense pixel-level pseudo-label position i in the semantic segmentation result image, e i is the feature embedding expression corresponding to the f-th pixel position, For the corresponding e i A set of negative sample pairs, each of which has and e i Different categories, for The elements in for e i is the transpose of , τ is a preset temperature parameter used to adjust the sensitivity of the third loss function.

[0029] In a second aspect, an embodiment of the present application provides a training device for a semantic segmentation model, comprising:

[0030] An acquisition module is used to acquire an image training set based on sparse point annotation, wherein the image training set includes images and their corresponding sparse point labels, and the sparse point labels corresponding to any image include the two-dimensional coordinates of the sparse points and the categories of the sparse points;

[0031] A first processing module, used for inputting the images in the image training set into a semantic segmentation model to obtain semantic segmentation results and feature data of the images;

[0032] A second processing module is used to mark the sparse point positions in the semantic segmentation result and update the model parameters of the semantic segmentation model for the first time based on the first loss function of the semantic segmentation model;

[0033] A third processing module is used to obtain a dense pixel-level pseudo-label corresponding to the image based on the semantic segmentation result and the sparse point label corresponding to the image, and update the model parameters of the semantic segmentation model for the second time based on the dense pixel-level pseudo-label and the second loss function of the semantic segmentation model;

[0034] The fourth processing module is used to update the model parameters of the semantic segmentation model for the third time based on the feature data of the image, the dense pixel-level pseudo-label and the third loss function of the semantic segmentation model.

[0035] Optionally, the semantic segmentation model includes a basic network model f=F(I; θ f ), used to map the input image I to the intermediate layer feature f, where θ f is the learnable model parameter of the basic network model, F is the basic network model; the basic network model is connected to a semantic segmentation branch model s=S(f; θ s ), used to map the intermediate layer feature f to the semantic segmentation result s, where θ s is the learnable model parameter of the semantic segmentation branch model, S is the semantic segmentation branch model; the basic network model is also connected to a feature embedding branch model e=E(f; θ e), used to map the intermediate layer feature f to the embedded feature space for intermediate layer feature contrast learning, θ e are the learnable model parameters of the feature embedding branch model, and E is the feature embedding branch model.

[0036] Optionally, the second processing module is specifically used to:

[0037] The sparse point marking position in the semantic segmentation result is trained based on the first loss function of the semantic segmentation model, and the model parameter θ of the semantic segmentation model is updated. f and θ s ;

[0038] Among them, the first loss function L pnt (θ s ,θ f ; 1) is:

[0039]

[0040] Among them, Y is the sparse point label |Y| is the number of sparse point labels, (h k , w k ) is the two-dimensional coordinate of the sparse point, L k is the category of sparse points, The semantic segmentation model predicts the image at sparse points (h k , w k ) The category of the position is L k The probability of k refers to the kth sparse point label.

[0041] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the training method for the semantic segmentation model as described in the first aspect is implemented.

[0042] In a fourth aspect, an embodiment of the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for training a semantic segmentation model as described in the first aspect.

[0043] It can be seen from the above technical scheme that the embodiment of the present application first obtains an image training set based on sparse point annotation, and then inputs the image in the image training set into the semantic segmentation model to obtain the semantic segmentation result and feature data of the image, so that at the sparse point annotation position in the semantic segmentation result, based on the first loss function of the semantic segmentation model, the model parameters of the semantic segmentation model are updated for the first time, and, based on the semantic segmentation result and the sparse point label corresponding to the image, the dense pixel-level pseudo-label corresponding to the image is obtained, and based on the dense pixel-level pseudo-label and the second loss function of the semantic segmentation model, the model parameters of the semantic segmentation model are updated for the second time, and based on the feature data of the image, the dense pixel-level pseudo-label and the third loss function of the semantic segmentation model, the model parameters of the semantic segmentation model are updated for the third time. It can be seen from this that in the process of training the semantic segmentation model based on the sparse point annotated image training set and the segmentation loss function, the embodiment of the present application generates dense pixel-level pseudo-annotations according to the online prediction results of the semantic segmentation model to supervise the training of the semantic segmentation model. In addition, the embodiment of the present application also performs constrained training on the semantic segmentation model on the intermediate layer features through methods such as contrastive learning, thereby making up for the disadvantage of the lack of supervised information in the sparse point annotations, and fully mining and utilizing the implicit information in the sparse point annotations to train the deep model, so as to achieve relatively good weak supervision model performance at the smallest possible annotation cost, thereby achieving a good balance between the annotation cost and the model performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0045] Figure 1 This is one of the step flow charts of the training method of the semantic segmentation model provided in the embodiment of the present application;

[0046] Figure 2 This is the second step flow chart of the method for training a semantic segmentation model provided in an embodiment of the present application;

[0047] Figure 3 is a schematic diagram of a sparse point annotated image provided in an embodiment of the present application;

[0048] Figure 4 It is a schematic diagram of the training framework of the semantic segmentation model provided in the embodiment of the present application;

[0049] Figure 5It is a structural schematic diagram of a training device for a semantic segmentation model provided in an embodiment of the present application;

[0050] Figure 6 It is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0052] Figure 1 One of the flow charts of the steps of the training method of the semantic segmentation model provided in the embodiment of the present application is shown. Figure 2 This is the second step flow chart of the method for training a semantic segmentation model provided in an embodiment of the present application. Figure 3 is a schematic diagram of a sparse point annotated image provided in an embodiment of the present application, Figure 4 Schematic diagram of the training framework of the semantic segmentation model provided in the embodiment of the present application. Figures 1 to 4 The training method of the semantic segmentation model provided in the embodiment of the present application is explained and illustrated in detail. Figure 1 As shown, a method for training a semantic segmentation model provided in an embodiment of the present application includes:

[0053] Step 101: obtaining an image training set based on sparse point annotation, wherein the image training set includes images and their corresponding sparse point labels, and the sparse point labels corresponding to any image include the two-dimensional coordinates of the sparse points and the categories of the sparse points;

[0054] In this step, we first construct an image training set based on sparse point annotation for semantic segmentation model training, including images and their corresponding sparse point labels. Each sparse point label includes a two-dimensional coordinate to describe the position of the point and a category label to describe the category of the point, and an image can contain any number of sparse point labels.

[0055] Step 102: inputting the images in the image training set into a semantic segmentation model to obtain semantic segmentation results and feature data of the images;

[0056] In this step, a semantic segmentation model is first constructed, and the semantic segmentation model can adopt any deep model architecture based on gradient back propagation for parameter update and learning. Then, the images in the image training set are input into the semantic segmentation model to obtain the semantic segmentation results and feature data of the images.

[0057] Step 103: marking the sparse point positions in the semantic segmentation result, and updating the model parameters of the semantic segmentation model for the first time based on the first loss function of the semantic segmentation model;

[0058] In this step, the sparse point annotation positions in the semantic segmentation results are trained based on the first loss function of the semantic segmentation model to update the model parameters θ of the semantic segmentation model. f and θ s ;

[0059] Among them, the first loss function L pnt (θ s ,θ f ; 1) is:

[0060]

[0061] Among them, Y is the sparse point label |Y| is the number of sparse point labels, (h k , w k ) is the two-dimensional coordinate of the sparse point, L k is the category of sparse points, The semantic segmentation model predicts the image at sparse points (h k , w k ) The category of the position is L k The probability of k refers to the kth sparse point label.

[0062] Step 104: obtaining dense pixel-level pseudo labels corresponding to the image based on the semantic segmentation result and the sparse point labels corresponding to the image, and updating the model parameters of the semantic segmentation model for the second time based on the dense pixel-level pseudo labels and the second loss function of the semantic segmentation model;

[0063] In this step, training is performed based on the dense pixel-level pseudo-label and the second loss function of the semantic segmentation model, and the model parameters θ of the semantic segmentation model are updated for the second time. f and θ s ;

[0064] Among them, the second loss function L dse (θ s ,θ f ; 1) is:

[0065]

[0066] Among them, Y′ is a dense pixel-level pseudo label, |Y′| is the number of dense pixel-level pseudo labels, (h k , w k) is the two-dimensional coordinate of the pixel point, L k is the category of the pixel, The semantic segmentation model predicts the image at the pixel point (h k , w k ) The category of the position is L k The probability of k is k, and k refers to the kth dense pixel-level pseudo label.

[0067] Step 105: Based on the feature data of the image, the dense pixel-level pseudo-labels and the third loss function of the semantic segmentation model, the model parameters of the semantic segmentation model are updated for the third time.

[0068] In this step, training is performed based on the feature data of the image, the dense pixel-level pseudo-labels, and the third loss function of the semantic segmentation model, and the model parameters θ of the semantic segmentation model are updated for the third time. f and θ e ;

[0069] Among them, the third loss function L cl (θ e ,θ f ; 1) is:

[0070]

[0071] Where H and W are the height and width of the semantic segmentation result image output by the model, respectively, and e is the feature embedding expression of the semantic segmentation model. is the positive sample pair of features at each dense pixel-level pseudo-label position i in the semantic segmentation result image, e i is the feature embedding expression corresponding to the i-th pixel position, For the corresponding e i A set of negative sample pairs, each of which has and e i Different categories, for The elements in for e i is the transpose of , τ is a preset temperature parameter used to adjust the sensitivity of the third loss function.

[0072] It can be seen from the above technical scheme that the embodiment of the present application first obtains an image training set based on sparse point annotation, and then inputs the image in the image training set into the semantic segmentation model to obtain the semantic segmentation result and feature data of the image, so that at the sparse point annotation position in the semantic segmentation result, based on the first loss function of the semantic segmentation model, the model parameters of the semantic segmentation model are updated for the first time, and, based on the semantic segmentation result and the sparse point label corresponding to the image, the dense pixel-level pseudo-label corresponding to the image is obtained, and based on the dense pixel-level pseudo-label and the second loss function of the semantic segmentation model, the model parameters of the semantic segmentation model are updated for the second time, and based on the feature data of the image, the dense pixel-level pseudo-label and the third loss function of the semantic segmentation model, the model parameters of the semantic segmentation model are updated for the third time. It can be seen from this that in the process of training the semantic segmentation model based on the sparse point annotated image training set and the segmentation loss function, the embodiment of the present application generates dense pixel-level pseudo-annotations according to the online prediction results of the semantic segmentation model to supervise the training of the semantic segmentation model. In addition, the embodiment of the present application also performs constrained training on the semantic segmentation model on the intermediate layer features through methods such as contrastive learning, thereby making up for the disadvantage of the lack of supervised information in the sparse point annotations, and fully mining and utilizing the implicit information in the sparse point annotations to train the deep model, so as to achieve relatively good weak supervision model performance at the smallest possible annotation cost, thereby achieving a good balance between the annotation cost and the model performance.

[0073] Based on the content of the above embodiment, in this embodiment, the semantic segmentation model includes a basic network model f=F(I; θ f ), used to map the input image I to the intermediate layer feature f, where θ f is the learnable model parameter of the basic network model, F is the basic network model; the basic network model is connected to a semantic segmentation branch model s=S(f; θ s ), used to map the intermediate layer feature f to the semantic segmentation result s, where θ s is the learnable model parameter of the semantic segmentation branch model, S is the semantic segmentation branch model; the basic network model is also connected to a feature embedding branch model e=E(f; θ e ), used to map the intermediate layer feature f to the embedded feature space for intermediate layer feature contrast learning, θ e are the learnable model parameters of the feature embedding branch model, and E is the feature embedding branch model.

[0074] The present application is described in detail below through specific embodiments.

[0075] First embodiment:

[0076] like Figure 2As shown, a method for training a semantic segmentation model provided in an embodiment of the present application includes:

[0077] Step 201: Construct a sparse point labeled image training set for semantic segmentation model training, including image I and its corresponding sparse point label Where |Y| is the number of sparse point labels corresponding to the image. Each sparse point label f ​​includes a two-dimensional coordinate (h k , w k ) is used to describe the location of the point, and a category label L k Used to describe the category of the point. An image can contain more than one sparse point label of any number, such as Figure 3 As shown;

[0078] Step 202: construct a semantic segmentation model, which may adopt any deep model architecture based on gradient backpropagation for parameter update and learning. In this embodiment, the semantic segmentation model used for training includes a basic network model f = F(I; θ f ), used to map the input image I to the intermediate layer feature f, where θ f is the model parameter that can be learned; including a segmentation branch s = S(f; θ s ) is used to map the intermediate layer feature f to the segmentation prediction result s, where θ s is the model parameter that can be learned; including a feature embedding branch e=E(f;θ e ), which is used to map the intermediate layer feature f to the embedded feature space to implement intermediate layer feature contrast learning, etc. For the overall model structure, see Figure 4 ;

[0079] Step 203: For the semantic segmentation model constructed in step 202, the data image I in step 201 is input to obtain the prediction s of the semantic segmentation result. At the corresponding position with sparse point annotation, the model is trained using the segmentation loss function according to formula 1, and the model parameter θ is f and θ s Update. In formula 1, The model predicts that the image is in (h k , w k ) The category of the position is L k The probability of

[0080]

[0081] Step 204: Based on the output prediction s of the semantic segmentation model in step 202, dense pixel-level pseudo labels are generated online according to Formula 2, and the pseudo labels are combined with a segmentation loss function L similar to Formula 1. dsePerform model training. H and W are the height and width of the predicted image respectively;

[0082]

[0083] Step 205: Based on the intermediate feature expression e of the semantic segmentation model in step 202 and the pixel-level pseudo-annotation obtained in step 204, the model is trained using contrastive learning according to formula 3 to adjust the model parameters θ f and θ e To update. is a positive sample pair corresponding to the feature at each position i, with and e i The same category, For the corresponding e i A set of negative sample pairs, each of which has and e i Different categories. Participating in the loss function calculation and It can be obtained through various ways, including but not limited to sampling from the intermediate layer feature expression corresponding to the current image, or by reading the reference features corresponding to other processed images in the storage queue;

[0084]

[0085] Step 206: After completing the semantic segmentation model trained in the above steps, only the basic network model and the segmentation branch can be retained for testing and deployment of semantic segmentation tasks. Figure 2 .

[0086] Second embodiment:

[0087] In this embodiment, optionally, step 204 in the above embodiment can also calculate the super pixel point set according to the original input image I. Among them, |R| is the number of superpixels, and each superpixel r i is a containing |r i |A collection of pixels According to the output prediction s of the semantic segmentation model in step 202, the category of each superpixel is calculated according to Formula 4, and then a dense pixel-level pseudo-label is obtained. The pseudo-label is used in combination with the segmentation loss function to train the model, where Y′(r i ) is r i The category to which it belongs;

[0088]

[0089] In this embodiment, optionally, step 205 in the above embodiment can also be based on the intermediate layer feature expression e of the semantic segmentation model in step 202, combined with the pixel-level pseudo-annotation obtained in step 204, and use contrastive learning to perform model training according to formula 3 to train the model parameters θ f and θ e To update. and is the reference feature, is the expectation of embedding all features of the same category corresponding to a single input image, whose corresponding categories are the same as and different from e respectively. i Specifically, it corresponds to the reference feature with category c. It can be calculated according to Formula 5, where is the expected function. and It can be obtained through various ways, including but not limited to sampling from the intermediate layer feature expression corresponding to the current image, or by reading the reference features corresponding to other processed images in the storage queue;

[0090]

[0091] It can be seen that the embodiment of the present application trains the semantic segmentation model through weak supervision information of sparse point annotations; during the training process, dense pixel-level pseudo-labels are generated online to assist in training based on the model's predictions and input images and sparse point information; during the training process, the intermediate feature layer is subjected to constraint-assisted training using contrastive learning and other methods, and the finally trained model can be used for testing and deployment of semantic segmentation tasks, thereby solving the problem that traditionally training semantic segmentation models requires a large amount of costly dense pixel-level annotations. The embodiment of the present application can be widely used in data sets with various characteristics.

[0092] Based on the same inventive concept, another embodiment of the present invention provides a training device for a semantic segmentation model, such as Figure 5 As shown, the device comprises:

[0093] Acquisition module 1, used to acquire an image training set based on sparse point annotation, wherein the image training set includes images and their corresponding sparse point labels, and the sparse point labels corresponding to any image include the two-dimensional coordinates of the sparse points and the categories of the sparse points;

[0094] The first processing module 2 is used to input the image in the image training set into the semantic segmentation model to obtain the semantic segmentation result and feature data of the image;

[0095] A second processing module 3 is used to mark the sparse point positions in the semantic segmentation result and update the model parameters of the semantic segmentation model for the first time based on the first loss function of the semantic segmentation model;

[0096] A third processing module 4 is used to obtain a dense pixel-level pseudo-label corresponding to the image based on the semantic segmentation result and the sparse point label corresponding to the image, and update the model parameters of the semantic segmentation model for the second time based on the dense pixel-level pseudo-label and the second loss function of the semantic segmentation model;

[0097] The fourth processing module 5 is used to update the model parameters of the semantic segmentation model for the third time based on the feature data of the image, the dense pixel-level pseudo-label and the third loss function of the semantic segmentation model.

[0098] In this embodiment, a sparse point annotation-based image training set for semantic segmentation model training is first constructed, including images and their corresponding sparse point labels. Each sparse point label includes a two-dimensional coordinate to describe the position of the point and a category label to describe the category of the point, and an image can contain any number of sparse point labels.

[0099] In this embodiment, a semantic segmentation model is first constructed, and the semantic segmentation model can adopt any deep model architecture based on gradient back propagation for parameter updating and learning. Then, an image in the image training set is input into the semantic segmentation model to obtain the semantic segmentation result and feature data of the image.

[0100] In this embodiment, at the sparse point annotation position in the semantic segmentation result, training is performed based on the first loss function of the semantic segmentation model to update the model parameter θ of the semantic segmentation model. f and θ s ;

[0101] Among them, the first loss function L pnt (θ s ,θ f ; 1) is:

[0102]

[0103] Among them, Y is the sparse point label |Y| is the number of sparse point labels, (h k , w k ) is the two-dimensional coordinate of the sparse point, L k is the category of sparse points, The semantic segmentation model predicts the image at sparse points (h k , w k ) The category of the position is L kThe probability of , k is...

[0104] In this embodiment, training is performed based on the dense pixel-level pseudo-label and the second loss function of the semantic segmentation model, and the model parameter θ of the semantic segmentation model is updated for the second time. f and θ s ;

[0105] Among them, the second loss function L dse (θ s ,θ f ; 1) is:

[0106]

[0107] Among them, Y′ is a dense pixel-level pseudo label, |Y′| is the number of dense pixel-level pseudo labels, (h k , w k ) is the two-dimensional coordinate of the pixel point, L k is the category of the pixel, The semantic segmentation model predicts the image at the pixel point (h k , w k ) The category of the position is L k The probability of k is k, and k refers to the kth dense pixel-level pseudo label.

[0108] In this embodiment, training is performed based on the feature data of the image, the dense pixel-level pseudo-labels, and the third loss function of the semantic segmentation model, and the model parameter θ of the semantic segmentation model is updated for the third time. f and θ e ;

[0109] Among them, the third loss function L cl (θ e ,θ f ; 1) is:

[0110]

[0111] Where H and W are the height and width of the semantic segmentation result image output by the model, respectively, and e is the feature embedding expression of the semantic segmentation model. is the positive sample pair of features at each dense pixel-level pseudo-label position i in the semantic segmentation result image, e i is the feature embedding expression corresponding to the i-th pixel position, For the corresponding e i A set of negative sample pairs, each of which has and e i Different categories, for The elements in for ei is the transpose of , τ is a preset temperature parameter used to adjust the sensitivity of the third loss function.

[0112] It can be seen from the above technical scheme that the embodiment of the present application first obtains an image training set based on sparse point annotation, and then inputs the image in the image training set into the semantic segmentation model to obtain the semantic segmentation result and feature data of the image, so that at the sparse point annotation position in the semantic segmentation result, based on the first loss function of the semantic segmentation model, the model parameters of the semantic segmentation model are updated for the first time, and, based on the semantic segmentation result and the sparse point label corresponding to the image, the dense pixel-level pseudo-label corresponding to the image is obtained, and based on the dense pixel-level pseudo-label and the second loss function of the semantic segmentation model, the model parameters of the semantic segmentation model are updated for the second time, and based on the feature data of the image, the dense pixel-level pseudo-label and the third loss function of the semantic segmentation model, the model parameters of the semantic segmentation model are updated for the third time. It can be seen from this that in the process of training the semantic segmentation model based on the sparse point annotated image training set and the segmentation loss function, the embodiment of the present application generates dense pixel-level pseudo-annotations according to the online prediction results of the semantic segmentation model to supervise the training of the semantic segmentation model. In addition, the embodiment of the present application also performs constrained training on the semantic segmentation model on the intermediate layer features through methods such as contrastive learning, thereby making up for the disadvantage of the lack of supervised information in the sparse point annotations, and fully mining and utilizing the implicit information in the sparse point annotations to train the deep model, so as to achieve relatively good weak supervision model performance at the smallest possible annotation cost, thereby achieving a good balance between the annotation cost and the model performance.

[0113] The training device of the semantic segmentation model described in this embodiment can be used to execute the above method embodiments. Its principles and technical effects are similar and will not be repeated here.

[0114] Based on the same inventive concept, another embodiment of the present invention provides an electronic device, see Figure 6 The structural diagram of the electronic device specifically includes the following contents: a processor 601, a memory 602, a communication interface 603 and a communication bus 604;

[0115] The processor 601, the memory 602, and the communication interface 603 communicate with each other through the communication bus 604; the communication interface 603 is used to realize information transmission between various devices;

[0116] The processor 601 is used to call the computer program in the memory 602, and when the processor executes the computer program, all steps of the training method of the above-mentioned semantic segmentation model are implemented, for example: obtaining an image training set based on sparse point annotation, the image training set includes images and their corresponding sparse point labels, and the sparse point labels corresponding to any image include the two-dimensional coordinates of the sparse points and the categories of the sparse points; inputting the images in the image training set into the semantic segmentation model to obtain the semantic segmentation results and feature data of the images; at the sparse point annotation positions in the semantic segmentation results, based on the first loss function of the semantic segmentation model, updating the model parameters of the semantic segmentation model for the first time; based on the semantic segmentation results and the sparse point labels corresponding to the images, obtaining the dense pixel-level pseudo-labels corresponding to the images, and based on the dense pixel-level pseudo-labels and the second loss function of the semantic segmentation model, updating the model parameters of the semantic segmentation model for the second time; based on the feature data of the images, the dense pixel-level pseudo-labels and the third loss function of the semantic segmentation model, updating the model parameters of the semantic segmentation model for the third time.

[0117] Based on the same inventive concept, another embodiment of the present invention provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, all steps of the training method of the above-mentioned semantic segmentation model are implemented, for example: obtaining an image training set based on sparse point annotation, the image training set includes images and their corresponding sparse point labels, and the sparse point labels corresponding to any image include the two-dimensional coordinates of the sparse points and the categories of the sparse points; inputting the images in the image training set into the semantic segmentation model to obtain the semantic segmentation results and feature data of the images; at the sparse point annotation positions in the semantic segmentation results, based on the first loss function of the semantic segmentation model, updating the model parameters of the semantic segmentation model for the first time; based on the semantic segmentation results and the sparse point labels corresponding to the images, obtaining the dense pixel-level pseudo-labels corresponding to the images, and based on the dense pixel-level pseudo-labels and the second loss function of the semantic segmentation model, updating the model parameters of the semantic segmentation model for the second time; based on the feature data of the images, the dense pixel-level pseudo-labels and the third loss function of the semantic segmentation model, updating the model parameters of the semantic segmentation model for the third time.

[0118] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program code.

[0119] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the embodiments of the present invention. Those of ordinary skill in the art may understand and implement them without creative effort.

[0120] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus the necessary general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the training method of the semantic segmentation model described in each embodiment or some parts of the embodiment.

[0121] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A training method for a semantic segmentation model, It is characterized in that include: Acquire an image training set based on sparse point annotation, wherein the image training set includes images and their corresponding sparse point labels, and the sparse point labels corresponding to any image include two-dimensional coordinates of the sparse points and categories of the sparse points; Inputting the images in the image training set into the semantic segmentation model to obtain the semantic segmentation results and feature data of the images; At the sparse point marking position in the semantic segmentation result, based on the first loss function of the semantic segmentation model, updating the model parameters of the semantic segmentation model for the first time; Based on the semantic segmentation result and the sparse point label corresponding to the image, a dense pixel-level pseudo label corresponding to the image is obtained, and based on the dense pixel-level pseudo label and a second loss function of the semantic segmentation model, a model parameter of the semantic segmentation model is updated for a second time; Based on the feature data of the image, the dense pixel-level pseudo-label and a third loss function of the semantic segmentation model, updating the model parameters of the semantic segmentation model for a third time; The semantic segmentation model includes a basic network model f=F(I; θ f ), used to map the input image I to the intermediate layer feature f, where θ f is the learnable model parameter of the basic network model, F is the basic network model; the basic network model is connected to a semantic segmentation branch model s=S(f; θ s ), used to map the intermediate layer feature f to the semantic segmentation result s, where θ s is the learnable model parameter of the semantic segmentation branch model, S is the semantic segmentation branch model; the basic network model is also connected to a feature embedding branch model e=E(f; θ e ), used to map the intermediate layer feature f to the embedded feature space for intermediate layer feature contrast learning, θ e are the learnable model parameters of the feature embedding branch model, and E is the feature embedding branch model.

2. The method for training a semantic segmentation model according to claim 1, It is characterized in that At the sparse point marking position in the semantic segmentation result, based on the first loss function of the semantic segmentation model, the model parameters of the semantic segmentation model are updated for the first time, including: The sparse point marking position in the semantic segmentation result is trained based on the first loss function of the semantic segmentation model, and the model parameter θ of the semantic segmentation model is updated. f and θ s ; Among them, the first loss function L pnt (θ s ,θ f ; 1) is: Among them, Y is the sparse point label |Y| is the number of sparse point labels, (h k ,w k ) is the two-dimensional coordinate of the sparse point, L k is the category of sparse points, The semantic segmentation model predicts the image at sparse points (h k ,w k ) The category of the position is L k The probability of k refers to the kth sparse point label.

3. The method for training a semantic segmentation model according to claim 1, It is characterized in that Based on the dense pixel-level pseudo-label and the second loss function of the semantic segmentation model, updating the model parameters of the semantic segmentation model for the second time includes: Training is performed based on the dense pixel-level pseudo-label and the second loss function of the semantic segmentation model, and the model parameters θ of the semantic segmentation model are updated for the second time. f and θ s ; Among them, the second loss function L dse (θ s ,θ f ; 1) is: Among them, Y′ is a dense pixel-level pseudo label, |Y′| is the number of dense pixel-level pseudo labels, (h k ,w k ) is the two-dimensional coordinate of the pixel point, L k is the category of the pixel, The semantic segmentation model predicts the image at the pixel point (h k ,w k ) The category of the position is L k The probability of k is k, and k refers to the kth dense pixel-level pseudo label.

4. The method for training a semantic segmentation model according to claim 1, It is characterized in that Updating the model parameters of the semantic segmentation model for the third time based on the feature data of the image, the dense pixel-level pseudo-label, and a third loss function of the semantic segmentation model includes: Training is performed based on the feature data of the image, the dense pixel-level pseudo-labels and the third loss function of the semantic segmentation model, and the model parameter θ of the semantic segmentation model is updated for the third time. f and θ e ; Among them, the third loss function L cl (θ e ,θ f ; 1) is: Where H and W are the height and width of the semantic segmentation result image output by the model, respectively, and e is the feature embedding expression of the semantic segmentation model. is the positive sample pair of features at each dense pixel-level pseudo-label position i in the semantic segmentation result image, e i is the feature embedding expression corresponding to the i-th pixel position, For the corresponding e i A set of negative sample pairs, each of which has and e i Different categories, for The elements in for e i is the transpose of , τ is a preset temperature parameter used to adjust the sensitivity of the third loss function.

5. A training device for a semantic segmentation model, It is characterized in that include: An acquisition module is used to acquire an image training set based on sparse point annotation, wherein the image training set includes images and their corresponding sparse point labels, and the sparse point labels corresponding to any image include the two-dimensional coordinates of the sparse points and the categories of the sparse points; A first processing module, used for inputting the images in the image training set into a semantic segmentation model to obtain semantic segmentation results and feature data of the images; A second processing module is used to mark the sparse point positions in the semantic segmentation result and update the model parameters of the semantic segmentation model for the first time based on the first loss function of the semantic segmentation model; A third processing module is used to obtain a dense pixel-level pseudo-label corresponding to the image based on the semantic segmentation result and the sparse point label corresponding to the image, and update the model parameters of the semantic segmentation model for the second time based on the dense pixel-level pseudo-label and the second loss function of the semantic segmentation model; a fourth processing module, configured to update the model parameters of the semantic segmentation model for a third time based on the feature data of the image, the dense pixel-level pseudo-label and a third loss function of the semantic segmentation model; The semantic segmentation model includes a basic network model f=F(I; θ f ), used to map the input image I to the intermediate layer feature f, where θ f is the learnable model parameter of the basic network model, F is the basic network model; the basic network model is connected to a semantic segmentation branch model s=S(f; θ s ), used to map the intermediate layer feature f to the semantic segmentation result s, where θ s is the learnable model parameter of the semantic segmentation branch model, S is the semantic segmentation branch model; the basic network model is also connected to a feature embedding branch model e=E(f; θ e ), used to map the intermediate layer feature f to the embedded feature space for intermediate layer feature contrast learning, θ e are the learnable model parameters of the feature embedding branch model, and E is the feature embedding branch model.

6. The training device for the semantic segmentation model according to claim 5, It is characterized in that The second processing module is specifically used for: The sparse point marking position in the semantic segmentation result is trained based on the first loss function of the semantic segmentation model, and the model parameter θ of the semantic segmentation model is updated. f and θ s ; Among them, the first loss function L pnt (θ s ,θ f ; 1) is: Among them, Y is the sparse point label |Y| is the number of sparse point labels, (h k ,w k ) is the two-dimensional coordinate of the sparse point, L k is the category of sparse points, The semantic segmentation model predicts the image at sparse points (h k ,w k ) The category of the position is L k The probability of k refers to the kth sparse point label.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the program, the steps of the training method of the semantic segmentation model as described in any one of claims 1 to 4 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the method for training a semantic segmentation model as described in any one of claims 1 to 4 are implemented.