Model training method and apparatus, semantic segmentation method and apparatus, and device and medium
Through the combination of classifier-free model and class representation features, the semantic segmentation model is updated, which solves the problem of poor model flexibility in the prior art, and realizes adaptability and efficient semantic segmentation to any category.
Patent Information
- Application Number
- PCT/CN2024/090121
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-12
- Filing Date
- 2024-04-26
- Publication Date
- 2024-11-21
AI Technical Summary
The existing semantic segmentation models have poor flexibility when the number of categories changes and cannot quickly adapt to any number of categories, resulting in insufficient model plasticity and flexibility.
A classifier-free segmentation model is adopted, and the image is processed by obtaining training data and semantic segmentation labels, and the classifier-free model is used to process the image, and the model updates are performed in combination with category characterization features to realize the semantic segmentation task, and the model structure does not change with the increase of categories.
The plasticity and flexibility of the semantic segmentation model is improved, allowing it to adapt to any number of new categories, and the flexibility and performance of semantic segmentation is enhanced.
Smart Images

Figure CN2024090121_21112024_PF_FP_ABST
Abstract
Description
Model training method, semantic segmentation method, device, equipment, and medium
[0001] This application claims priority to Chinese Patent Application No. 202310538859.7 filed on May 12, 2023, and the contents of the above-mentioned Chinese patent application disclosure are hereby incorporated by reference in their entirety as a part of this application. Technical Field
[0002] The embodiments of the present disclosure relate to a model training method, a semantic segmentation method, an apparatus, a device, and a medium. Background Art
[0003] Semantic segmentation is a fundamental task in computer vision and can be applied to many fields (for example, autonomous driving and image analysis). Semantic segmentation is used to determine the classification information of each pixel in an image.
[0004] However, how to implement semantic segmentation tasks is a technical problem that needs to be solved urgently.
[0005] Summary of the Invention
[0006] The present disclosure provides a model training method, semantic segmentation method, apparatus, device, and medium, which can implement semantic segmentation tasks.
[0007] In order to achieve the above objectives, the technical solutions provided by the present disclosure are as follows:
[0008] The present disclosure provides a model training method, the method comprising:
[0009] Acquire training data corresponding to a first category, the training data including a training image and a semantic segmentation label of the training image, the semantic segmentation label including the first category;
[0010] Processing the training image using a classifier-free segmentation model to obtain a first segmentation mask and a first semantic feature;
[0011] The segmentation model is updated based on the semantic segmentation label, the first segmentation mask, and the similarity between the first semantic feature and the category representation feature, where the category representation feature is obtained by encoding a category identification set using a pre-built text encoder, and the category identification set is determined based on the first category.
[0012] In a possible implementation manner, updating the segmentation model according to the semantic segmentation label, the first segmentation mask, and the similarity between the first semantic feature and the category representation feature includes:
[0013] Determining classification prediction information based on the similarity between the first semantic feature and the category representation feature;
[0014] The segmentation model is updated according to the semantic segmentation label, the first segmentation mask, and the classification prediction information.
[0015] In a possible implementation manner, updating the segmentation model according to the semantic segmentation label, the first segmentation mask, and the classification prediction information includes:
[0016] Determining a semantic segmentation result of the training image according to the first segmentation mask and the classification prediction information;
[0017] Determining a semantic segmentation loss based on the semantic segmentation label and the semantic segmentation result;
[0018] The segmentation model is updated according to the semantic segmentation loss.
[0019] In one possible implementation manner, before determining the classification prediction information based on the similarity between the first semantic feature and the category representation feature, the method further includes:
[0020] Determining an image corresponding to the first segmentation mask using the training image;
[0021] Inputting the image corresponding to the first segmentation mask into a pre-built image classification model to obtain an image classification result output by the image classification model;
[0022] The determining of classification prediction information according to the similarity between the first semantic feature and the category representation feature includes:
[0023] The classification prediction information is determined according to the image classification result and the similarity between the first semantic feature and the category representation feature.
[0024] In a possible implementation manner, updating the segmentation model according to the semantic segmentation label, the first segmentation mask, and the classification prediction information includes:
[0025] Determining a classification loss based on the semantic segmentation label and the classification prediction information;
[0026] Determining a mask loss based on the semantic segmentation label and the first segmentation mask;
[0027] The segmentation model is updated according to the classification loss and the mask loss.
[0028] In one possible implementation, the segmentation model is initialized using a reference model; the reference model is a classifier-free segmentation model trained using training data corresponding to the second category;
[0029] The segmentation model includes a normalization layer;
[0030] When updating the segmentation model, parameters of the normalization layer of the segmentation model are fixed.
[0031] In one possible implementation, the segmentation model is initialized using a reference model; the reference model is a classifier-free segmentation model trained using training data corresponding to the second category;
[0032] The reference model is used to process the training image to obtain a second segmentation mask and a second semantic feature;
[0033] Before updating the segmentation model according to the semantic segmentation label, the first segmentation mask, and the similarity between the first semantic feature and the category representation feature, the method further includes:
[0034] Determining a semantic distillation loss based on the first semantic feature and the second semantic feature;
[0035] determining a mask distillation loss based on the first segmentation mask and the second segmentation mask;
[0036] Updating the segmentation model according to the semantic segmentation label, the first segmentation mask, and the similarity between the first semantic feature and the category representation feature includes:
[0037] The segmentation model is updated according to the semantic segmentation label, the first segmentation mask, the similarity between the first semantic feature and the category representation feature, the semantic distillation loss, and the mask distillation loss.
[0038] In one possible implementation, the method further includes:
[0039] The training data set is divided into T training data subsets, where different training data subsets include training data corresponding to different categories, and T is a positive integer;
[0040] The obtaining of training data corresponding to the first category includes:
[0041] The training data corresponding to the first category is obtained from the t-th training data subset, where t is a positive integer and t≤T.
[0042] In one possible implementation, the method further includes:
[0043] Initialize t to 1;
[0044] After updating the segmentation model according to the semantic segmentation label, the first segmentation mask, and the similarity between the first semantic feature and the category representation feature, the method further includes:
[0045] Continue to execute the step of obtaining the training data corresponding to the first category from the tth training data subset until a preset stopping condition is reached, update t, and continue to execute the step of obtaining the training data corresponding to the first category from the tth training data subset until the segmentation model is trained using the T training data subsets.
[0046] In one possible implementation, if 2≤t≤T, the segmentation model is initialized using a reference model; the reference model is a classifier-free segmentation model trained using the first t-1 training data subsets.
[0047] The present disclosure provides a semantic segmentation method, the method comprising:
[0048] Obtain the image to be segmented;
[0049] Processing the image to be segmented using a segmentation model without a classifier to obtain a third segmentation mask and a third semantic feature; the segmentation model is obtained using the model training method provided by the present disclosure;
[0050] The semantic segmentation result of the image to be segmented is determined based on the third segmentation mask and the similarity between the third semantic feature and the category representation feature corresponding to the segmentation model; the category representation feature corresponding to the segmentation model is obtained by encoding the category identification set corresponding to the segmentation model using a pre-built text encoder.
[0051] In one possible implementation, before determining the semantic segmentation result of the image to be segmented based on the third segmentation mask and the similarity between the third semantic feature and the category representation feature corresponding to the segmentation model, the method further includes:
[0052] The process of determining the semantic segmentation result of the image to be segmented includes:
[0053] Determining an image corresponding to the third segmentation mask using the image to be segmented;
[0054] Inputting the image corresponding to the third segmentation mask into a pre-built image classification model to obtain an image classification result output by the image classification model;
[0055] Determining a semantic segmentation result of the image to be segmented according to the third segmentation mask and the similarity between the third semantic feature and the category representation feature corresponding to the segmentation model includes:
[0056] A semantic segmentation result of the image to be segmented is determined according to the image classification result, the third segmentation mask, and the similarity between the third semantic feature and the category representation feature corresponding to the segmentation model.
[0057] In one possible implementation, the image classification model includes an image encoder; the image encoder is configured to perform encoding processing on the image corresponding to the third segmentation mask to obtain an image code corresponding to the third segmentation mask; and the image classification result is determined based on a similarity between the image code corresponding to the third segmentation mask and a category representation feature corresponding to the segmentation model:
[0058] and / or,
[0059] Determining a semantic segmentation result of the image to be segmented based on the image classification result, the third segmentation mask, and the similarity between the third semantic feature and the category representation feature corresponding to the segmentation model includes:
[0060] Determining the classification prediction information according to the image classification result and the similarity between the third semantic feature and the category representation feature corresponding to the segmentation model;
[0061] Determine a semantic segmentation result of the image to be segmented according to the classification prediction information and the third segmentation mask.
[0062] The present disclosure provides a model training device, comprising:
[0063] A first acquisition unit is configured to acquire training data corresponding to a first category, the training data including a training image and a semantic segmentation label of the training image, the semantic segmentation label including the first category;
[0064] a first processing unit configured to process the training image using a classifier-free segmentation model to obtain a first segmentation mask and a first semantic feature;
[0065] A model updating unit is configured to update the segmentation model based on the semantic segmentation label, the first segmentation mask, and the similarity between the first semantic feature and the category representation feature, where the category representation feature is obtained by encoding a category identification set using a pre-built text encoder, and the category identification set is determined based on the first category.
[0066] The present disclosure provides a semantic segmentation device, comprising:
[0067] A second acquisition unit is configured to acquire an image to be segmented;
[0068] a second processing unit configured to process the image to be segmented using a segmentation model without a classifier to obtain a third segmentation mask and a third semantic feature; the segmentation model is obtained using the model training method provided by the present disclosure;
[0069] A result determination unit is configured to determine the semantic segmentation result of the image to be segmented based on the third segmentation mask and the similarity between the third semantic feature and the category representation feature corresponding to the segmentation model; the category representation feature corresponding to the segmentation model is obtained by encoding the category identification set corresponding to the segmentation model using a pre-built text encoder.
[0070] The present disclosure provides an electronic device, the device comprising: a processor and a memory;
[0071] The memory is configured to store instructions or computer programs;
[0072] The processor is configured to execute the instructions or computer program in the memory so that the electronic device executes the model training method or semantic segmentation method provided by the present disclosure.
[0073] The present disclosure provides a computer-readable medium having instructions or computer programs stored therein. When the instructions or computer programs are executed on a device, the device executes the model training method or semantic segmentation method provided by the present disclosure.
[0074] The present disclosure provides a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the model training method or semantic segmentation method provided by the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0075] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.
[0076] FIG1 is a flow chart of a model training method provided by an embodiment of the present disclosure;
[0077] FIG2 is a schematic diagram of a training process for a segmentation model without a classifier provided in an embodiment of the present disclosure;
[0078] FIG3 is a flowchart of a semantic segmentation method provided by an embodiment of the present disclosure;
[0079] FIG4 is a schematic structural diagram of a model training device provided by an embodiment of the present disclosure;
[0080] FIG5 is a schematic structural diagram of a semantic segmentation device provided by an embodiment of the present disclosure;
[0081] FIG6 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION
[0082] Research has found that in some technical solutions, a pre-trained semantic segmentation model can be used to implement semantic segmentation processing for a single image. However, because the structure of the classifier in the semantic segmentation model is determined by the number of categories, the structure of the semantic segmentation model changes as the number of categories changes. This makes the model unable to quickly adapt to any number of categories, resulting in poor model plasticity and, in turn, poor semantic segmentation flexibility.
[0083] Based on the above research content, it can be seen that in order to better improve the flexibility of semantic segmentation, the present disclosure provides a classifier-free segmentation model and its related technical solutions (for example, a model training method and a semantic segmentation method, etc.). For the segmentation model, the training process involved in any learning stage of the segmentation model is specifically as follows: if the learning stage is used to learn a first category (for example, a new category), then after obtaining the training data corresponding to the first category (that is, the training image and the semantic segmentation label of the training image), the training image is first processed using the segmentation model to obtain a first segmentation mask and a first semantic feature, so that the first segmentation mask and the first semantic feature can represent the visual information carried by the training image; then, based on the similarity between the semantic segmentation label, the first segmentation mask, and the first semantic feature and the category representation feature corresponding to the category identification set involved in the learning stage, the segmentation model is updated so that the updated segmentation model has better performance. In this way, the training process for a classifier-free segmentation model can be realized, so that the semantic segmentation task can be achieved subsequently with the help of the trained segmentation model.
[0084] In addition, since there is no classifier in the segmentation model provided by the present invention, the model structure of the segmentation model does not need to change as the number of categories increases, so that the segmentation model can be adapted to learn any number of new categories, which can effectively improve the plasticity of the segmentation model and thus help improve the flexibility of semantic segmentation.
[0085] In order to enable those skilled in the art to better understand the solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the embodiments described are only part of the embodiments of the present disclosure, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present disclosure without making any creative efforts shall fall within the scope of protection of the present disclosure.
[0086] In order to better understand the technical solution provided by the present disclosure, the model training method provided by the present disclosure is described below with reference to some drawings. As shown in Figure 1, the model training method provided by the embodiment of the present disclosure includes the following S101-S103. Among them, Figure 1 is a flowchart of a model training method provided by the embodiment of the present disclosure.
[0087] S101: Acquire training data corresponding to a first category, where the training data includes a training image and a semantic segmentation label of the training image, and the semantic segmentation label includes the first category.
[0088] Among them, the first category refers to the category that needs to be learned in the current learning stage (for example, the "t-th learning stage" shown below); and the present disclosure does not limit the first category. For example, when the classifier-free segmentation model provided by the present disclosure is trained using an incremental learning method, if the current learning stage is used to implement the first learning process (that is, the first training) for the segmentation model, then the first category is the category involved in the first learning process; if the current learning stage is used to implement the t-th learning process (that is, the t-th training) for the segmentation model, t is a positive integer, 2≤t, then the first category is the new category that needs to be learned in the t-th learning process.
[0089] The training data corresponding to the first category refers to the training data determined in advance for the first category, so that the semantic segmentation label in the training data involves the first category. It can be seen that when the training data corresponding to the first category includes a training image and a semantic segmentation label of the training image, the semantic segmentation label includes the first category. The training image is at least used to describe the object under the first category (for example, when the first category is a cat, then at least the object of cat exists in the training image). The semantic segmentation label of the training image is used to describe the actual classification information of each pixel in the training image; and the present disclosure does not limit the acquisition process of the semantic segmentation label of the training image. For example, it can be implemented by means of manual annotation.
[0090] In addition, in the incremental learning stage, the old categories that have been learned are not very important, so in order to save training costs, the old categories in the semantic segmentation labels of the above training image data can be regarded as background. Based on this, the present disclosure also provides a processing process for the semantic segmentation label, which can be specifically: for the t-th learning process, 2≤t, after obtaining the semantic segmentation label corresponding to a training data, if the label information corresponding to some pixels in the semantic segmentation label is the old category label, then the label information corresponding to these pixels in the semantic segmentation label can be directly adjusted to the background category label, so that only the new category label exists in the adjusted semantic segmentation label. Among them, the old category label is used to identify the category that has been learned in the previous t-1 learning process. The background category label is used to identify the category to which the background pixel in an image data belongs.
[0091] It can be seen that for the t-th learning process, 2≤t, when the new category to be learned in the t-th learning process is the "plant" category, and the old category learned in the previous t-1 learning process is the "furniture" category, if the training data required for the t-th learning process includes training image A, and the training image A contains both plants and furniture, then the semantic segmentation label of the training image A contains a label corresponding to the "plant" category and a label corresponding to the "furniture" category. At this time, it is necessary to remove the "furniture" category in the semantic segmentation label of the training image A (for example, replace the "furniture" category in the semantic segmentation label of the training image A with the background category label). In this way, when the loss is calculated in the t-th learning process, the old category will not participate in the loss calculation process, which can save training costs.
[0092] S102: Processing the training image using a segmentation model without a classifier to obtain a first segmentation mask and a first semantic feature.
[0093] Among them, the classifier-free segmentation model is used to perform visual processing (such as segmentation mask determination processing of unknown categories and high-dimensional semantic feature determination processing, etc.) on the input data of the segmentation model (such as image data).
[0094] In addition, the present disclosure does not limit the implementation method of the above-mentioned "segmentation model without classifier". For example, it can be implemented using any existing or future visual model (for example, a semantic segmentation model without the classification layer).
[0095] For example, in some application scenarios, the above “segmentation model without classifier” can have an encoder-decoder structure, so that the “segmentation model without classifier” can extract visual features from an image data and generate a segmentation mask M∈R of unknown category. N×H×WAnd high-dimensional semantic features F semantic ∈R N×D . Among them, the segmentation mask M∈R N×H×W Represents the mask state of the image data under N feature channel layers, N represents the number of feature channel layers in the segmentation model, and H×W represents the resolution of the mask (that is, the resolution of the image data). It can be seen that the segmentation mask M∈R N×H×W It includes N masks, each of which has a size of H×W, and the nth mask is used to represent the mask state of the image data at the nth feature channel layer. semantic ∈R N×D It represents the semantic information presented by the image data under N feature channel layers, and D represents the number of feature dimensions of the semantic features under each feature channel layer. semantic ∈R N×D It can include N feature vectors, each feature vector is a D-dimensional vector, and the nth feature vector is used to represent the semantic information presented by the image data under the nth feature channel layer, so that the nth feature vector can represent the image semantic information corresponding to the nth mask (for example, information used to represent categories, etc.), n is a positive integer, n≤N.
[0096] It should be noted that the "mask" in the previous paragraph is used to describe the position of one or more image areas within the image data in the previous paragraph (for example, the position of certain objects in the image data, etc.); and the present disclosure does not limit the implementation method of the mask. For example, it can be implemented using any existing or future image data that can represent the mask result.
[0097] In addition, the present disclosure does not limit the initialization method of the above-mentioned "classifier-free segmentation model". For example, when the classifier-free segmentation model provided by the present disclosure is trained using an incremental learning method, if the current learning stage is used to implement the first learning process for the segmentation model, then the segmentation model involved in the current learning stage can be initialized using an initially constructed, untrained classifier-free segmentation model; if the current learning stage is used to implement the t-th learning process for the segmentation model, t is a positive integer, 2≤t (that is, the segmentation model has learned some categories in the historical learning stage), then the segmentation model involved in the current learning stage can be initialized using a reference model, which refers to a classifier-free segmentation model that has learned a second category in the historical learning stage. The historical learning stage is used to describe one or more learning stages that have been completed for the segmentation model. The second category refers to the category that the segmentation model has learned in the historical learning stage (that is, the old category).
[0098] It can be seen that for the current learning stage, if the current learning stage refers to the first training performed on the segmentation model without classifier, an initially constructed segmentation model without classifier that has not yet been trained can be set as the initial value of the segmentation model involved in the current learning stage (that is, the "segmentation model without classifier" in S102 above), so that the first training of the segmentation model can be completed based on the initial value subsequently; however, if the current learning stage refers to the t-th training performed on the segmentation model without classifier, t≥2, then the segmentation model that has been trained in the t-1-th training can be set as the initial value of the segmentation model involved in the current learning stage (that is, the "segmentation model without classifier" in S102 above), so that the non-first training of the segmentation model can be completed based on the initial value subsequently.
[0099] It should be noted that the “t-th learning process” and “t-th training” involved in the present disclosure refer to a training stage, and the training stage is usually achieved with the help of multiple rounds of training, t is a positive integer, 1≤t≤T.
[0100] The first segmentation mask refers to a segmentation mask of unknown category obtained by processing the training image by the classifier-free segmentation model, so that the first segmentation mask can represent the mask state presented by the training image at the N feature channel layers. For example, when the training image is the image shown in Figure 2, and the current learning phase is used to implement the t-th training, t ≥ 2, the first segmentation mask can refer to "mask 2 of unknown category" shown in Figure 2.
[0101] In addition, the present disclosure does not limit the implementation of the first segmentation mask above. For example, in some application scenarios, the first segmentation mask may include at least one mask. It should be noted that the relevant content of the mask is referred to above and will not be repeated here for the sake of brevity.
[0102] The first semantic feature refers to a high-dimensional semantic feature obtained by processing the training image by the classifier-free segmentation model, so that the first semantic feature can represent the semantic information presented by the training image at the N feature channel layers. For example, when the training image is the image shown in Figure 2, and the current learning phase is for the t-th training, where t ≥ 2, the first semantic feature may refer to "semantic feature 2" shown in Figure 2.
[0103] In addition, the present disclosure does not limit the implementation of the first semantic feature above. For example, when the first segmentation mask above includes at least one mask, the first semantic feature may include a semantic feature vector corresponding to each mask, so that the semantic feature vector corresponding to the mask is used to represent the image semantic information corresponding to the mask (for example, information used to represent a category, etc.). It can be seen that in one possible implementation, when the first segmentation mask above includes N masks, the first semantic feature may include N semantic feature vectors, and the nth semantic feature vector is used to represent the image semantic information corresponding to the nth mask, where n is a positive integer and n≤N.
[0104] S103: Update the segmentation model based on the semantic segmentation label, the first segmentation mask, and the similarity between the first semantic feature and the category representation feature, where the category representation feature is obtained by encoding the category identification set using a pre-built text encoder, and the category identification set is determined based on the first category.
[0105] The category characterization feature is used to characterize the categories involved in the current learning stage (for example, new categories that need to be learned in the current learning stage, etc.).
[0106] In addition, for the above-mentioned category characterization feature, the category characterization feature is obtained by encoding the category identification set using a pre-built text encoder. The category identification set is used to record the categories involved in the current learning stage; and the present disclosure does not limit the category identification set. For example, the category identification set may at least include the category identification of the category that needs to be learned in the current learning stage (for example, the first category above). For another example, in some application scenarios, the category identification set may not only include at least the category identification of the category that needs to be learned in the current learning stage, but also the category identification of the category that has been learned in the historical learning stage. The category identification is used to uniquely identify a category; and the present disclosure does not limit the implementation method of the category identification. For example, it can be implemented using a class name.
[0107] In fact, in the incremental learning phase, the old categories that have been learned are not very important. Therefore, in order to simplify training, multiple old categories can be inaccurately distinguished, but instead referred to by the same identifier. Based on this, the present disclosure also provides a possible implementation of the above category identifier set. In this implementation, the category identifier set can not only include at least the category identifiers of the categories that need to be learned in the current learning phase, but also include a preset category identifier (for example, the identifier string of "old category", etc.), so that the preset category identifier is used to uniformly represent all categories that have been learned in the historical learning phase.
[0108] Based on the above two paragraphs, it can be seen that if the first category needs to be learned in the current learning stage, the above category identification set can be determined based on the first category, so that the category identification set can at least include the category identification of the first category. In addition, if the first category needs to be learned in the current learning stage, the above segmentation model has learned the second category (that is, the old category) in the historical learning stage, and the second category is different from the first category, then the category identification set can be determined based on the first category and the second category, so that the category identification set can include the category identification of the first category and the category identification of the second category, or the category identification set can include the category identification of the first category and the above background category identification.
[0109] It can be seen that in one possible implementation, if the current learning stage is used to implement the t-th training, t≥2, then the category identification set may include all categories required to be learned in the 1st training to all categories required to be learned in the t-th training.
[0110] The above-mentioned "text encoder" is used to perform text feature extraction processing on a text data; and the present disclosure does not limit the implementation method of the text encoder. For example, it can be implemented using any existing or future encoder with a text feature extraction function. For another example, in some application scenarios, the text encoder can be implemented using the text encoder existing in the Contrastive Language-Image Pre-Training (CLIP) model. For another example, the text encoder can be implemented using the text encoder shown in Figure 2.
[0111] Based on the relevant content of the "text encoder" above, it can be known that because the text encoder has been trained, there is no need to adjust the parameters in the text encoder during the training process of the above classifier-free segmentation model. This ensures that the text encoders used in different learning stages remain consistent, thereby ensuring that the representation methods used for category identification in different learning stages remain consistent, which is conducive to improving the training effect for the segmentation model. Also, because the text encoder has the function of encoding any number of texts, the present disclosure can use the text encoder and the classifier-free segmentation model to achieve semantic segmentation processing for any number of categories, so that the classifier-free segmentation model can quickly adapt to any number of new categories without adjusting the structure.
[0112] In addition, the present disclosure does not limit the determination process of the above category characterization features. For example, when the above category identification set is {category identification 1, category identification 2, ..., category identification C t This set, C tWhen representing the number of all categories involved in the current learning stage, the process of determining the category representation feature can be specifically as follows: first, the category identifier m is inserted into a pre-built text template (for example, the template "a photo of [category identifier]") to obtain text m (for example, the text string "a photo of [category identifier m]"), so that the text m carries the category identifier m, where m is a positive integer and m≤C t ; Then, use the pre-built text encoder for text 1 to text C t Perform encoding processing to obtain category representation features To characterize the category It can represent each category identifier in the category identifier set. It can be seen that the category representation feature Including the text feature vectors corresponding to each category identifier in the category identifier set, the text feature vector corresponding to each category identifier is a D-dimensional vector, and the text feature vector corresponding to the mth category identifier is used to represent the category identifier m, m is a positive integer, m≤C t It should be noted that the text template refers to a template pre-set based on an application scenario, so that the text template is used to generate text data carrying a category identifier; and the present disclosure does not limit the text template.
[0113] Based on the relevant content of the category representation features above, it can be seen that for the current learning stage, if the category identification set involved in the learning stage is used to record the categories involved in the learning stage (for example, new categories, etc.), then the category identification set can be encoded using a pre-built text encoder to obtain the category representation features corresponding to the learning stage, so that the category representation features can represent the categories involved in the learning stage.
[0114] The above “similarity between the first semantic feature and the category representation feature” is used to represent the classification prediction information for the above training image, for example, the classification prediction information for the above first segmentation mask; and the present disclosure does not limit the method for determining the “similarity between the first semantic feature and the category representation feature”. For example, it can be implemented using any existing or future calculation method that can calculate the similarity between two data (for example, the method shown in formula (1) below).
[0115] Where S(i,j) refers to the similarity between the i-th feature vector in the first semantic feature and the j-th feature vector in the category representation feature, so that S(i,j) can represent the possibility that the i-th mask in the first segmentation mask belongs to the category identified by the j-th category identifier in the category identifier set; F semanticrepresents the first semantic feature; refers to the i-th feature vector in the first semantic feature, so that represents the semantic information presented by the training image at the i-th feature channel layer, and The size of F is 1×D, i is a positive integer, i≤N; text Indicates the characteristics of this category; Refers to the jth feature vector in the category representation feature, so that represents the jth category identifier in the category identifier set, and The size is 1×D, j is a positive integer, j≤C t ; Indicates that The transpose of And the size is D×1.
[0116] Based on the relevant content of the above "similarity between the first semantic feature and the category representation feature", it can be known that when the above first segmentation mask includes N masks, and the first semantic feature includes semantic feature vectors corresponding to each mask, the "similarity between the first semantic feature and the category representation feature" can be used to indicate the possibility that each mask belongs to each category identified by the above category identification set, so that the semantic segmentation loss or classification loss for the training image can be determined based on the "similarity between the first semantic feature and the category representation feature" in the future.
[0117] In addition, the present disclosure does not limit the implementation of the above S103. For example, in one possible implementation, the above S103 may specifically include the following steps 11 and 12.
[0118] Step 11: Determine classification prediction information based on the similarity between the first semantic feature and the category representation feature.
[0119] The classification prediction information is used to represent the classification information determined for the training image. For example, the classification prediction information can be used to represent the possibility that each mask in the first segmentation mask belongs to each category identified by the category identification set above. Moreover, the present disclosure does not limit the implementation method of the classification prediction information. For example, it can adopt N×C t The matrix represents .
[0120] In addition, the present disclosure does not limit the implementation of the above step 11. For example, it can be specifically: directly determining the similarity between the above first semantic feature and the category representation feature as classification prediction information. It can be seen that in one possible implementation, the classification prediction information can be and Please refer to the above formula (1) for the determination process of .
[0121] In fact, in order to better improve the classification prediction effect, the present disclosure also provides a possible implementation method of the above classification prediction information determination process, which may specifically include the following steps 111 to 113.
[0122] Step 111: Determine the image corresponding to the first segmentation mask using the training image.
[0123] The image corresponding to the first segmentation mask refers to an image extracted from the training image using the first segmentation mask, so that the "image corresponding to the first segmentation mask" can represent the image information in the image area corresponding to the first segmentation mask in the training image.
[0124] In addition, the present disclosure does not limit the implementation method of the above step 111. For example, in some application scenarios (for example, when the above mask is used to represent the position of one or more objects in the image data), the step 111 can be specifically: each mask in the above first segmentation mask is fused with the above training image to obtain the image corresponding to the first segmentation mask, so that the "image corresponding to the first segmentation mask" includes the fusion results corresponding to each mask, so that the "image corresponding to the first segmentation mask" can represent the image information in the image area corresponding to each mask in the first segmentation mask in the training image.
[0125] Step 112: Input the image corresponding to the first segmentation mask into a pre-built image classification model to obtain an image classification result output by the image classification model.
[0126] The image classification model is used to perform image classification processing on input data (e.g., image data) of the image classification model. Furthermore, the present disclosure does not limit the implementation of the image classification model. For example, the image classification model can be implemented using any existing or future model with image classification capabilities. For another example, the image classification model can be implemented using the visual encoder in CLIP.
[0127] The image classification result is used to represent the category prediction information of the image corresponding to the first segmentation mask above; and the present disclosure does not limit the implementation method of the image classification result. For example, it can adopt the classification probability matrix Implement so that the It can indicate the possibility that the image corresponding to each mask in the first segmentation mask belongs to each category identified by the above category identification set.
[0128] In addition, the present disclosure does not limit the determination process of the above image classification result. For example, when the above step 112 is implemented with the help of CLIP, the determination process of the image classification result can be specifically as follows: first, the image corresponding to the above first segmentation mask is encoded using the visual encoder in CLIP to obtain the image encoding of the image corresponding to the first segmentation mask; then, the similarity between the image encoding of the image corresponding to the first segmentation mask and the above category representation feature is determined as the image classification result corresponding to the first segmentation mask. So that the image classification results can be used later Determine the following classification prediction information S all .
[0129] Step 113: Determine classification prediction information based on the above image classification result and the similarity between the above first semantic feature and the category representation feature.
[0130] It should be noted that the present disclosure does not limit the implementation of step 113. For example, in some application scenarios, step 113 may be: c , and the similarity S between the first semantic feature and the category representation feature are weighted and summed to obtain classification prediction information. For example, in some application scenarios, step 113 can be implemented using the following formula (2).
[0131] Where S all Represents the classification prediction information above; S represents the similarity between the first semantic feature and the category representation feature above; S c Represents the image classification result above; α represents the hyperparameter between different prediction results, which can be set in advance.
[0132] Based on the relevant content of steps 111 to 113 above, it can be known that in some application scenarios, the classification prediction information can be determined based on the first segmentation mask above and the similarity between the first semantic feature above and the category representation feature, so that the classification prediction information can better represent the category prediction results corresponding to each mask in the first segmentation mask, so that the segmentation model can be updated later by referring to the classification prediction information.
[0133] Step 12: Update the segmentation model based on the semantic segmentation label, the first segmentation mask, and the classification prediction information.
[0134] It should be noted that the embodiment of the present disclosure does not limit the implementation of the above step 12. For ease of understanding, two examples are used for illustration below.
[0135] Example 1: Step 12 above may specifically include steps 121 to 123 below.
[0136] Step 121: Determine a semantic segmentation result of the training image according to the first segmentation mask and the classification prediction information, so that the semantic segmentation result is used to represent the predicted category of each pixel in the training image.
[0137] It should be noted that the present disclosure does not limit the specific implementation of the above step 121. For example, it can be implemented by using any existing or future method that can calculate the semantic segmentation result of image data based on the mask and its classification prediction result (such as the method shown in formula (3) below). Predict ) T *M (3)
[0138] Where ResultSemantic_segmentation represents the semantic segmentation result of the training image above, so that the semantic segmentation result is used to represent the predicted category of each pixel in the training image, that is, C t Indicates the number of categories involved in the t-th learning stage, H×W represents the size of the training image; Class Predict Represents the above classification prediction information, This disclosure does not limit the Class Predict , for example, Class Predict It can be the above S all or S; (Class Predict ) T Indicates the Class Predict The transpose of M represents the first segmentation mask above, and M∈R N×H×W , N represents the number of feature channel layers in the segmentation model.
[0139] Step 122: Determine a semantic segmentation loss based on the semantic segmentation label and the semantic segmentation result, so that the semantic segmentation loss can represent the performance of the segmentation model in semantic segmentation.
[0140] It should be noted that the present disclosure does not limit the implementation of step 122. For example, it can be implemented by using any existing or future method that can determine the semantic segmentation loss based on the difference between the semantic segmentation label and the semantic segmentation result (for example, the cross entropy loss calculation method shown in formula (4) below). SS =CE(ResultSemantic_segmentation,Yt ) (4)
[0141] Among them, L SS Represents the semantic segmentation loss above; ResultSemantic_segmentation represents the semantic segmentation result of the training image above; Y t represents the semantic segmentation label in the training data at the t-th learning stage (for example, the semantic segmentation label of the training image above); CE(ResultSemantic_segmentation,Y t ) indicates ResultSemantic_segmentation and Y t The cross entropy loss between .
[0142] Step 123: Update the segmentation model based on the semantic segmentation loss above.
[0143] It should be noted that the present disclosure does not limit the implementation of the above step 123. For example, it can be implemented by using any existing or future method that can update the model based on semantic segmentation loss.
[0144] For example, in order to better improve the model training effect, the present disclosure also provides a possible implementation of the above step 123. In this implementation, when the segmentation model includes a normalization layer, the step 123 can be specifically as follows: if the current learning stage is used for the first training of the segmentation model without a classifier (that is, the current learning stage is the first learning stage), then all parameters in the segmentation model are updated using the above semantic segmentation loss; however, if the current learning stage is used for non-first training of the segmentation model without a classifier (that is, the current learning stage is the tth learning stage, t≥2), then when using the semantic segmentation loss to update the segmentation model, the parameters of the normalization layer are fixed (for example, using the semantic segmentation loss to update the parameters of other network layers in the segmentation model except the normalization layer, etc.).
[0145] Based on the relevant content of steps 121 to 123 above, it can be seen that in some application scenarios, for any round of training process in the current learning stage, after obtaining the first segmentation mask and classification prediction information corresponding to the training image, the semantic segmentation loss of the training image can be determined based on the first segmentation mask, the classification prediction information and the semantic segmentation label of the training image; and then the segmentation model is updated based on the semantic segmentation loss so that the updated segmentation model has better performance.
[0146] Example 2: Step 12 above may specifically include steps 124 to 126 below.
[0147] Step 124: Determine the classification loss based on the above semantic segmentation label and the above classification prediction information.
[0148] Among them, the classification loss is used to represent the performance of the above segmentation model in classification.
[0149] In addition, the present disclosure does not limit the implementation of the above step 124. For example, it can be implemented using any existing or future method for determining the classification loss based on the above semantic segmentation label and the above classification prediction information.
[0150] For another example, the above step 124 may specifically include the following steps 1241 to 1243.
[0151] Step 1241: Using the above semantic segmentation labels, determine a category label so that the category label can indicate which categories appear in the semantic segmentation label, thereby enabling the category label to indicate which type of object exists in the training image.
[0152] It should be noted that the present disclosure does not limit the implementation of the above step 1241.
[0153] Step 1242: Encode the above category label to obtain an encoding vector corresponding to the category label, so that the encoding vector can represent the category label.
[0154] It should be noted that the present disclosure does not limit the implementation method of the above "encoding vector corresponding to the category label". For example, it can be implemented in the form of a one-hot vector.
[0155] It should also be noted that the present disclosure does not limit the implementation method of the "encoding process" in the above step 1242.
[0156] Step 1243: Calculate the cross entropy between the above classification prediction information and the encoding vector corresponding to the above category label as the above classification loss (as shown in Formula (5) below).
[0157] Among them, L cls Represents the above classification loss; Class Predict Represents the above classification prediction information, This disclosure does not limit the Class Predict , for example, Class Predict It can be the above S all or S; Represents the encoding vector corresponding to the above category label; Y t represents the semantic segmentation label in the training data at the t-th learning stage (for example, the semantic segmentation label of the training image above).
[0158] Based on the relevant content of steps 1241 to 1243 above, it can be known that in some application scenarios, the classification loss can be determined with the help of the category labels involved in the above semantic segmentation labels and the above classification prediction information, so that the classification loss can represent the performance of the above segmentation model in classification.
[0159] Step 125: Determine a mask loss based on the semantic segmentation label and the first segmentation mask, so that the mask loss can represent the performance of the segmentation model in terms of masking.
[0160] It should be noted that the present disclosure does not limit the implementation of the above step 125. For example, the step 125 can be implemented by using any existing or future method that can determine the mask loss based on the semantic segmentation label and the first segmentation mask (for example, the mask loss calculation method shown in formula (6) below). mask =L Dice +β×L Focal (6)
[0161] Where, L mask Indicates the above mask loss; L Dice Represents the Dice loss between the semantic segmentation label and the first segmentation mask above, and the L Dice is determined based on the semantic segmentation label and the first segmentation mask; L Focal Represents the Focal loss between the semantic segmentation label and the first segmentation mask, and the L Focal is determined based on the semantic segmentation label and the first segmentation mask; β represents a pre-set loss coefficient. It should be noted that the present disclosure does not limit the calculation method of the Focal loss or the calculation method of the Dice loss.
[0162] Step 126: Update the segmentation model based on the above classification loss and the above mask loss.
[0163] It should be noted that the present disclosure does not limit the implementation of the above step 126. For example, the step 126 can be implemented by using any existing or future method that can update the model based on classification loss and mask loss.
[0164] For example, in one possible implementation, step 126 may specifically be: first determine the model loss of the segmentation model based on the above classification loss and the above mask loss (for example, directly perform a weighted summation of the classification loss and the mask loss to obtain the model loss of the segmentation model); then update the segmentation model based on the model loss.
[0165] For example, in order to better improve the model training effect, the present disclosure also provides a possible implementation of the above step 126. In this implementation, when the segmentation model includes a normalization layer, the step 126 can be specifically as follows: if the current learning stage is used for the first training of the segmentation model without a classifier (that is, the current learning stage is the first learning stage), then the above classification loss and the above mask loss are used to update all parameters in the segmentation model; however, if the current learning stage is used for non-first training of the segmentation model without a classifier (that is, the current learning stage is the t-th learning stage, t≥2), then when the classification loss and the mask loss are used to update the segmentation model, the parameters of the normalization layer are fixed (for example, the classification loss and the mask loss are used to update the parameters of other network layers in the segmentation model except the normalization layer).
[0166] Based on the relevant content of steps 124 to 126 above, it can be seen that in some application scenarios, for any round of training process in the current learning stage, after obtaining the first segmentation mask and classification prediction information corresponding to the training image, the classification loss and mask loss of the training image can be determined based on the first segmentation mask, the classification prediction information and the semantic segmentation label of the training image; and then the segmentation model is updated based on the classification loss and mask loss so that the updated segmentation model has better performance.
[0167] Based on the relevant content of steps 11 to 12 above, it can be known that in some application scenarios, for any round of training process in the current learning stage, after obtaining the first segmentation mask and the first semantic feature corresponding to the training image, some loss information (for example, semantic segmentation loss or classification loss + mask loss) can be determined based on the semantic segmentation label of the training image, the first segmentation mask, and the similarity between the first semantic feature and the above category representation feature; and then the segmentation model is updated based on these loss information so that the updated segmentation model has better performance.
[0168] In fact, for the t-th learning stage, t≥2, in order to better avoid forgetting old categories, the present disclosure also provides another possible implementation of the above model update process. Under this implementation, when the segmentation model involved in the t-th learning stage is initialized using a reference model, and the reference model is a classifier-free segmentation model trained using training data corresponding to the second category, that is, the reference model can refer to a classifier-free segmentation model that has learned the second category in the historical learning stage (that is, the first t-1 learning stages), the model update process can specifically include the following steps 21 to 24. It should be noted that the present disclosure is not limited to the second category. For example, in some embodiments, for the training of the t-th stage, the category learned in the first t-1 stages is the second category.
[0169] Step 21: Use the above reference model to process the training image to obtain the second segmentation mask and the second semantic feature.
[0170] The reference model is a classifier-free segmentation model trained using the training data corresponding to the second category, that is, the reference model may refer to a classifier-free segmentation model that has been trained in a historical learning phase (for example, the segmentation model of phase t-1 shown in FIG2 ). That is, if the current learning phase is used to perform the t-th training on the classifier-free segmentation model (that is, the current learning phase is the t-th learning phase), t≥2, then the reference model refers to the classifier-free segmentation model that has been trained in the t-1-th learning phase (that is, the classifier-free segmentation model that has been trained t-1 times).
[0171] The second segmentation mask refers to a segmentation mask of unknown category (e.g., the unknown category mask 1 shown in FIG2 ) obtained by processing the training image using the reference model described above, so that the second segmentation mask can represent the mask state presented by the training image at the N feature channel layers. It should be noted that the second segmentation mask is similar to the first segmentation mask described above and will not be described again here for the sake of brevity.
[0172] The second semantic feature is a high-dimensional semantic feature (e.g., semantic feature 1 shown in Figure 2 ) obtained by processing the training image using the reference model described above. This second semantic feature represents the semantic information presented by the training image at the N feature channel levels. It should be noted that this second semantic feature is similar to the first semantic feature described above and will not be further described here for the sake of brevity.
[0173] It should also be noted that the present disclosure does not limit the execution time of step 21, as long as the execution time of step 21 is ensured to be earlier than the following step 22 or the following step 23.
[0174] Step 22: Determine a semantic distillation loss based on the first semantic feature and the second semantic feature, so that the semantic distillation loss can represent the differences in semantic prediction between the segmentation models involved in two adjacent learning stages.
[0175] It should be noted that the present disclosure does not limit the implementation method of step 22. For example, step 22 can be implemented using any existing or future method that can determine the distillation loss between two data (for example, the distillation loss calculation method shown in formula (7) below).
[0176] Where, represents the semantic distillation loss above; F semantic_t represents the first semantic feature above (for example, the high-dimensional semantic feature output by the segmentation model being used for the training image at the t-th learning stage); F semantic_t (n, d) represents the value of the nth row and dth column in the first semantic feature; F semantic_t-1 represents the second semantic feature above (for example, the high-dimensional semantic feature output by the segmentation model trained at the t-1th learning stage for the training image); F semantic_t-1 (n, d) represents the value of the nth row and dth column in the second semantic feature; (F semantic_t (n,d)-F semantic_t-1 (n,d)) 2 represents the distance between the value of the nth row and dth column in the first semantic feature and the value of the nth row and dth column in the second semantic feature, so that (F semantic_t (n,d)-F semantic_t-1 (n,d)) 2 It can represent the similarity between the value in the nth row and the dth column of the first semantic feature and the value in the nth row and the dth column of the second semantic feature.
[0177] Step 23: Determine a mask distillation loss based on the first segmentation mask and the second segmentation mask, so that the mask distillation loss can represent the differences in mask prediction between the segmentation models involved in two adjacent learning stages.
[0178] It should be noted that the present disclosure does not limit the implementation method of step 23. For example, step 23 can be implemented using any existing or future method that can determine the distillation loss between two data (for example, the distillation loss calculation method shown in formula (8) below).
[0179] Where, represents the mask distillation loss; M tM represents the first segmentation mask above (for example, the unknown category segmentation mask output by the segmentation model being used for the training image at the t-th learning stage); t (n,i) represents the value of the i-th pixel in the n-th mask in the first segmentation mask; M t-1 M represents the second segmentation mask above (for example, the unknown category segmentation mask output by the segmentation model trained at the t-1th learning stage for the training image); t-1 (n,i) represents the value of the i-th pixel in the n-th mask in the second segmentation mask; (M t (n,i)-M t-1 (n,i)) 2 It refers to the distance between the value of the i-th pixel in the n-th mask in the first segmentation mask and the value of the i-th pixel in the n-th mask in the second segmentation mask, so that (M t (n,i)-M t-1 (n,i)) 2 It can represent the similarity between the value of the i-th pixel in the n-th mask in the first segmentation mask and the value of the i-th pixel in the n-th mask in the second segmentation mask.
[0180] Step 24: Update the segmentation model according to the semantic segmentation label, the first segmentation mask, the similarity between the first semantic feature and the category representation feature, the semantic distillation loss, and the mask distillation loss.
[0181] It should be noted that the present disclosure is not limited to the implementation method of 24. For ease of understanding, two examples are used for illustration below.
[0182] Example 1: Step 24 above may specifically include steps 241 and 242 below.
[0183] Step 241: Determine the classification loss and the mask loss based on the semantic segmentation label, the first segmentation mask, and the similarity between the first semantic feature and the category representation feature.
[0184] It should be noted that the present disclosure does not limit the implementation of the above step 241. For example, it can be implemented using the determination process of the classification loss and mask loss shown above.
[0185] Step 242: Update the segmentation model according to the above classification loss, the above mask loss, the above semantic distillation loss, and the above mask distillation loss.
[0186] It should be noted that the embodiment of the present disclosure does not limit the implementation of the above step 242. For example, it can be implemented by using any existing or future method for updating the model based on multiple losses.
[0187] For example, in one possible implementation, the above step 242 may be: first perform weighted summation on the above classification loss, the above mask loss, the above semantic distillation loss, and the above mask distillation loss to obtain the model loss of the segmentation model (as shown in formula (9) below); and then update the segmentation model based on the model loss.
[0188] Where, L all Represents the model loss of the above segmentation model; L cls Represents the above classification loss; L mask Represents the above mask loss; Represents the semantic distillation loss above; represents the mask distillation loss; λ1, λ2, and λ3 are preset weighting coefficients.
[0189] Based on the relevant content of steps 241 to 242 above, it can be seen that in some application scenarios, the segmentation model can be updated with the help of the above classification loss, the above mask loss, the above semantic distillation loss, and the above mask distillation loss, so that the updated segmentation model has better performance.
[0190] Example 2: Step 24 above may specifically include steps 243 and 244 below.
[0191] Step 243: Determine the semantic segmentation loss based on the semantic segmentation label, the first segmentation mask, and the similarity between the first semantic feature and the category representation feature.
[0192] It should be noted that the present disclosure does not limit the implementation of the above step 243. For example, it can be implemented using the determination process of the semantic segmentation loss shown above.
[0193] Step 244: Update the segmentation model according to the above semantic segmentation loss, the above semantic distillation loss, and the above mask distillation loss.
[0194] It should be noted that the embodiment of the present disclosure does not limit the implementation method of the above step 244. For example, it can be implemented by adopting any existing or future method for updating the model based on multiple losses.
[0195] For example, in one possible implementation, the above step 244 can be: first perform a weighted summation of the above semantic segmentation, the above semantic distillation loss, and the above mask distillation loss to obtain the model loss of the segmentation model (as shown in formula (10) below); and then update the segmentation model based on the model loss.
[0196] Where, Lall Represents the model loss of the above segmentation model; L SS Represents the semantic segmentation loss above; Represents the semantic distillation loss above; represents the mask distillation loss; λ4 and λ5 are preset weighting coefficients.
[0197] Based on the relevant content of steps 243 to 244 above, it can be seen that in some application scenarios, the segmentation model can be updated with the help of the above semantic segmentation loss, the above semantic distillation loss, and the above mask distillation loss, so that the updated segmentation model has better performance.
[0198] In fact, in order to better improve the model training effect, the present disclosure also provides a possible implementation of the above step 24. In this implementation, step 24 can be specifically as follows: if the current learning stage is used for the first training of the segmentation model without a classifier (that is, the current learning stage is the first learning stage), then the above semantic segmentation label, the above first segmentation mask, the similarity between the above first semantic feature and the category representation feature, the above semantic distillation loss, and the above mask distillation loss are used to update all parameters in the segmentation model; however, if the current learning stage is used for non-first training of the segmentation model without a classifier (that is, the current learning stage is the t-th learning stage, t≥2), then when using the semantic segmentation label, the first segmentation mask, the similarity between the first semantic feature and the category representation feature, the semantic distillation loss, and the mask distillation loss to update the segmentation model, the parameters of the normalization layer are fixed (for example, the parameters of other network layers in the segmentation model except the normalization layer are updated using the semantic segmentation label, the first segmentation mask, the similarity between the first semantic feature and the category representation feature, the semantic distillation loss, and the mask distillation loss).
[0199] Based on the relevant contents of steps 21 to 24 above, it can be seen that in some application scenarios, for any round of training process in the current learning stage, the training process of the segmentation model in the current learning stage can be guided by the segmentation model without classifier trained in the previous learning stage, so as to achieve learning new categories without forgetting the old categories. This can effectively avoid the adverse effects caused by forgetting the old categories, thereby helping to improve the stability of the model and further improve the model training effect. It can be seen that the present disclosure effectively retains the old knowledge information in the old model by limiting the update of the new model parameters, so that a powerful and stable incremental segmentation model can be obtained.
[0200] Based on the above model update process for the segmentation model, it can be seen that in one possible implementation, in order to ensure that different learning stages are normalized using the same standard, the present disclosure also provides a possible implementation of the above S103. In this implementation, if the segmentation model being used in the current learning stage (for example, the t-th learning stage) has learned the second category in the historical learning stage (for example, the previous t-1 learning stages), then the S103 can be specifically: updating the segmentation model according to the above semantic segmentation label, the above first segmentation mask, and the similarity between the above first semantic feature and the category representation feature, and when updating the segmentation model, fixing the parameters of the normalization layer of the segmentation model (for example, updating the parameters of other network layers in the segmentation model except the normalization layer according to the above semantic segmentation label, the above first segmentation mask, and the similarity between the above first semantic feature and the category representation feature, etc.), so that the adverse effects caused by changes in the normalization layer as the model is updated can be effectively avoided, thereby helping to improve the model training effect.
[0201] Based on the relevant contents of S101 to S103 above, it can be known that for a segmentation model without a classifier, the training process involved in the segmentation model in any learning stage is specifically as follows: if the learning stage is used to learn the first category (for example, a new category), then after obtaining the training data corresponding to the first category (that is, the training image and the semantic segmentation label of the training image), the training image is first processed using the segmentation model to obtain a first segmentation mask and a first semantic feature, so that the first segmentation mask and the first semantic feature can represent the visual information carried by the training image; then, based on the semantic segmentation label, the first segmentation mask, and the similarity between the first semantic feature and the category representation feature corresponding to the category identification set involved in the learning stage, the segmentation model is updated so that the updated segmentation model has better performance. In this way, the training process for a segmentation model without a classifier can be realized, so that the semantic segmentation task can be achieved subsequently with the help of the trained segmentation model.
[0202] In addition, since there is no classifier in the segmentation model provided by the present invention, the model structure of the segmentation model does not need to change as the number of categories increases, so that the segmentation model can be adapted to learn any number of new categories, which can effectively improve the plasticity of the segmentation model and thus help improve the flexibility of semantic segmentation.
[0203] In addition, the present disclosure does not limit the execution subject of the above model training method. For example, the model training method provided in the embodiment of the present disclosure can be applied to devices with data processing functions such as terminal devices or servers. For another example, the model training method provided in the embodiment of the present disclosure can also be implemented with the help of data communication processes between different devices (for example, a terminal device and a server, two terminal devices, or two servers). Among them, the terminal device can be a smart phone, a computer, a personal digital assistant (PDA) or a tablet computer, etc. The server can be a stand-alone server, a cluster server or a cloud server.
[0204] In fact, in order to better understand the technical solution provided by the present disclosure, the following is an example of a model training process implemented in an incremental learning manner.
[0205] As an example, when the classifier-free segmentation model provided by the present disclosure is trained using an incremental learning method, the model training method provided by the present disclosure may include the following steps 31 to 37.
[0206] Step 31: Divide the training data set into T training data subsets, where different training data subsets include training data corresponding to different categories, and T is a positive integer.
[0207] The training dataset refers to a dataset required for reference when training a segmentation model without a classifier; and the present disclosure is not limited to the training dataset, for example, it includes a large amount of image data and semantic segmentation labels of each image data.
[0208] The t-th training data subset is used to participate in the t-th training (i.e., the t-th learning phase) for the above segmentation model without a classifier, where t is a positive integer, 1≤t≤T.
[0209] It should be noted that the present disclosure does not limit the implementation of the above step 31. For example, the training data set may be divided according to categories so that different training data subsets obtained by division include training data corresponding to different categories.
[0210] Step 32: Initialize t to 1 (ie, t=1).
[0211] Step 33: Acquire training data corresponding to the first category from the tth training data subset, where the training data includes a training image and a semantic segmentation label of the training image, and the semantic segmentation label includes the first category.
[0212] It should be noted that the present disclosure does not limit the implementation method of "obtaining training data corresponding to the first category from the tth training data subset" in step 33 above. For example, it can be implemented using any existing or future method that can extract training data from a data set.
[0213] It should also be noted that the relevant content of the “training data corresponding to the first category” in the above step 33 can be found in the above S101, and for the sake of brevity, it will not be repeated here.
[0214] Step 34: Process the training image using the classifier-free segmentation model to obtain a first segmentation mask and a first semantic feature.
[0215] It should be noted that the relevant content of step 34 can be found in S102 above, and for the sake of brevity, it will not be repeated here.
[0216] Step 35: Determine whether the preset stop condition is met. If so, execute step 37 below; if not, execute step 36 below.
[0217] Among them, the preset stopping condition refers to the condition that needs to be met when ending the training process for the segmentation model in the current learning stage; and the present disclosure does not limit the preset stopping condition. For example, it can be: the model loss of the segmentation model in the current learning stage is lower than the preset first threshold, or the rate of change of the model loss of the segmentation model in the current learning stage is lower than the preset second threshold, or the number of updates of the segmentation model in the current learning stage is higher than the preset third threshold.
[0218] Step 36: If the preset stopping condition is not met, the segmentation model is updated based on the semantic segmentation label, the first segmentation mask, and the similarity between the first semantic feature and the category representation feature, and the process returns to step 33 and subsequent steps. The category representation feature is obtained by encoding a set of category identifiers determined based on the first category using a pre-built text encoder.
[0219] In the present disclosure, for a round of training process being executed in the tth learning stage, if it is determined that the preset stopping condition has not been reached, it can be determined that the segmentation model without a classifier has not yet learned the category (for example, the first category) that needs to be learned in the tth learning stage. Therefore, the segmentation model can be updated based on the above semantic segmentation label, the first segmentation mask, and the similarity between the first semantic feature and the category representation feature, so that the updated segmentation model has better performance, so that it can subsequently return to execute the above step 33 and its subsequent steps based on the updated segmentation model to start the next round of training process for the segmentation model, and iterate until the preset stopping condition is reached.
[0220] It should be noted that the relevant content of the step "updating the segmentation model according to the semantic segmentation label, the first segmentation mask, and the similarity between the first semantic feature and the category representation feature" in step 36 can be found in S103 above. For the sake of brevity, it will not be repeated here.
[0221] Step 37: If the preset stopping condition is reached, determine whether T learning stages for the segmentation model are completed. If so, end the training process for the segmentation model without a classifier; if not, update t and return to continue executing the above step 33 and its subsequent steps.
[0222] Among them, the T learning stages include the first learning stage implemented using the first training data subset, the second learning stage implemented using the second training data subset,... (and so on), and the Tth learning stage implemented using the Tth training data subset.
[0223] In addition, the present disclosure does not limit the implementation method of "determining whether T learning stages for the segmentation model have been completed" in step 37 above. For example, it can be specifically: determine whether t is equal to T. If it is determined that t is equal to T, it can be determined that T learning stages for the segmentation model have been completed; if it is determined that t is less than T, it can be determined that T learning stages for the segmentation model have not yet been completed.
[0224] It can be seen that in some application scenarios, for a round of training process being executed in the t-th learning stage, if it is determined that the preset stop condition has been reached, it can be determined that the segmentation model without a classifier has learned the category (for example, the first category) required to be learned in the t-th learning stage, so that it can be determined that the learning task of the t-th learning stage has been completed, so the t-th training process for the segmentation model can be directly ended, and it is judged whether t is equal to T. If t is equal to T, it can be determined that T learning stages for the segmentation model have been completed, so that it can be determined that the incremental learning process for the segmentation model has been completed, so the training process for the segmentation model without a classifier can be directly ended; however, if t<T, it can be determined that the incremental learning process for the segmentation model has not been completed, so t can be updated using the following formula (11), and based on the updated t, return to execute the above step 33 and its subsequent steps to start the next learning stage for the segmentation model, and iterate until t=T. t'=t+1 (11)
[0225] Where t' represents the updated t.
[0226] Based on the relevant content of steps 31 to 37 above, it can be seen that in one possible implementation, for the classifier-free segmentation model, it can be trained using the incremental learning method shown in steps 31 to 37, so that the segmentation model can be prompted to learn new categories in the future using the incremental learning method. In addition, for the t-th learning stage, if 2≤t≤T, then the segmentation model used in the t-th learning stage is initialized using the reference model, and the reference model refers to the classifier-free segmentation model trained using the first t-1 training data subsets (that is, the classifier-free segmentation model obtained after completing the first t-1 learning stages).
[0227] In fact, for a segmentation model that has been trained without a classifier (for example, a segmentation model trained using the process shown in steps 31 to 37 above), the segmentation model can continue to learn new categories, and the learning process for the new category is similar to the learning process shown in the non-first learning stage above (for example, the learning process shown in the tth learning stage, and t≥2). For the sake of brevity, it will not be repeated here.
[0228] Based on the relevant content of the model training method above, the present disclosure also provides a semantic segmentation method, which is described below with reference to the accompanying drawings for ease of understanding. As shown in Figure 3, the semantic segmentation method provided by the embodiment of the present disclosure includes the following S301-S303. Among them, Figure 3 is a flowchart of a semantic segmentation method provided by an embodiment of the present disclosure.
[0229] S301: Obtain an image to be segmented.
[0230] The image to be segmented refers to image data that needs to be semantically segmented; and the present disclosure is not limited to the image to be segmented.
[0231] S302: Process the image to be segmented using a segmentation model without a classifier to obtain a third segmentation mask and a third semantic feature; the segmentation model is obtained using any implementation of the model training method provided in the embodiments of the present disclosure.
[0232] Among them, the classifier-free segmentation model is used to perform visual processing (such as segmentation mask determination processing of unknown categories and high-dimensional semantic feature determination processing, etc.) on the input data of the segmentation model (such as image data); and the segmentation model is obtained by using any implementation method of the model training method provided in the embodiments of the present disclosure. For the sake of brevity, it will not be repeated here.
[0233] The third segmentation mask is a segmentation mask of unknown category obtained by processing the image to be segmented using a segmentation model without a classifier, so that the third segmentation mask can represent the mask state presented by the image to be segmented at the N feature channel layers. It should be noted that this third segmentation mask is similar to the first segmentation mask above and will not be repeated here for the sake of brevity.
[0234] The third semantic feature is a high-dimensional semantic feature obtained by processing the image to be segmented using a classifier-free segmentation model. This feature represents the semantic information presented by the image to be segmented at the N feature channel levels. It should be noted that this third semantic feature is similar to the first semantic feature described above and will not be further described here for the sake of brevity.
[0235] S303: Determine the semantic segmentation result of the image to be segmented based on the third segmentation mask and the similarity between the third semantic feature and the category representation feature corresponding to the segmentation model; the category representation feature corresponding to the segmentation model is obtained by encoding the category identification set corresponding to the segmentation model using a pre-built text encoder.
[0236] The category representation feature corresponding to the segmentation model refers to the category representation feature corresponding to the trained segmentation model without a classifier, so that the category representation feature can represent all categories involved in training the segmentation model.
[0237] In addition, the "category representation features corresponding to the segmentation model" mentioned above are obtained by encoding the category identifier set corresponding to the segmentation model using a pre-built text encoder. The "category identifier set corresponding to the segmentation model" is used to record all categories involved in training the segmentation model (for example, all categories involved in the training dataset mentioned above).
[0238] In addition, the process of determining the “category representation features corresponding to the segmentation model” is similar to the process of determining the “category representation features” above. For the sake of brevity, it will not be repeated here.
[0239] The semantic segmentation result of the image to be segmented is used to indicate the category to which each pixel in the image to be segmented belongs. The process of determining the semantic segmentation result of the image to be segmented is similar to the process of determining the semantic segmentation result of the training image above. For the sake of brevity, we will not elaborate on this here. For easier understanding, we will explain it with examples below.
[0240] As an example, the process of determining the semantic segmentation result of the image to be segmented includes the following steps 41 to 43.
[0241] Step 41: Determine an image corresponding to a third segmentation mask using the image to be segmented.
[0242] The image corresponding to the third segmentation mask refers to the image extracted from the image to be segmented using the third segmentation mask, so that the "image corresponding to the third segmentation mask" can represent the image information in the image area corresponding to the third segmentation mask in the image to be segmented.
[0243] In addition, the present disclosure does not limit the implementation of the above step 41. For example, the implementation of the above step 41 is similar to the implementation of the above step 111.
[0244] Step 42: Input the image corresponding to the third segmentation mask into a pre-built image classification model to obtain an image classification result output by the image classification model, so that the image classification result can represent category prediction information for the image corresponding to the third segmentation mask.
[0245] For details about the image classification model, please refer to step 112 above.
[0246] In addition, the present disclosure does not limit the implementation of the above step 412. For example, when the above image classification model includes an image encoder (for example, a visual encoder in CLIP), the step 412 can be specifically as follows: using the image encoder to perform encoding processing on the image corresponding to the above third segmentation mask to obtain the image code corresponding to the third segmentation mask, so that the image code can represent the image information carried by the image corresponding to the third segmentation mask; then, based on the similarity between the image code corresponding to the third segmentation mask and the category representation feature corresponding to the above segmentation model, determine the image classification result, so that the image classification result can represent the category prediction information for the image corresponding to the third segmentation mask. The image encoder is used to perform encoding processing on an image data; and the present disclosure does not limit the image encoder.
[0247] It can be seen that in a possible implementation, when the above step 412 is implemented with the help of CLIP, the step 412 can be specifically as follows: first, the image corresponding to the above third segmentation mask is encoded using the visual encoder in CLIP to obtain the image encoding of the image corresponding to the third segmentation mask; then, the similarity between the image encoding of the image corresponding to the third segmentation mask and the above category representation feature is determined as the image classification result corresponding to the third segmentation mask. So that the image classification results can be used later Determine the following classification prediction information S all .
[0248] Step 43: Determine a semantic segmentation result of the image to be segmented based on the above image classification result, the above third segmentation mask, and the similarity between the above third semantic feature and the category representation feature corresponding to the segmentation model.
[0249] It should be noted that the present disclosure does not limit the implementation of step 43. For example, step 43 may specifically include the following steps 431 and 432.
[0250] Step 431: Determine classification prediction information based on the above image classification result and the similarity between the above third semantic feature and the category representation feature corresponding to the segmentation model, so that the classification prediction information is used to represent the classification information determined for the image to be segmented.
[0251] It should be noted that the implementation of step 431 is similar to the implementation of step 113 above, and for the sake of brevity, it will not be repeated here.
[0252] Step 432: Determine a semantic segmentation result of the image to be segmented based on the above classification prediction information and the above third segmentation mask, so that the semantic segmentation result is used to represent the predicted category of each pixel in the image to be segmented.
[0253] It should be noted that the implementation of step 432 is similar to the implementation of step 121 above, and for the sake of brevity, it will not be repeated here.
[0254] Based on the relevant content of steps 41 to 43 above, it can be seen that in some application scenarios, after determining the third segmentation mask and the third semantic feature from the image to be segmented, the image classification result corresponding to the image to be segmented can be determined based on the third segmentation mask and the visual encoder in CLIP; then, using the image classification result, the similarity between the third semantic feature and the category representation feature corresponding to the segmentation model is corrected to obtain the classification prediction information corresponding to the image to be segmented; finally, based on the classification prediction information and the third segmentation mask, the semantic segmentation result of the image to be segmented is determined, so that the semantic segmentation result is used to represent the predicted category of each pixel point in the image to be segmented, which is conducive to improving the accuracy of semantic segmentation.
[0255] Based on the relevant contents of S301 to S303 above, it can be known that for the semantic segmentation method provided by the embodiment of the present disclosure, after the classifier-free segmentation model is trained, the segmentation model can be directly used to process any image to be segmented to obtain a third segmentation mask and a third semantic feature; then, based on the third segmentation mask and the similarity between the third semantic feature and the category representation feature corresponding to the segmentation model, the semantic segmentation result of the image to be segmented is determined, so that the semantic segmentation result can represent the category to which each pixel in the image to be segmented belongs, thereby enabling semantic segmentation to be performed with the help of a pre-built classifier-free segmentation model. Among them, since there is no classifier in the segmentation model, the model structure of the segmentation model will not change with the increase of categories, so that the segmentation model can be adapted to learn any number of new categories, which can effectively improve the plasticity of the segmentation model and thus effectively improve the flexibility of the semantic segmentation processing.
[0256] In addition, the present disclosure does not limit the execution subject of the above semantic segmentation method. For example, the semantic segmentation method provided in the embodiments of the present disclosure can be applied to a device with data processing capabilities, such as a terminal device or a server. For another example, the semantic segmentation method provided in the embodiments of the present disclosure can also be implemented through a data communication process between different devices (e.g., a terminal device and a server, two terminal devices, or two servers).
[0257] Based on the model training method provided in the embodiment of the present disclosure, the embodiment of the present disclosure also provides a model training device, which is explained and illustrated below in conjunction with Figure 4. Figure 4 is a schematic diagram of the structure of a model training device provided in the embodiment of the present disclosure. It should be noted that for the technical details of the model training device provided in the embodiment of the present disclosure, please refer to the relevant content of the model training method above.
[0258] As shown in FIG4 , the model training device 400 provided in an embodiment of the present disclosure includes:
[0259] A first acquisition unit 401 is configured to acquire training data corresponding to a first category, wherein the training data includes a training image and a semantic segmentation label of the training image, and the semantic segmentation label includes the first category;
[0260] A first processing unit 402 is configured to process the training image using a segmentation model without a classifier to obtain a first segmentation mask and a first semantic feature;
[0261] The model updating unit 403 is used to update the segmentation model based on the semantic segmentation label, the first segmentation mask, and the similarity between the first semantic feature and the category representation feature, where the category representation feature is obtained by encoding a category identification set using a pre-built text encoder, and the category identification set is determined based on the first category.
[0262] In a possible implementation manner, the model updating unit 403 includes:
[0263] a first determining subunit, configured to determine classification prediction information based on a similarity between the first semantic feature and a category representation feature;
[0264] A first updating subunit is configured to update the segmentation model according to the semantic segmentation label, the first segmentation mask, and the classification prediction information.
[0265] In one possible implementation, the first update subunit is specifically used to: determine the semantic segmentation result of the training image based on the first segmentation mask and the classification prediction information; determine the semantic segmentation loss based on the semantic segmentation label and the semantic segmentation result; and update the segmentation model based on the semantic segmentation loss.
[0266] In one possible implementation, the model training device 400 further includes:
[0267] an image determining unit, configured to determine an image corresponding to the first segmentation mask using the training image;
[0268] an image classification unit, configured to input the image corresponding to the first segmentation mask into a pre-built image classification model, and obtain an image classification result output by the image classification model;
[0269] The first determining subunit is specifically configured to determine the classification prediction information based on the image classification result and the similarity between the first semantic feature and the category representation feature.
[0270] In one possible implementation, the first update subunit is specifically used to: determine the classification loss based on the semantic segmentation label and the classification prediction information; determine the mask loss based on the semantic segmentation label and the first segmentation mask; and update the segmentation model based on the classification loss and the mask loss.
[0271] In one possible implementation, the segmentation model is initialized using a reference model; the reference model is a classifier-free segmentation model trained using training data corresponding to the second category; the segmentation model includes a normalization layer; and when updating the segmentation model, the parameters of the normalization layer of the segmentation model are fixed.
[0272] In one possible implementation, the segmentation model is initialized using a reference model; the reference model is a segmentation model trained using training data corresponding to the second category; the reference model is used to process the training image to obtain a second segmentation mask and a second semantic feature;
[0273] The model updating unit 403 is specifically used to: determine the semantic distillation loss based on the first semantic feature and the second semantic feature; determine the mask distillation loss based on the first segmentation mask and the second segmentation mask; and update the segmentation model based on the semantic segmentation label, the first segmentation mask, the similarity between the first semantic feature and the category representation feature, the semantic distillation loss, and the mask distillation loss.
[0274] In a possible implementation, the model training device 400 further includes:
[0275] A data partitioning unit is used to divide the training data set into T training data subsets, where different training data subsets include training data corresponding to different categories, and T is a positive integer;
[0276] The first acquisition unit 401 is specifically configured to acquire training data corresponding to the first category from the t-th training data subset, where t is a positive integer and t≤T.
[0277] In a possible implementation, the model training device 400 further includes:
[0278] Initialization unit, used to initialize t to 1;
[0279] An iterative unit is used to, after updating the segmentation model based on the semantic segmentation label, the first segmentation mask, and the similarity between the first semantic feature and the category representation feature, return to the first acquisition unit 401 to continue executing the step of acquiring the training data corresponding to the first category from the tth training data subset, until a preset stopping condition is reached, update t, and return to the first acquisition unit 401 to continue executing the step of acquiring the training data corresponding to the first category from the tth training data subset, until the segmentation model is trained using the T training data subsets.
[0280] In one possible implementation, if 2≤t≤T, the segmentation model is initialized using a reference model; the reference model is a classifier-free segmentation model trained using the first t-1 training data subsets.
[0281] Based on the relevant content of the above-mentioned model training device 400, it can be known that for the model training device 400 provided in the embodiment of the present disclosure, the model training device 400 is used to implement a training process for a segmentation model without a classifier, and the model training device 400 is specifically used to: if the segmentation model is used to learn a first category (for example, a new category), then after obtaining the training data corresponding to the first category (that is, the training image and the semantic segmentation label of the training image), the training image is first processed using the segmentation model to obtain a first segmentation mask and a first semantic feature, so that the first segmentation mask and the first semantic feature can represent the visual information carried by the training image; then, based on the semantic segmentation label, the first segmentation mask, and the similarity between the first semantic feature and the category representation feature corresponding to the category identification set involved in the learning stage, the segmentation model is updated so that the updated segmentation model has better performance, so that the training process for a segmentation model without a classifier can be implemented, so that the semantic segmentation task can be implemented subsequently with the help of the trained segmentation model.
[0282] In addition, since there is no classifier in the segmentation model provided by the present invention, the model structure of the segmentation model will not change with the increase of categories, so that the segmentation model can be adapted to learn any number of new categories, which can effectively improve the plasticity of the segmentation model, thereby facilitating the improvement of the flexibility of semantic segmentation.
[0283] Based on the semantic segmentation method provided in the embodiments of the present disclosure, the embodiments of the present disclosure also provide a semantic segmentation device, which will be explained and illustrated below in conjunction with Figure 5. Figure 5 is a schematic diagram of the structure of a semantic segmentation device provided in the embodiments of the present disclosure. It should be noted that for the technical details of the semantic segmentation device provided in the embodiments of the present disclosure, please refer to the relevant content of the semantic segmentation method above.
[0284] As shown in FIG5 , the semantic segmentation apparatus 500 provided in an embodiment of the present disclosure includes:
[0285] The second acquisition unit 501 is used to acquire the image to be segmented;
[0286] a second processing unit 502 configured to process the image to be segmented using a classifier-free segmentation model to obtain a third segmentation mask and a third semantic feature; wherein the segmentation model is obtained using the model training method according to any one of claims 1 to 12;
[0287] The result determination unit 503 is used to determine the semantic segmentation result of the image to be segmented based on the third segmentation mask and the similarity between the third semantic feature and the category representation feature corresponding to the segmentation model; the category representation feature corresponding to the segmentation model is obtained by encoding the category identification set corresponding to the segmentation model using a pre-built text encoder.
[0288] In a possible implementation manner, the semantic segmentation device 500 further includes:
[0289] a data determining unit, configured to determine an image corresponding to the third segmentation mask using the image to be segmented;
[0290] a data classification unit, configured to input the image corresponding to the third segmentation mask into a pre-built image classification model to obtain an image classification result output by the image classification model;
[0291] The result determination unit 503 is configured to determine a semantic segmentation result of the image to be segmented based on the image classification result, the third segmentation mask, and the similarity between the third semantic feature and the category representation feature corresponding to the segmentation model.
[0292] In one possible implementation, the image classification model includes an image encoder; the image encoder is configured to perform encoding processing on the image corresponding to the third segmentation mask to obtain an image code corresponding to the third segmentation mask; and the image classification result is determined based on a similarity between the image code corresponding to the third segmentation mask and a category representation feature corresponding to the segmentation model:
[0293] The result determination unit 503 includes:
[0294] a second determining subunit, configured to determine the classification prediction information based on the image classification result and the similarity between the third semantic feature and the category representation feature corresponding to the segmentation model;
[0295] The third determining subunit is configured to determine a semantic segmentation result of the image to be segmented according to the classification prediction information and the third segmentation mask.
[0296] Based on the relevant content of the above-mentioned semantic segmentation device 500, it can be known that for the semantic segmentation device 500 provided by the embodiment of the present disclosure, after obtaining the trained classifier-free segmentation model, the segmentation model can be directly used to process any image to be segmented to obtain a third segmentation mask and a third semantic feature; then, based on the third segmentation mask and the similarity between the third semantic feature and the category representation feature corresponding to the segmentation model, the semantic segmentation result of the image to be segmented is determined, so that the semantic segmentation result can represent the category to which each pixel point in the image to be segmented belongs, thereby realizing semantic segmentation with the help of a pre-built classifier-free segmentation model. Among them, because there is no classifier in the segmentation model, the model structure of the segmentation model will not change with the increase of categories, so that the segmentation model can be adapted to learn any number of new categories, which can effectively improve the plasticity of the segmentation model, thereby effectively improving the flexibility of the semantic segmentation processing.
[0297] In addition, an embodiment of the present disclosure also provides an electronic device, which includes a processor and a memory: the memory is used to store instructions or computer programs; the processor is used to execute the instructions or computer programs in the memory, so that the electronic device executes any implementation of the model training method provided by the embodiment of the present disclosure or executes any implementation of the semantic segmentation method provided by the embodiment of the present disclosure.
[0298] Referring to FIG6 , a schematic diagram of the structure of an electronic device 600 suitable for implementing embodiments of the present disclosure is shown. Terminal devices in embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. The electronic device shown in FIG6 is merely an example and should not limit the functionality or scope of use of embodiments of the present disclosure.
[0299] As shown in Figure 6, electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. Various programs and data required for the operation of electronic device 600 are also stored in RAM 603. Processing device 601, ROM 602, and RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to bus 604.
[0300] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although FIG. 6 shows the electronic device 600 with various devices, it should be understood that not all of the devices shown are required to be implemented or present. More or fewer devices may alternatively be implemented or present.
[0301] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.
[0302] The electronic device provided by the embodiment of the present disclosure and the method provided by the above embodiment belong to the same inventive concept. For technical details not fully described in this embodiment, please refer to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.
[0303] The embodiments of the present disclosure also provide a computer-readable medium, which stores instructions or computer programs. When the instructions or computer programs are executed on a device, the device executes any implementation of the model training method provided by the embodiments of the present disclosure or executes any implementation of the semantic segmentation method provided by the embodiments of the present disclosure.
[0304] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.
[0305] In some embodiments, the client and server can communicate using any currently known or later developed network protocol, such as HTTP (Hypertext Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or later developed network.
[0306] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.
[0307] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device can perform the method.
[0308] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including, but not limited to, object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).
[0309] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0310] The units involved in the embodiments described in this disclosure may be implemented in software or hardware, wherein the name of a unit / module does not, in some cases, limit the unit itself.
[0311] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.
[0312] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0313] It should be noted that the various embodiments of this disclosure are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the descriptions of the systems or devices disclosed in the embodiments for similarities and differences between them. Since the systems or devices disclosed in the embodiments correspond to the methods disclosed in the embodiments, their descriptions are relatively simple, and reference can be made to the descriptions of the methods for any related details.
[0314] It should be understood that in the present disclosure, "at least one (item)" refers to one or more, and "plurality" refers to two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.
[0315] It should also be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.
[0316] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
[0317] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present disclosure. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure is not limited to the embodiments shown herein, but is intended to be construed in the widest manner consistent with the principles and novel features disclosed herein.
Claims
1. A model training method, comprising: Acquire training data corresponding to the first category, the training data including a training image and a semantic segmentation label of the training image, the semantic segmentation label including the first category; Processing the training image using a segmentation model without a classifier to obtain a first segmentation mask and a first semantic feature; The segmentation model is updated according to the semantic segmentation label, the first segmentation mask, and the similarity between the first semantic feature and the category characterization feature, wherein the category characterization feature is obtained by encoding a category identification set using a pre-built text encoder, and the category identification set is determined based on the first category.
2. The method according to claim 1, wherein: The updating of the segmentation model according to the semantic segmentation label, the first segmentation mask, and the similarity between the first semantic feature and the category representation feature includes: Determining classification prediction information according to the similarity between the first semantic feature and the category representation feature; The segmentation model is updated according to the semantic segmentation label, the first segmentation mask and the classification prediction information.
3. The method according to claim 2, wherein: The updating of the segmentation model according to the semantic segmentation label, the first segmentation mask and the classification prediction information includes: Determining a semantic segmentation result of the training image according to the first segmentation mask and the classification prediction information; Determining a semantic segmentation loss according to the semantic segmentation label and the semantic segmentation result; The segmentation model is updated according to the semantic segmentation loss.
4. The method according to claim 2, wherein: Before determining the classification prediction information according to the similarity between the first semantic feature and the category representation feature, the method further includes: Determining an image corresponding to the first segmentation mask using the training image; Inputting the image corresponding to the first segmentation mask into a pre-built image classification model to obtain an image classification result output by the image classification model; The method further comprises determining a classification prediction based on the similarity between the first semantic feature and the category representation feature. Measurement information, including: The classification prediction information is determined according to the image classification result and the similarity between the first semantic feature and the category representation feature.
5. The method according to claim 2, wherein: The updating of the segmentation model according to the semantic segmentation label, the first segmentation mask and the classification prediction information includes: Determining a classification loss according to the semantic segmentation label and the classification prediction information; Determining a mask loss according to the semantic segmentation label and the first segmentation mask; The segmentation model is updated according to the classification loss and the mask loss.
6. The method according to claim 1, wherein: The segmentation model is initialized using a reference model; the reference model is a segmentation model without a classifier obtained by training using training data corresponding to the second category; The segmentation model includes a normalization layer; When the segmentation model is updated, parameters of a normalization layer of the segmentation model are fixed.
7. The method according to claim 1, wherein: The segmentation model is initialized using a reference model; the reference model is a segmentation model without a classifier obtained by training using training data corresponding to the second category; The reference model is used to process the training image to obtain a second segmentation mask and a second semantic feature; Before updating the segmentation model according to the semantic segmentation label, the first segmentation mask, and the similarity between the first semantic feature and the category representation feature, the method further includes: Determining a semantic distillation loss according to the first semantic feature and the second semantic feature; Determining a mask distillation loss according to the first segmentation mask and the second segmentation mask; The updating of the segmentation model according to the semantic segmentation label, the first segmentation mask, and the similarity between the first semantic feature and the category representation feature includes: The segmentation model is updated according to the semantic segmentation label, the first segmentation mask, the similarity between the first semantic feature and the category representation feature, the semantic distillation loss, and the mask distillation loss.
8. The method according to any one of claims 1 to 7, further comprising: The training data set is divided into T training data subsets, where different training data subsets include training data corresponding to different categories, and T is a positive integer; The obtaining of training data corresponding to the first category includes: The training data corresponding to the first category is obtained from the t-th training data subset, where t is a positive integer and t≤T.
9. The method according to claim 8, further comprising: Initialize t to 1; After updating the segmentation model according to the semantic segmentation label, the first segmentation mask, and the similarity between the first semantic feature and the category representation feature, the method further includes: Continue to execute the step of obtaining the training data corresponding to the first category from the tth training data subset until a preset stopping condition is reached, update t, and continue to execute the step of obtaining the training data corresponding to the first category from the tth training data subset until the segmentation model is trained using the T training data subsets.
10. The method according to claim 8, wherein: If 2≤t≤T, the segmentation model is initialized using a reference model; the reference model refers to a segmentation model without a classifier trained using the first t-1 training data subsets.
11. A semantic segmentation method, comprising: Obtain the image to be segmented; Processing the image to be segmented by using a segmentation model without a classifier to obtain a third segmentation mask and a third semantic feature; The segmentation model is obtained by using the model training method described in any one of claims 1 to 10; The semantic segmentation result of the image to be segmented is determined based on the third segmentation mask and the similarity between the third semantic feature and the category representation feature corresponding to the segmentation model; the category representation feature corresponding to the segmentation model is obtained by encoding the category identification set corresponding to the segmentation model using a pre-built text encoder.
12. The method according to claim 11, wherein: Before determining the semantic segmentation result of the image to be segmented according to the third segmentation mask and the similarity between the third semantic feature and the category representation feature corresponding to the segmentation model, the method further includes: The process of determining the semantic segmentation result of the image to be segmented includes: Determining an image corresponding to the third segmentation mask using the image to be segmented; Inputting the image corresponding to the third segmentation mask into a pre-built image classification model to obtain an image classification result output by the image classification model; The determining, according to the third segmentation mask and the similarity between the third semantic feature and the category representation feature corresponding to the segmentation model, a semantic segmentation result of the image to be segmented includes: A semantic segmentation result of the image to be segmented is determined according to the image classification result, the third segmentation mask, and the similarity between the third semantic feature and the category representation feature corresponding to the segmentation model.
13. The method according to claim 12, wherein: The image classification model includes an image encoder; the image encoder is used to encode the image corresponding to the third segmentation mask to obtain the image code corresponding to the third segmentation mask; the image classification result is determined according to the similarity between the image code corresponding to the third segmentation mask and the category representation feature corresponding to the segmentation model: and / or, Determining the semantic segmentation result of the image to be segmented according to the image classification result, the third segmentation mask, and the similarity between the third semantic feature and the category representation feature corresponding to the segmentation model includes: Determining the classification prediction information according to the image classification result and the similarity between the third semantic feature and the category representation feature corresponding to the segmentation model; A semantic segmentation result of the image to be segmented is determined according to the classification prediction information and the third segmentation mask.
14. A model training device, comprising: A first acquisition unit is configured to acquire training data corresponding to a first category, wherein the training data includes a training image and a semantic segmentation label of the training image, and the semantic segmentation label includes the first category; A first processing unit is configured to process the training image using a segmentation model without a classifier to obtain a first segmentation mask and a first semantic feature; A model updating unit is configured to update the segmentation model based on the semantic segmentation label, the first segmentation mask, and the similarity between the first semantic feature and a category representation feature, wherein the category representation feature is obtained by encoding a category identification set using a pre-built text encoder, and the category identification set is determined based on the first category.
15. A semantic segmentation device, comprising: A second acquisition unit is configured to acquire an image to be segmented; A second processing unit is configured to process the image to be segmented by using a segmentation model without a classifier to obtain a third segmentation mask and a third semantic feature; The segmentation model is obtained by using the model training method described in any one of claims 1 to 10; A result determination unit is configured to determine the semantic segmentation result of the image to be segmented based on the third segmentation mask and the similarity between the third semantic feature and the category representation feature corresponding to the segmentation model; the category representation feature corresponding to the segmentation model is obtained by encoding the category identification set corresponding to the segmentation model using a pre-built text encoder.
16. An electronic device, comprising: processor and memory; wherein, The memory is configured to store instructions or computer programs; The processor is configured to execute the instructions or computer programs in the memory so that the electronic device performs the method according to any one of claims 1 to 13.
17. A computer readable medium, wherein: The computer-readable medium stores instructions or computer programs, and when the instructions or computer programs are executed on a device, the device executes the method according to any one of claims 1 to 13.
Citation Information
Patent Citations
Semantic segmentation model training method, semantic segmentation method and related devices
CN112465840A
Semantic segmentation model training method and semantic segmentation method and device
CN114648638A
Data processing method and device
CN115375781A
Zero sample semantic segmentation method, system and equipment based on knowledge distillation and medium
CN115761235A
Method and apparatus for training semantic segmentation model, computer device, and storage medium
US20200294240A1
Cited By
Room reconstruction method and device
CN120543760A
