Model training method and device, semantic segmentation method and device, equipment and medium
By using a classifier-free segmentation model and updating the model with segmentation masks and semantic features from the training images, the problem of model structure changing with category in existing technologies is solved. This achieves adaptability and flexibility to any number of categories and improves the performance of semantic segmentation.
Patent Information
- Application Number
- CN202310538859.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-12
- Publication Date
- 2024-11-12
AI Technical Summary
Existing semantic segmentation models contain classifiers, and their structure changes with the number of categories, making it difficult for the models to quickly adapt to any number of categories, resulting in poor adaptability and flexibility.
A classifier-free segmentation model is adopted. By acquiring the segmentation mask and semantic features of the training image, the model is updated using the similarity between the semantic segmentation label and the category representation feature. This avoids the model structure changing with the increase of categories and adapts to any number of new categories.
It improves the plasticity of the segmentation model and the flexibility of semantic segmentation, enabling the model to adapt to any number of new categories and enhancing the performance of semantic segmentation.
Smart Images

Figure CN118941784A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a model training method, semantic segmentation method, device, equipment, and medium. Background Art
[0002] Semantic segmentation is a basic task in computer vision and can be applied in many fields (for example, autonomous driving and image analysis). Semantic segmentation is used to determine the classification information of each pixel in an image data.
[0003] However, how to implement the semantic segmentation task is a technical problem that needs to be solved urgently. Summary of the invention
[0004] The present application provides a model training method, semantic segmentation method, device, equipment, and medium, which can realize semantic segmentation tasks.
[0005] In order to achieve the above objectives, the technical solutions provided by this application are as follows:
[0006] The present application provides a model training method, the method comprising:
[0007] Acquire training data corresponding to the first category, the training data including a training image and a semantic segmentation label of the training image, the semantic segmentation label including the first category;
[0008] Processing the training image using a segmentation model without a classifier to obtain a first segmentation mask and a first semantic feature;
[0009] The segmentation model is updated according to the semantic segmentation label, the first segmentation mask, and the similarity between the first semantic feature and the category characterization feature, wherein the category characterization feature is obtained by encoding a category identification set using a pre-built text encoder, and the category identification set is determined based on the first category.
[0010] In a possible implementation manner, updating the segmentation model according to the semantic segmentation label, the first segmentation mask, and the similarity between the first semantic feature and the category representation feature includes:
[0011] Determining classification prediction information according to the similarity between the first semantic feature and the category representation feature;
[0012] The segmentation model is updated according to the semantic segmentation label, the first segmentation mask and the classification prediction information.
[0013] In a possible implementation manner, updating the segmentation model according to the semantic segmentation label, the first segmentation mask, and the classification prediction information includes:
[0014] Determining a semantic segmentation result of the training image according to the first segmentation mask and the classification prediction information;
[0015] Determining a semantic segmentation loss according to the semantic segmentation label and the semantic segmentation result;
[0016] The segmentation model is updated according to the semantic segmentation loss.
[0017] In a possible implementation manner, before determining the classification prediction information according to the similarity between the first semantic feature and the category representation feature, the method further includes:
[0018] Determining an image corresponding to the first segmentation mask using the training image;
[0019] Inputting the image corresponding to the first segmentation mask into a pre-built image classification model to obtain an image classification result output by the image classification model;
[0020] The determining of classification prediction information according to the similarity between the first semantic feature and the category representation feature includes:
[0021] The classification prediction information is determined according to the image classification result and the similarity between the first semantic feature and the category representation feature.
[0022] In a possible implementation manner, updating the segmentation model according to the semantic segmentation label, the first segmentation mask, and the classification prediction information includes:
[0023] Determining a classification loss according to the semantic segmentation label and the classification prediction information;
[0024] Determining a mask loss according to the semantic segmentation label and the first segmentation mask;
[0025] The segmentation model is updated according to the classification loss and the mask loss.
[0026] In a possible implementation manner, the segmentation model is initialized using a reference model; the reference model is a segmentation model without a classifier obtained by training using training data corresponding to the second category;
[0027] The segmentation model includes a normalization layer;
[0028] When the segmentation model is updated, parameters of a normalization layer of the segmentation model are fixed.
[0029] In a possible implementation manner, the segmentation model is initialized using a reference model; the reference model is a segmentation model without a classifier obtained by training using training data corresponding to the second category;
[0030] The reference model is used to process the training image to obtain a second segmentation mask and a second semantic feature;
[0031] Before updating the segmentation model according to the semantic segmentation label, the first segmentation mask, and the similarity between the first semantic feature and the category representation feature, the method further includes:
[0032] Determining a semantic distillation loss according to the first semantic feature and the second semantic feature;
[0033] Determining a mask distillation loss according to the first segmentation mask and the second segmentation mask;
[0034] The updating of the segmentation model according to the semantic segmentation label, the first segmentation mask, and the similarity between the first semantic feature and the category representation feature includes:
[0035] The segmentation model is updated according to the semantic segmentation label, the first segmentation mask, the similarity between the first semantic feature and the category representation feature, the semantic distillation loss, and the mask distillation loss.
[0036] In a possible implementation manner, the method further includes:
[0037] The training data set is divided into T training data subsets, where different training data subsets include training data corresponding to different categories, and T is a positive integer;
[0038] The obtaining of training data corresponding to the first category includes:
[0039] The training data corresponding to the first category is obtained from the t-th training data subset, where t is a positive integer and t≤T.
[0040] In a possible implementation manner, the method further includes:
[0041] Initialize t to 1;
[0042] After updating the segmentation model according to the semantic segmentation label, the first segmentation mask, and the similarity between the first semantic feature and the category representation feature, the method further includes:
[0043] Continue to execute the step of obtaining the training data corresponding to the first category from the tth training data subset until a preset stopping condition is reached, update t, and continue to execute the step of obtaining the training data corresponding to the first category from the tth training data subset until the segmentation model is trained using the T training data subsets.
[0044] In a possible implementation, if 2≤t≤T, the segmentation model is initialized using a reference model; the reference model refers to a classifier-free segmentation model trained using the first t-1 training data subsets.
[0045] The present application provides a semantic segmentation method, the method comprising:
[0046] Obtain the image to be segmented;
[0047] Processing the image to be segmented using a segmentation model without a classifier to obtain a third segmentation mask and a third semantic feature; the segmentation model is obtained using the model training method provided in the present application;
[0048] The semantic segmentation result of the image to be segmented is determined based on the third segmentation mask and the similarity between the third semantic feature and the category representation feature corresponding to the segmentation model; the category representation feature corresponding to the segmentation model is obtained by encoding the category identification set corresponding to the segmentation model using a pre-built text encoder.
[0049] In a possible implementation manner, before determining the semantic segmentation result of the image to be segmented according to the third segmentation mask and the similarity between the third semantic feature and the category representation feature corresponding to the segmentation model, the method further includes:
[0050] The process of determining the semantic segmentation result of the image to be segmented includes:
[0051] Determining an image corresponding to the third segmentation mask using the image to be segmented;
[0052] Inputting the image corresponding to the third segmentation mask into a pre-built image classification model to obtain an image classification result output by the image classification model;
[0053] The determining, according to the third segmentation mask and the similarity between the third semantic feature and the category representation feature corresponding to the segmentation model, a semantic segmentation result of the image to be segmented includes:
[0054] A semantic segmentation result of the image to be segmented is determined according to the image classification result, the third segmentation mask, and the similarity between the third semantic feature and the category representation feature corresponding to the segmentation model.
[0055] In a possible implementation manner, the image classification model includes an image encoder; the image encoder is used to encode the image corresponding to the third segmentation mask to obtain the image code corresponding to the third segmentation mask; the image classification result is determined according to the similarity between the image code corresponding to the third segmentation mask and the category representation feature corresponding to the segmentation model:
[0056] and / or,
[0057] Determining the semantic segmentation result of the image to be segmented according to the image classification result, the third segmentation mask, and the similarity between the third semantic feature and the category representation feature corresponding to the segmentation model includes:
[0058] Determining the classification prediction information according to the image classification result and the similarity between the third semantic feature and the category representation feature corresponding to the segmentation model;
[0059] A semantic segmentation result of the image to be segmented is determined according to the classification prediction information and the third segmentation mask.
[0060] The present application provides a model training device, comprising:
[0061] A first acquisition unit, configured to acquire training data corresponding to a first category, wherein the training data includes a training image and a semantic segmentation label of the training image, and the semantic segmentation label includes the first category;
[0062] A first processing unit, configured to process the training image using a segmentation model without a classifier to obtain a first segmentation mask and a first semantic feature;
[0063] A model updating unit is used to update the segmentation model according to the semantic segmentation label, the first segmentation mask, and the similarity between the first semantic feature and a category characterization feature, wherein the category characterization feature is obtained by encoding a category identification set using a pre-built text encoder, and the category identification set is determined according to the first category.
[0064] The present application provides a semantic segmentation device, comprising:
[0065] A second acquisition unit, used for acquiring an image to be segmented;
[0066] A second processing unit is used to process the image to be segmented using a segmentation model without a classifier to obtain a third segmentation mask and a third semantic feature; the segmentation model is obtained using the model training method provided in the present application;
[0067] A result determination unit is used to determine the semantic segmentation result of the image to be segmented based on the third segmentation mask and the similarity between the third semantic feature and the category representation feature corresponding to the segmentation model; the category representation feature corresponding to the segmentation model is obtained by encoding the category identification set corresponding to the segmentation model using a pre-built text encoder.
[0068] The present application provides an electronic device, the device comprising: a processor and a memory;
[0069] The memory is used to store instructions or computer programs;
[0070] The processor is used to execute the instructions or computer programs in the memory so that the electronic device executes the model training method or semantic segmentation method provided in this application.
[0071] The present application provides a computer-readable medium, which stores instructions or computer programs. When the instructions or computer programs are executed on a device, the device executes the model training method or semantic segmentation method provided in the present application.
[0072] The present application provides a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the model training method or semantic segmentation method provided by the present application.
[0073] Compared with the prior art, this application has at least the following advantages:
[0074] The present application provides a classifier-free segmentation model and related technical solutions (for example, a model training method and a semantic segmentation method, etc.). For the segmentation model, the training process involved in any learning stage of the segmentation model is specifically as follows: if the learning stage is used to learn a first category (for example, a new category), then after obtaining the training data corresponding to the first category (that is, a training image and a semantic segmentation label of the training image), the training image is first processed using the segmentation model to obtain a first segmentation mask and a first semantic feature, so that the first segmentation mask and the first semantic feature can represent the visual information carried by the training image; then, based on the similarity between the semantic segmentation label, the first segmentation mask, and the first semantic feature and the category representation feature corresponding to the category identification set involved in the learning stage, the segmentation model is updated so that the updated segmentation model has better performance, so that the training process for a classifier-free segmentation model can be implemented, so that the semantic segmentation task can be implemented subsequently with the help of the trained segmentation model.
[0075] In addition, since there is no classifier in the segmentation model provided in the present application, the model structure of the segmentation model does not need to change with the increase of categories, so that the segmentation model can be adapted to learn any number of new categories, which can effectively improve the plasticity of the segmentation model, thereby facilitating improving the flexibility of semantic segmentation. BRIEF DESCRIPTION OF THE DRAWINGS
[0076] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0077] Figure 1 A flowchart of a model training method provided in an embodiment of the present application;
[0078] Figure 2 A schematic diagram of a training process for a segmentation model without a classifier provided in an embodiment of the present application;
[0079] Figure 3 A flowchart of a semantic segmentation method provided in an embodiment of the present application;
[0080] Figure 4 A schematic diagram of the structure of a model training device provided in an embodiment of the present application;
[0081] Figure 5 A schematic diagram of the structure of a semantic segmentation device provided in an embodiment of the present application;
[0082] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0083] Through research, it is found that in some technical solutions, a pre-trained semantic segmentation model can be used to implement semantic segmentation processing for an image data. Among them, because the structure of the classifier in the semantic segmentation model is determined according to the number of categories, the structure of the semantic segmentation model will change with the change of the number of categories, which makes the model unable to quickly adapt to any number of categories, resulting in poor plasticity of the model, and further resulting in poor flexibility of semantic segmentation.
[0084] Based on the above research content, it can be known that in order to better improve the flexibility of semantic segmentation, the present application provides a classifier-free segmentation model and its related technical solutions (for example, a model training method and a semantic segmentation method, etc.). For the segmentation model, the training process involved in any learning stage of the segmentation model is specifically as follows: if the learning stage is used to learn a first category (for example, a new category), then after obtaining the training data corresponding to the first category (that is, a training image and a semantic segmentation label of the training image), the training image is first processed using the segmentation model to obtain a first segmentation mask and a first semantic feature, so that the first segmentation mask and the first semantic feature can represent the visual information carried by the training image; then, based on the semantic segmentation label, the first segmentation mask, and the similarity between the first semantic feature and the category representation feature corresponding to the category identification set involved in the learning stage, the segmentation model is updated so that the updated segmentation model has better performance, so that the training process for a classifier-free segmentation model can be realized, so that the semantic segmentation task can be achieved subsequently with the help of the trained segmentation model.
[0085] In addition, since there is no classifier in the segmentation model provided in the present application, the model structure of the segmentation model does not need to change with the increase of categories, so that the segmentation model can be adapted to learn any number of new categories, which can effectively improve the plasticity of the segmentation model, thereby facilitating improving the flexibility of semantic segmentation.
[0086] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0087] In order to better understand the technical solution provided by this application, the model training method provided by this application is described below with reference to some drawings. Figure 1 As shown, the model training method provided in the embodiment of the present application includes the following S101-S103. Figure 1 A flowchart of a model training method provided in an embodiment of the present application.
[0088] S101: Acquire training data corresponding to a first category, where the training data includes a training image and a semantic segmentation label of the training image, and the semantic segmentation label includes the first category.
[0089] Among them, the first category refers to the category that needs to be learned in the current learning stage (for example, the "tth learning stage" shown below); and the present application does not limit the first category. For example, when the classifier-free segmentation model provided in the present application is trained using an incremental learning method, if the current learning stage is used to implement the first learning process (that is, the first training) for the segmentation model, then the first category is the category involved in the first learning process; if the current learning stage is used to implement the tth learning process (that is, the tth training) for the segmentation model, t is a positive integer, 2≤t, then the first category is the new category that needs to be learned in the tth learning process.
[0090] The training data corresponding to the first category refers to the training data determined in advance for the first category, so that the semantic segmentation label in the training data involves the first category. It can be seen that when the training data corresponding to the first category includes a training image and a semantic segmentation label of the training image, the semantic segmentation label includes the first category. Among them, the training image is at least used to describe the object under the first category (for example, when the first category is a cat, then at least the object of cat exists in the training image). The semantic segmentation label of the training image is used to describe the actual classification information of each pixel in the training image; and the present application does not limit the acquisition process of the semantic segmentation label of the training image, for example, it can be implemented by means of manual annotation.
[0091] In addition, in the incremental learning stage, the old categories that have been learned are not very important, so in order to save training costs, the old categories in the semantic segmentation labels of the above training image data can be regarded as background. Based on this, the present application also provides a processing process for the semantic segmentation label, which can be specifically: for the t-th learning process, 2≤t, after obtaining a semantic segmentation label corresponding to a training data, if the label information corresponding to some pixels in the semantic segmentation label is an old category label, then the label information corresponding to these pixels in the semantic segmentation label can be directly adjusted to the background category label, so that only the new category label exists in the adjusted semantic segmentation label. Among them, the old category label is used to identify the category that has been learned in the previous t-1 learning processes. The background category label is used to identify the category to which the background pixel in an image data belongs.
[0092] It can be seen that for the t-th learning process, 2≤t, when the new category to be learned in the t-th learning process is the "plant" category, and the old category learned in the previous t-1 learning process is the "furniture" category, if the training data required for the t-th learning process includes training image A, and the training image A contains both plants and furniture, then the semantic segmentation label of the training image A contains labels corresponding to the "plant" category and labels corresponding to the "furniture" category. At this time, it is necessary to remove the "furniture" category in the semantic segmentation label of the training image A (for example, replace the "furniture" category in the semantic segmentation label of the training image A with the background category label). In this way, when the loss is calculated in the t-th learning process, the old category will not be involved in the loss calculation process, which can save training costs.
[0093] S102: Processing the training image using a segmentation model without a classifier to obtain a first segmentation mask and a first semantic feature.
[0094] Among them, the classifier-free segmentation model is used to perform visual processing (such as segmentation mask determination processing of unknown categories and high-dimensional semantic feature determination processing, etc.) on the input data (such as image data) of the segmentation model.
[0095] In addition, the present application does not limit the implementation method of the above-mentioned "segmentation model without classifier". For example, it can be implemented using any existing or future visual model (for example, a semantic segmentation model without the classification layer).
[0096] For example, in some application scenarios, the above “segmentation model without classifier” can have an encoder-decoder structure so that the “segmentation model without classifier” can extract visual features from an image data and generate a segmentation mask M∈R of unknown category. N×H×WAnd the high-dimensional semantic features F semantic ∈R N×D . Among them, the segmentation mask M∈R N×H×W Represents the mask state of the image data under N feature channel layers, N represents the number of feature channel layers in the segmentation model, and H×W represents the resolution of the mask (that is, the resolution of the image data). It can be seen that the segmentation mask M∈R N×H×W It includes N masks, each of which has a size of H×W, and the nth mask is used to represent the mask state of the image data at the nth feature channel layer. Semantic feature F semantic ∈R N×D represents the semantic information presented by the image data at N feature channel layers, and D represents the number of feature dimensions of the semantic features at each feature channel layer. semantic ∈R N×D It can include N feature vectors, each feature vector is a D-dimensional vector, and the nth feature vector is used to represent the semantic information presented by the image data under the nth feature channel layer, so that the nth feature vector can represent the image semantic information corresponding to the nth mask (for example, information used to represent categories, etc.), n is a positive integer, n≤N.
[0097] It should be noted that the "mask" in the above paragraph is used to describe the position of one or more image areas in the image data in the above paragraph (for example, the position of certain objects in the image data, etc.); and the present application does not limit the implementation method of the mask. For example, it can be implemented using any existing or future image data that can represent the mask result.
[0098] In addition, the present application does not limit the initialization method of the above-mentioned "classifier-free segmentation model". For example, when the classifier-free segmentation model provided by the present application is trained by incremental learning, if the current learning stage is used to implement the first learning process for the segmentation model, the segmentation model involved in the current learning stage can be initialized using an initially constructed, untrained classifier-free segmentation model; if the current learning stage is used to implement the t-th learning process for the segmentation model, t is a positive integer, 2≤t (that is, the segmentation model has learned some categories in the historical learning stage), then the segmentation model involved in the current learning stage can be initialized using a reference model, and the reference model refers to a classifier-free segmentation model that has learned the second category in the historical learning stage. Among them, the historical learning stage is used to describe one or more learning stages that have been completed for the segmentation model. The second category refers to the category that the segmentation model has learned in the historical learning stage (that is, the old category).
[0099] It can be seen that for the current learning stage, if the current learning stage refers to the first training performed on the segmentation model without classifier, an initially constructed segmentation model without classifier that has not yet been trained can be set as the initial value of the segmentation model involved in the current learning stage (that is, the "segmentation model without classifier" in S102 above), so that the first training of the segmentation model can be completed based on the initial value later; however, if the current learning stage refers to the t-th training performed on the segmentation model without classifier, t≥2, then the segmentation model that has been trained in the t-1-th training can be set as the initial value of the segmentation model involved in the current learning stage (that is, the "segmentation model without classifier" in S102 above), so that the non-first training of the segmentation model can be completed based on the initial value later.
[0100] It should be noted that the “t-th learning process” and “t-th training” involved in this application refer to a training stage, and the training stage is usually achieved with the help of multiple rounds of training, t is a positive integer, 1≤t≤T.
[0101] The first segmentation mask refers to a segmentation mask of unknown category obtained by processing the training image by the segmentation model without a classifier, so that the first segmentation mask can represent the mask state presented by the above training image under the N feature channel layers. For example, when the training image is Figure 2 The image shown in FIG. 1 and the current learning stage is used to implement the t-th training, when t ≥ 2, the first segmentation mask may refer to Figure 2 “Mask 2 of unknown class” is shown.
[0102] In addition, the present application does not limit the implementation of the first segmentation mask above. For example, in some application scenarios, the first segmentation mask may include at least one mask. It should be noted that the relevant content of the mask can be found above, and for the sake of brevity, it will not be repeated here.
[0103] The first semantic feature refers to a high-dimensional semantic feature obtained by processing the training image by the segmentation model without a classifier, so that the first semantic feature can represent the semantic information presented by the training image above at the N feature channel layers. For example, when the training image is Figure 2 The image shown in the figure, and the current learning stage is used to implement the t-th training, when t≥2, the first semantic feature can refer to Figure 2 “Semantic Feature 2” shown.
[0104] In addition, the present application does not limit the implementation method of the first semantic feature above. For example, when the first segmentation mask above includes at least one mask, the first semantic feature may include semantic feature vectors corresponding to each mask, so that the semantic feature vector corresponding to the mask is used to represent the image semantic information corresponding to the mask (for example, information used to represent a category, etc.). It can be seen that in one possible implementation method, when the first segmentation mask above includes N masks, the first semantic feature may include N semantic feature vectors, and the nth semantic feature vector is used to represent the image semantic information corresponding to the nth mask, where n is a positive integer and n≤N.
[0105] S103: updating the segmentation model according to the semantic segmentation label, the first segmentation mask, and the similarity between the first semantic feature and the category characterization feature, wherein the category characterization feature is obtained by encoding a category identification set using a pre-built text encoder, and the category identification set is determined based on the first category.
[0106] The category characterization feature is used to characterize the categories involved in the current learning stage (for example, new categories that need to be learned in the current learning stage, etc.).
[0107] In addition, for the above-mentioned category characterization feature, the category characterization feature is obtained by encoding a category identification set using a pre-built text encoder. Among them, the category identification set is used to record the categories involved in the current learning stage; and the present application does not limit the category identification set. For example, the category identification set may at least include the category identification of the category that needs to be learned in the current learning stage (for example, the first category above). For another example, in some application scenarios, the category identification set may not only include at least the category identification of the category that needs to be learned in the current learning stage, but also the category identification of the category that has been learned in the historical learning stage. Among them, the category identification is used to uniquely identify a category; and the present application does not limit the implementation method of the category identification. For example, it can be implemented using a class name.
[0108] In fact, in the incremental learning stage, the old categories that have been learned are not very important, so in order to simplify the training, multiple old categories can be inaccurately distinguished, but the same identifier can be used to refer to them. Based on this, the present application also provides a possible implementation of the above category identifier set, in which the category identifier set can not only include at least the category identifiers of the categories that need to be learned in the current learning stage, but also a preset category identifier (for example, the identifier string of "old category", etc.), so that the preset category identifier is used to uniformly represent all categories that have been learned in the historical learning stage.
[0109] Based on the above two paragraphs, it can be known that if the first category needs to be learned in the current learning stage, the above category identification set can be determined based on the first category, so that the category identification set can at least include the category identification of the first category. In addition, if the first category needs to be learned in the current learning stage, the above segmentation model has learned the second category (that is, the old category) in the historical learning stage, and the second category is different from the first category, then the category identification set can be determined based on the first category and the second category, so that the category identification set can include the category identification of the first category and the category identification of the second category, or the category identification set can include the category identification of the first category and the above background category identification.
[0110] It can be seen that in a possible implementation, if the current learning stage is used to implement the t-th training, t≥2, then the category identification set may include all categories required to be learned in the 1st training to all categories required to be learned in the t-th training.
[0111] The above “text encoder” is used to perform text feature extraction processing on a text data; and the present application does not limit the implementation method of the text encoder. For example, it can be implemented by using any existing or future encoder with text feature extraction function. For another example, in some application scenarios, the text encoder can be implemented by using the text encoder in the Contrastive Language-Image Pre-Training (CLIP) model. For another example, the text encoder can be implemented by using Figure 2 The text encoder shown is implemented.
[0112] Based on the above content related to the "text encoder", it can be known that because the text encoder has been trained, there is no need to adjust the parameters in the text encoder during the training process of the above classifier-free segmentation model. This ensures that the text encoder used in different learning stages remains consistent, thereby ensuring that the representation method used for category identification in different learning stages remains consistent, which is beneficial to improving the training effect for the segmentation model. Also, because the text encoder has the function of encoding any number of texts, the present application can use the text encoder and the classifier-free segmentation model to achieve semantic segmentation processing for any number of categories, so that the classifier-free segmentation model can quickly adapt to any number of new categories without adjusting the structure.
[0113] In addition, the present application does not limit the determination process of the above category characterization features. For example, when the above category identifier set is {category identifier 1, category identifier 2, ..., category identifier C t This set, Ct When the number of all categories involved in the current learning stage is represented, the process of determining the category representation feature can be specifically as follows: first, the category identifier m is inserted into a pre-constructed text template (e.g., the template "a photo of [category identifier]", etc.) to obtain a text m (e.g., the text string "a photoof [category identifier m]", etc.), so that the text m carries the category identifier m, where m is a positive integer and m≤C t ; Then, use the pre-built text encoder for text 1 to text C t Perform encoding processing to obtain category representation features To characterize the category It can represent each category identifier in the category identifier set. It can be seen that the category representation feature It includes the text feature vectors corresponding to each category identifier in the category identifier set, and the text feature vector corresponding to each category identifier is a D-dimensional vector, and the text feature vector corresponding to the mth category identifier is used to represent the category identifier m, where m is a positive integer, and m≤C t It should be noted that the text template refers to a template that is pre-set based on an application scenario, so that the text template is used to generate text data carrying a category identifier; and the present application does not limit the text template.
[0114] Based on the relevant content of the category representation features in the above text, it can be known that for the current learning stage, if the category identification set involved in the learning stage is used to record the categories involved in the learning stage (for example, new categories, etc.), then the category identification set can be encoded using a pre-built text encoder to obtain the category representation features corresponding to the learning stage, so that the category representation features can represent the categories involved in the learning stage.
[0115] The above “similarity between the first semantic feature and the category representation feature” is used to represent the classification prediction information for the above training image, for example, the classification prediction information for the above first segmentation mask; and the present application does not limit the method for determining the “similarity between the first semantic feature and the category representation feature”, for example, it can be implemented using any existing or future calculation method that can calculate the similarity between two data (for example, the method shown in formula (1) below).
[0116]
[0117] Where S(i,j) refers to the similarity between the i-th feature vector in the first semantic feature and the j-th feature vector in the category representation feature, so that S(i,j) can represent the possibility that the i-th mask in the first segmentation mask belongs to the category identified by the j-th category identifier in the category identifier set; F semantic represents the first semantic feature; refers to the i-th feature vector in the first semantic feature, so that represents the semantic information presented by the training image at the i-th feature channel layer, and The size of F is 1×D, i is a positive integer, i≤N; text Indicates the characteristics of this category; refers to the jth feature vector in the feature representation of this category, so that represents the jth category identifier in the category identifier set, and The size is 1×D, j is a positive integer, j≤C t ; Indicates for The transpose of And the size is D×1.
[0118] Based on the relevant content of "the similarity between the first semantic feature and the category representation feature" above, it can be known that when the first segmentation mask above includes N masks, and the first semantic feature includes semantic feature vectors corresponding to each mask, the "similarity between the first semantic feature and the category representation feature" can be used to indicate the possibility that each mask belongs to each category identified by the above category identification set, so that the semantic segmentation loss or classification loss for the training image can be determined based on the "similarity between the first semantic feature and the category representation feature" later.
[0119] In addition, the present application does not limit the implementation of the above S103. For example, in a possible implementation, the above S103 may specifically include the following steps 11 and 12.
[0120] Step 11: Determine classification prediction information based on the similarity between the first semantic feature and the category representation feature.
[0121] The classification prediction information is used to represent the classification information determined for the training image. For example, the classification prediction information can be used to represent the possibility that each mask in the first segmentation mask belongs to each category identified by the category identification set above. Moreover, the present application does not limit the implementation method of the classification prediction information. For example, it can adopt N×C t The matrix represents .
[0122] In addition, the present application does not limit the implementation of the above step 11. For example, it can be specifically: directly determining the similarity between the above first semantic feature and the category representation feature as the classification prediction information. It can be seen that in a possible implementation, the classification prediction information can be and For the determination process of , please refer to formula (1) above.
[0123] In fact, in order to better improve the classification prediction effect, the present application also provides a possible implementation method of the above classification prediction information determination process, which may specifically include the following steps 111 to 113.
[0124] Step 111: using the training image to determine the image corresponding to the first segmentation mask.
[0125] The image corresponding to the first segmentation mask refers to an image extracted from the training image using the first segmentation mask, so that the “image corresponding to the first segmentation mask” can represent the image information in the image area corresponding to the first segmentation mask in the training image.
[0126] In addition, the present application does not limit the implementation method of the above step 111. For example, in some application scenarios (for example, when the above mask is used to indicate the position of one or more objects in the image data), the step 111 may specifically be: fusing each mask in the above first segmentation mask with the above training image respectively to obtain the image corresponding to the first segmentation mask, so that the "image corresponding to the first segmentation mask" includes the fusion results corresponding to each mask, so that the "image corresponding to the first segmentation mask" can represent the image information in the image area corresponding to each mask in the first segmentation mask in the training image.
[0127] Step 112: Input the image corresponding to the first segmentation mask into a pre-built image classification model to obtain an image classification result output by the image classification model.
[0128] The image classification model is used to perform image classification processing on the input data (e.g., an image data) of the image classification model; and the present application does not limit the implementation method of the image classification model. For example, the image classification model can be implemented using any existing or future model with image classification function. For another example, the image classification model can be implemented using the visual encoder in CLIP.
[0129] The image classification result is used to represent the category prediction information of the image corresponding to the first segmentation mask above; and the present application does not limit the implementation method of the image classification result. For example, it can adopt the classification probability matrix Implement so that the It can indicate the possibility that the images corresponding to each mask in the first segmentation mask belong to each category identified by the above category identification set.
[0130] In addition, the present application does not limit the determination process of the above image classification result. For example, when the above step 112 is implemented with the aid of CLIP, the determination process of the image classification result can be specifically as follows: first, the image corresponding to the above first segmentation mask is encoded using the visual encoder in CLIP to obtain the image encoding of the image corresponding to the first segmentation mask; then, the similarity between the image encoding of the image corresponding to the first segmentation mask and the above category representation feature is determined as the image classification result corresponding to the first segmentation mask. So that the image classification results can be used later Determine the following classification prediction information S all .
[0131] Step 113: Determine classification prediction information according to the above image classification result and the similarity between the above first semantic feature and the category representation feature.
[0132] It should be noted that the present application does not limit the implementation of step 113. For example, in some application scenarios, step 113 may be: c , and the similarity S between the first semantic feature and the category representation feature are weighted summed to obtain classification prediction information. For example, in some application scenarios, step 113 can be implemented using the following formula (2).
[0133]
[0134] In the formula, S all represents the classification prediction information of the above text; S represents the similarity between the first semantic feature of the above text and the category representation feature; S c Represents the image classification result above; α represents the hyperparameter between different prediction results, which can be set in advance.
[0135] Based on the relevant contents of steps 111 to 113 above, it can be known that in some application scenarios, the classification prediction information can be determined based on the first segmentation mask above and the similarity between the first semantic feature above and the category representation feature, so that the classification prediction information can better represent the category prediction results corresponding to each mask in the first segmentation mask, so that the segmentation model can be updated later by referring to the classification prediction information.
[0136] Step 12: Update the segmentation model according to the above semantic segmentation label, the above first segmentation mask and the above classification prediction information.
[0137] It should be noted that the embodiment of the present application does not limit the implementation method of the above step 12. For ease of understanding, two examples are combined for explanation below.
[0138] Example 1: Step 12 above may specifically include steps 121 to 123 below.
[0139] Step 121: Determine a semantic segmentation result of the training image according to the first segmentation mask and the classification prediction information, so that the semantic segmentation result is used to represent the predicted category of each pixel in the training image.
[0140] It should be noted that the present application does not limit the specific implementation of the above step 121. For example, it can be implemented by using any existing or future method that can calculate the semantic segmentation result of the image data based on the mask and its classification prediction result (such as the method shown in formula (3) below).
[0141] Result Semantic_segmentation =(Class Predict ) T *M (3)
[0142] In the formula, Result Semantic_segmentation represents the semantic segmentation result of the training image above, so that the semantic segmentation result is used to represent the predicted category of each pixel in the training image, that is, C t represents the number of categories involved in the tth learning stage, H×W represents the size of the training image; Class Predict Represents the above classification prediction information. And this application does not limit the Class Predict , for example, Class Predict It can be the above S all or S; (Class Predict ) T Indicates that the Class Predict The transpose of M represents the first segmentation mask above, and M∈R N×H×W , N represents the number of feature channel layers in the segmentation model.
[0143] Step 122: Determine a semantic segmentation loss according to the above semantic segmentation label and the above semantic segmentation result, so that the semantic segmentation loss can represent the performance of the above segmentation model in semantic segmentation.
[0144] It should be noted that the present application does not limit the implementation method of step 122. For example, it can be implemented by any existing or future method that can determine the semantic segmentation loss based on the difference between the semantic segmentation label and the semantic segmentation result (for example, the cross entropy loss calculation method shown in formula (4) below).
[0145] L SS =CE(Result Semantic_segmentation ,Y t ) (4)
[0146] Among them, L SS Represents the semantic segmentation loss above; Result Semantic_segmentation Represents the semantic segmentation result of the above training image; Y t represents the semantic segmentation label in the training data at the t-th learning stage (for example, the semantic segmentation label of the training image above); CE (Result Semantic_segmentation ,Y t ) indicates Result Semantic_segmentation With Y t The cross entropy loss between .
[0147] Step 123: Update the segmentation model according to the above semantic segmentation loss.
[0148] It should be noted that the present application does not limit the implementation method of the above step 123. For example, it can be implemented by any existing or future method that can update the model based on semantic segmentation loss.
[0149] For example, in order to better improve the model training effect, the present application also provides a possible implementation of step 123 above. In this implementation, when the segmentation model includes a normalization layer, step 123 can be specifically as follows: if the current learning stage is used for the first training of the segmentation model without a classifier (that is, the current learning stage is the first learning stage), then all parameters in the segmentation model are updated using the above semantic segmentation loss; however, if the current learning stage is used for non-first training of the segmentation model without a classifier (that is, the current learning stage is the tth learning stage, t≥2), then when using the semantic segmentation loss to update the segmentation model, the parameters of the normalization layer are fixed (for example, using the semantic segmentation loss to update the parameters of other network layers in the segmentation model except the normalization layer, etc.).
[0150] Based on the relevant contents of steps 121 to 123 above, it can be known that in some application scenarios, for any round of training process in the current learning stage, after obtaining the first segmentation mask and classification prediction information corresponding to the training image, the semantic segmentation loss of the training image can be determined based on the first segmentation mask, the classification prediction information and the semantic segmentation label of the training image; and then the segmentation model is updated based on the semantic segmentation loss so that the updated segmentation model has better performance.
[0151] Example 2: Step 12 above may specifically include steps 124 to 126 below.
[0152] Step 124: Determine the classification loss based on the above semantic segmentation label and the above classification prediction information.
[0153] Among them, the classification loss is used to represent the performance of the above segmentation model in classification.
[0154] In addition, the present application does not limit the implementation method of the above step 124. For example, it can be implemented by using any existing or future method for determining the classification loss based on the above semantic segmentation label and the above classification prediction information.
[0155] For another example, the above step 124 may specifically include the following steps 1241 to 1243.
[0156] Step 1241: Using the above semantic segmentation labels, determine a category label so that the category label can indicate which categories appear in the semantic segmentation label, thereby enabling the category label to indicate which type of object exists in the training image.
[0157] It should be noted that the present application does not limit the implementation method of the above step 1241.
[0158] Step 1242: Encode the above category label to obtain a coding vector corresponding to the category label, so that the coding vector can represent the category label.
[0159] It should be noted that the present application does not limit the implementation method of the above "encoding vector corresponding to the category label", for example, it can be implemented in the form of a one-hot vector.
[0160] It should also be noted that the present application does not limit the implementation method of the "encoding processing" in the above step 1242.
[0161] Step 1243: Calculate the cross entropy between the above classification prediction information and the encoding vector corresponding to the above category label as the above classification loss (as shown in formula (5) below).
[0162]
[0163] Among them, L cls Represents the above classification loss; Class Predict Represents the above classification prediction information. And this application does not limit the Class Predict , for example, Class Predict It can be the above S all or S; Represents the encoding vector corresponding to the above category label; Y t represents the semantic segmentation label in the training data at the t-th learning stage (e.g., the semantic segmentation label of the training image above).
[0164] Based on the relevant contents of steps 1241 to 1243 above, it can be known that in some application scenarios, the classification loss can be determined with the help of the category labels involved in the above semantic segmentation labels and the above classification prediction information, so that the classification loss can represent the performance of the above segmentation model in classification.
[0165] Step 125: Determine a mask loss according to the semantic segmentation label and the first segmentation mask, so that the mask loss can represent the performance of the segmentation model in terms of mask.
[0166] It should be noted that the present application does not limit the implementation method of the above step 125. For example, the step 125 can be implemented by using any existing or future method that can determine the mask loss based on the semantic segmentation label and the first segmentation mask (for example, the mask loss calculation method shown in formula (6) below).
[0167] L mask =L Dice +β×L Focal (6)
[0168] Where, L mask represents the above mask loss; L Dice represents the Dice loss between the semantic segmentation label above and the first segmentation mask above, and the L Dice is determined according to the semantic segmentation label and the first segmentation mask; L Focal represents the focal loss between the semantic segmentation label and the first segmentation mask, and the L Focal is determined according to the semantic segmentation label and the first segmentation mask; β represents a pre-set loss coefficient. It should be noted that the present application does not limit the calculation method of the Focal loss, nor does it limit the calculation method of the Dice loss.
[0169] Step 126: Update the segmentation model based on the above classification loss and the above mask loss.
[0170] It should be noted that the present application does not limit the implementation method of the above step 126. For example, the step 126 can be implemented by using any existing or future method that can update the model based on classification loss and mask loss.
[0171] For example, in one possible implementation, step 126 can specifically be: first determine the model loss of the segmentation model based on the above classification loss and the above mask loss (for example, directly perform a weighted summation of the classification loss and the mask loss to obtain the model loss of the segmentation model); and then update the segmentation model based on the model loss.
[0172] For example, in order to better improve the model training effect, the present application also provides a possible implementation of step 126 above. In this implementation, when the segmentation model includes a normalization layer, step 126 can be specifically as follows: if the current learning stage is used for the first training of the segmentation model without a classifier (that is, the current learning stage is the first learning stage), then the above classification loss and the above mask loss are used to update all parameters in the segmentation model; however, if the current learning stage is used for non-first training of the segmentation model without a classifier (that is, the current learning stage is the tth learning stage, t≥2), then when the classification loss and the mask loss are used to update the segmentation model, the parameters of the normalization layer are fixed (for example, the classification loss and the mask loss are used to update the parameters of other network layers in the segmentation model except the normalization layer).
[0173] Based on the relevant contents of steps 124 to 126 above, it can be known that in some application scenarios, for any round of training process in the current learning stage, after obtaining the first segmentation mask and classification prediction information corresponding to the training image, the classification loss and mask loss of the training image can be determined based on the first segmentation mask, the classification prediction information and the semantic segmentation label of the training image; and then the segmentation model is updated based on the classification loss and mask loss so that the updated segmentation model has better performance.
[0174] Based on the relevant contents of steps 11 to 12 above, it can be known that in some application scenarios, for any round of training process in the current learning stage, after obtaining the first segmentation mask and the first semantic feature corresponding to the training image, some loss information (for example, semantic segmentation loss or classification loss + mask loss) can be determined based on the semantic segmentation label of the training image, the first segmentation mask, and the similarity between the first semantic feature and the above category representation feature; and then the segmentation model is updated based on the loss information so that the updated segmentation model has better performance.
[0175] In fact, for the t-th learning stage, t≥2, in order to better avoid forgetting old categories, the present application also provides another possible implementation of the above model update process. Under this implementation, when the segmentation model involved in the t-th learning stage is initialized using a reference model, and the reference model is a classifier-free segmentation model trained using training data corresponding to the second category, that is, the reference model may refer to a classifier-free segmentation model that has learned the second category in the historical learning stage (that is, the first t-1 learning stages), the model update process may specifically include the following steps 21-24. It should be noted that the present application does not limit the second category. For example, in some embodiments, for the training of the t-th stage, the category learned in the first t-1 stages is the second category.
[0176] Step 21: Use the above reference model to process the training image to obtain a second segmentation mask and a second semantic feature.
[0177] The reference model is a segmentation model without a classifier trained using the training data corresponding to the second category. That is, the reference model may refer to a segmentation model without a classifier that has been trained in the historical learning stage (for example, Figure 2 That is, if the current learning stage is used to perform the t-th training on the segmentation model without classifier (that is, the current learning stage is the t-th learning stage), t≥2, then the reference model refers to the segmentation model without classifier that has been trained in the t-1-th learning stage (that is, the segmentation model without classifier that has been trained t-1 times).
[0178] The second segmentation mask refers to a segmentation mask of unknown category obtained by processing the training image using the above reference model (for example, Figure 2 The unknown class mask 1 is shown, so that the second segmentation mask can represent the mask state presented by the training image above under the N feature channel layers. It should be noted that the second segmentation mask is similar to the first segmentation mask above, and for the sake of brevity, it will not be repeated here.
[0179] The second semantic feature refers to the high-dimensional semantic feature obtained by processing the training image using the above reference model (for example, Figure 2 The semantic feature 1 shown is used to make the second semantic feature represent the semantic information presented by the training image in the N feature channel layers. It should be noted that the second semantic feature is similar to the first semantic feature above, and for the sake of brevity, it will not be repeated here.
[0180] It should also be noted that the present application does not limit the execution time of step 21, as long as the execution time of step 21 is ensured to be earlier than the following step 22 or the following step 23.
[0181] Step 22: Determine a semantic distillation loss based on the first semantic feature and the second semantic feature, so that the semantic distillation loss can represent the difference in semantic prediction between the segmentation models involved in two adjacent learning stages.
[0182] It should be noted that the present application does not limit the implementation method of step 22. For example, step 22 can be implemented by using any existing or future method that can determine the distillation loss between two data (for example, the distillation loss calculation method shown in formula (7) below).
[0183]
[0184] In the formula, represents the semantic distillation loss above; F semantic_t represents the first semantic feature above (for example, the high-dimensional semantic feature output by the segmentation model being used in the t-th learning stage for the training image); F semantic_t (n, d) represents the value of the nth row and dth column in the first semantic feature; F semantic_t-1 F represents the second semantic feature above (for example, the high-dimensional semantic feature output by the segmentation model trained in the t-1th learning stage for the training image); semantic_t-1 (n, d) represents the value of the nth row and dth column in the second semantic feature; (F semantic_t (n,d)-F semantic_t-1 (n,d)) 2 represents the distance between the value of the nth row and the dth column in the first semantic feature and the value of the nth row and the dth column in the second semantic feature, so that (F semantic_t (n,d)-F semantic_t-1 (n,d)) 2 It can represent the similarity between the value of the nth row and the dth column in the first semantic feature and the value of the nth row and the dth column in the second semantic feature.
[0185] Step 23: Determine a mask distillation loss based on the first segmentation mask and the second segmentation mask, so that the mask distillation loss can represent the difference in mask prediction between the segmentation models involved in two adjacent learning stages.
[0186] It should be noted that the present application does not limit the implementation method of step 23. For example, step 23 can be implemented by using any existing or future method that can determine the distillation loss between two data (for example, the distillation loss calculation method shown in formula (8) below).
[0187]
[0188] In the formula, represents the mask distillation loss; M t M represents the first segmentation mask above (e.g., the unknown category segmentation mask output by the segmentation model being used at the t-th learning stage for the training image); t (n,i) represents the value of the i-th pixel in the n-th mask in the first segmentation mask; M t-1 M represents the second segmentation mask above (for example, the unknown category segmentation mask output by the segmentation model trained in the t-1th learning stage for the training image); t-1 (n,i) represents the value of the i-th pixel in the n-th mask in the second segmentation mask; (M t (n,i)-M t-1 (n,i)) 2 It refers to the distance between the value of the i-th pixel in the n-th mask in the first segmentation mask and the value of the i-th pixel in the n-th mask in the second segmentation mask, so that (M t (n,i)-M t-1 (n,i)) 2 The similarity between the value of the i-th pixel in the n-th mask in the first segmentation mask and the value of the i-th pixel in the n-th mask in the second segmentation mask can be expressed.
[0189] Step 24: Update the segmentation model according to the above semantic segmentation label, the above first segmentation mask, the similarity between the above first semantic feature and the category representation feature, the above semantic distillation loss, and the above mask distillation loss.
[0190] It should be noted that the present application is not limited to the implementation method of 24. For ease of understanding, two examples are provided below for illustration.
[0191] Example 1: Step 24 above may specifically include steps 241 and 242 below.
[0192] Step 241: Determine the classification loss and the mask loss according to the above semantic segmentation label, the above first segmentation mask, and the similarity between the above first semantic feature and the category representation feature.
[0193] It should be noted that the present application does not limit the implementation method of step 241 above. For example, it can be implemented using the determination process of the classification loss and mask loss shown above.
[0194] Step 242: Update the segmentation model according to the above classification loss, the above mask loss, the above semantic distillation loss, and the above mask distillation loss.
[0195] It should be noted that the embodiment of the present application does not limit the implementation method of the above step 242. For example, it can be implemented by using any existing or future method for updating the model based on multiple losses.
[0196] For example, in one possible implementation, the above step 242 may be: first perform weighted summation on the above classification loss, the above mask loss, the above semantic distillation loss, and the above mask distillation loss to obtain the model loss of the segmentation model (as shown in formula (9) below); and then update the segmentation model based on the model loss.
[0197]
[0198] Where, L all represents the model loss of the above segmentation model; L cls Represents the above classification loss; L mask Indicates the above mask loss; Represents the semantic distillation loss above; represents the mask distillation loss; λ1, λ2 and λ3 are preset weighting coefficients.
[0199] Based on the relevant contents of steps 241 to 242 above, it can be known that in some application scenarios, the segmentation model can be updated with the help of the above classification loss, the above mask loss, the above semantic distillation loss, and the above mask distillation loss, so that the updated segmentation model has better performance.
[0200] Example 2: Step 24 above may specifically include steps 243 and 244 below.
[0201] Step 243: Determine the semantic segmentation loss according to the above semantic segmentation label, the above first segmentation mask, and the similarity between the above first semantic feature and the category representation feature.
[0202] It should be noted that the present application does not limit the implementation method of step 243 above. For example, it can be implemented using the process of determining the semantic segmentation loss shown above.
[0203] Step 244: Update the segmentation model according to the above semantic segmentation loss, the above semantic distillation loss, and the above mask distillation loss.
[0204] It should be noted that the embodiment of the present application does not limit the implementation method of the above step 244. For example, it can be implemented by adopting any existing or future method for updating the model based on multiple losses.
[0205] For example, in one possible implementation, the above step 244 may be: first perform weighted summation on the above semantic segmentation, the above semantic distillation loss, and the above mask distillation loss to obtain the model loss of the segmentation model (as shown in formula (10) below); and then update the segmentation model based on the model loss.
[0206]
[0207] Where, L all represents the model loss of the above segmentation model; L SS Represents the semantic segmentation loss above; Represents the semantic distillation loss above; represents the mask distillation loss; λ4 and λ5 are preset weighting coefficients.
[0208] Based on the relevant contents of steps 243 to 244 above, it can be known that in some application scenarios, the segmentation model can be updated with the help of the above semantic segmentation loss, the above semantic distillation loss, and the above mask distillation loss so that the updated segmentation model has better performance.
[0209] In fact, in order to better improve the model training effect, the present application also provides a possible implementation method of step 24 above, in which step 24 can be specifically as follows: if the current learning stage is used for the first training of the segmentation model without a classifier (that is, the current learning stage is the first learning stage), then the above semantic segmentation label, the above first segmentation mask, the similarity between the above first semantic feature and the category representation feature, the above semantic distillation loss, and the above mask distillation loss are used to update all parameters in the segmentation model; however, if the current learning stage is used for non-first training of the segmentation model without a classifier (that is, the current learning stage is the tth learning stage, t≥2), then when using the semantic segmentation label, the first segmentation mask, the similarity between the first semantic feature and the category representation feature, the semantic distillation loss, and the mask distillation loss to update the segmentation model, the parameters of the normalization layer are fixed (for example, the parameters of other network layers in the segmentation model except the normalization layer are updated using the semantic segmentation label, the first segmentation mask, the similarity between the first semantic feature and the category representation feature, the semantic distillation loss, and the mask distillation loss).
[0210] Based on the relevant contents of steps 21 to 24 above, it can be known that in some application scenarios, for any round of training process in the current learning stage, the training process of the segmentation model in the current learning stage can be guided by the segmentation model without classifier trained in the previous learning stage, so as to achieve learning new categories without forgetting the old categories. This can effectively avoid the adverse effects caused by forgetting the old categories, which is beneficial to improve the stability of the model and further improve the model training effect. It can be seen that the present application effectively retains the old knowledge information in the old model by limiting the update of the new model parameters, so that a powerful and stable incremental segmentation model can be obtained.
[0211] Based on the above model update process for the segmentation model, it can be known that, under a possible implementation, in order to ensure that different learning stages are normalized using the same standard, the present application also provides a possible implementation of S103 above. In this implementation, if the segmentation model being used in the current learning stage (for example, the tth learning stage) has learned the second category in the historical learning stage (for example, the first t-1 learning stages), then S103 can be specifically: updating the segmentation model according to the above semantic segmentation label, the above first segmentation mask, and the similarity between the above first semantic feature and the category representation feature, and when updating the segmentation model, fixing the parameters of the normalization layer of the segmentation model (for example, updating the parameters of other network layers in the segmentation model except the normalization layer according to the above semantic segmentation label, the above first segmentation mask, and the similarity between the above first semantic feature and the category representation feature, etc.), so that the adverse effects caused by the changes in the normalization layer as the model is updated can be effectively avoided, thereby helping to improve the model training effect.
[0212] Based on the relevant contents of S101 to S103 above, it can be known that for a segmentation model without a classifier, the training process involved in the segmentation model in any learning stage is specifically as follows: if the learning stage is used to learn a first category (for example, a new category), then after obtaining the training data corresponding to the first category (that is, a training image and a semantic segmentation label of the training image), the training image is first processed using the segmentation model to obtain a first segmentation mask and a first semantic feature, so that the first segmentation mask and the first semantic feature can represent the visual information carried by the training image; then, based on the semantic segmentation label, the first segmentation mask, and the similarity between the first semantic feature and the category representation feature corresponding to the category identification set involved in the learning stage, the segmentation model is updated so that the updated segmentation model has better performance, thereby realizing the training process for a segmentation model without a classifier, so that the semantic segmentation task can be subsequently achieved with the help of the trained segmentation model.
[0213] In addition, since there is no classifier in the segmentation model provided in the present application, the model structure of the segmentation model does not need to change with the increase of categories, so that the segmentation model can be adapted to learn any number of new categories, which can effectively improve the plasticity of the segmentation model, thereby facilitating improving the flexibility of semantic segmentation.
[0214] In addition, the present application does not limit the execution subject of the above model training method. For example, the model training method provided in the embodiment of the present application can be applied to a device with a data processing function such as a terminal device or a server. For example, the model training method provided in the embodiment of the present application can also be implemented with the help of a data communication process between different devices (for example, a terminal device and a server, two terminal devices, or two servers). Among them, the terminal device can be a smart phone, a computer, a personal digital assistant (PDA) or a tablet computer. The server can be a stand-alone server, a cluster server or a cloud server.
[0215] In fact, in order to better understand the technical solution provided by this application, a model training process implemented in an incremental learning manner is used as an example to illustrate.
[0216] As an example, when the classifier-free segmentation model provided in the present application is trained using an incremental learning method, the model training method provided in the present application may include the following steps 31 to 37.
[0217] Step 31: Divide the training data set into T training data subsets, where different training data subsets include training data corresponding to different categories, and T is a positive integer.
[0218] The training data set refers to a data set required for reference when training a segmentation model without a classifier; and the present application does not limit the training data set, for example, it includes a large amount of image data and semantic segmentation labels of each image data.
[0219] The t-th training data subset is used to participate in the t-th training (that is, the t-th learning stage) for the above segmentation model without a classifier. Wherein, t is a positive integer, 1≤t≤T.
[0220] It should be noted that the present application does not limit the implementation method of the above step 31. For example, the training data set can be divided according to categories so that different training data subsets obtained by division include training data corresponding to different categories.
[0221] Step 32: Initialize t to 1 (ie, t=1).
[0222] Step 33: Obtain training data corresponding to the first category from the tth training data subset, where the training data includes a training image and a semantic segmentation label of the training image, and the semantic segmentation label includes the first category.
[0223] It should be noted that the present application is not limited to the implementation method of "obtaining training data corresponding to the first category from the tth training data subset" in step 33 above. For example, it can be implemented by any existing or future method that can extract training data from a data set.
[0224] It should also be noted that the relevant content of the "training data corresponding to the first category" in the above step 33 can be found in the above S101, and for the sake of brevity, it will not be repeated here.
[0225] Step 34: Process the training image using the segmentation model without a classifier to obtain a first segmentation mask and a first semantic feature.
[0226] It should be noted that the relevant contents of step 34 can be found in S102 above, and for the sake of brevity, they will not be repeated here.
[0227] Step 35: Determine whether the preset stop condition is reached, if so, execute the following step 37; if not, execute the following step 36.
[0228] Among them, the preset stopping condition refers to the condition that needs to be met when ending the training process for the segmentation model in the current learning stage; and the present application does not limit the preset stopping condition. For example, it can be: the model loss of the segmentation model in the current learning stage is lower than a preset first threshold, or the rate of change of the model loss of the segmentation model in the current learning stage is lower than a preset second threshold, or the number of updates of the segmentation model in the current learning stage is higher than a preset third threshold.
[0229] Step 36: If the preset stop condition is not met, the segmentation model is updated according to the semantic segmentation label, the first segmentation mask, and the similarity between the first semantic feature and the category representation feature, and the process returns to continue to execute the above step 33 and its subsequent steps. The category representation feature is obtained by encoding a category identification set using a pre-built text encoder, and the category identification set is determined according to the first category.
[0230] In the present application, for a round of training process being executed in the tth learning stage, if it is determined that the preset stop condition has not been reached, it can be determined that the segmentation model without a classifier has not yet learned the category (for example, the first category) required to be learned in the tth learning stage. Therefore, the segmentation model can be updated based on the above semantic segmentation label, the first segmentation mask, and the similarity between the first semantic feature and the category representation feature, so that the updated segmentation model has better performance, so that it can subsequently return to execute the above step 33 and its subsequent steps based on the updated segmentation model to start the next round of training process for the segmentation model, and iterate in this way until the preset stop condition is reached.
[0231] It should be noted that the relevant content of the step "updating the segmentation model according to the semantic segmentation label, the first segmentation mask, and the similarity between the first semantic feature and the category representation feature" in step 36 can be found in S103 above. For the sake of brevity, it will not be repeated here.
[0232] Step 37: If the preset stop condition is reached, determine whether T learning stages for the segmentation model are completed. If so, end the training process for the segmentation model without a classifier; if not, update t and return to continue executing the above step 33 and its subsequent steps.
[0233] Among them, the T learning stages include the first learning stage implemented using the first training data subset, the second learning stage implemented using the second training data subset, ... (and so on), and the Tth learning stage implemented using the Tth training data subset.
[0234] In addition, the present application does not limit the implementation method of "determining whether T learning stages for the segmentation model have been completed" in step 37 above. For example, it can be specifically: determine whether t is equal to T. If it is determined that t is equal to T, it can be determined that T learning stages for the segmentation model have been completed; if it is determined that t is less than T, it can be determined that T learning stages for the segmentation model have not yet been completed.
[0235] It can be seen that in some application scenarios, for a round of training process being executed in the tth learning stage, if it is determined that the preset stop condition has been reached, it can be determined that the segmentation model without classifier has learned the category (for example, the first category) required to be learned in the tth learning stage, so it can be determined that the learning task of the tth learning stage has been completed, so the tth training process for the segmentation model can be directly ended, and it is judged whether t is equal to T. If t is equal to T, it can be determined that T learning stages for the segmentation model have been completed, so it can be determined that the incremental learning process for the segmentation model has been completed, so the training process for the segmentation model without classifier can be directly ended; however, if t<T, it can be determined that the incremental learning process for the segmentation model has not been completed, so t can be updated using the following formula (11), and based on the updated t, return to execute the above step 33 and its subsequent steps to start the next learning stage for the segmentation model, and iterate until t=T.
[0236] t'=t+1 (11)
[0237] Where t' represents the updated t.
[0238] Based on the relevant contents of steps 31 to 37 above, it can be known that in a possible implementation, for the segmentation model without a classifier, it can be trained with the incremental learning method shown in steps 31 to 37, so that the segmentation model can be prompted to learn new categories with the help of incremental learning in the future. In addition, for the t-th learning stage, if 2≤t≤T, the segmentation model used in the t-th learning stage is initialized using a reference model, and the reference model refers to a segmentation model without a classifier trained with the first t-1 training data subsets (that is, the segmentation model without a classifier obtained after completing the first t-1 learning stages).
[0239] In fact, for a trained segmentation model without a classifier (for example, a segmentation model trained by the process shown in steps 31 to 37 above), the segmentation model can continue to learn new categories, and the learning process for the new category is similar to the learning process shown in the non-first learning stage above (for example, the learning process shown in the tth learning stage, and t≥2). For the sake of brevity, it will not be repeated here.
[0240] Based on the above model training method, this application also provides a semantic segmentation method, which is described below with reference to the accompanying drawings for easy understanding. Figure 3 As shown, the semantic segmentation method provided in the embodiment of the present application includes the following S301-S303. Figure 3 A flowchart of a semantic segmentation method provided in an embodiment of the present application.
[0241] S301: Obtain an image to be segmented.
[0242] The image to be segmented refers to image data that needs to be processed for semantic segmentation; and the present application does not limit the image to be segmented.
[0243] S302: Process the image to be segmented using a segmentation model without a classifier to obtain a third segmentation mask and a third semantic feature; the segmentation model is obtained using any implementation of the model training method provided in the embodiments of the present application.
[0244] Among them, the classifier-free segmentation model is used to perform visual processing (such as segmentation mask determination processing of unknown categories and high-dimensional semantic feature determination processing, etc.) on the input data of the segmentation model (such as image data); and the segmentation model is obtained by using any implementation method of the model training method provided in the embodiments of the present application. For the sake of brevity, it will not be repeated here.
[0245] The third segmentation mask refers to a segmentation mask of unknown category obtained by processing the image to be segmented by the segmentation model without a classifier, so that the third segmentation mask can represent the mask state presented by the image to be segmented under the N feature channel layers. It should be noted that the third segmentation mask is similar to the first segmentation mask above, and for the sake of brevity, it will not be repeated here.
[0246] The third semantic feature refers to a high-dimensional semantic feature obtained by processing the image to be segmented by the segmentation model without a classifier, so that the third semantic feature can represent the semantic information presented by the image to be segmented under the N feature channel layers. It should be noted that the third semantic feature is similar to the first semantic feature above, and for the sake of brevity, it will not be repeated here.
[0247] S303: Determine the semantic segmentation result of the image to be segmented based on the third segmentation mask and the similarity between the third semantic feature and the category representation feature corresponding to the segmentation model; the category representation feature corresponding to the segmentation model is obtained by encoding the category identification set corresponding to the segmentation model using a pre-built text encoder.
[0248] The category representation feature corresponding to the segmentation model refers to the category representation feature corresponding to the trained segmentation model without a classifier, so that the category representation feature can represent all categories involved in training the segmentation model.
[0249] In addition, the above “category representation features corresponding to the segmentation model” are obtained by encoding the category identification set corresponding to the segmentation model using a pre-built text encoder. The “category identification set corresponding to the segmentation model” is used to record all categories involved in training the segmentation model (for example, all categories involved in the training data set above).
[0250] In addition, the process of determining the “category representation features corresponding to the segmentation model” above is similar to the process of determining the “category representation features” above, and for the sake of brevity, it will not be repeated here.
[0251] The semantic segmentation result of the image to be segmented is used to indicate the category to which each pixel in the image to be segmented belongs; and the determination process of the "semantic segmentation result of the image to be segmented" is similar to the determination process of the "semantic segmentation result of the training image" above, and for the sake of brevity, it will not be repeated here. For ease of understanding, the following is an explanation with examples.
[0252] As an example, the process of determining the semantic segmentation result of the image to be segmented above includes the following steps 41 to 43.
[0253] Step 41: using the image to be segmented, determine an image corresponding to the third segmentation mask.
[0254] The image corresponding to the third segmentation mask refers to an image extracted from the image to be segmented using the third segmentation mask, so that the "image corresponding to the third segmentation mask" can represent the image information in the image area corresponding to the third segmentation mask in the image to be segmented.
[0255] In addition, the present application does not limit the implementation of the above step 41. For example, the implementation of the step 41 is similar to the implementation of the above step 111.
[0256] Step 42: Input the image corresponding to the third segmentation mask into a pre-constructed image classification model to obtain an image classification result output by the image classification model, so that the image classification result can represent the category prediction information for the image corresponding to the third segmentation mask.
[0257] For the relevant content of the image classification model, please refer to step 112 above.
[0258] In addition, the present application does not limit the implementation of the above step 412. For example, when the above image classification model includes an image encoder (for example, a visual encoder in CLIP), the step 412 can be specifically: using the image encoder to encode the image corresponding to the third segmentation mask above to obtain the image code corresponding to the third segmentation mask, so that the image code can represent the image information carried by the image corresponding to the third segmentation mask; and then determining the image classification result based on the similarity between the image code corresponding to the third segmentation mask and the category representation feature corresponding to the above segmentation model, so that the image classification result can represent the category prediction information for the image corresponding to the third segmentation mask. The image encoder is used to encode an image data; and the present application does not limit the image encoder.
[0259] It can be seen that in a possible implementation mode, when the above step 412 is implemented with the help of CLIP, the step 412 can be specifically as follows: firstly, the image corresponding to the above third segmentation mask is encoded using the visual encoder in CLIP to obtain the image encoding of the image corresponding to the third segmentation mask; then, the similarity between the image encoding of the image corresponding to the third segmentation mask and the above category representation feature is determined as the image classification result corresponding to the third segmentation mask. So that the image classification results can be used later Determine the following classification prediction information S all .
[0260] Step 43: Determine a semantic segmentation result of the image to be segmented according to the above image classification result, the above third segmentation mask, and the similarity between the above third semantic feature and the category representation feature corresponding to the segmentation model.
[0261] It should be noted that the present application does not limit the implementation method of step 43. For example, step 43 may specifically include the following steps 431 and 432.
[0262] Step 431: Determine classification prediction information based on the above image classification result and the similarity between the above third semantic feature and the category representation feature corresponding to the segmentation model, so that the classification prediction information is used to represent the classification information determined for the image to be segmented.
[0263] It should be noted that the implementation of step 431 is similar to the implementation of step 113 above, and for the sake of brevity, it will not be repeated here.
[0264] Step 432: Determine a semantic segmentation result of the image to be segmented according to the above classification prediction information and the above third segmentation mask, so that the semantic segmentation result is used to represent the predicted category of each pixel point in the image to be segmented.
[0265] It should be noted that the implementation of step 432 is similar to the implementation of step 121 above, and for the sake of brevity, it will not be repeated here.
[0266] Based on the relevant contents of steps 41 to 43 above, it can be known that in some application scenarios, after determining the third segmentation mask and the third semantic feature from the image to be segmented, the image classification result corresponding to the image to be segmented can be determined based on the third segmentation mask and the visual encoder in CLIP; then, using the image classification result, correction processing is performed on the similarity between the third semantic feature and the category representation feature corresponding to the segmentation model to obtain the classification prediction information corresponding to the image to be segmented; finally, based on the classification prediction information and the third segmentation mask, the semantic segmentation result of the image to be segmented is determined, so that the semantic segmentation result is used to represent the predicted category of each pixel point in the image to be segmented, which is conducive to improving the accuracy of semantic segmentation.
[0267] Based on the relevant contents of S301 to S303 above, it can be known that for the semantic segmentation method provided in the embodiment of the present application, after the segmentation model without a classifier is trained, the segmentation model can be directly used to process any image to be segmented to obtain a third segmentation mask and a third semantic feature; then, based on the third segmentation mask and the similarity between the third semantic feature and the category representation feature corresponding to the segmentation model, the semantic segmentation result of the image to be segmented is determined, so that the semantic segmentation result can represent the category to which each pixel point in the image to be segmented belongs, so that semantic segmentation can be achieved with the help of a pre-built segmentation model without a classifier. Among them, because there is no classifier in the segmentation model, the model structure of the segmentation model will not change with the increase of categories, so that the segmentation model can be adapted to learn any number of new categories, so that the plasticity of the segmentation model can be effectively improved, thereby effectively improving the flexibility of the semantic segmentation processing.
[0268] In addition, the present application does not limit the execution subject of the above semantic segmentation method. For example, the semantic segmentation method provided in the embodiment of the present application can be applied to a device with data processing function such as a terminal device or a server. For another example, the semantic segmentation method provided in the embodiment of the present application can also be implemented by means of a data communication process between different devices (for example, a terminal device and a server, two terminal devices, or two servers).
[0269] Based on the model training method provided in the embodiment of the present application, the embodiment of the present application also provides a model training device. Figure 4 Explain and illustrate. Figure 4 This is a schematic diagram of the structure of a model training device provided in an embodiment of the present application. It should be noted that for the technical details of the model training device provided in an embodiment of the present application, please refer to the relevant content of the model training method above.
[0270] like Figure 4 As shown, the model training device 400 provided in the embodiment of the present application includes:
[0271] A first acquisition unit 401 is used to acquire training data corresponding to a first category, wherein the training data includes a training image and a semantic segmentation label of the training image, and the semantic segmentation label includes the first category;
[0272] A first processing unit 402 is used to process the training image using a segmentation model without a classifier to obtain a first segmentation mask and a first semantic feature;
[0273] The model updating unit 403 is used to update the segmentation model according to the semantic segmentation label, the first segmentation mask, and the similarity between the first semantic feature and the category characterization feature, wherein the category characterization feature is obtained by encoding a category identification set using a pre-built text encoder, and the category identification set is determined according to the first category.
[0274] In a possible implementation manner, the model updating unit 403 includes:
[0275] A first determining subunit, configured to determine classification prediction information according to a similarity between the first semantic feature and a category representation feature;
[0276] A first updating subunit is used to update the segmentation model according to the semantic segmentation label, the first segmentation mask and the classification prediction information.
[0277] In one possible implementation, the first updating subunit is specifically used to: determine the semantic segmentation result of the training image based on the first segmentation mask and the classification prediction information; determine the semantic segmentation loss based on the semantic segmentation label and the semantic segmentation result; and update the segmentation model based on the semantic segmentation loss.
[0278] In a possible implementation manner, the model training device 400 further includes:
[0279] an image determining unit, configured to determine an image corresponding to the first segmentation mask using the training image;
[0280] An image classification unit, configured to input the image corresponding to the first segmentation mask into a pre-built image classification model to obtain an image classification result output by the image classification model;
[0281] The first determination subunit is specifically used to determine the classification prediction information according to the image classification result and the similarity between the first semantic feature and the category representation feature.
[0282] In one possible implementation, the first updating subunit is specifically used to: determine the classification loss based on the semantic segmentation label and the classification prediction information; determine the mask loss based on the semantic segmentation label and the first segmentation mask; and update the segmentation model based on the classification loss and the mask loss.
[0283] In one possible implementation, the segmentation model is initialized using a reference model; the reference model is a classifier-free segmentation model trained using training data corresponding to the second category; the segmentation model includes a normalization layer; and when updating the segmentation model, the parameters of the normalization layer of the segmentation model are fixed.
[0284] In a possible implementation manner, the segmentation model is initialized using a reference model; the reference model is a segmentation model trained using training data corresponding to the second category; the reference model is used to process the training image to obtain a second segmentation mask and a second semantic feature;
[0285] The model updating unit 403 is specifically used to: determine the semantic distillation loss according to the first semantic feature and the second semantic feature; determine the mask distillation loss according to the first segmentation mask and the second segmentation mask; and update the segmentation model according to the semantic segmentation label, the first segmentation mask, the similarity between the first semantic feature and the category representation feature, the semantic distillation loss, and the mask distillation loss.
[0286] In a possible implementation manner, the model training device 400 further includes:
[0287] A data partitioning unit, used to partition the training data set into T training data subsets, where different training data subsets include training data corresponding to different categories, and T is a positive integer;
[0288] The first acquisition unit 401 is specifically configured to: acquire training data corresponding to the first category from the t-th training data subset, where t is a positive integer and t≤T.
[0289] In a possible implementation manner, the model training device 400 further includes:
[0290] An initialization unit, used to initialize t to 1;
[0291] An iterative unit is used to return to the first acquisition unit 401 to continue to execute the step of acquiring the training data corresponding to the first category from the tth training data subset after updating the segmentation model according to the semantic segmentation label, the first segmentation mask, and the similarity between the first semantic feature and the category representation feature, until a preset stop condition is reached, update t, and return to the first acquisition unit 401 to continue to execute the step of acquiring the training data corresponding to the first category from the tth training data subset until the segmentation model is trained using the T training data subsets.
[0292] In one possible implementation, if 2≤t≤T, the segmentation model is initialized using a reference model; the reference model refers to a classifier-free segmentation model trained using the first t-1 training data subsets.
[0293] Based on the relevant content of the above-mentioned model training device 400, it can be known that for the model training device 400 provided in the embodiment of the present application, the model training device 400 is used to implement a training process for a segmentation model without a classifier, and the model training device 400 is specifically used for: if the segmentation model is used to learn a first category (for example, a new category), then after obtaining the training data corresponding to the first category (that is, a training image and a semantic segmentation label of the training image), first use the segmentation model to process the training image to obtain a first segmentation mask and a first semantic feature, so that the first segmentation mask and the first semantic feature can represent the visual information carried by the training image; then update the segmentation model based on the similarity between the semantic segmentation label, the first segmentation mask, and the first semantic feature and the category representation feature corresponding to the category identification set involved in the learning stage, so that the updated segmentation model has better performance, so that the training process for a segmentation model without a classifier can be implemented, so that the semantic segmentation task can be implemented subsequently with the help of the trained segmentation model.
[0294] In addition, since there is no classifier in the segmentation model provided in the present application, the model structure of the segmentation model will not change with the increase of categories, so that the segmentation model can be adapted to learn any number of new categories, which can effectively improve the plasticity of the segmentation model, thereby facilitating improving the flexibility of semantic segmentation.
[0295] Based on the semantic segmentation method provided in the embodiment of the present application, the embodiment of the present application also provides a semantic segmentation device. Figure 5 Explain and illustrate. Figure 5 This is a schematic diagram of the structure of a semantic segmentation device provided in an embodiment of the present application. It should be noted that for the technical details of the semantic segmentation device provided in an embodiment of the present application, please refer to the relevant content of the semantic segmentation method above.
[0296] like Figure 5 As shown, the semantic segmentation device 500 provided in the embodiment of the present application includes:
[0297] The second acquisition unit 501 is used to acquire the image to be segmented;
[0298] A second processing unit 502 is used to process the image to be segmented using a segmentation model without a classifier to obtain a third segmentation mask and a third semantic feature; the segmentation model is obtained using the model training method according to any one of claims 1 to 12;
[0299] The result determination unit 503 is used to determine the semantic segmentation result of the image to be segmented based on the third segmentation mask and the similarity between the third semantic feature and the category representation feature corresponding to the segmentation model; the category representation feature corresponding to the segmentation model is obtained by encoding the category identification set corresponding to the segmentation model using a pre-built text encoder.
[0300] In a possible implementation manner, the semantic segmentation device 500 further includes:
[0301] A data determining unit, configured to determine an image corresponding to the third segmentation mask using the image to be segmented;
[0302] A data classification unit, used for inputting the image corresponding to the third segmentation mask into a pre-built image classification model to obtain an image classification result output by the image classification model;
[0303] The result determination unit 503 is used to determine the semantic segmentation result of the image to be segmented according to the image classification result, the third segmentation mask, and the similarity between the third semantic feature and the category representation feature corresponding to the segmentation model.
[0304] In a possible implementation manner, the image classification model includes an image encoder; the image encoder is used to encode the image corresponding to the third segmentation mask to obtain the image code corresponding to the third segmentation mask; the image classification result is determined according to the similarity between the image code corresponding to the third segmentation mask and the category representation feature corresponding to the segmentation model:
[0305] The result determination unit 503 includes:
[0306] A second determining subunit, configured to determine the classification prediction information according to the image classification result and the similarity between the third semantic feature and the category representation feature corresponding to the segmentation model;
[0307] The third determining subunit is used to determine the semantic segmentation result of the image to be segmented according to the classification prediction information and the third segmentation mask.
[0308] Based on the relevant contents of the above-mentioned semantic segmentation device 500, it can be known that for the semantic segmentation device 500 provided in the embodiment of the present application, after obtaining the trained classifier-free segmentation model, the segmentation model can be directly used to process any image to be segmented to obtain a third segmentation mask and a third semantic feature; then, based on the third segmentation mask and the similarity between the third semantic feature and the category representation feature corresponding to the segmentation model, the semantic segmentation result of the image to be segmented is determined, so that the semantic segmentation result can represent the category to which each pixel point in the image to be segmented belongs, so that semantic segmentation can be achieved with the help of a pre-built classifier-free segmentation model. Among them, because there is no classifier in the segmentation model, the model structure of the segmentation model will not change with the increase of categories, so that the segmentation model can be adapted to learn any number of new categories, so that the plasticity of the segmentation model can be effectively improved, thereby effectively improving the flexibility of the semantic segmentation processing.
[0309] In addition, an embodiment of the present application also provides an electronic device, which includes a processor and a memory: the memory is used to store instructions or computer programs; the processor is used to execute the instructions or computer programs in the memory, so that the electronic device executes any implementation of the model training method provided in the embodiment of the present application or executes any implementation of the semantic segmentation method provided in the embodiment of the present application.
[0310] See also Figure 6 , which shows a schematic diagram of the structure of an electronic device 600 suitable for implementing the embodiment of the present disclosure. The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.
[0311] like Figure 6 As shown, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.
[0312] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.
[0313] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.
[0314] The electronic device provided by the embodiment of the present disclosure and the method provided by the above embodiment belong to the same inventive concept, and the technical details not fully described in this embodiment can be referred to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.
[0315] An embodiment of the present application also provides a computer-readable medium, which stores instructions or computer programs. When the instructions or computer programs are executed on a device, the device executes any implementation of the model training method provided in the embodiment of the present application or executes any implementation of the semantic segmentation method provided in the embodiment of the present application.
[0316] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.
[0317] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (Hyper Text Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.
[0318] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.
[0319] The computer-readable medium carries one or more programs, and when the one or more programs are executed by the electronic device, the electronic device can execute the method.
[0320] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).
[0321] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0322] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit / module does not, in some cases, constitute a limitation on the unit itself.
[0323] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.
[0324] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0325] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the system or device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.
[0326] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.
[0327] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.
[0328] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
[0329] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A model training method, characterized in that: The method comprises: Acquire training data corresponding to the first category, the training data including a training image and a semantic segmentation label of the training image, the semantic segmentation label including the first category; Processing the training image using a segmentation model without a classifier to obtain a first segmentation mask and a first semantic feature; The segmentation model is updated according to the semantic segmentation label, the first segmentation mask, and the similarity between the first semantic feature and the category characterization feature, wherein the category characterization feature is obtained by encoding a category identification set using a pre-built text encoder, and the category identification set is determined based on the first category.
2. The method according to claim 1, characterized in that The updating of the segmentation model according to the semantic segmentation label, the first segmentation mask, and the similarity between the first semantic feature and the category representation feature includes: Determining classification prediction information according to the similarity between the first semantic feature and the category representation feature; The segmentation model is updated according to the semantic segmentation label, the first segmentation mask and the classification prediction information.
3. The method according to claim 2, characterized in that The updating of the segmentation model according to the semantic segmentation label, the first segmentation mask and the classification prediction information includes: Determining a semantic segmentation result of the training image according to the first segmentation mask and the classification prediction information; Determining a semantic segmentation loss according to the semantic segmentation label and the semantic segmentation result; The segmentation model is updated according to the semantic segmentation loss.
4. The method according to claim 2, characterized in that: Before determining the classification prediction information according to the similarity between the first semantic feature and the category representation feature, the method further includes: Determining an image corresponding to the first segmentation mask using the training image; Inputting the image corresponding to the first segmentation mask into a pre-built image classification model to obtain an image classification result output by the image classification model; The determining of classification prediction information according to the similarity between the first semantic feature and the category representation feature includes: The classification prediction information is determined according to the image classification result and the similarity between the first semantic feature and the category representation feature.
5. The method according to claim 2, characterized in that: The updating of the segmentation model according to the semantic segmentation label, the first segmentation mask and the classification prediction information includes: Determining a classification loss according to the semantic segmentation label and the classification prediction information; Determining a mask loss according to the semantic segmentation label and the first segmentation mask; The segmentation model is updated according to the classification loss and the mask loss.
6. The method according to claim 1, characterized in that The segmentation model is initialized using a reference model; the reference model is a segmentation model without a classifier obtained by training using training data corresponding to the second category; The segmentation model includes a normalization layer; When the segmentation model is updated, parameters of a normalization layer of the segmentation model are fixed.
7. The method according to claim 1, characterized in that The segmentation model is initialized using a reference model; the reference model is a segmentation model without a classifier obtained by training using training data corresponding to the second category; The reference model is used to process the training image to obtain a second segmentation mask and a second semantic feature; Before updating the segmentation model according to the semantic segmentation label, the first segmentation mask, and the similarity between the first semantic feature and the category representation feature, the method further includes: Determining a semantic distillation loss according to the first semantic feature and the second semantic feature; Determining a mask distillation loss according to the first segmentation mask and the second segmentation mask; The updating of the segmentation model according to the semantic segmentation label, the first segmentation mask, and the similarity between the first semantic feature and the category representation feature includes: The segmentation model is updated according to the semantic segmentation label, the first segmentation mask, the similarity between the first semantic feature and the category representation feature, the semantic distillation loss, and the mask distillation loss.
8. The method according to any one of claims 1 to 7, characterized in that: The method further comprises: The training data set is divided into T training data subsets, where different training data subsets include training data corresponding to different categories, and T is a positive integer; The obtaining of training data corresponding to the first category includes: The training data corresponding to the first category is obtained from the t-th training data subset, where t is a positive integer and t≤T.
9. The method according to claim 8, characterized in that The method further comprises: Initialize t to 1; After updating the segmentation model according to the semantic segmentation label, the first segmentation mask, and the similarity between the first semantic feature and the category representation feature, the method further includes: Continue to execute the step of obtaining the training data corresponding to the first category from the tth training data subset until a preset stopping condition is reached, update t, and continue to execute the step of obtaining the training data corresponding to the first category from the tth training data subset until the segmentation model is trained using the T training data subsets.
10. The method according to claim 8, characterized in that If 2≤t≤T, the segmentation model is initialized using a reference model; the reference model refers to a segmentation model without a classifier trained using the first t-1 training data subsets.
11. A semantic segmentation method, characterized in that: The method comprises: Obtain the image to be segmented; Processing the image to be segmented using a segmentation model without a classifier to obtain a third segmentation mask and a third semantic feature; the segmentation model is obtained using the model training method according to any one of claims 1 to 10; The semantic segmentation result of the image to be segmented is determined based on the third segmentation mask and the similarity between the third semantic feature and the category representation feature corresponding to the segmentation model; the category representation feature corresponding to the segmentation model is obtained by encoding the category identification set corresponding to the segmentation model using a pre-built text encoder.
12. The method according to claim 11, characterized in that Before determining the semantic segmentation result of the image to be segmented according to the third segmentation mask and the similarity between the third semantic feature and the category representation feature corresponding to the segmentation model, the method further includes: The process of determining the semantic segmentation result of the image to be segmented includes: Determining an image corresponding to the third segmentation mask using the image to be segmented; Inputting the image corresponding to the third segmentation mask into a pre-built image classification model to obtain an image classification result output by the image classification model; The determining, according to the third segmentation mask and the similarity between the third semantic feature and the category representation feature corresponding to the segmentation model, a semantic segmentation result of the image to be segmented includes: A semantic segmentation result of the image to be segmented is determined according to the image classification result, the third segmentation mask, and the similarity between the third semantic feature and the category representation feature corresponding to the segmentation model.
13. The method according to claim 12, characterized in that The image classification model includes an image encoder; the image encoder is used to encode the image corresponding to the third segmentation mask to obtain the image code corresponding to the third segmentation mask; the image classification result is determined according to the similarity between the image code corresponding to the third segmentation mask and the category representation feature corresponding to the segmentation model: and / or, Determining the semantic segmentation result of the image to be segmented according to the image classification result, the third segmentation mask, and the similarity between the third semantic feature and the category representation feature corresponding to the segmentation model includes: Determining the classification prediction information according to the image classification result and the similarity between the third semantic feature and the category representation feature corresponding to the segmentation model; A semantic segmentation result of the image to be segmented is determined according to the classification prediction information and the third segmentation mask.
14. A model training device, characterized in that: include: A first acquisition unit, configured to acquire training data corresponding to a first category, wherein the training data includes a training image and a semantic segmentation label of the training image, and the semantic segmentation label includes the first category; A first processing unit, configured to process the training image using a segmentation model without a classifier to obtain a first segmentation mask and a first semantic feature; A model updating unit is used to update the segmentation model according to the semantic segmentation label, the first segmentation mask, and the similarity between the first semantic feature and a category characterization feature, wherein the category characterization feature is obtained by encoding a category identification set using a pre-built text encoder, and the category identification set is determined according to the first category.
15. A semantic segmentation device, characterized in that: include: A second acquisition unit, used for acquiring an image to be segmented; A second processing unit is used to process the image to be segmented by using a segmentation model without a classifier to obtain a third segmentation mask and a third semantic feature; The segmentation model is obtained by using the model training method described in any one of claims 1 to 10; A result determination unit is used to determine the semantic segmentation result of the image to be segmented based on the third segmentation mask and the similarity between the third semantic feature and the category representation feature corresponding to the segmentation model; the category representation feature corresponding to the segmentation model is obtained by encoding the category identification set corresponding to the segmentation model using a pre-built text encoder.
16. An electronic device, characterized in that: The device comprises: a processor and a memory; The memory is used to store instructions or computer programs; The processor is used to execute the instructions or computer programs in the memory so that the electronic device executes the method according to any one of claims 1 to 13.
17. A computer readable medium, characterized in that The computer-readable medium stores instructions or computer programs, and when the instructions or computer programs are executed on a device, the device executes the method according to any one of claims 1 to 13.
