Segmentation model training method, image segmentation method, device, equipment and medium

Abstract

The present disclosure provides a training method for a segmentation model, which relates to the fields of artificial intelligence technology, especially computer vision technology and deep learning technology, and can be applied in smart city scenarios. The specific implementation scheme is: inputting a first sample image into a first segmentation model to obtain the gradient information of the first intermediate network of the first segmentation model; obtaining a first intermediate mask feature corresponding to the first intermediate network based on the gradient information of the first intermediate network and the first intermediate feature output by the first intermediate network; determining a first distillation loss based on at least one second intermediate mask feature and at least one first intermediate mask feature, wherein the at least one second intermediate mask feature is obtained based on the gradient information of the second intermediate network and at least one second intermediate feature output by the second intermediate network; and training the first segmentation model based on the first distillation loss. The present disclosure also provides an image segmentation method, apparatus, electronic device, and storage medium.

Description

Technical Field

[0001] The present disclosure relates to the fields of artificial intelligence technology, particularly computer vision technology and deep learning technology, and can be applied in smart city scenarios. More specifically, the present disclosure provides a training method for a segmentation model, an image segmentation method, an apparatus, an electronic device, and a storage medium. Background Art

[0002] With the development of artificial intelligence technology, the application scenarios of deep learning models are constantly increasing. Model distillation technology can be used to improve the performance of image segmentation models. Summary of the Invention

[0003] The present disclosure provides a training method for a segmentation model, an image segmentation method, an apparatus, a device, and a storage medium.

[0004] According to one aspect of the present disclosure, a training method for a segmentation model is provided, the method comprising: inputting a first sample image into a first segmentation model to obtain gradient information of a first intermediate network of the first segmentation model; obtaining at least one first intermediate mask feature corresponding to the first intermediate network based on the gradient information of the first intermediate network and at least one first intermediate feature output by the first intermediate network; determining a first distillation loss based on at least one second intermediate mask feature and at least one first intermediate mask feature, wherein the at least one second intermediate mask feature corresponds to a second intermediate network of a second segmentation model, the at least one second intermediate mask feature is obtained based on the gradient information of the second intermediate network and at least one second intermediate feature output by the second intermediate network, the gradient information of the second intermediate network is obtained by inputting a second sample image into the second segmentation model, and the parameter amount of the second segmentation model is greater than the parameter amount of the first segmentation model; and training the first segmentation model based on the first distillation loss.

[0005] According to another aspect of the present disclosure, an image segmentation method is provided, the method comprising: inputting a target image into a first segmentation model to obtain a target segmentation result, wherein the target segmentation result includes a target mask of a target instance in the target image and a category of the target instance, and the first segmentation model is trained using the method provided by the present disclosure.

[0006] According to another aspect of the present disclosure, a training device for a segmentation model is provided, which includes: a first acquisition module for inputting a first sample image into the first segmentation model to obtain gradient information of a first intermediate network of the first segmentation model; a second acquisition module for obtaining at least one first intermediate mask feature corresponding to the first intermediate network based on the gradient information of the first intermediate network and at least one first intermediate feature output by the first intermediate network; a first determination module for determining a first distillation loss based on at least one second intermediate mask feature and at least one first intermediate mask feature, wherein the at least one second intermediate mask feature corresponds to the second intermediate network of the second segmentation model, the at least one second intermediate mask feature is obtained based on the gradient information of the second intermediate network and the at least one second intermediate feature output by the second intermediate network, the gradient information of the second intermediate network is obtained by inputting the second sample image into the second segmentation model, and the parameter amount of the second segmentation model is greater than the parameter amount of the first segmentation model; and a training module for training the first segmentation model based on the first distillation loss.

[0007] According to another aspect of the present disclosure, an image segmentation device is provided, which includes: a third acquisition module, used to input the target image into a first segmentation model to obtain a target segmentation result, wherein the target segmentation result includes a target mask of a target instance in the target image and a category of the target instance, and the first segmentation model is trained using the device provided by the present disclosure.

[0008] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method provided according to the present disclosure.

[0009] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided. The computer instructions are used to cause a computer to execute the method provided according to the present disclosure.

[0010] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements the method provided according to the present disclosure when executed by a processor.

[0011] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings are used to better understand the present invention and do not constitute a limitation of the present invention.

[0013] Figure 1 is a schematic diagram of a segmentation model according to one embodiment of the present disclosure;

[0014] Figure 2 is a flowchart of a method for training a segmentation model according to one embodiment of the present disclosure;

[0015] Figure 3 is a schematic diagram of a first distillation loss according to one embodiment of the present disclosure;

[0016] Figure 4 is a schematic diagram of determining the second distillation loss and the attention distillation loss according to one embodiment of the present disclosure;

[0017] Figure 5 is a schematic diagram of an attention mechanism according to an embodiment of the present disclosure;

[0018] Figure 6A is a schematic diagram of a first output network according to one embodiment of the present disclosure;

[0019] Figure 6B is a schematic diagram of determining distillation loss according to one embodiment of the present disclosure;

[0020] Figure 7 is a flowchart of an image segmentation method according to another embodiment of the present disclosure

[0021] Figure 8 is a block diagram of a training apparatus for a segmentation model according to one embodiment of the present disclosure;

[0022] Figure 9 is a block diagram of an image segmentation apparatus according to an embodiment of the present disclosure; and

[0023] Figure 10 is a block diagram of an electronic device to which an image segmentation method can be applied according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0025] Large models can have a large number of parameters and can preserve prior knowledge about images. Their application scenarios are constantly expanding. Large models can rapidly improve the accuracy of image processing results. However, their large number of parameters leads to slower response times. Large models are therefore difficult to adapt to scenarios that require high processing speed. Therefore, knowledge distillation techniques can be used to train smaller models based on the knowledge learned by the large model, enabling model compression without significantly sacrificing performance.

[0026] Image segmentation technology can include semantic segmentation technology and instance segmentation technology. The purpose of instance segmentation includes: distinguishing all objects in the image and determining the instances corresponding to pixels in the image. The category of the instance can be pre-set. For example, taking the example of an image containing multiple people, the instance can correspond to a single person or to multiple people who are close to each other (i.e., a crowd). Compared with semantic segmentation, instance segmentation is more difficult to implement. The category of objects in the image can belong to a set of semantic categories, but the number of instances is variable. Semantic segmentation can be achieved through dense, pixel-by-pixel classification technology. However, instance segmentation is difficult to achieve through this classification technology.

[0027] In some embodiments, instance segmentation can be performed using a top-down approach. For example, an image can be subjected to object detection to obtain a detection box for the object. The local image within the detection box can then be segmented to obtain a mask for the object. However, if the object detection results are accurate, instance segmentation using a top-down approach can yield more accurate results.

[0028] In some embodiments, instance segmentation can also be performed based on a bottom-up approach. For example, an embedding vector can be used to process pixels in an image so that the distance between multiple pixels of the same instance is reduced and the distance between multiple pixels of different instances is increased. After the embedding process is performed, an aggregation process can be performed to obtain instances in the image. However, the bottom-up approach requires higher accuracy in the embedding and aggregation processes.

[0029] Deep learning models can be used to implement instance segmentation. For example, based on a top-down approach, a multi-stage instance segmentation model can be trained to perform detection followed by segmentation. Alternatively, a single-stage instance segmentation model can be trained to accurately determine the instance corresponding to each pixel in an image. Single-stage instance segmentation models can achieve faster inference speeds. However, knowledge distillation techniques are difficult to use to improve the performance of single-stage instance segmentation models.

[0030] A single-stage instance segmentation model with a large number of parameters can be used as the teacher model, while a single-stage instance segmentation model with a small number of parameters can be used as the student model. A sample image is fed into both the teacher and student models, resulting in two segmentation results. The segmentation results indicate whether a pixel corresponds to an instance and the pixel's class. The student model's parameters can then be adjusted to reduce the discrepancy between the two segmentation results. However, aligning the model outputs and using distillation methods are relatively simple and difficult to train efficiently.

[0031] Furthermore, in instance segmentation scenarios, the backbone network of the teacher model can include multiple Transformer blocks. To reduce the number of parameters, the backbone network of the student model can include a convolutional neural network (CNN). However, the different backbone network structures of the teacher and student models make it difficult to improve the performance of the student model using the teacher model.

[0032] In order to improve the performance of the instance segmentation model, the present disclosure provides a training method for the segmentation model, which will be described below.

[0033] Figure 1 is a schematic diagram of a segmentation model according to an embodiment of the present disclosure.

[0034] In some embodiments, the segmentation model may include a backbone network (Backbone), an intermediate network (Neck), and an output network (Head). Figure 1 As shown, the first segmentation model 110 may include a first backbone network 111, a first intermediate network 112, and a first output network 113. The second segmentation model 120 may include a second backbone network 121, a second intermediate network 122, and a second output network 123. For example, the first backbone network 111 may be a convolutional neural network. The second backbone network 121 may include multiple Transformer modules. It is understood that the structure of the first backbone network 111 and the structure of the second backbone network 121 are different. It is also understood that the first segmentation model 110 can serve as a student model. The second segmentation model 120 can serve as a teacher model. The number of parameters in the second segmentation model 120 may be greater than the number of parameters in the first segmentation model 110.

[0035] In the embodiment of the present disclosure, the first sample image can be input into the first backbone network to obtain the first backbone feature. The first backbone feature can be input into the first intermediate network to obtain at least one first intermediate feature. Based on the at least one first intermediate feature, the first input feature of the first output network can be determined. The first input feature can be input into the first output network to obtain the first segmentation result. Figure 1As shown, sample image 1001 is input into the first backbone network 111 to obtain a first backbone feature. The first backbone feature is input into the first intermediate network 112 to obtain at least one first intermediate feature. The at least one first intermediate feature is input as a first input feature into the first output network 113 to obtain a first segmentation result. Based on the first segmentation result and the first label of the first sample image, a first segmentation loss of the first segmentation model can be determined.

[0036] In the embodiment of the present disclosure, the second sample image can be input into the second backbone network to obtain the second backbone feature. The second backbone feature can be input into the second intermediate network to obtain at least one second intermediate feature. Based on the at least one second intermediate feature, the second input feature of the second output network can be determined. The second input feature can be input into the second output network to obtain a second segmentation result. Figure 1 As shown, the sample image 1002 is input into the second backbone network 121 to obtain a second backbone feature. The second backbone feature is input into the second intermediate network 122 to obtain at least one second intermediate feature. The at least one second intermediate feature is input as a second input feature into the second output network 123 to obtain a second segmentation result. The segmentation loss of the second segmentation model can be determined based on the second segmentation result and the second label of the second sample image.

[0037] It can be understood that the above describes the structures of the first segmentation model and the second style model of the present disclosure, and the training method of the segmentation model will be further described below.

[0038] Figure 2 4 is a flowchart of a method for training a segmentation model according to an embodiment of the present disclosure.

[0039] like Figure 2 As shown, the method 200 may include operations S210 to S240.

[0040] In operation S210, a first sample image is input into a first segmentation model to obtain gradient information of a first intermediate network of the first segmentation model.

[0041] For example, the first sample image may be the aforementioned first sample image 1001. The first segmentation model may be the aforementioned first segmentation model 110. The first intermediate network may be the aforementioned first intermediate network 112. The first segmentation model may serve as a student model.

[0042] In an embodiment of the present disclosure, a first segmentation loss of the first segmentation model can be determined based on the first segmentation result and the first label of the first sample image. Based on the first segmentation loss, gradient information of the first segmentation model can be determined. The gradient information of the first segmentation model can include gradient information of the first intermediate network. It is understood that the gradient information of the first segmentation model can also include gradient information of the first backbone network and gradient information of the first output network.

[0043] In operation S220, at least one first intermediate mask feature corresponding to the first intermediate network is obtained according to the gradient information of the first intermediate network and at least one first intermediate feature output by the first intermediate network.

[0044] In an embodiment of the present disclosure, the first intermediate network may include a Feature Pyramid Network (FPN). There may be at least one first intermediate feature. The gradient information of the first intermediate network may include gradient information of at least one first intermediate feature. For example, the first intermediate network may output four levels of features, each of which may serve as a first intermediate feature. The gradient information of the first intermediate network may include gradient information of each of the four first intermediate features.

[0045] In operation S230 , a first distillation loss is determined based on the at least one second intermediate mask feature and the at least one first intermediate mask feature.

[0046] In the disclosed embodiment, the at least one second intermediate mask feature corresponds to a second intermediate network of a second segmentation model. For example, the second segmentation model may be the aforementioned second segmentation model 120. The second intermediate network may be the aforementioned second intermediate network 122. The second segmentation model may serve as a teacher model.

[0047] In an embodiment of the present disclosure, at least one second intermediate mask feature is obtained based on the gradient information of the second intermediate network and at least one second intermediate feature output by the second intermediate network, the gradient information of the second intermediate network is obtained by inputting the second sample image into the second segmentation model, and the parameter amount of the second segmentation model is greater than the parameter amount of the first segmentation model.

[0048] For example, the second sample image may be the second sample image 1001 described above. Based on the second segmentation result and the second label of the second sample image, the segmentation loss of the second segmentation model may be determined. Based on the second segmentation loss, gradient information of the second segmentation model may be determined. The gradient information of the second segmentation model may include gradient information of the second intermediate network. It is understood that the gradient information of the second segmentation model may also include gradient information of the second backbone network and gradient information of the second output network. The second sample image may be the second sample image 1002 described above. It is also understood that information related to the second segmentation model may be obtained before or simultaneously with obtaining relevant information about the first segmentation model.

[0049] The second intermediate network may also include a feature pyramid network. There may be at least one second intermediate feature. The gradient information of the second intermediate network may include gradient information of at least one second intermediate feature. For example, the second intermediate network may output at least four levels of features, each of which may serve as a second intermediate feature. The gradient information of the second intermediate network may include gradient information of each of the four second intermediate features.

[0050] In operation S240 , a first segmentation model is trained according to the first distillation loss.

[0051] For example, according to the first distillation loss, the parameters of the first segmentation model may be adjusted to make the first distillation loss converge.

[0052] Through the disclosed embodiments, a first distillation loss is determined based on the first intermediate features output by the first segmentation model and the second intermediate features of the second segmentation model. This allows the second segmentation model's feature extraction capabilities to be transferred to the first segmentation model, effectively improving the capabilities of the first segmentation model. Thus, if the second segmentation model is a single-stage instance segmentation model, the first segmentation model trained using the first distillation loss can efficiently perform single-stage instance segmentation on images.

[0053] It can be understood that the training method of the present disclosure has been described above, and the first distillation loss of the present disclosure will be further described below.

[0054] Figure 3 is a schematic diagram of the first distillation loss according to one embodiment of the present disclosure.

[0055] like Figure 3 As shown, the first intermediate network can output four first intermediate features, namely the first intermediate feature F311, the first intermediate feature F312, the first intermediate feature F313, and the first intermediate feature F314. The second intermediate network can output four second intermediate features, namely the second intermediate feature F321, the first intermediate feature F322, the first intermediate feature F323, and the first intermediate feature F324.

[0056] In an embodiment of the present disclosure, the gradient information of the first intermediate feature includes the gradients of each of the multiple first intermediate feature values ​​of the first intermediate feature. Obtaining at least one first intermediate mask feature corresponding to the first intermediate network based on the gradient information of the first intermediate network and at least one first intermediate feature output by the first intermediate network may include: obtaining the first intermediate mask feature based on a preset gradient threshold and the gradients of each of the multiple first intermediate feature values ​​of the first intermediate feature. For example, for the multiple first intermediate feature values ​​of the first intermediate feature, first intermediate feature values ​​greater than or equal to the preset gradient threshold are replaced with a first preset value, and first intermediate feature values ​​less than the preset gradient threshold are replaced with a second preset value, to obtain the first intermediate mask feature.

[0057] like Figure 3 As shown, based on the preset gradient threshold Th31 and the first intermediate feature F311, a first intermediate mask feature M311 can be obtained. Based on the preset gradient threshold Th31 and the first intermediate feature F312, a first intermediate mask feature M312 can be obtained. Based on the preset gradient threshold Th31 and the first intermediate feature F313, a first intermediate mask feature M313 can be obtained. Based on the preset gradient threshold Th31 and the first intermediate feature 314, a first intermediate mask feature M314 can be obtained. Taking the first intermediate feature F311 as an example, for multiple first intermediate eigenvalues ​​of the first intermediate feature F311, if the gradient of a first intermediate eigenvalue is determined to be less than the preset gradient threshold Th31, the first intermediate eigenvalue can be replaced with a second preset value (e.g., 0). If the gradient of another first intermediate eigenvalue is determined to be greater than the preset gradient threshold Th31, the first intermediate eigenvalue can be replaced with the first preset value (e.g., 1). After replacing the multiple first intermediate eigenvalues ​​with the first and second preset values, the first intermediate mask feature M311 can be obtained. The preset gradient threshold Th1 can be, for example, 0.3. It can be understood that the gradient of the first intermediate eigenvalue can be a normalized gradient. It can also be understood that the way of obtaining the first intermediate mask feature M312, the first intermediate mask feature M313 and the first intermediate mask feature M314 respectively according to the first intermediate feature F312, the first intermediate feature F313 and the first intermediate feature F314 is the same or similar to the way of obtaining the first intermediate mask feature M311 according to the first intermediate feature F311, and the present disclosure will not go into details here. Through the embodiment of the present disclosure, the first intermediate mask feature is determined by utilizing the gradient of the first intermediate eigenvalue and the preset gradient threshold, which can make full use of the knowledge of the image features extracted by the second segmentation model, help improve the ability of the first intermediate network to extract image features, and thereby improve the single-stage instance segmentation capability of the first segmentation model.

[0058] In an embodiment of the present disclosure, the gradient information of the second intermediate network includes gradient information of at least one second intermediate feature, and the gradient information of the second intermediate feature includes the gradient of each of multiple second intermediate feature values ​​of the second intermediate feature. The second intermediate mask feature is obtained based on a preset gradient threshold and the gradient of each of the multiple second intermediate feature values ​​of the second intermediate feature. For example, for the multiple second intermediate feature values ​​of the second intermediate feature, the second intermediate mask feature is obtained by replacing second intermediate feature values ​​greater than or equal to the preset gradient threshold with the first preset value and replacing second intermediate feature values ​​less than the preset gradient threshold with the second preset value.

[0059] like Figure 3 As shown, based on the preset gradient threshold Th32 and the second intermediate feature F321, a second intermediate mask feature M321 can be obtained. Based on the preset gradient threshold Th32 and the second intermediate feature F322, a second intermediate mask feature M322 can be obtained. Based on the preset gradient threshold Th32 and the second intermediate feature F323, a second intermediate mask feature M323 can be obtained. Based on the preset gradient threshold Th32 and the second intermediate feature 324, a second intermediate mask feature M324 can be obtained. Taking the second intermediate feature F321 as an example, for multiple second intermediate eigenvalues ​​of the second intermediate feature F321, if the gradient of a second intermediate eigenvalue is determined to be less than the preset gradient threshold Th32, the second intermediate eigenvalue can be replaced with the second preset value (e.g., 0). If the gradient of another second intermediate eigenvalue is determined to be greater than the preset gradient threshold Th32, the second intermediate eigenvalue can be replaced with the first preset value (e.g., 1). After replacing the multiple second intermediate eigenvalues ​​with the first and second preset values, a second intermediate mask feature M311 can be obtained. The preset gradient threshold Th2 can be, for example, 0.3. It is understood that the gradient of the second intermediate feature value can be a normalized gradient. It is also understood that the method for obtaining the second intermediate mask feature M322, the second intermediate mask feature M323, and the second intermediate mask feature M324 based on the second intermediate feature F322, the second intermediate feature F323, and the second intermediate feature F324, respectively, is the same as or similar to the method for obtaining the second intermediate mask feature M321 based on the second intermediate feature F321, and this disclosure will not repeat it here.

[0060] Next, in the embodiment of the present disclosure, a first distillation loss may be determined based on at least one first intermediate mask feature and at least one second intermediate mask feature. For example, at least one correlation loss may be determined based on at least one first intermediate mask feature and at least one second intermediate mask feature. The first distillation loss may be determined based on at least one correlation loss. Figure 3As shown, a Pearson Correlation Coefficient (PCC) can be determined based on the first intermediate mask feature M311 and the second intermediate mask feature M321 as a first correlation loss. A Pearson Correlation Coefficient can be determined based on the second intermediate mask feature M312 and the second intermediate mask feature M322 as a second correlation loss. A Pearson Correlation Coefficient can be determined based on the first intermediate mask feature M313 and the second intermediate mask feature M323 as a third correlation loss. A Pearson Correlation Coefficient can be determined based on the first intermediate mask feature M314 and the second intermediate mask feature M324 as a fourth correlation loss. A first distillation loss L301 can be determined based on the first to fourth correlation losses. Through the disclosed embodiment, the correlation loss between the first intermediate mask feature and the second intermediate mask feature is determined. Using the first distillation loss training model obtained based on this correlation loss can improve the correlation between the features output by the first intermediate network and the features output by the second intermediate network, helping to further transfer the capabilities of the second segmentation model to the first segmentation model, thereby improving the instance segmentation capability of the first segmentation model.

[0061] It is understood that the first distillation loss of the present disclosure is described above, but the distillation loss of the present disclosure is not limited thereto. The second distillation loss and attention distillation loss of the present disclosure will be described below.

[0062] Figure 4 2 is a schematic diagram of determining the second distillation loss and the attention distillation loss according to an embodiment of the present disclosure.

[0063] like Figure 4As shown, the first intermediate network can output four first intermediate features, namely the first intermediate feature F411, the first intermediate feature F412, the first intermediate feature F413, and the first intermediate feature F414. The second intermediate network can output four second intermediate features, namely the second intermediate feature F421, the second intermediate feature F422, the first intermediate feature F423, and the second intermediate feature F424. Based on the preset gradient threshold Th41, the first intermediate features F411, the first intermediate features F412, the first intermediate features F413, and the first intermediate features F414, the first intermediate mask features M311, M312, M313, and M314 can be obtained. It can be understood that for the detailed description of the first intermediate features F411 to the first intermediate features F414, the preset gradient threshold Th41, and the first intermediate mask features M411 to the first intermediate mask features M414, reference can be made to the first intermediate features F311 to the first intermediate features F314, the preset gradient threshold Th31, and the first intermediate mask features M311 to the first intermediate mask features M314, and the present disclosure will not repeat them here.

[0064] like Figure 4 As shown, the second intermediate network can output four second intermediate features, namely, the second intermediate feature F421, the second intermediate feature F422, the second intermediate feature F423, and the second intermediate feature F424. The second intermediate network can output four second intermediate features, namely, the second intermediate feature F421, the second intermediate feature F422, the second intermediate feature F423, and the second intermediate feature F424. Based on the preset gradient threshold Th42, the second intermediate features F421, the second intermediate features F422, the second intermediate features F423, and the second intermediate features F424, the second intermediate mask features M421, the second intermediate mask features M422, the second intermediate mask features M423, and the second intermediate mask features M424 can be obtained. It can be understood that for the detailed description of the second intermediate features F421 to the second intermediate features F424, the preset gradient threshold Th42, and the second intermediate mask features M421 to the second intermediate mask features M424, reference can be made to the second intermediate features F321 to the second intermediate features F324, the preset gradient threshold Th32, and the second intermediate mask features M321 to the second intermediate mask features M324, and the present disclosure will not repeat them here.

[0065] In an embodiment of the present disclosure, training a first segmentation model based on a first distillation loss includes normalizing at least one first intermediate mask feature to obtain at least one first intermediate normalized feature. Determining a second distillation loss based on at least one second intermediate normalized feature and the at least one first intermediate normalized feature. The at least one second intermediate normalized feature is obtained by normalizing the at least one second intermediate mask feature.

[0066] like Figure 4 As shown, the first intermediate mask feature M411, the first intermediate mask feature M412, the first intermediate mask feature M413, and the first intermediate mask feature M414 can be respectively input into the linear processing layer 414 to obtain four first intermediate normalized features. The second intermediate mask feature M421, the second intermediate mask feature M422, the second intermediate mask feature M423, and the second intermediate mask feature M424 can also be respectively input into the linear processing layer 414 to obtain four second intermediate normalized features. Next, four binary cross entropy losses can be determined based on the four first intermediate normalized features and the four second intermediate normalized features using a binary cross entropy loss (BCE) function. Based on the four binary cross entropy losses, the second distillation loss L402 can be determined using various methods (such as summation or weighted summation). For example, a binary cross entropy loss may be determined based on the first intermediate normalized features obtained from the first intermediate mask features M411 and the second intermediate normalized features obtained from the second intermediate mask features M421.

[0067] In the embodiment of the present disclosure, the first segmentation model can be trained based on the first distillation loss and the second distillation loss. For example, based on the first distillation loss and the second distillation loss, the total loss can be determined in various ways (such as summation or weighted summation). Next, the parameters of the first segmentation model can be adjusted based on the total loss to converge the total loss. Through the embodiment of the present disclosure, the intermediate mask features are normalized, and the first segmentation model is trained using the first distillation loss and the second distillation loss, so that the performance of the second segmentation model can be more comprehensively transferred to the first segmentation model.

[0068] It can be understood that some methods for determining the second distillation loss are described above, and some methods for determining the attention distillation loss will be described below.

[0069] In an embodiment of the present disclosure, training a first segmentation model based on a first distillation loss may include: processing at least one first intermediate mask feature using an attention mechanism to obtain at least one first attention feature. Determining an attention distillation loss based on at least one second attention feature and the at least one first attention feature. The at least one second attention feature is obtained by processing the at least one second intermediate mask feature using the attention mechanism.

[0070] like Figure 4 As shown, the first intermediate mask features M411, M412, M413, and M414 can be input into the linear processing layer 414 to obtain four first intermediate normalized features. The second intermediate mask features M421, M422, M423, and M424 can also be input into the linear processing layer 414 to obtain four second intermediate normalized features. Next, the four first intermediate normalized features can be processed separately using the attention mechanism to obtain four first attention features. The four second intermediate normalized features can also be processed separately using the attention mechanism to obtain four second attention features. Next, four attention distillation sub-losses can be determined based on the four first attention features and the four second attention features using a binary cross-entropy loss function. Based on the four attention distillation sub-losses, the attention distillation loss can be determined. For example, an attention distillation sub-loss may be determined based on a first attention feature obtained from the first intermediate mask feature F411 and a second attention feature obtained from the second intermediate mask feature F421.

[0071] In the embodiment of the present disclosure, the first segmentation model can be trained based on the first distillation loss and the attention distillation loss. For example, based on the first distillation loss and the attention distillation loss, the total loss can be determined in various ways (such as summation or weighted summation). Next, the parameters of the first segmentation model can be adjusted based on the total loss to converge the total loss. Through the embodiment of the present disclosure, the first segmentation model is trained using the attention distillation loss, which can make the first segmentation model pay more attention to the information related to instance segmentation in the image, which helps to improve the instance segmentation ability of the first segmentation model.

[0072] In the embodiments of the present disclosure, the attention mechanism can be various attention mechanisms. For example, the attention mechanism can be a self-attention mechanism or a multi-head self-attention mechanism. Figure 5 Describe an attention mechanism.

[0073] Figure 5 is a schematic diagram of an attention mechanism according to an embodiment of the present disclosure.

[0074] like Figure 5 As shown, the first intermediate normalized feature N51 can be input into the first attention unit 5151 to obtain the first attention feature A51. The second intermediate normalized feature can be input into the second attention unit 5152 to obtain the second attention feature A52. It can be understood that the first intermediate normalized feature N51 can be any one of the four first intermediate normalized features mentioned above. The second intermediate normalized feature can be any one of the four second intermediate normalized features mentioned above.

[0075] The first attention unit 5151 may include a convolutional layer 51511, a normalization layer 51512, a convolutional layer 51513, an excitation layer 51514, and a convolutional layer 51515. The first intermediate normalized feature N51 may be input into the convolutional layer 51511 to obtain a first convolutional feature. The first convolutional feature may be input into the normalization layer 51512 to obtain a first normalized feature. The first normalized feature may be multiplied by the first intermediate normalized feature N51 to obtain a first product feature. The first product feature may be input into the convolutional layer 51513 to obtain a first intermediate output feature. The first intermediate output feature may be input into the excitation layer 51514 to obtain a first excitation feature. The first excitation feature may be input into the convolutional layer 51515 to obtain a first processed feature. The first processed feature may be added to the first intermediate normalized feature N51 to obtain a first attention feature A51.

[0076] The second attention unit 5152 may include a convolutional layer 51521, a normalization layer 51522, a convolutional layer 51523, an excitation layer 51524, and a convolutional layer 51525. The second intermediate normalized feature N52 may be input into the convolutional layer 51521 to obtain a second convolutional feature. The second convolutional feature may be input into the normalization layer 51522 to obtain a second normalized feature. The second normalized feature may be multiplied with the second intermediate normalized feature N52 to obtain a second product feature. The second product feature may be input into the convolutional layer 51523 to obtain a second intermediate output feature. The second intermediate output feature may be input into the excitation layer 51524 to obtain a second excitation feature. The second excitation feature may be input into the convolutional layer 51525 to obtain a second processed feature. The second processed feature and the second intermediate normalized feature N52 may be added together to obtain a second attention feature A52.

[0077] Next, based on the first attention feature A51 and the second attention feature A52, the attention distillation sub-loss Lattl can be determined.

[0078] It can be understood that the first attention feature and the second attention feature are obtained above based on the first intermediate normalized feature and the second intermediate normalized feature. However, the present disclosure is not limited to this. The first attention feature and the second attention feature can also be obtained based on the first intermediate mask feature and the second intermediate mask feature. For example, the first intermediate mask feature can be input into the above-mentioned first attention unit to obtain the first attention feature. The second intermediate mask feature can be input into the above-mentioned second attention unit to obtain the second attention feature.

[0079] It can be understood that the above describes the attention mechanism of the present invention, and some methods of training the first segmentation model of the present invention will be further described below.

[0080] Figure 6A is a schematic diagram of a first output network according to one embodiment of the present disclosure.

[0081] In the embodiment of the present disclosure, inputting the first sample image into the first segmentation model may further include: inputting the first sample image into the first segmentation model to obtain the first segmentation feature output by the first output network. Figure 6A As shown, the first input feature is input into the first output network 613 of the first segmentation model to obtain the first segmentation feature Seg61.

[0082] In the embodiment of the present disclosure, inputting the first sample image into the first segmentation model may further include: inputting the first sample image into the first segmentation model to obtain the first instance feature and the first category feature output by the first output network. Figure 6A As shown, the above-mentioned first input feature is input into the first output network 613 of the first segmentation model, and the first instance feature Ins61 and the first category feature lable61 can also be obtained.

[0083] In an embodiment of the present disclosure, a first instance feature may indicate whether a first sample pixel in a first sample image belongs to a sample instance. For example, the first instance feature Ins61 may include multiple first instance feature values. The first instance feature value may be 0 or 1. If the first instance feature value is 0, the first sample pixel corresponding to the first instance feature value does not belong to a sample instance. If the first instance feature value is 1, the first sample pixel corresponding to the first instance feature value belongs to a sample instance.

[0084] In the embodiment of the present disclosure, the first category feature may indicate the category of at least one sample instance in the first sample image. For example, the first category feature label 61 includes multiple first category feature values, which may indicate the category corresponding to the first sample pixel.

[0085] In the disclosed embodiment, a first sample mask can be determined based on the first segmentation feature and the first instance feature. A first classification result can be determined based on the first category feature. The first classification result can indicate the category corresponding to the sample instance. For example, a dot product operation can be performed based on the first segmentation feature seg61 and the first instance feature Ins61 to obtain a first sample mask mask61. The first sample mask mask61 and the first classification result Rc61 can serve as the first segmentation result described above.

[0086] It can be understood that the first output network of the present disclosure is described above, and some methods of training the first segmentation model of the present disclosure will be further described below.

[0087] Figure 6B is a schematic diagram of determining distillation loss according to one embodiment of the present disclosure.

[0088] In the disclosed embodiment, training the first segmentation model according to the first distillation loss includes determining a third distillation loss based on the second segmentation feature and the first segmentation feature. The second segmentation feature corresponds to the second output network, and the second segmentation feature is obtained by inputting the second sample image into the second segmentation model. For example, the second input feature is input into the second output network of the second segmentation model to obtain the second segmentation feature Seg62. Based on the first segmentation feature Seg61 and the second segmentation feature Seg62, a third distillation loss L603 can be determined using a pair-wise distillation function.

[0089] In the disclosed embodiment, the first segmentation model is trained based on the first distillation loss and the third distillation loss. For example, the total loss can be determined using various methods (e.g., summation or weighted summation) based on the first distillation loss and the third distillation loss L603. Next, the parameters of the first segmentation model can be adjusted based on the total loss to converge the total loss. Through the disclosed embodiment, the ability of the second segmentation model to extract local image features can be transferred to the first segmentation model, thereby improving the feature extraction capability of the first segmentation model.

[0090] It can be understood that the third distillation loss of the present disclosure is described above, but the present disclosure is not limited thereto, which will be further described below.

[0091] In the disclosed embodiment, training the first segmentation model according to the first distillation loss includes determining a first channel dimension feature according to the first instance feature and the first category feature. For example, the first channel dimension feature can be obtained by fusing the first instance feature Ins61 with the first category feature label61.

[0092] In the embodiment of the present disclosure, a channel-dimensional distillation loss is determined based on the second channel-dimensional feature and the first channel-dimensional feature. The second channel-dimensional feature is determined based on the second instance feature and the second category feature, and the second instance feature and the second category feature are obtained by inputting the second sample image into the second segmentation model. For example, the second instance feature Ins62 and the second category feature label62 can be fused to obtain the second channel-dimensional feature. Based on the first channel-dimensional feature and the second channel-dimensional feature, the channel-dimensional distillation loss Lchannel can be determined using the Jensen-Shannon Divergence function.

[0093] In the embodiment of the present disclosure, the first segmentation model can be trained based on the first distillation loss and the channel-dimensional distillation loss. For example, based on the first distillation loss and the channel-dimensional distillation loss, the total loss can be determined using various methods (such as summation or weighted summation). Next, the parameters of the first segmentation model can be adjusted based on the total loss to converge the total loss. Through the embodiment of the present disclosure, features of different dimensions extracted by different models can be fully utilized for training, which helps to improve the global feature extraction capability of the first segmentation model.

[0094] It can be understood that the third distillation loss and channel dimension distillation loss are explained above, and the method of training the first segmentation model will be further explained below.

[0095] In an embodiment of the present disclosure, a segmentation loss is determined based on a first segmentation result output by a first segmentation model and a first label of a first sample image. The first segmentation model is trained based on the first distillation loss and the segmentation loss. For example, the first segmentation loss can be determined using various loss functions based on the first label and the first segmentation result. A total loss can be determined based on the first distillation loss and the first segmentation loss using various methods (e.g., summation or weighted summation). Next, the parameters of the first segmentation model can be adjusted based on the total loss to converge the total loss.

[0096] It will be appreciated that the first segmentation model is trained above based on the first segmentation loss, the second distillation loss, the third distillation loss, the attention distillation loss, and the channel-divided distillation loss, as well as the first distillation loss. However, the present disclosure is not limited thereto. The first segmentation model may also be trained based on at least two of the first segmentation loss, the second distillation loss, the third distillation loss, the attention distillation loss, and the channel-divided distillation loss, as well as the first distillation loss. In other words, the first segmentation model may be trained based on at least one of the first segmentation loss, the second distillation loss, the third distillation loss, the attention distillation loss, and the channel-divided distillation loss, as well as the first distillation loss.

[0097] It can be understood that the above describes the model training method of the present disclosure, and the following describes the image segmentation method of the present disclosure.

[0098] Figure 7 is a flowchart of an image segmentation method according to another embodiment of the present disclosure.

[0099] like Figure 7 As shown, the method 700 may include operation S710.

[0100] In operation S710 , a target image is input into a first segmentation model to obtain a target segmentation result.

[0101] In the embodiment of the present disclosure, the target segmentation result includes the target mask of the target instance in the target image and the category of the target instance.

[0102] In an embodiment of the present disclosure, the first segmentation model may be trained using the method provided by the present disclosure. For example, the first segmentation model may be trained according to the above-mentioned method 200.

[0103] Figure 8 4 is a block diagram of a training apparatus for a segmentation model according to an embodiment of the present disclosure.

[0104] like Figure 8 As shown, the apparatus 800 may include a first obtaining module 810 , a second obtaining module 820 , a determining module 830 and a training module 840 .

[0105] The first obtaining module 810 is used to input the first sample image into the first segmentation model to obtain gradient information of the first intermediate network of the first segmentation model.

[0106] The second obtaining module 820 is configured to obtain at least one first intermediate mask feature corresponding to the first intermediate network based on the gradient information of the first intermediate network and at least one first intermediate feature output by the first intermediate network.

[0107] Determination module 830 is configured to determine a first distillation loss based on at least one second intermediate mask feature and at least one first intermediate mask feature. The at least one second intermediate mask feature corresponds to a second intermediate network of the second segmentation model. The at least one second intermediate mask feature is obtained based on gradient information of the second intermediate network and at least one second intermediate feature output by the second intermediate network. The gradient information of the second intermediate network is obtained by inputting the second sample image into the second segmentation model. The second segmentation model has a greater number of parameters than the first segmentation model.

[0108] The training module 840 is configured to train a first segmentation model according to the first distillation loss.

[0109] In some embodiments, the gradient information of the first intermediate network includes gradient information of at least one first intermediate feature, and the gradient information of the first intermediate feature includes the gradient of each of multiple first intermediate feature values ​​of the first intermediate feature. The second obtaining module includes a first obtaining submodule for obtaining a first intermediate mask feature based on a preset gradient threshold and the gradient of each of the multiple first intermediate feature values ​​of the first intermediate feature.

[0110] In some embodiments, the first acquisition submodule includes: a replacement unit, which is used to replace the first intermediate feature value whose gradient is greater than or equal to the preset gradient threshold with the first preset value and replace the first intermediate feature value whose gradient is less than the preset gradient threshold with the second preset value for multiple first intermediate feature values ​​of the first intermediate feature, to obtain the first intermediate mask feature.

[0111] In some embodiments, the gradient information of the second intermediate network includes gradient information of at least one second intermediate feature, and the gradient information of the second intermediate feature includes the gradient of each of multiple second intermediate feature values ​​of the second intermediate feature. The second intermediate mask feature is derived based on a preset gradient threshold and the gradient of each of the multiple second intermediate feature values ​​of the second intermediate feature.

[0112] In some embodiments, the training module includes: a first attention processing submodule for processing at least one first intermediate mask feature using an attention mechanism to obtain at least one first attention feature; a first determination submodule for determining an attention distillation loss based on at least one second attention feature and at least one first attention feature. The at least one second attention feature is obtained by processing the at least one second intermediate mask feature using the attention mechanism; and a first training submodule for training a first segmentation model based on the first distillation loss and the attention distillation loss.

[0113] In some embodiments, the training module includes: a normalization submodule for normalizing the at least one first intermediate mask feature to obtain the at least one first intermediate normalized feature; a second determination submodule for determining a second distillation loss based on the at least one second intermediate normalized feature and the at least one first intermediate normalized feature. The at least one second intermediate normalized feature is obtained by normalizing the at least one second intermediate mask feature; and a second training submodule for training the first segmentation model based on the first distillation loss and the second distillation loss.

[0114] In some embodiments, the first segmentation model further includes a first output network, and the first acquisition module is further used to: input the first sample image into the first segmentation model to obtain the first segmentation feature output by the first output network.

[0115] In some embodiments, the second segmentation model further includes a second output network, and the training module includes a third determination submodule configured to determine a third distillation loss based on the second segmentation features and the first segmentation features. The second segmentation features correspond to the second output network and are obtained by inputting the second sample image into the second segmentation model. The third training submodule is configured to train the first segmentation model based on the first distillation loss and the third distillation loss.

[0116] In some embodiments, the first segmentation model further includes a first output network, and the first acquisition module is further configured to input the first sample image into the first segmentation model and obtain a first instance feature and a first category feature output by the first output network. The first instance feature is configured to indicate whether a first sample pixel in the first sample image belongs to a sample instance, and the first category feature is configured to indicate a category of at least one sample instance in the first sample image.

[0117] In some embodiments, the training module includes: a fourth determination submodule for determining a first channel-dimensional feature based on the first instance feature and the first category feature. A fifth determination submodule for determining a channel-dimensional distillation loss based on the second channel-dimensional feature and the first channel-dimensional feature. The second channel-dimensional feature is determined based on the second instance feature and the second category feature, where the second instance feature and the second category feature are obtained by inputting the second sample image into the second segmentation model. A fourth training submodule for training the first segmentation model based on the first distillation loss and the channel-dimensional distillation loss.

[0118] In some embodiments, the training module includes: a sixth determination submodule for determining a segmentation loss based on a first segmentation result output by the first segmentation model and a first label of the first sample image; and a fifth training submodule for training the first segmentation model based on the first distillation loss and the segmentation loss.

[0119] In some embodiments, the first segmentation model is a first instance segmentation model, and the second segmentation model is a second instance segmentation model.

[0120] Figure 9 is a block diagram of an image segmentation apparatus according to another embodiment of the present disclosure.

[0121] like Figure 9 As shown, the apparatus 900 may include a third obtaining module 910 .

[0122] The third obtaining module 930 is configured to input the target image into the first segmentation model to obtain a target segmentation result, which includes a target mask of a target instance in the target image and a category of the target instance.

[0123] In the embodiment of the present disclosure, the first segmentation model is trained using the apparatus provided by the present disclosure. For example, the first segmentation model is trained using the apparatus 800

[0124] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0125] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0126] Figure 10 A schematic block diagram of an example electronic device 1000 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0127] like Figure 10 As shown, the device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. Various programs and data required for the operation of the device 1000 can also be stored in the RAM 1003. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0128] Various components in device 1000 are connected to I / O interface 1005, including an input unit 1006, such as a keyboard, mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a magnetic disk, optical disk, etc.; and a communication unit 1009, such as a network card, modem, wireless communication transceiver, etc. The communication unit 1009 allows device 1000 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0129] The computing unit 1001 can be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1001 performs the various methods and processes described above, such as the training method of the segmentation model and / or the image segmentation method. For example, in some embodiments, the training method of the segmentation model and / or the image segmentation method can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as the storage unit 1008. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into the RAM 1003 and executed by the computing unit 1001, one or more steps of the training method of the segmentation model and / or the image segmentation method described above can be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to execute the segmentation model training method and / or the image segmentation method in any other appropriate manner (eg, by means of firmware).

[0130] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0131] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0132] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0133] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) display or an LCD (liquid crystal display)) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0134] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0135] Computer systems may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The client and server relationship arises through computer programs running on the respective computers and having a client-server relationship to each other.

[0136] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0137] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A training method for a segmentation model, comprising: Inputting a first sample image into a first segmentation model to obtain gradient information of a first intermediate network of the first segmentation model; obtaining, based on the gradient information of the first intermediate network and at least one first intermediate feature output by the first intermediate network, at least one first intermediate mask feature corresponding to the first intermediate network, wherein the first segmentation model further includes a first output network; determining a first distillation loss based on at least one second intermediate mask feature and at least one first intermediate mask feature, wherein the at least one second intermediate mask feature corresponds to a second intermediate network of a second segmentation model, the at least one second intermediate mask feature is obtained based on gradient information of the second intermediate network and at least one second intermediate feature output by the second intermediate network, the gradient information of the second intermediate network is obtained by inputting a second sample image into the second segmentation model, the number of parameters of the second segmentation model is greater than the number of parameters of the first segmentation model, and the second segmentation model further includes a second output network; and Training the first segmentation model according to the first distillation loss includes: Normalizing at least one of the first intermediate mask features to obtain at least one first intermediate normalized feature; determining a second distillation loss based on at least one second intermediate normalized feature and at least one first intermediate normalized feature, wherein the at least one second intermediate normalized feature is obtained by normalizing the at least one second intermediate mask feature; determining a third distillation loss according to a second segmentation feature and a first segmentation feature, wherein the first segmentation feature is output by the first output network, and the second segmentation feature is output by the second output network; The first segmentation model is trained according to the first distillation loss, the second distillation loss, and the third distillation loss.

2. The method according to claim 1, wherein The gradient information of the first intermediate network includes gradient information of at least one of the first intermediate features, and the gradient information of the first intermediate feature includes the gradients of multiple first intermediate feature values ​​of the first intermediate feature. The obtaining, based on the gradient information of the first intermediate network and the at least one first intermediate feature output by the first intermediate network, at least one first intermediate mask feature corresponding to the first intermediate network includes: The first intermediate mask feature is obtained according to a preset gradient threshold and the gradients of the plurality of first intermediate feature values ​​of the first intermediate feature.

3. The method according to claim 2, wherein: The step of obtaining the first intermediate mask feature according to a preset gradient threshold and the gradients of the plurality of first intermediate feature values ​​of the first intermediate feature includes: For multiple first intermediate feature values ​​of the first intermediate feature, the first intermediate feature values ​​whose gradient is greater than or equal to the preset gradient threshold are replaced with a first preset value, and the first intermediate feature values ​​whose gradient is less than the preset gradient threshold are replaced with a second preset value, to obtain the first intermediate mask feature.

4. The method according to claim 1, wherein The gradient information of the second intermediate network includes gradient information of at least one second intermediate feature, and the gradient information of the second intermediate feature includes the gradient of each of a plurality of second intermediate feature values ​​of the second intermediate feature. The second intermediate mask feature is obtained according to a preset gradient threshold and the gradient of each of a plurality of second intermediate feature values ​​of the second intermediate feature.

5. The method according to claim 1, wherein The training of the first segmentation model according to the first distillation loss includes: Processing at least one of the first intermediate mask features using an attention mechanism to obtain at least one first attention feature; determining an attention distillation loss based on at least one second attention feature and at least one first attention feature, wherein at least one second attention feature is obtained by processing at least one second intermediate mask feature using an attention mechanism; The first segmentation model is trained according to the first distillation loss and the attention distillation loss.

6. The method according to claim 1, wherein The inputting the first sample image into the first segmentation model further comprises: The first sample image is input into the first segmentation model to obtain a first segmentation feature output by the first output network.

7. The method according to claim 1, wherein The inputting the first sample image into the first segmentation model further comprises: The first sample image is input into the first segmentation model to obtain a first instance feature and a first category feature output by the first output network, wherein the first instance feature is used to indicate whether a first sample pixel in the first sample image belongs to a sample instance, and the first category feature is used to indicate the category of at least one sample instance in the first sample image.

8. The method according to claim 7, wherein: The training of the first segmentation model according to the first distillation loss includes: Determining a first channel dimension feature according to the first instance feature and the first category feature; determining a channel-dimensional distillation loss based on a second channel-dimensional feature and the first channel-dimensional feature, wherein the second channel-dimensional feature is determined based on a second instance feature and a second category feature, and the second instance feature and the second category feature are obtained by inputting the second sample image into the second segmentation model; and The first segmentation model is trained according to the first distillation loss and the channel dimension distillation loss.

9. The method according to claim 1, wherein: The training of the first segmentation model according to the first distillation loss includes: determining a segmentation loss according to a first segmentation result output by the first segmentation model and a first label of the first sample image; The first segmentation model is trained according to the first distillation loss and the segmentation loss.

10. The method according to claim 1, wherein The first segmentation model is a first instance segmentation model, and the second segmentation model is a second instance segmentation model.

11. An image segmentation method, comprising: Inputting the target image into the first segmentation model to obtain a target segmentation result, wherein the target segmentation result includes a target mask of a target instance in the target image and a category of the target instance, The first segmentation model is trained using the method according to any one of claims 1 to 10.

12. A training device for a segmentation model, comprising: A first obtaining module, configured to input a first sample image into a first segmentation model to obtain gradient information of a first intermediate network of the first segmentation model; a second obtaining module, configured to obtain at least one first intermediate mask feature corresponding to the first intermediate network based on the gradient information of the first intermediate network and at least one first intermediate feature output by the first intermediate network, wherein the first segmentation model further includes a first output network; a determination module, configured to determine a first distillation loss based on at least one second intermediate mask feature and at least one first intermediate mask feature, wherein the at least one second intermediate mask feature corresponds to a second intermediate network of a second segmentation model, the at least one second intermediate mask feature is obtained based on gradient information of the second intermediate network and at least one second intermediate feature output by the second intermediate network, the gradient information of the second intermediate network is obtained by inputting a second sample image into the second segmentation model, the number of parameters of the second segmentation model is greater than the number of parameters of the first segmentation model, and the second segmentation model further includes a second output network; and A training module, configured to train the first segmentation model according to the first distillation loss, comprising: a normalization processing submodule, configured to perform normalization processing on at least one of the first intermediate mask features to obtain at least one first intermediate normalized feature; a second determination submodule, configured to determine a second distillation loss based on at least one second intermediate normalized feature and at least one first intermediate normalized feature, wherein the at least one second intermediate normalized feature is obtained by normalizing the at least one second intermediate mask feature; a third determination submodule, configured to determine a third distillation loss based on a second segmentation feature and a first segmentation feature, wherein the first segmentation feature is output by the first output network, and the second segmentation feature is output by the second output network; The training module is further configured to train the first segmentation model according to the first distillation loss, the second distillation loss, and the third distillation loss.

13. The device according to claim 12, wherein The gradient information of the first intermediate network includes gradient information of at least one of the first intermediate features, and the gradient information of the first intermediate feature includes the gradients of multiple first intermediate feature values ​​of the first intermediate feature. The second obtaining module includes: The first obtaining submodule is configured to obtain the first intermediate mask feature according to a preset gradient threshold and the gradients of the plurality of first intermediate feature values ​​of the first intermediate feature.

14. The device according to claim 13, wherein The first obtaining submodule includes: A replacement unit is used to replace, for multiple first intermediate feature values ​​of the first intermediate feature, the first intermediate feature values ​​whose gradient is greater than or equal to the preset gradient threshold with a first preset value, and to replace the first intermediate feature values ​​whose gradient is less than the preset gradient threshold with a second preset value, to obtain the first intermediate mask feature.

15. The device according to claim 12, wherein The gradient information of the second intermediate network includes gradient information of at least one second intermediate feature, and the gradient information of the second intermediate feature includes the gradient of each of a plurality of second intermediate feature values ​​of the second intermediate feature. The second intermediate mask feature is obtained according to a preset gradient threshold and the gradient of each of a plurality of second intermediate feature values ​​of the second intermediate feature.

16. The device according to claim 12, wherein The training module includes: A first attention processing submodule, configured to process at least one of the first intermediate mask features using an attention mechanism to obtain at least one first attention feature; a first determination submodule, configured to determine an attention distillation loss based on at least one second attention feature and at least one first attention feature, wherein the at least one second attention feature is obtained by processing at least one second intermediate mask feature using an attention mechanism; A first training submodule is configured to train the first segmentation model based on the first distillation loss and the attention distillation loss.

17. The device according to claim 12, wherein The first obtaining module is further configured to: The first sample image is input into the first segmentation model to obtain a first segmentation feature output by the first output network.

18. The device according to claim 12, wherein The first obtaining module is further configured to: The first sample image is input into the first segmentation model to obtain a first instance feature and a first category feature output by the first output network, wherein the first instance feature is used to indicate whether a first sample pixel in the first sample image belongs to a sample instance, and the first category feature is used to indicate the category of at least one sample instance in the first sample image.

19. The device according to claim 18, wherein The training module includes: a fourth determining submodule, configured to determine a first channel dimension feature based on the first instance feature and the first category feature; a fifth determination submodule, configured to determine a channel-dimensional distillation loss based on a second channel-dimensional feature and the first channel-dimensional feature, wherein the second channel-dimensional feature is determined based on a second instance feature and a second category feature, and the second instance feature and the second category feature are obtained by inputting the second sample image into the second segmentation model; and A fourth training submodule is configured to train the first segmentation model based on the first distillation loss and the channel dimension distillation loss.

20. The device according to claim 12, wherein The training module includes: a sixth determination submodule, configured to determine a segmentation loss according to a first segmentation result output by the first segmentation model and a first label of the first sample image; A fifth training submodule is configured to train the first segmentation model according to the first distillation loss and the segmentation loss.

21. The device according to claim 12, wherein The first segmentation model is a first instance segmentation model, and the second segmentation model is a second instance segmentation model.

22. An image segmentation device, comprising: The third obtaining module is configured to input the target image into the first segmentation model to obtain a target segmentation result, wherein the target segmentation result includes a target mask of a target instance in the target image and a category of the target instance. The first segmentation model is trained using the apparatus according to any one of claims 12 to 21.

23. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 11.

24. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 11.

25. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 11.

Images