Image segmentation method, image segmentation model training method and device

Abstract

The present disclosure relates to an image segmentation method, an image segmentation model training method and a device. The method comprises: obtaining a first feature of an image to be segmented and an initial segmentation result, wherein the initial segmentation result is obtained by segmenting the first feature according to a second prototype vector of a labeled image, and the labeled object in the labeled image and the object to be segmented in the image to be segmented belong to the same category; determining the first prototype vector of the image to be segmented according to the first feature and the initial segmentation result; and segmenting the first feature according to the second prototype vector and the first prototype vector to obtain a segmentation result of the image to be segmented. By utilizing the first prototype vector in combination with the second prototype vector, accurate segmentation can be achieved even when the labeled object of the labeled image and the object to be segmented of the image to be segmented are quite different.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision technology, and in particular to an image segmentation method, and a training method and device for an image segmentation model. Background Art

[0002] With the development of computer vision technology, image segmentation technology has emerged. Through image segmentation technology, images can be separated into non-overlapping areas with the same properties, which has a wide range of application value, such as autonomous driving, target detection, medical imaging, etc. In related technologies, image segmentation models are used to process images to be segmented. However, in order to obtain a highly accurate image segmentation model, a lot of manpower and material resources are often required to label training samples. Even so, the generalization ability of the trained image segmentation model is still not high, and it is impossible to accurately segment new categories of images. Summary of the invention

[0003] The present disclosure provides an image segmentation method, an image segmentation model training method and a device, so as to at least solve the problem that the image segmentation model in the related art cannot accurately segment new categories of images. The technical solution of the present disclosure is as follows:

[0004] According to a first aspect of an embodiment of the present disclosure, there is provided an image segmentation method, comprising:

[0005] Acquire a first feature and an initial segmentation result of the image to be segmented, wherein the initial segmentation result is obtained by segmenting the first feature according to the second prototype vector of the marked image, and the marked object in the marked image and the object to be segmented in the image to be segmented belong to the same category;

[0006] Determining a first prototype vector of the image to be segmented according to the first feature and the initial segmentation result;

[0007] The first feature is segmented according to the second prototype vector and the first prototype vector to obtain a segmentation result of the image to be segmented.

[0008] In a possible implementation, the initial segmentation result includes an initial foreground segmentation result and an initial background segmentation result, and the first prototype vector includes a foreground first prototype vector and a background first prototype vector; and determining the first prototype vector of the image to be segmented according to the first feature and the initial segmentation result includes:

[0009] Determine a first feature of the object to be segmented according to the first feature and the initial foreground segmentation result, wherein the first feature of the object to be segmented is a feature belonging to the object to be segmented in the first feature of the image to be segmented;

[0010] Determine, according to the first feature and the initial background segmentation result, a first feature of a background image in the image to be segmented, wherein the background image of the image to be segmented is an image in the image to be segmented excluding the object to be segmented;

[0011] Performing global averaging processing on the first feature of the object to be segmented to obtain a foreground first prototype vector;

[0012] A global averaging process is performed on the first feature of the background image in the image to be segmented to obtain a background first prototype vector.

[0013] In a possible implementation, performing global averaging processing on the first feature of the object to be segmented in the image to be segmented to obtain a foreground first prototype vector includes:

[0014] Obtaining a prediction confidence of a first feature of the object to be segmented;

[0015] Performing global averaging processing on the first feature of the object to be segmented whose prediction confidence is greater than a preset threshold to obtain a foreground first prototype vector;

[0016] The step of performing global averaging processing on the first feature of the background image in the image to be segmented to obtain a first background prototype vector includes:

[0017] Obtaining a prediction confidence of a first feature of a background image in the image to be segmented;

[0018] A global averaging process is performed on the first feature of the background image in the image to be segmented whose prediction confidence is greater than a preset threshold to obtain a background first prototype vector.

[0019] In a possible implementation manner, performing segmentation processing on the first feature according to the second prototype vector and the first prototype vector to obtain a segmentation result of the image to be segmented includes:

[0020] Determine a third prototype vector according to the second prototype vector and the first prototype vector;

[0021] Calculating the similarity between the third prototype vector and the first feature;

[0022] A segmentation result of the image to be segmented is determined according to the similarity between the third prototype vector and the first feature.

[0023] In a possible implementation, the second prototype vector includes a foreground second prototype vector and a background second prototype vector, the first prototype vector includes a foreground first prototype vector and a background first prototype vector, and the segmentation processing is performed on the first feature according to the second prototype vector and the first prototype vector to obtain a segmentation result of the image to be segmented, including:

[0024] Determine a foreground third prototype vector according to the foreground second prototype vector and the foreground first prototype vector;

[0025] Determine a background third prototype vector according to the background second prototype vector and the background first prototype vector;

[0026] Calculating the similarity between the foreground third prototype vector and the first feature to obtain a first similarity;

[0027] Calculating the similarity between the background third prototype vector and the first feature to obtain a second similarity;

[0028] A segmentation result of the image to be segmented is determined according to the first similarity and the second similarity.

[0029] In a possible implementation manner, a method of obtaining the second prototype vector of the marked image includes:

[0030] Acquire a second feature of the marked image and a mask corresponding to the marked image;

[0031] A second prototype vector of the marked image is determined according to the second feature and the mask.

[0032] In a possible implementation, the mask includes a foreground mask and a background mask, and the second prototype vector includes a foreground second prototype vector and a background second prototype vector; the method for obtaining the second prototype vector of the marked image includes:

[0033] Determining the second feature of the marked object according to the second feature of the marked image and the foreground mask corresponding to the marked image, wherein the second feature of the marked object is a feature belonging to the marked object in the second feature of the marked image;

[0034] Determine, according to the second feature of the marked image and the background mask corresponding to the marked image, the second feature of the background image in the marked image, wherein the background image of the marked image is an image in the marked image excluding the marked object;

[0035] Performing global averaging processing on the second feature of the marked object to obtain a foreground second prototype vector;

[0036] A global averaging process is performed on the second feature of the background image in the marked image to obtain a background second prototype vector.

[0037] In a possible implementation, the initial segmentation result is obtained by performing segmentation processing on the first feature according to the second prototype vector of the marked image, including:

[0038] Calculating the similarity between the second prototype vector and the first feature;

[0039] An initial segmentation result of the sample image to be segmented is determined according to the similarity between the second prototype vector and the first feature.

[0040] In a possible implementation, the segmentation result is obtained by an image segmentation model, and the training method of the image segmentation model includes:

[0041] Acquire a sample image set, wherein the sample image set includes a sample labeled image and a sample to-be-segmented image with an object to be segmented;

[0042] Determine a sample second prototype vector according to the sample second feature of the sample labeled image and the mask corresponding to the sample labeled image;

[0043] According to the sample second prototype vector, segmenting the sample first feature of the sample image to be segmented, to obtain an initial segmentation result of the sample image to be segmented;

[0044] Determine a first prototype vector of the sample according to the first feature and the initial segmentation result;

[0045] According to the sample second prototype vector and the sample first prototype vector, segmenting the sample first feature to obtain a segmentation result of the sample image to be segmented;

[0046] Based on the difference between the segmentation result and the marked object to be segmented, the training parameters of the image segmentation model are iteratively adjusted until the difference meets the preset requirement.

[0047] According to a second aspect of an embodiment of the present disclosure, a method for training an image segmentation model is provided, comprising:

[0048] Acquire a sample image set, wherein the sample image set includes a sample labeled image and a sample to-be-segmented image with an object to be segmented;

[0049] Determine a sample second prototype vector according to a sample second feature of the sample labeled image and a mask corresponding to the sample labeled image;

[0050] According to the sample second prototype vector, segmenting the sample first feature of the sample image to be segmented, to obtain an initial segmentation result of the sample image to be segmented;

[0051] Determining a first prototype vector of the sample according to the first feature of the sample and the initial segmentation result;

[0052] According to the sample second prototype vector and the sample first prototype vector, segmenting the sample first feature to obtain a segmentation result of the sample image to be segmented;

[0053] Based on the difference between the segmentation result and the marked object to be segmented, the training parameters of the image segmentation model are iteratively adjusted until the difference meets the preset requirement.

[0054] According to a third aspect of an embodiment of the present disclosure, there is provided an image segmentation device, including:

[0055] A first acquisition module is used to acquire a first feature of the image to be segmented and an initial segmentation result, wherein the initial segmentation result is obtained by segmenting the first feature according to the second prototype vector of the marked image, and the marked object in the marked image and the object to be segmented in the image to be segmented belong to the same category;

[0056] A first determining module, used for determining a first prototype vector of the image to be segmented according to the first feature and the initial segmentation result;

[0057] The first segmentation module is used to perform segmentation processing on the first feature according to the second prototype vector and the first prototype vector to obtain a segmentation result of the image to be segmented.

[0058] In a possible implementation, the initial segmentation result includes an initial foreground segmentation result and an initial background segmentation result, the first prototype vector includes a foreground first prototype vector and a background first prototype vector; and the first determining module includes:

[0059] A first determination submodule is used to determine a first feature of the object to be segmented according to the first feature and an initial foreground segmentation result, wherein the first feature of the object to be segmented is a feature belonging to the object to be segmented in the first feature of the image to be segmented;

[0060] A second determination submodule is used to determine a first feature of a background image in the image to be segmented according to the first feature and an initial background segmentation result, wherein the background image of the image to be segmented is an image in the image to be segmented excluding the object to be segmented;

[0061] A first processing submodule is used to perform global averaging processing on the first feature of the object to be segmented to obtain a first foreground prototype vector;

[0062] The second processing submodule is used to perform global averaging processing on the first feature of the background image in the image to be segmented to obtain a first background prototype vector.

[0063] In a possible implementation, the first processing submodule includes:

[0064] A first acquisition unit, used to acquire a prediction confidence of a first feature of the object to be segmented;

[0065] A first processing unit is used to perform global averaging processing on the first feature of the object to be segmented whose prediction confidence is greater than a preset threshold, so as to obtain a foreground first prototype vector;

[0066] The second processing submodule includes:

[0067] A second acquisition unit, used to acquire a prediction confidence of a first feature of a background image in the image to be segmented;

[0068] The second processing unit is used to perform global averaging processing on the first feature of the background image in the image to be segmented whose prediction confidence is greater than a preset threshold, so as to obtain a background first prototype vector.

[0069] In a possible implementation, the segmentation module includes:

[0070] A third determining submodule, configured to determine a third prototype vector according to the second prototype vector and the first prototype vector;

[0071] A first calculation submodule, used for calculating the similarity between the third prototype vector and the first feature;

[0072] The fourth determination submodule is used to determine the segmentation result of the image to be segmented according to the similarity between the third prototype vector and the first feature.

[0073] In a possible implementation, the second prototype vector includes a foreground second prototype vector and a background second prototype vector, the first prototype vector includes a foreground first prototype vector and a background first prototype vector, and the segmentation module includes:

[0074] a fifth determining submodule, configured to determine a foreground third prototype vector according to the foreground second prototype vector and the foreground first prototype vector;

[0075] A sixth determining submodule, configured to determine a background third prototype vector according to the background second prototype vector and the background first prototype vector;

[0076] A second calculation submodule is used to calculate the similarity between the foreground third prototype vector and the first feature to obtain a first similarity;

[0077] A third calculation submodule is used to calculate the similarity between the background third prototype vector and the first feature to obtain a second similarity;

[0078] The seventh determination submodule is used to determine a segmentation result of the image to be segmented according to the first similarity and the second similarity.

[0079] In a possible implementation, the method further includes:

[0080] A second acquisition module, used to acquire a second feature of the marked image and a mask corresponding to the marked image;

[0081] The second determining module is used to determine a second prototype vector of the marked image according to the second feature and the mask.

[0082] In a possible implementation, the mask includes a foreground mask and a background mask, and the second prototype vector includes a foreground second prototype vector and a background second prototype vector; and the device further includes:

[0083] A third determining module is used to determine the second feature of the marked object according to the second feature of the marked image and the foreground mask corresponding to the marked image, wherein the second feature of the marked object is a feature of the second feature of the marked image belonging to the marked object;

[0084] a fourth determining module, configured to determine a second feature of a background image in the marked image according to the second feature of the marked image and a background mask corresponding to the marked image, wherein the background image of the marked image is an image in the marked image excluding the marked object;

[0085] A first processing module, configured to perform global averaging processing on the second feature of the marked object to obtain a foreground second prototype vector;

[0086] The second processing module is used to perform global averaging processing on the second feature of the background image in the marked image to obtain a background second prototype vector.

[0087] In a possible implementation, the method further includes:

[0088] A calculation module, used for calculating the similarity between the second prototype vector and the first feature;

[0089] The fifth determination module is used to determine the initial segmentation result of the sample image to be segmented according to the similarity between the second prototype vector and the first feature.

[0090] In a possible implementation, the method further includes:

[0091] A third acquisition module is used to acquire a sample image set, wherein the sample image set includes a sample marked image and a sample to-be-segmented image with an object to be segmented;

[0092] a sixth determination module, configured to determine a sample second prototype vector according to the sample second feature of the sample labeled image and a mask corresponding to the sample labeled image;

[0093] A second segmentation module is used to perform segmentation processing on the sample first feature of the sample image to be segmented according to the sample second prototype vector to obtain an initial segmentation result of the sample image to be segmented;

[0094] a seventh determination module, configured to determine a first prototype vector of the sample according to the first feature and the initial segmentation result;

[0095] A third segmentation module is used to perform segmentation processing on the first feature of the sample according to the second prototype vector of the sample and the first prototype vector of the sample to obtain a segmentation result of the sample image to be segmented;

[0096] The adjustment module is used to iteratively adjust the training parameters of the image segmentation model based on the difference between the segmentation result and the marked object to be segmented, until the difference meets the preset requirements.

[0097] According to a sixth aspect of an embodiment of the present disclosure, a training device for an image segmentation model is provided, comprising:

[0098] A third acquisition module is used to acquire a sample image set, wherein the sample image set includes a sample marked image and a sample to-be-segmented image with an object to be segmented;

[0099] a sixth determination module, configured to determine a sample second prototype vector according to the sample second feature of the sample labeled image and a mask corresponding to the sample labeled image;

[0100] A second segmentation module is used to perform segmentation processing on the sample first feature of the sample image to be segmented according to the sample second prototype vector to obtain an initial segmentation result of the sample image to be segmented;

[0101] a seventh determination module, configured to determine a first prototype vector of the sample according to the first feature and the initial segmentation result;

[0102] A third segmentation module is used to perform segmentation processing on the first feature of the sample according to the second prototype vector of the sample and the first prototype vector of the sample to obtain a segmentation result of the sample image to be segmented;

[0103] The adjustment module is used to iteratively adjust the training parameters of the image segmentation model based on the difference between the segmentation result and the marked object to be segmented, until the difference meets the preset requirements.

[0104] According to a fifth aspect of an embodiment of the present disclosure, a server is provided, including:

[0105] processor;

[0106] a memory for storing instructions executable by the processor;

[0107] The processor is configured to execute the instructions to implement the image segmentation method as described in any one of the embodiments of the present disclosure or the training method of the image segmentation model as described in any one of the embodiments of the present disclosure.

[0108] According to the sixth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided. When the instructions in the computer-readable storage medium are executed by the processor of the server, the server is enabled to execute the image segmentation method as described in any one of the embodiments of the present disclosure or the training method of the image segmentation model as described in any one of the embodiments of the present disclosure.

[0109] According to the seventh aspect of the embodiments of the present disclosure, a computer program product is provided, which includes instructions. When the instructions are executed by a processor of a server, the server can execute the image segmentation method as described in any one of the embodiments of the present disclosure or the training method of the image segmentation model as described in any one of the embodiments of the present disclosure.

[0110] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects: In the embodiments of the present disclosure, the first prototype vector of the image to be segmented is determined according to the first feature of the image to be segmented and the initial segmentation result of the image to be segmented, and the first feature is segmented according to the first prototype vector and the second prototype vector of the labeled image to obtain the segmentation result of the image to be segmented. Therefore, for new categories of objects to be segmented, the present disclosure does not need to retrain the image segmentation model, and can also accurately segment new categories of objects to be segmented. Since the first prototype vector derived from the image to be segmented is used in the embodiments of the present disclosure, and since the first prototype vector is derived from the features of the image to be segmented itself, the first prototype vector is closer to the object to be segmented. By combining the first prototype vector with the second prototype vector, accurate segmentation can be achieved even when the labeled object of the labeled image is quite different from the object to be segmented of the image to be segmented.

[0111] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0112] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0113] Figure 1The figure is a flowchart of an image segmentation method according to an exemplary embodiment.

[0114] Figure 2 The figure is a flowchart of an image segmentation method according to another exemplary embodiment.

[0115] Figure 3 The figure is a flowchart of a method for generating a first prototype vector in an image segmentation method according to an exemplary embodiment.

[0116] Figure 4 The figure is a flowchart of an image segmentation model training method according to an exemplary embodiment.

[0117] Figure 5 The figure is a block diagram of an image segmentation device according to an exemplary embodiment.

[0118] Figure 6 A training device for an image segmentation model is shown according to an exemplary embodiment.

[0119] Figure 7 The figure is a block diagram of a server for an image segmentation method according to an exemplary embodiment. DETAILED DESCRIPTION

[0120] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0121] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0122] It should also be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0123] Figure 1 is a flowchart of an image segmentation method according to an exemplary embodiment. Figure 1 As shown, the method is used in a server and includes the following steps.

[0124] Step S101, obtaining a first feature of an image to be segmented and an initial segmentation result, wherein the initial segmentation result is obtained by segmenting the first feature according to a second prototype vector of a marked image, and the marked object in the marked image and the object to be segmented in the image to be segmented belong to the same category.

[0125] In the disclosed embodiment, the image to be segmented includes at least an object to be segmented, wherein the area where the object to be segmented is located can be the foreground image of the image to be segmented, and the area other than the object to be segmented in the image to be segmented can be the background image of the image to be segmented. In the image to be segmented, the number of objects to be segmented can include one or more. In the disclosed embodiment, the marked image includes a marked object, and the marked object belongs to the same category as the object to be segmented in the object to be segmented. For example, if the object to be segmented is a cat, then the marked object is also a cat. For another example, if the object to be segmented is a ship, then the marked object is also a ship. In the disclosed embodiment, the marked image includes an image containing a marked object, wherein the marked object needs to be marked in advance, that is, through the marked object, the image containing the marked object and the foreground mask corresponding to the marked object can be extracted. Of course, by performing relevant processing on the foreground mask, such as inversion processing, the background mask can also be obtained.

[0126] Figure 2 is a flowchart of an image segmentation method according to an exemplary embodiment. Figure 2 As shown. A convolutional neural network can be used to extract features of an image, for example, to extract features of the segmented image 202 to obtain a first feature, and to extract features of the marked image 201 to obtain a second feature. In one example, a twin network can be used to extract features of an image, for example, the network for extracting features of the segmented image is the same as the network for extracting features of the marked image, and the network parameters are shared. In one example, a mask of the marked image, such as a foreground mask, can be multiplied with the second feature of the marked image to obtain a second feature corresponding to the marked object. In another example, the background mask of the marked image can also be multiplied with the second feature of the marked image to obtain a second feature corresponding to the area other than the marked object in the marked image. According to the second feature of the marked image and the second feature corresponding to the area other than the marked object, the second feature corresponding to the marked object is obtained. In one example, the second feature corresponding to the marked object is globally averaged to obtain a foreground second prototype vector. The second feature of the area other than the marked object in the marked image is globally averaged to obtain a background second prototype vector.

[0127] In the embodiment of the present disclosure, segmenting the first feature according to the second prototype vector of the marked image may include: calculating the similarity between the second prototype vector and the first feature of the image to be segmented, and if the similarity is greater than a preset value, determining that the first feature is a feature of the foreground image, such as Figure 2 The initial foreground segmentation result 206 in the example; if the similarity is less than or equal to the preset value, the first feature is a feature of the background image, such as Figure 2 The initial background segmentation result 205 in .

[0128] Step S103: determining a first prototype vector of the image to be segmented according to the first feature and the initial segmentation result.

[0129] In the disclosed embodiment, the first prototype vector may include a foreground first prototype vector and / or a background first prototype vector. The initial segmentation result may include an initial foreground segmentation result and / or an initial background segmentation result. In one example, an initial foreground mask of the image to be segmented may be obtained from the initial foreground segmentation result, and the initial foreground mask is multiplied with the first feature of the image to be segmented to obtain the first feature corresponding to the object to be segmented, and the first feature corresponding to the object to be segmented is globally averaged to obtain the foreground first prototype vector. Figure 2 In another example, an initial background mask of the image to be segmented can be obtained from the initial background segmentation result, and the initial background mask is multiplied with the first feature of the image to be segmented to obtain the first feature corresponding to the area other than the object to be segmented in the image to be segmented, and the first feature corresponding to the area other than the object to be segmented is globally averaged to obtain the background first prototype vector 208. Figure 2 The background first prototype vector 207 in .

[0130] Step S103: performing segmentation processing on the first feature according to the second prototype vector and the first prototype vector to obtain a segmentation result of the image to be segmented.

[0131] In the disclosed embodiment, the foreground second prototype vector in the second prototype vector and the foreground first prototype vector in the first prototype vector can be used to perform segmentation processing on the first feature of the image to be segmented, and the segmentation result of the image to be segmented can be obtained. The background second prototype vector in the second prototype vector and the background first prototype vector in the first prototype vector can also be used to perform segmentation processing on the first feature of the image to be segmented, and the segmentation result of the image to be segmented can be obtained. The foreground second prototype vector, the background second prototype vector, the foreground first prototype vector and the background first prototype vector can also be used to perform segmentation processing on the first feature of the image to be segmented, and the segmentation result of the image to be segmented can be obtained. Specifically, taking the foreground second prototype vector and the foreground first prototype vector as an example, the foreground second prototype vector and the foreground first prototype vector can be added to the foreground first prototype vector to obtain the foreground third prototype vector. The similarity between the foreground third prototype vector and the first feature in the image to be segmented is calculated, and based on the similarity, the foreground and background in the image to be segmented are determined to obtain the segmentation result. It should be noted that the processing based on the second prototype vector and the first prototype vector is not limited to the addition of the first prototype vector and the second prototype vector, but may also include other methods, such as weighted summation of the first prototype vector and the second prototype vector. Technical personnel in the relevant field may make other changes inspired by the technical essence of the present application, but as long as the functions and effects achieved are the same or similar to those of the present application, they should be covered within the scope of protection of the present application.

[0132] In the disclosed embodiment, the first prototype vector of the image to be segmented is determined based on the first feature of the image to be segmented and the initial segmentation result of the image to be segmented, and the first feature is segmented based on the first prototype vector and the second prototype vector of the labeled image to obtain the segmentation result of the image to be segmented. Therefore, for new categories of objects to be segmented, the disclosed embodiment does not need to retrain the image segmentation model, and can accurately segment new categories of objects to be segmented. Since the first prototype vector derived from the image to be segmented is used in the disclosed embodiment, and since the first prototype vector is derived from the features of the image to be segmented itself, the first prototype vector is closer to the object to be segmented. By utilizing the first prototype vector in combination with the second prototype vector, accurate segmentation can be achieved even when the labeled object of the labeled image is quite different from the object to be segmented of the image to be segmented.

[0133] In a possible implementation, the initial segmentation result includes an initial foreground segmentation result and an initial background segmentation result, and the first prototype vector includes a foreground first prototype vector and a background first prototype vector; and determining the first prototype vector of the image to be segmented according to the first feature and the initial segmentation result includes:

[0134] Determine a first feature of the object to be segmented according to the first feature and the initial foreground segmentation result, wherein the first feature of the object to be segmented is a feature belonging to the object to be segmented in the first feature of the image to be segmented;

[0135] Determine, according to the first feature and the initial background segmentation result, a first feature of a background image in the image to be segmented, wherein the background image of the image to be segmented is an image in the image to be segmented excluding the object to be segmented;

[0136] Performing global averaging processing on the first feature of the object to be segmented in the image to be segmented to obtain a foreground first prototype vector;

[0137] A global averaging process is performed on the first feature of the background image in the image to be segmented to obtain a background first prototype vector.

[0138] In the embodiments of the present disclosure, reference Figure 2 As shown, the specific generation method of the self-supporting foreground prototype vector generation module (SSFP) can be referred to Figure 3 , a masked average pooling (MAP) operation is performed on the first feature 204 of the image to be segmented and the initial foreground segmentation result 206, that is, a global average processing is performed on the first feature of the segmented object to obtain a foreground first prototype vector 208. Similarly, a masked average pooling operation is performed on the first feature 204 of the image to be segmented and the initial background segmentation result 205, that is, a global average processing is performed on the first feature of the background image in the segmented image to obtain a background first prototype vector 207.

[0139] By using the method of the embodiment of the present disclosure, the foreground first prototype vector and the background first prototype vector of the image to be segmented can be accurately and quickly obtained. The similarity between the obtained first prototype vector and the first feature of the image to be segmented is higher than the similarity between the second prototype vector and the first feature of the image to be segmented, thus facilitating subsequent accurate segmentation.

[0140] In a possible implementation, performing global averaging processing on the first feature of the object to be segmented in the image to be segmented to obtain a foreground first prototype vector includes:

[0141] Obtaining a prediction confidence of a first feature of an object to be segmented in the image to be segmented;

[0142] Performing global averaging processing on the first feature of the object to be segmented in the image to be segmented whose prediction confidence is greater than a preset threshold, to obtain a foreground first prototype vector;

[0143] The step of performing global averaging processing on the first feature of the background image in the image to be segmented to obtain a first background prototype vector includes:

[0144] Obtaining the confidence of a first feature of a background image in the image to be segmented;

[0145] A global averaging process is performed on the first feature of the background image in the image to be segmented whose confidence is greater than a preset threshold to obtain a background first prototype vector.

[0146] In the disclosed embodiment, the first feature is segmented according to the second prototype vector of the marked image to obtain an initial segmentation result, which can be obtained by training an artificial neural network, thereby obtaining the predicted confidence that the first feature of the image to be segmented falls into the initial segmentation result. The predicted confidence of the first feature of the object to be segmented in the image to be segmented is obtained, that is, the predicted confidence of the first feature in the initial foreground segmentation result is obtained, the first feature of the object to be segmented whose predicted confidence is greater than a preset threshold is selected, and the first feature is globally averaged to obtain the foreground first prototype vector. Similarly, the disclosed embodiment can obtain the predicted confidence of the first feature of the background image in the image to be segmented. The first feature of the background image whose predicted confidence is greater than a preset threshold is globally averaged to obtain the background first prototype vector.

[0147] In the disclosed embodiment, according to the prediction confidence of the first feature of the image to be segmented, a global averaging operation is performed on the first feature with high prediction confidence to obtain the foreground first prototype vector and the background first prototype vector respectively. By screening the prediction confidence, the similarity between the first prototype vector and the first feature can be improved, which is conducive to subsequent accurate image segmentation.

[0148] In a possible implementation manner, performing segmentation processing on the first feature according to the second prototype vector and the first prototype vector to obtain a segmentation result of the image to be segmented includes:

[0149] Determine a third prototype vector according to the second prototype vector and the first prototype vector;

[0150] Calculating the similarity between the third prototype vector and the first feature;

[0151] A segmentation result of the image to be segmented is determined according to the similarity between the third prototype vector and the first feature.

[0152] In the disclosed embodiment, the first prototype vector may include a foreground first prototype vector or a background first prototype vector. The second prototype vector may include a foreground second prototype vector or a background second prototype vector. The determining of the third prototype vector according to the second prototype vector and the first prototype vector may include: in one example, adding the first prototype to the second prototype vector to obtain the third prototype vector. In another example, different weights may be set for the first prototype vector and the second prototype vector, and the third prototype vector is the weighted sum of the first prototype vector and the second prototype vector. In one example, the third prototype vector includes a foreground third prototype vector, that is, the foreground third prototype vector is determined according to the foreground second prototype vector and the foreground first prototype vector. The similarity between the foreground third prototype vector and the first feature of the image to be segmented is calculated. If the similarity is greater than a preset value, the first feature is the first feature corresponding to the object to be segmented. If the similarity is less than or equal to a preset value, the first feature is the first feature of the background image of the image to be segmented. In another example, the third prototype vector includes a background third prototype vector, that is, the background third prototype vector is determined according to the background second prototype vector and the background first prototype vector. The similarity between the background third prototype vector and the first feature of the image to be segmented is calculated. If the similarity is greater than a preset value, the first feature is the first feature of the background image in the image to be segmented. If the similarity is less than or equal to the preset value, the first feature is the first feature of the object to be segmented.

[0153] In the disclosed embodiment, the first prototype vector derived from the image to be segmented is used. Since the first prototype vector is derived from the characteristics of the image to be segmented, the first prototype vector is closer to the object to be segmented. By combining the first prototype vector with the second prototype vector, accurate segmentation can be performed even when the marked object of the marked image is significantly different from the object to be segmented of the image to be segmented.

[0154] In a possible implementation, the second prototype vector includes a foreground second prototype vector and a background second prototype vector, the first prototype vector includes a foreground first prototype vector and a background first prototype vector, and the segmentation processing is performed on the first feature according to the second prototype vector and the first prototype vector to obtain a segmentation result of the image to be segmented, including:

[0155] Determine a foreground third prototype vector according to the foreground second prototype vector and the foreground first prototype vector;

[0156] Determine a background third prototype vector according to the background second prototype vector and the background first prototype vector;

[0157] Calculating the similarity between the foreground third prototype vector and the first feature to obtain a first similarity;

[0158] Calculating the similarity between the background third prototype vector and the first feature to obtain a second similarity;

[0159] A segmentation result of the image to be segmented is determined according to the first similarity and the second similarity.

[0160] In the disclosed embodiment, the foreground third prototype vector is determined based on the foreground second prototype vector and the foreground first prototype vector, and the specific method for determining the background third prototype vector based on the background second prototype vector and the background first prototype vector is the same as the method of determining the third prototype vector based on the second prototype vector and the first prototype vector in the above-mentioned embodiment, and will not be repeated here. In the disclosed embodiment, the foreground third prototype vector and the background third prototype vector are used simultaneously to calculate the similarity between the foreground third prototype vector and the first feature as the first similarity. The similarity between the background third prototype vector and the first feature is calculated as the second similarity. The first similarity and the second similarity are compared. If the first similarity is greater than the second similarity, the first feature is the first feature of the foreground image; if the first similarity is less than the second similarity, the first feature is the first feature of the background image.

[0161] The disclosed embodiment uses the third foreground prototype vector and the third background prototype vector in combination to perform image segmentation on the first feature of the image to be segmented, and compares the similarity between the first feature and the third foreground prototype vector and the first feature and the third background prototype vector, and the one with the greater similarity is taken as the segmentation result. Since the first feature has only two options, the foreground feature and the background feature, segmentation is performed based on the similarity, which can improve the accuracy of segmentation.

[0162] In a possible implementation manner, a method of obtaining the second prototype vector of the marked image includes:

[0163] Acquire a second feature of the marked image and a mask corresponding to the marked image;

[0164] A second prototype vector of the marked image is determined according to the second feature and the mask.

[0165] In the disclosed embodiment, the second feature of the marked image can be obtained by an image feature extraction algorithm, such as a histogram of oriented gradients feature, a local binary pattern feature, a Haar-like feature, etc.; it can also be extracted by a convolutional neural network. The mask corresponding to the marked image can be obtained by the marked image. Since the marked image is a label of the marked object, in one example, the area within the marked outline can be set to 1, and the marked image area other than the marked object can be set to 0 to obtain a foreground mask corresponding to the marked image. In one example, the area outside the marked outline in the marked image can also be set to 1, and the area within the marked outline can be set to 0 to obtain a background mask corresponding to the marked image.

[0166] In the disclosed embodiment, the second prototype vector may include a foreground second prototype vector or a background second prototype vector. In one example, the foreground mask corresponding to the marked image may be multiplied by the second feature of the marked image to obtain the second feature corresponding to the marked object, and the second feature corresponding to the marked object may be globally averaged to obtain the foreground second prototype vector. In one example, the background mask corresponding to the marked image may be multiplied by the second feature of the marked image to obtain the second feature corresponding to the area other than the marked object in the marked image, and the second feature corresponding to the area other than the marked object in the marked image may be globally averaged to obtain the background second prototype vector.

[0167] In the disclosed embodiment, the first feature of the image to be segmented can be segmented using the foreground second prototype vector and the first prototype vector to obtain the segmentation result of the image to be segmented. The first feature of the image to be segmented can also be segmented using the background second prototype vector and the first prototype vector to obtain the segmentation result of the image to be segmented. By marking the second feature of the image and the corresponding mask, the second prototype vector of the marked image can be easily and accurately obtained.

[0168] In a possible implementation, the mask includes a foreground mask and a background mask, and the second prototype vector includes a foreground second prototype vector and a background second prototype vector; the method for obtaining the second prototype vector of the marked image includes:

[0169] determining, according to the second feature of the marked image and the foreground mask corresponding to the marked image, a second feature corresponding to the marked object in the marked image;

[0170] Determining, according to the second feature of the marked image and the background mask corresponding to the marked image, the second feature corresponding to the area in the marked image excluding the marked object;

[0171] Performing global averaging processing on the second feature corresponding to the marked object to obtain a foreground second prototype vector;

[0172] A global averaging process is performed on the second feature corresponding to the area other than the marked object to obtain a background second prototype vector.

[0173] In the disclosed embodiment, the foreground mask and the background mask corresponding to the marked image can be obtained in the same manner as in the above-mentioned embodiment. Among them, the background mask can be obtained by negating the foreground mask, and the foreground mask can also be obtained by negating the background mask. Different from the above-mentioned embodiment, in the disclosed embodiment, the foreground second prototype vector, the background second prototype vector and the first prototype vector can be used simultaneously to perform segmentation processing on the first feature of the image to be segmented. For example, the foreground first prototype vector and the foreground second prototype vector are added to obtain the foreground third prototype vector, and the background first prototype vector and the background second prototype vector are added to obtain the background third prototype vector. Calculate the similarity between the first feature of the image to be segmented and the foreground third prototype vector, as well as the similarity between the first feature of the image to be segmented and the background third prototype vector, compare the two similarities, and take the type with the higher similarity as the segmentation result.

[0174] The disclosed embodiment utilizes the foreground second prototype vector and the background second prototype vector simultaneously. Compared with using only one of the second prototype vectors to calculate the similarity and the threshold value to determine the segmentation result, the setting of the threshold can be avoided, and the segmentation results caused by inaccurate threshold setting are reduced. Therefore, the disclosed embodiment can further improve the accuracy of image segmentation.

[0175] In a possible implementation, the initial segmentation result is obtained by performing segmentation processing on the first feature according to the second prototype vector of the marked image, including:

[0176] Calculating the similarity between the second prototype vector and the first feature;

[0177] An initial segmentation result of the sample image to be segmented is determined according to the similarity between the second prototype vector and the first feature.

[0178] In the disclosed embodiment, the similarity is used to compare whether two features belong to the same category, and the method of representing the similarity may be, for example, cosine distance or Euclidean distance. In one example, when the second prototype vector is a foreground second prototype vector, the similarity between the second prototype vector and the first feature is calculated, and if the similarity is greater than a preset value, the first feature is a foreground feature of the image to be segmented; otherwise, if the similarity is less than or equal to the preset value, the first feature is a background feature of the image to be segmented. In another example, when the second prototype vector is a background second prototype vector, the similarity between the second prototype vector and the first feature is calculated, and if the similarity is greater than a preset value, the first feature is a background feature of the image to be segmented; otherwise, if the similarity is less than or equal to the preset value, the first feature is a foreground feature of the image to be segmented.

[0179] By using the method of the embodiment of the present disclosure, an initial segmentation result of a sample image to be segmented can be obtained conveniently and accurately.

[0180] In a possible implementation, the segmentation result is obtained by an image segmentation model, and the training method of the image segmentation model includes:

[0181] Acquire a sample image set, wherein the sample image set includes a sample labeled image and a sample to-be-segmented image with an object to be segmented;

[0182] Determine a sample second prototype vector according to the sample second feature of the sample labeled image and the mask corresponding to the sample labeled image;

[0183] According to the sample second prototype vector, segmenting the sample first feature of the sample image to be segmented, to obtain an initial segmentation result of the sample image to be segmented;

[0184] Determine a first prototype vector of the sample according to the first feature and the initial segmentation result;

[0185] According to the sample second prototype vector and the sample first prototype vector, segmenting the sample first feature to obtain a segmentation result of the sample image to be segmented;

[0186] Based on the difference between the segmentation result and the marked object to be segmented, the training parameters of the image segmentation model are iteratively adjusted until the difference meets the preset requirement.

[0187] In the disclosed embodiment, the sample labeled image to be segmented may include multiple categories of objects to be segmented, and for each category of objects to be segmented, there may be a corresponding labeled object in the sample labeled image. In one example, a twin network may be used to extract the second feature of the sample labeled image and the sample first feature of the sample image to be segmented. The twin network may include a ResNet-50 network, a ResNet101 network, a ResNet200 network, a ResNeXt101 network, a ResNeSt101 network, or a faster ResNet18 network, a MobileNet network, a SqueezeNet network, etc. During training, a sample labeled image and a sample image to be segmented may be selected by random sampling each time, and these two images may be input into the segmentation model for training. The segmentation model will process the sample image to be segmented and the sample labeled image as follows: determine the sample second prototype vector according to the second feature of the sample labeled image and the mask corresponding to the sample labeled image; segment the sample first feature according to the sample second prototype vector to obtain the initial segmentation result of the sample image to be segmented; determine the first prototype vector according to the sample first feature and the initial segmentation result; segment the sample first feature according to the second prototype vector and the first prototype vector, and finally output the segmentation result of the sample image to be segmented. At this time, based on the difference between the segmentation result and the labeled object to be segmented, the training parameters of the image segmentation model are iteratively adjusted until the difference meets the preset requirements. In an example, the gradient descent method of SGD (Stochastic Gradient Descent) can also be used to train the network. The initial learning rate of the network is 0.002, the entire network is trained for 6000 rounds, and the learning rate is reduced by ten times at the 2000th and 4000th rounds respectively.

[0188] In the disclosed embodiment, an image segmentation model is used to obtain an image segmentation result. During the training process of the image segmentation model, a self-supporting sample first prototype vector is generated according to the initial segmentation result of the sample image to be segmented. Compared with the sample second prototype vector generated by using the traditional mask based on the sample labeled image alone. Since the sample first prototype vector is derived from the image to be segmented itself, it is closer to the sample first feature of the image to be segmented, so that the trained model can still accurately segment the image when the difference between the labeled image and the image to be segmented is relatively large.

[0189] Figure 4 FIG. 1 is a flowchart of an image segmentation model training method according to an exemplary embodiment. Figure 4 As shown, the method includes:

[0190] Step S401 : obtaining a sample image set, wherein the sample image set includes a sample labeled image and a sample image to be segmented which is labeled with an object to be segmented.

[0191] In the embodiment of the present disclosure, the sample labeled image to be segmented may include multiple categories of objects to be segmented, and for each category of objects to be segmented, there may be a corresponding labeled object in the sample labeled image. In the sample image to be segmented, the number of objects to be segmented may include one or more.

[0192] Step S403: determining a sample second prototype vector according to the sample second feature of the sample labeled image and the mask corresponding to the sample labeled image.

[0193] In the disclosed embodiment, a convolutional neural network can be used to extract features of an image, for example, feature extraction is performed on a sample image to be segmented to obtain a first feature of the sample, and feature extraction is performed on a sample marked image to obtain a second feature of the sample. The convolutional neural network may include: a ResNet-50 network, a ResNet101 network, a ResNet200 network, a ResNeXt101 network, a ResNeSt101 network, or a faster ResNet18 network, a MobileNet network, a SqueezeNet network, etc. In the disclosed embodiment, a global average processing is performed on the second feature corresponding to the marked object to obtain a sample foreground second prototype vector. A global average processing is performed on the sample second feature of the background image in the marked image to obtain a sample background second prototype vector.

[0194] Step S405 : performing segmentation processing on the first feature according to the sample second prototype vector to obtain an initial segmentation result of the sample image to be segmented.

[0195] In the disclosed embodiment, when the sample second prototype vector is a sample foreground second prototype vector, the similarity between the sample second prototype vector and the sample first feature is calculated, and if the similarity is greater than a preset value, the sample first feature is a foreground feature of the image to be segmented; otherwise, if the similarity is less than or equal to the preset value, the sample first feature is a background feature of the image to be segmented. In another example, when the sample second prototype vector is a sample background second prototype vector, the similarity between the sample second prototype vector and the sample first feature is calculated, and if the similarity is greater than a preset value, the sample first feature is a background feature of the sample image to be segmented; otherwise, if the similarity is less than or equal to the preset value, the sample first feature is a foreground feature of the sample image to be segmented.

[0196] Step S407, determining a first prototype vector of the sample according to the first feature of the sample image to be segmented and the initial segmentation result.

[0197] In the disclosed embodiment, the sample first prototype vector may include a sample foreground first prototype vector and / or a sample background first prototype vector. The initial segmentation result may include an initial foreground segmentation result and / or an initial background segmentation result. In one example, an initial foreground mask of the sample image to be segmented may be obtained from the initial foreground segmentation result, and the initial foreground mask is multiplied with the sample first feature of the sample image to be segmented to obtain the sample first feature corresponding to the sample object to be segmented, and the first feature of the sample object to be segmented is globally averaged to obtain the sample foreground first prototype vector.

[0198] Step S409: performing segmentation processing on the sample first feature according to the sample second prototype vector and the sample first prototype vector to obtain a segmentation result of the sample image to be segmented.

[0199] In the disclosed embodiments, in one example, a sample third prototype vector can be determined based on the sample second prototype vector and the sample first prototype vector; the similarity between the sample third prototype vector and the sample first feature is calculated; and the segmentation result of the image to be segmented is determined based on the similarity between the sample third prototype vector and the sample first feature. In another example, a sample foreground third prototype vector can be determined based on the sample foreground second prototype vector and the sample foreground first prototype vector; a sample background third prototype vector can be determined based on the sample background second prototype vector and the sample background first prototype vector; the similarity between the sample foreground third prototype vector and the sample first feature is calculated to obtain a first similarity; the similarity between the sample background third prototype vector and the sample first feature is calculated to obtain a second similarity; and the segmentation result of the sample image to be segmented is determined based on the first similarity and the second similarity.

[0200] Step S411: Based on the difference between the segmentation result and the marked object to be segmented, iteratively adjust the training parameters of the image segmentation model until the difference meets the preset requirement.

[0201] In the disclosed embodiment, the network can be trained using the SGD (Stochastic Gradient Descent) gradient descent method, the initial learning rate of the network is 0.002, the entire network is trained for 6000 rounds, and the learning rate is reduced tenfold at the 2000th and 4000th rounds respectively.

[0202] In the embodiment of the present disclosure, during the training process of the image segmentation model, a self-supporting first prototype vector is generated according to the initial segmentation result of the sample image to be segmented. Compared with the second prototype vector generated by using the traditional mask based on the sample labeled image alone, since the first prototype vector is derived from the image to be segmented itself, it is closer to the first feature of the image to be segmented, so that the trained model can still accurately segment the image when the difference between the labeled image and the image to be segmented is relatively large.

[0203] In a possible implementation, the training step in the embodiment of the present disclosure can be expanded by using any of the above-mentioned image segmentation methods. Under the inspiration of the technical essence of the present application, technicians in the relevant field may also make other changes, but as long as the functions and effects achieved are the same or similar to those of the present application, they should be included in the protection scope of the present application. A detailed explanation will not be given here.

[0204] It should be understood that, although the various steps in the flow chart are shown in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear description in this article, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a part of the steps in the figure may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0205] It can be understood that the same / similar parts between the various embodiments of the above method in this specification can refer to each other, and each embodiment focuses on the differences from other embodiments. For related points, please refer to the description of other method embodiments.

[0206] Figure 5 is a block diagram of an image segmentation device according to an exemplary embodiment. Figure 5 , the device 500 comprises:

[0207] An acquisition module 501 is used to acquire a first feature of the image to be segmented and an initial segmentation result, wherein the initial segmentation result is obtained by segmenting the first feature according to the second prototype vector of the marked image, and the marked object in the marked image and the object to be segmented in the image to be segmented belong to the same category;

[0208] A first determining module 503, configured to determine a first prototype vector of the image to be segmented according to the first feature and the initial segmentation result;

[0209] The segmentation module 505 is used to perform segmentation processing on the first feature according to the second prototype vector and the first prototype vector to obtain a segmentation result of the image to be segmented.

[0210] In a possible implementation, the initial segmentation result includes an initial foreground segmentation result and an initial background segmentation result, the first prototype vector includes a foreground first prototype vector and a background first prototype vector; and the first determining module includes:

[0211] A first determination submodule is used to determine a first feature of the object to be segmented according to the first feature and an initial foreground segmentation result, wherein the first feature of the object to be segmented is a feature belonging to the object to be segmented in the first feature of the image to be segmented;

[0212] A second determination submodule is used to determine a first feature of a background image in the image to be segmented according to the first feature and an initial background segmentation result, wherein the background image of the image to be segmented is an image in the image to be segmented excluding the object to be segmented;

[0213] A first processing submodule is used to perform global averaging processing on the first feature of the object to be segmented to obtain a first foreground prototype vector;

[0214] The second processing submodule is used to perform global averaging processing on the first feature of the background image in the image to be segmented to obtain a first background prototype vector.

[0215] In a possible implementation, the first processing submodule includes:

[0216] A first acquisition unit, used to acquire a prediction confidence of a first feature of the object to be segmented;

[0217] A first processing unit is used to perform global averaging processing on the first feature of the object to be segmented whose prediction confidence is greater than a preset threshold, so as to obtain a foreground first prototype vector;

[0218] The second processing submodule includes:

[0219] A second acquisition unit, used to acquire a prediction confidence of a first feature of a background image in the image to be segmented;

[0220] The second processing unit is used to perform global averaging processing on the first feature of the background image in the image to be segmented whose prediction confidence is greater than a preset threshold, so as to obtain a background first prototype vector.

[0221] In a possible implementation, the segmentation module includes:

[0222] A third determining submodule, configured to determine a third prototype vector according to the second prototype vector and the first prototype vector;

[0223] A first calculation submodule, used for calculating the similarity between the third prototype vector and the first feature;

[0224] The fourth determination submodule is used to determine the segmentation result of the image to be segmented according to the similarity between the third prototype vector and the first feature.

[0225] In a possible implementation, the second prototype vector includes a foreground second prototype vector and a background second prototype vector, the first prototype vector includes a foreground first prototype vector and a background first prototype vector, and the segmentation module includes:

[0226] a fifth determining submodule, configured to determine a foreground third prototype vector according to the foreground second prototype vector and the foreground first prototype vector;

[0227] A sixth determining submodule, configured to determine a background third prototype vector according to the background second prototype vector and the background first prototype vector;

[0228] A second calculation submodule is used to calculate the similarity between the foreground third prototype vector and the first feature to obtain a first similarity;

[0229] A third calculation submodule is used to calculate the similarity between the background third prototype vector and the first feature to obtain a second similarity;

[0230] The seventh determination submodule is used to determine a segmentation result of the image to be segmented according to the first similarity and the second similarity.

[0231] In a possible implementation, the method further includes:

[0232] A second acquisition module, used to acquire a second feature of the marked image and a mask corresponding to the marked image;

[0233] The second determining module is used to determine a second prototype vector of the marked image according to the second feature and the mask.

[0234] In a possible implementation, the mask includes a foreground mask and a background mask, and the second prototype vector includes a foreground second prototype vector and a background second prototype vector; and the device further includes:

[0235] A third determining module is used to determine the second feature of the marked object according to the second feature of the marked image and the foreground mask corresponding to the marked image, wherein the second feature of the marked object is a feature of the second feature of the marked image belonging to the marked object;

[0236] a fourth determining module, configured to determine a second feature of a background image in the marked image according to the second feature of the marked image and a background mask corresponding to the marked image, wherein the background image of the marked image is an image in the marked image excluding the marked object;

[0237] A first processing module, configured to perform global averaging processing on the second feature of the marked object to obtain a foreground second prototype vector;

[0238] The second processing module is used to perform global averaging processing on the second feature of the background image in the marked image to obtain a background second prototype vector.

[0239] In a possible implementation, the method further includes:

[0240] A calculation module, used for calculating the similarity between the second prototype vector and the first feature;

[0241] The fifth determination module is used to determine the initial segmentation result of the sample image to be segmented according to the similarity between the second prototype vector and the first feature.

[0242] In a possible implementation, the method further includes:

[0243] A third acquisition module is used to acquire a sample image set, wherein the sample image set includes a sample marked image and a sample to-be-segmented image with an object to be segmented;

[0244] a sixth determination module, configured to determine a sample second prototype vector according to the sample second feature of the sample labeled image and a mask corresponding to the sample labeled image;

[0245] A second segmentation module is used to perform segmentation processing on the sample first feature of the sample image to be segmented according to the sample second prototype vector to obtain an initial segmentation result of the sample image to be segmented;

[0246] a seventh determination module, configured to determine a first prototype vector of the sample according to the first feature and the initial segmentation result;

[0247] A third segmentation module is used to perform segmentation processing on the first feature of the sample according to the second prototype vector of the sample and the first prototype vector of the sample to obtain a segmentation result of the sample image to be segmented;

[0248] The adjustment module is used to iteratively adjust the training parameters of the image segmentation model based on the difference between the segmentation result and the marked object to be segmented, until the difference meets the preset requirements.

[0249] According to a sixth aspect of an embodiment of the present disclosure, a training device for an image segmentation model is provided, comprising:

[0250] A third acquisition module 601 is used to acquire a sample image set, where the sample image set includes a sample marked image and a sample to-be-segmented image with an object to be segmented;

[0251] A sixth determination module 603, configured to determine a sample second prototype vector according to the sample second feature of the sample labeled image and a mask corresponding to the sample labeled image;

[0252] The second segmentation module 605 is used to perform segmentation processing on the sample first feature of the sample image to be segmented according to the sample second prototype vector to obtain an initial segmentation result of the sample image to be segmented;

[0253] A seventh determination module 607, configured to determine a first prototype vector of a sample according to the first feature and the initial segmentation result;

[0254] A third segmentation module 609 is used to perform segmentation processing on the first feature of the sample according to the second prototype vector of the sample and the first prototype vector of the sample to obtain a segmentation result of the sample image to be segmented;

[0255] The adjustment module 611 is used to iteratively adjust the training parameters of the image segmentation model based on the difference between the segmentation result and the marked object to be segmented, until the difference meets the preset requirement.

[0256] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0257] Figure 7 is a block diagram of a server for an image segmentation method according to an exemplary embodiment. Figure 7 , the electronic device 700 includes a processing component 720, which further includes one or more processors, and a memory resource represented by a memory 722 for storing instructions that can be executed by the processing component 720, such as an application. The application stored in the memory 722 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 720 is configured to execute the instructions to perform the above method.

[0258] The electronic device 700 may further include a power supply component 724 configured to perform power management of the electronic device 700, a wired or wireless network interface 726 configured to connect the electronic device 700 to a network, and an input / output (I / O) interface 728. The electronic device 700 may operate based on an operating system stored in the memory 722, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, or the like.

[0259] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 722 including instructions, and the instructions can be executed by a processor of the electronic device 700 to perform the above method. The storage medium can be a computer-readable storage medium, for example, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0260] In an exemplary embodiment, a computer program product is further provided. The computer program product includes instructions. The instructions can be executed by a processor of the electronic device 700 to complete the above method.

[0261] It should be noted that the above-mentioned devices, electronic devices, computer-readable storage media, computer program products, etc. may also include other implementation methods according to the description of the method embodiments. The specific implementation methods can refer to the description of the relevant method embodiments, which will not be described one by one here.

[0262] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0263] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image segmentation method, characterized in that: include: Acquire a first feature and an initial segmentation result of the image to be segmented, wherein the initial segmentation result is obtained by segmenting the first feature according to the second prototype vector of the marked image, and the marked object in the marked image and the object to be segmented in the image to be segmented belong to the same category; Determine the first prototype vector of the image to be segmented according to the first feature and the initial segmentation result; the determining the first prototype vector of the image to be segmented according to the first feature and the initial segmentation result includes: determining the first feature of the object to be segmented according to the first feature and the initial foreground segmentation result, the first feature of the object to be segmented being the feature of the first feature of the image to be segmented belonging to the object to be segmented; determining the first feature of the background image in the image to be segmented according to the first feature and the initial background segmentation result, the background image of the image to be segmented being the image other than the object to be segmented in the image to be segmented; performing global averaging processing on the first feature of the object to be segmented to obtain a foreground first prototype vector; performing global averaging processing on the first feature of the background image in the image to be segmented to obtain a background first prototype vector; wherein the initial segmentation result includes an initial foreground segmentation result and an initial background segmentation result, and the first prototype vector includes a foreground first prototype vector and a background first prototype vector; The first feature is segmented according to the second prototype vector and the first prototype vector to obtain a segmentation result of the image to be segmented.

2. The method according to claim 1, characterized in that Performing global averaging processing on the first feature of the object to be segmented in the image to be segmented to obtain a foreground first prototype vector, including: Obtaining a prediction confidence of a first feature of the object to be segmented; Performing global averaging processing on the first feature of the object to be segmented whose prediction confidence is greater than a preset threshold to obtain a foreground first prototype vector; The step of performing global averaging processing on the first feature of the background image in the image to be segmented to obtain a first background prototype vector includes: Obtaining a prediction confidence of a first feature of a background image in the image to be segmented; A global averaging process is performed on the first feature of the background image in the image to be segmented whose prediction confidence is greater than a preset threshold to obtain a background first prototype vector.

3. The method according to claim 1, characterized in that: The first feature is segmented according to the second prototype vector and the first prototype vector to obtain a segmentation result of the image to be segmented, including: Determine a third prototype vector according to the second prototype vector and the first prototype vector; Calculating the similarity between the third prototype vector and the first feature; A segmentation result of the image to be segmented is determined according to the similarity between the third prototype vector and the first feature.

4. The method according to claim 1, characterized in that: The second prototype vector includes a foreground second prototype vector and a background second prototype vector, the first prototype vector includes a foreground first prototype vector and a background first prototype vector, and the segmentation processing is performed on the first feature according to the second prototype vector and the first prototype vector to obtain the segmentation result of the image to be segmented, including: Determine a foreground third prototype vector according to the foreground second prototype vector and the foreground first prototype vector; Determine a background third prototype vector according to the background second prototype vector and the background first prototype vector; Calculating the similarity between the foreground third prototype vector and the first feature to obtain a first similarity; Calculating the similarity between the background third prototype vector and the first feature to obtain a second similarity; A segmentation result of the image to be segmented is determined according to the first similarity and the second similarity.

5. The method according to claim 1, characterized in that The method of obtaining the second prototype vector of the marked image includes: Acquire a second feature of the marked image and a mask corresponding to the marked image; A second prototype vector of the marked image is determined according to the second feature and the mask.

6. The method according to claim 1, characterized in that The second prototype vector includes a foreground second prototype vector and a background second prototype vector; the method for obtaining the second prototype vector of the marked image includes: Determining the second feature of the marked object according to the second feature of the marked image and the foreground mask corresponding to the marked image, wherein the second feature of the marked object is a feature belonging to the marked object in the second feature of the marked image; Determine, according to the second feature of the marked image and the background mask corresponding to the marked image, the second feature of the background image in the marked image, wherein the background image of the marked image is an image in the marked image excluding the marked object; Performing global averaging processing on the second feature of the marked object to obtain a foreground second prototype vector; A global averaging process is performed on the second feature of the background image in the marked image to obtain a background second prototype vector.

7. The method according to claim 1, characterized in that The initial segmentation result is obtained by segmenting the first feature according to the second prototype vector of the marked image, including: Calculating the similarity between the second prototype vector and the first feature; An initial segmentation result of the image to be segmented is determined according to the similarity between the second prototype vector and the first feature.

8. The method according to claim 1, characterized in that The segmentation result is obtained through an image segmentation model, and the training method of the image segmentation model includes: Acquire a sample image set, wherein the sample image set includes a sample labeled image and a sample to-be-segmented image with an object to be segmented; Determine a sample second prototype vector according to the sample second feature of the sample labeled image and the mask corresponding to the sample labeled image; According to the sample second prototype vector, segmenting the sample first feature of the sample image to be segmented, to obtain an initial segmentation result of the sample image to be segmented; Determine a first prototype vector of the sample according to the first feature and the initial segmentation result; According to the sample second prototype vector and the sample first prototype vector, segmenting the sample first feature to obtain a segmentation result of the sample image to be segmented; Based on the difference between the segmentation result and the marked object to be segmented, the training parameters of the image segmentation model are iteratively adjusted until the difference meets the preset requirement.

9. A method for training an image segmentation model, comprising: Acquire a sample image set, wherein the sample image set includes a sample labeled image and a sample to-be-segmented image with an object to be segmented; Determine a sample second prototype vector according to a sample second feature of the sample labeled image and a mask corresponding to the sample labeled image; According to the sample second prototype vector, segmenting the sample first feature of the sample image to be segmented, to obtain an initial segmentation result of the sample image to be segmented; Determining a sample first prototype vector of the sample image to be segmented according to the sample first feature and the initial segmentation result; Determining the sample first prototype vector of the sample image to be segmented according to the sample first feature and the initial segmentation result includes: determining the first feature of the object to be segmented according to the sample first feature and the initial foreground segmentation result, wherein the first feature of the object to be segmented is a feature of the object to be segmented in the first feature of the image to be segmented; determining the first feature of the background image in the image to be segmented according to the first feature and the initial background segmentation result, wherein the background image of the image to be segmented is an image other than the object to be segmented in the image to be segmented; performing global averaging processing on the first feature of the object to be segmented to obtain a sample foreground first prototype vector; performing global averaging processing on the first feature of the background image in the image to be segmented to obtain a sample background first prototype vector; wherein the initial segmentation result includes an initial foreground segmentation result and an initial background segmentation result, and the sample first prototype vector includes a sample foreground first prototype vector and a sample background first prototype vector; According to the sample second prototype vector and the sample first prototype vector, segmenting the sample first feature to obtain a segmentation result of the sample image to be segmented; Based on the difference between the segmentation result and the marked object to be segmented, the training parameters of the image segmentation model are iteratively adjusted until the difference meets the preset requirement.

10. An image segmentation device, characterized in that: include: A first acquisition module is used to acquire a first feature of the image to be segmented and an initial segmentation result, wherein the initial segmentation result is obtained by segmenting the first feature according to the second prototype vector of the marked image, and the marked object in the marked image and the object to be segmented in the image to be segmented belong to the same category; A first determination module is used to determine the first prototype vector of the image to be segmented according to the first feature and the initial segmentation result; wherein the initial segmentation result includes an initial foreground segmentation result and an initial background segmentation result, and the first prototype vector includes a foreground first prototype vector and a background first prototype vector; the first determination module includes: a first determination submodule is used to determine the first feature of the object to be segmented according to the first feature and the initial foreground segmentation result, the first feature of the object to be segmented is the feature of the first feature of the image to be segmented belonging to the object to be segmented; a second determination submodule is used to determine the first feature of the background image in the image to be segmented according to the first feature and the initial background segmentation result, the background image of the image to be segmented is the image in the image to be segmented other than the object to be segmented; a first processing submodule is used to perform global averaging processing on the first feature of the object to be segmented to obtain a foreground first prototype vector; a second processing submodule is used to perform global averaging processing on the first feature of the background image in the image to be segmented to obtain a background first prototype vector; The first segmentation module is used to perform segmentation processing on the first feature according to the second prototype vector and the first prototype vector to obtain a segmentation result of the image to be segmented.

11. The device according to claim 10, characterized in that The first processing submodule includes: A first acquisition unit, used to acquire a prediction confidence of a first feature of the object to be segmented; A first processing unit is used to perform global averaging processing on the first feature of the object to be segmented whose prediction confidence is greater than a preset threshold, so as to obtain a foreground first prototype vector; The second processing submodule includes: A second acquisition unit, used to acquire a prediction confidence of a first feature of a background image in the image to be segmented; The second processing unit is used to perform global averaging processing on the first feature of the background image in the image to be segmented whose prediction confidence is greater than a preset threshold, so as to obtain a background first prototype vector.

12. The device according to claim 10, characterized in that The segmentation module comprises: A third determining submodule, configured to determine a third prototype vector according to the second prototype vector and the first prototype vector; A first calculation submodule, used for calculating the similarity between the third prototype vector and the first feature; The fourth determination submodule is used to determine the segmentation result of the image to be segmented according to the similarity between the third prototype vector and the first feature.

13. The device according to claim 10, characterized in that The second prototype vector includes a foreground second prototype vector and a background second prototype vector, the first prototype vector includes a foreground first prototype vector and a background first prototype vector, and the segmentation module includes: a fifth determining submodule, configured to determine a foreground third prototype vector according to the foreground second prototype vector and the foreground first prototype vector; A sixth determining submodule, configured to determine a background third prototype vector according to the background second prototype vector and the background first prototype vector; A second calculation submodule is used to calculate the similarity between the foreground third prototype vector and the first feature to obtain a first similarity; A third calculation submodule is used to calculate the similarity between the background third prototype vector and the first feature to obtain a second similarity; The seventh determination submodule is used to determine a segmentation result of the image to be segmented according to the first similarity and the second similarity.

14. The device according to claim 10, characterized in that Also includes: A second acquisition module, used to acquire a second feature of the marked image and a mask corresponding to the marked image; The second determining module is used to determine a second prototype vector of the marked image according to the second feature and the mask.

15. The device according to claim 10, characterized in that The second prototype vector includes a foreground second prototype vector and a background second prototype vector; the device also includes: A third determining module is used to determine the second feature of the marked object according to the second feature of the marked image and the foreground mask corresponding to the marked image, wherein the second feature of the marked object is a feature belonging to the marked object in the second feature of the marked image; a fourth determining module, configured to determine a second feature of a background image in the marked image according to the second feature of the marked image and a background mask corresponding to the marked image, wherein the background image of the marked image is an image in the marked image excluding the marked object; A first processing module, configured to perform global averaging processing on the second feature of the marked object to obtain a foreground second prototype vector; The second processing module is used to perform global averaging processing on the second feature of the background image in the marked image to obtain a background second prototype vector.

16. The device according to claim 10, characterized in that Also includes: A calculation module, used for calculating the similarity between the second prototype vector and the first feature; The fifth determining module is used to determine the initial segmentation result of the image to be segmented according to the similarity between the second prototype vector and the first feature.

17. The device according to claim 10, characterized in that Also includes: A third acquisition module is used to acquire a sample image set, wherein the sample image set includes a sample marked image and a sample to-be-segmented image with an object to be segmented; a sixth determination module, configured to determine a sample second prototype vector according to the sample second feature of the sample labeled image and a mask corresponding to the sample labeled image; A second segmentation module is used to perform segmentation processing on the sample first feature of the sample image to be segmented according to the sample second prototype vector to obtain an initial segmentation result of the sample image to be segmented; a seventh determination module, configured to determine a first prototype vector of the sample according to the first feature and the initial segmentation result; A third segmentation module is used to perform segmentation processing on the first feature of the sample according to the second prototype vector of the sample and the first prototype vector of the sample to obtain a segmentation result of the sample image to be segmented; The adjustment module is used to iteratively adjust the training parameters of the image segmentation model based on the difference between the segmentation result and the marked object to be segmented, until the difference meets the preset requirements.

18. A training device for an image segmentation model, characterized in that: include: A third acquisition module is used to acquire a sample image set, wherein the sample image set includes a sample marked image and a sample to-be-segmented image with an object to be segmented; a sixth determination module, configured to determine a sample second prototype vector according to the sample second feature of the sample labeled image and a mask corresponding to the sample labeled image; A second segmentation module is used to perform segmentation processing on the sample first feature of the sample image to be segmented according to the sample second prototype vector to obtain an initial segmentation result of the sample image to be segmented; The seventh determination module is used to determine the sample first prototype vector according to the first feature and the initial segmentation result; the seventh determination module is also used to determine the first feature of the object to be segmented according to the sample first feature and the initial foreground segmentation result, the first feature of the object to be segmented is the feature of the first feature of the image to be segmented belonging to the object to be segmented; determine the first feature of the background image in the image to be segmented according to the first feature and the initial background segmentation result, the background image of the image to be segmented is the image in the image to be segmented other than the object to be segmented; perform global averaging processing on the first feature of the object to be segmented to obtain the sample foreground first prototype vector; perform global averaging processing on the first feature of the background image in the image to be segmented to obtain the sample background first prototype vector; wherein the initial segmentation result includes the initial foreground segmentation result and the initial background segmentation result, and the sample first prototype vector includes the sample foreground first prototype vector and the sample background first prototype vector; A third segmentation module is used to perform segmentation processing on the first feature of the sample according to the second prototype vector of the sample and the first prototype vector of the sample to obtain a segmentation result of the sample image to be segmented; The adjustment module is used to iteratively adjust the training parameters of the image segmentation model based on the difference between the segmentation result and the marked object to be segmented, until the difference meets the preset requirements.

19. A server, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the image segmentation method as described in any one of claims 1 to 8 or the image segmentation model training method as described in claim 9.

20. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of a server, the server is enabled to execute the image segmentation method according to any one of claims 1 to 8 or the image segmentation model training method according to claim 9.

21. A computer program product, comprising instructions, characterized in that: When the instruction is executed by the processor of the server, the server is enabled to execute the image segmentation method according to any one of claims 1 to 8 or the image segmentation model training method according to claim 9.

Images