Training method and device of image segmentation network, equipment and readable storage medium

Abstract

The present disclosure discloses a method, device and equipment for training an image segmentation network and a readable storage medium, the method comprising: inputting a sample image into a first image segmentation network to obtain feature information of the sample image, wherein the first sample image has annotation information; using the first image segmentation network to obtain content perception information of the sample image according to the feature information; obtaining an image segmentation result of the sample image according to the feature information and the content perception information; training the first image segmentation network according to the image segmentation result, the content perception information and the annotation information, and taking the trained first image segmentation network as a second image segmentation network. By learning the relevant information of the context of the sample image, the first image segmentation network can reduce the misclassification of foreground and background when used for image segmentation, and generate an image segmentation result with high precision.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of image processing, and particularly relates to a training method and device of an image segmentation network, an equipment and a readable storage medium. BACKGROUND

[0002] Salient object detection technology, that is, segmenting the most visually attractive foreground object in an image. This technology is widely used in many fields, such as lane detection for autonomous driving, application programs for changing backgrounds, and the like. At present, fully convolutional networks (FCN) have achieved great success in salient object detection, which can segment salient foreground objects without setting excessive experience parameters. However, since the fully convolutional network does not consider the scale information, the segmentation of the foreground is poor in fineness, and cannot meet the requirement for segmentation of fine granularity. In order to improve the fineness of segmentation, a network capable of extracting deep semantic information is used as a network backbone, such as Alexnet, VGG, ResNet, ResNeXt, DenseNet, and the like. However, these backbone networks do not consider the difference between the salient foreground and the background, and the foreground and the background may still be misclassified. SUMMARY

[0003] In view of this, the embodiments of the present disclosure at least provide a training method, device, equipment and readable storage medium of an image segmentation network.

[0004] According to a first aspect of the embodiments of the present disclosure, a training method of an image segmentation network is provided, and the method comprises:

[0005] inputting a sample image into a first image segmentation network to obtain feature information of the sample image, wherein the first sample image has label information;

[0006] obtaining content perception information of the sample image according to the feature information based on the first image segmentation network, the content perception information indicating the correlation between a first pixel and pixels other than the first pixel in the sample image, the first pixel being any pixel in the sample image;

[0007] obtaining an image segmentation result of the sample image according to the feature information and the content perception information;

[0008] training the first image segmentation network according to the image segmentation result, the content perception information and the label information, and taking the trained first image segmentation network as a second image segmentation network.

[0009] In some embodiments, the training of the first image segmentation network according to the segmentation result, the content-aware information, and the annotation information comprises:

[0010] adjusting network parameters of the first image segmentation network according to a difference between the image segmentation result and a real segmentation result indicated by the annotation information, and a difference between the content-aware information and real content-aware information, wherein the real content-aware information is obtained according to the annotation information.

[0011] In some embodiments, the method further comprises:

[0012] scaling the real segmentation result indicated by the annotation information, and the adjusted result has a same size as the content-aware information;

[0013] heat-encoding the adjusted result to obtain an encoded result;

[0014] using a cross-correlation matrix obtained by multiplying the encoded result and a transpose matrix of the encoded result as the real content-aware information.

[0015] In some embodiments, the obtaining of the content-aware information of the sample image according to the feature information comprises:

[0016] performing convolution operation on the feature information to obtain a convolution result, wherein the feature information is obtained by a backbone network in the first image segmentation network;

[0017] performing dimension reconstruction on the convolution result to obtain the content-aware information.

[0018] In some embodiments, the obtaining of the image segmentation result of the sample image according to the feature information and the content-aware information comprises:

[0019] performing activation operation on the content-aware information to obtain a first activation result, and a value of the first activation result is between 0 and 1;

[0020] using a difference between 1 and the first activation result as a second activation result;

[0021] multiplying the convolution result with the first activation result and the second activation result respectively to obtain a first multiplication result and a second multiplication result;

[0022] connecting the first multiplication result, the second multiplication result, and a feature output by a last convolution layer in the backbone network to obtain a merged result;

[0023] performing convolution operation and activation operation on the merged result to obtain the image segmentation result.

[0024] In some embodiments, the first image segmentation network is trained according to the image segmentation result, the content-aware information, and the annotation information, including:

[0025] The first image segmentation network is trained based on a first loss function, and the first loss function includes:

[0026] a first loss, used to indicate a cross entropy between the image segmentation result and a true segmentation result;

[0027] a second loss, used to indicate a cross entropy between the content-aware information and a true-aware information.

[0028] In some embodiments, the method further includes training the second image segmentation network based on a second loss function, and the second loss function includes the first loss, the second loss, and a third loss, wherein the third loss is used to indicate a mean square error between the image segmentation result and a true segmentation result.

[0029] According to a second aspect of embodiments of the present disclosure, an image segmentation method is provided, including:

[0030] inputting a to-be-processed image into an image segmentation network trained according to the training method of the image segmentation network according to any of the above embodiments, to obtain feature information of the to-be-processed image;

[0031] obtaining an image segmentation result of the to-be-processed image according to the feature information.

[0032] According to a third aspect of embodiments of the present disclosure, an image segmentation network training apparatus is provided, including:

[0033] a first obtaining unit, configured to input a sample image into a first image segmentation network to obtain feature information of the sample image, wherein the first sample image has annotation information;

[0034] a second obtaining unit, configured to obtain content-aware information of the sample image according to the feature information based on the first image segmentation network, the content-aware information indicating a correlation between a first pixel and pixels other than the first pixel in the sample image, and the first pixel being any pixel in the sample image;

[0035] a third obtaining unit, configured to obtain an image segmentation result of the sample image according to the feature information and the content-aware information;

[0036] a training unit configured to train the first image segmentation network according to the image segmentation result, the content-aware information and the annotation information, and take the trained first image segmentation network as a second image segmentation network.

[0037] In some embodiments, the training unit is specifically configured to adjust network parameters of the first image segmentation network according to a difference between the image segmentation result and a real segmentation result indicated by the annotation information, and a difference between the content-aware information and real content-aware information, wherein the real content-aware information is obtained according to the annotation information.

[0038] In some embodiments, the apparatus further comprises a real determining unit configured to:

[0039] performing scale adjustment on the real segmentation result indicated by the annotation information, so that the adjusted result has the same size as the content-aware information;

[0040] performing thermal coding on the adjusted result to obtain a coded result;

[0041] taking a cross-correlation matrix obtained by multiplying the coded result and a transpose matrix of the coded result as the real content-aware information.

[0042] In some embodiments, the second obtaining unit is specifically configured to:

[0043] performing convolution operation on the feature information to obtain a convolution result, wherein the feature information is obtained by a backbone network in the first image segmentation network;

[0044] performing dimension reconstruction on the convolution result to obtain the content-aware information.

[0045] In some embodiments, the third obtaining unit is specifically configured to:

[0046] performing activation operation on the content-aware information to obtain a first activation result, wherein a value of the first activation result is between 0 and 1;

[0047] taking a difference between 1 and the first activation result as a second activation result;

[0048] multiplying the convolution result with the first activation result and the second activation result respectively to obtain a first multiplication result and a second multiplication result;

[0049] connecting the first multiplication result, the second multiplication result and features output by a last convolution layer in the backbone network to obtain a merged result;

[0050] performing convolution operation and activation operation on the merged result to obtain the image segmentation result.

[0051] In some embodiments, the training unit is specifically configured to:

[0052] train the first image segmentation network based on a first loss function, the first loss function comprising:

[0053] a first loss for indicating a cross entropy between the image segmentation result and a ground truth segmentation result;

[0054] a second loss for indicating a cross entropy between the content-aware information and a ground truth content-aware information.

[0055] In some embodiments, the apparatus further comprises an adjusting unit configured to train the second image segmentation network based on a second loss function, the second loss function comprising the first loss, the second loss and a third loss, wherein the third loss is for indicating a mean squared error between the image segmentation result and a ground truth segmentation result.

[0056] According to a fourth aspect of embodiments of the present disclosure, an image segmentation apparatus is provided, the apparatus comprising:

[0057] a feature extraction unit configured to input a to-be-processed image into an image segmentation network trained by the training method of the image segmentation network according to any of the above embodiments, to obtain feature information of the to-be-processed image;

[0058] an image segmentation unit configured to obtain an image segmentation result of the to-be-processed image according to the feature information.

[0059] According to a fifth aspect of embodiments of the present disclosure, an electronic device is provided, comprising:

[0060] a memory configured to store executable instructions of the processor;

[0061] a processor configured to execute the executable instructions in the memory to implement the method according to any of the above embodiments.

[0062] According to a sixth aspect of embodiments of the present disclosure, a computer readable storage medium is provided, having stored thereon a computer program, which when executed by a processor, implements the method according to any of the above embodiments.

[0063] The embodiments of the present disclosure obtain content perception information of a sample image according to feature information of the sample image, obtain an image segmentation result of the sample image according to the feature information and the content perception information, and train the first image segmentation network according to the image segmentation result, the content perception information, and the label information, so that the first image segmentation network can reduce the misclassification of foreground and background when used for image segmentation, and generate an image segmentation result with high precision.

[0064] It should be understood that the general description above and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0065] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the disclosure.

[0066] Figure 1 A flowchart of a training method of an image segmentation network according to an embodiment of the present disclosure is schematically shown;

[0067] Figure 2 A system configuration diagram of a training method of an image segmentation network according to an embodiment of the present disclosure is schematically shown;

[0068] Figure 3 A comparison between an image segmentation result obtained by an image segmentation network according to an embodiment of the present disclosure and an image segmentation result obtained by an existing method is schematically shown;

[0069] Figure 4 A structure diagram of a training device of an image segmentation network according to an embodiment of the present disclosure is schematically shown;

[0070] Figure 5 A structure diagram of an electronic device according to an embodiment of the present disclosure is schematically shown.

[0071] In the drawings, the same or corresponding reference numbers represent the same or corresponding parts. DETAILED DESCRIPTION

[0072] The exemplary embodiments will be described in detail herein below with reference to the drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Rather, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0073] The terminology used in the present disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the present disclosure. As used in the present disclosure and the appended claims, the singular forms "a," "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0074] It is to be understood that, although the terms first, second, third, etc. can be used herein to describe various information, the information should not be limited to these terms. These terms are only used to differentiate one piece of information from another. For example, a first information can also be termed a second information, similarly, a second information can also be termed a first information without departing from the scope of the present disclosure. Depending on the context, the word "if' as used herein can be interpreted as meaning "when" or "in response to determining" or "in response to ascertaining".

[0075] Figure 1 A flowchart of a training method of an image segmentation network according to an embodiment of the present disclosure is schematically shown. The training method of the image segmentation network can be executed by a server, including a local server or a cloud server, etc., and can also be realized by a processor calling computer readable instructions stored in a memory. As shown in the figure, the method can include the following processing: Figure 1

[0076] In step 101, a sample image is input into a first image segmentation network to obtain feature information of the sample image.

[0077] The first image segmentation network is an image segmentation network to be trained, and the first sample image has annotation information, i.e., an image label. The annotation information includes information indicating that all or part of the pixels belong to the foreground or the background. In the case where the annotation information indicates that all the pixels belong to the foreground or the background, the ground truth of the sample image can be determined according to the annotation information.

[0078] In an embodiment of the present disclosure, the sample image can be feature-extracted by a backbone network. The backbone network can be the second layer of U2Net, and can also be ResNet, ResNeXt, DenseNet, VGG, MobileNet, ShuffleNet, etc. It should be understood by those skilled in the art that the specific result of the backbone network is not limited in the present disclosure.

[0079] In step 102, content-aware information of the sample image is obtained according to the feature information based on the first image segmentation network.​

[0080] The content-aware information indicates a correlation between a first pixel and other pixels in the sample image, the first pixel being any pixel in the sample image. The content-aware information can be in the form of a matrix, a size of the matrix being determined by a pixel matrix size of the sample image, each element in the matrix indicating correlation information between a pixel at a corresponding position and other pixels at other positions.

[0081] For example, in a case where the sample image contains 64 pixels, the content-aware information can have a size of [64, 64]. The first 64 represents a size of the sample image, and the second 64 represents a correlation between each pixel and the other 63 pixels and a correlation of the pixel itself (a value of 1).

[0082] In an example, a value corresponding to a pixel at a position can be multiplied by values of pixels at other positions respectively, and a mean value or a variance of all multiplied results is calculated, and a normalized result is taken as a value of the correlation.

[0083] In step 103, an image segmentation result of the sample image is obtained according to the feature information and the content-aware information.

[0084] In the embodiments of the present disclosure, each pixel in the sample image is predicted to belong to a foreground or a background according to the feature information and the content-aware information, and an image segmentation result of the sample image is obtained.

[0085] In step 104, the first image segmentation network is trained according to the image segmentation result, the content-aware information and the annotation information, and the trained first image segmentation network is taken as the second image segmentation network.

[0086] Since the annotation information of the sample image indicates that all or part of the pixels belong to the foreground or the background, that is, the annotation information is pixel-level, the true value corresponding to the segmentation result and the content perception information can be obtained through the annotation information, so that the first image segmentation network can be trained through the image segmentation result, the content perception information and the annotation information, and the trained first image segmentation network is used as the second image segmentation network. Through the disclosure, the content perception information of the sample image is obtained according to the feature information of the sample image, the image segmentation result of the sample image is obtained according to the feature information and the content perception information, and the first image segmentation network is trained according to the image segmentation result, the content perception information and the annotation information. By learning the related information of the context of the sample image, the first image segmentation network can reduce the misclassification of the foreground and the background when used for image segmentation, and generate an image segmentation result with high precision.

[0087] In some embodiments, the parameters of the first image segmentation network can be adjusted according to the difference between the prediction result output by the image segmentation network for the sample image and the true value corresponding to the prediction result, for example, gradient negative feedback adjustment, to realize the training of the first image segmentation network.

[0088] In one example, the network parameters of the first image segmentation network can be adjusted according to the difference between the image segmentation result and the true segmentation result indicated by the annotation information, and the difference between the content perception information and the true perception information, wherein the true perception information is obtained according to the annotation information.

[0089] In the embodiment of the disclosure, in addition to adjusting the network parameters according to the difference between the image segmentation result and the true segmentation result indicated by the annotation information, the network parameters are also adjusted according to the difference between the content perception information and the true perception information. Through continuous learning according to the content perception information and the corresponding true perception information, the content perception information finally approaches the true perception information, so that the loss of the two becomes smaller and smaller. In the case where the number of iterations reaches a set number, or the difference between the content perception information and the true perception information and / or the difference between the image segmentation result and the true segmentation result is less than a set threshold, the training is stopped, and the trained first image segmentation network is obtained.

[0090] In some embodiments, the real perception information can be determined by the following method: scaling the real segmentation result indicated by the annotation information, the adjusted result being the same size as the content perception information; performing one-hot encoding on the adjusted result to obtain an encoded result; and taking the cross-correlation matrix obtained by multiplying the encoded result and the transpose matrix of the encoded result as the real perception information. That is, the real segmentation result is scaled and one-hot encoded, and the encoded result is transposed and multiplied with itself to establish a correlation matrix (A = CC T , where C is the encoded result, and C T is the transpose matrix of the encoded result), which can also be referred to as a cross-correlation graph, i.e., the real perception information.

[0091] In some embodiments, the content perception information of the sample image can be obtained by the following method. First, the feature information is subjected to a convolution operation to obtain a convolution result. In one example, the convolution operation performed on the feature information can include a regular convolution operation and a separable convolution operation. Using a separable convolution can reduce learning parameters and speed up learning efficiency; next, the convolution result is subjected to dimension reconstruction to obtain the content perception information.

[0092] For example, assuming that the size of the feature information (feature map) is [8, 8, 64], the first 8 is the width of the feature map, the second 8 is the height of the feature map, and 64 represents the product of the width and the height. After dimension reconstruction, the size becomes [64, 64], where the first 64 is the size of the sample image, and the second 64 represents the correlation of each pixel with the remaining 63 pixels and the autocorrelation (the value is 1).

[0093] In some embodiments, the image segmentation result can be obtained according to the following manner: performing an activation operation on the content perception information to obtain a first activation result, the value of the first activation result being between 0 and 1; taking the difference between 1 and the first activation result as a second activation result; multiplying the convolution result with the first activation result and the second activation result respectively to obtain a first multiplication result and a second multiplication result; connecting the first multiplication result, the second multiplication result, and the feature output by the last convolution layer in the backbone network to obtain a merged result; and performing a convolution operation and an activation operation on the merged result to obtain the image segmentation result.

[0094] Taking the activation operation by the sigmoid function as an example, a first activation result obtained by performing the activation operation on the content-aware information can be represented as sigmoid(content-aware information), and a second activation result can be represented as (1-sigmoid(content-aware information)), the first multiplication result is a product of the convolution result obtained in the process of obtaining the content-aware information and sigmoid(content-aware information), and the second multiplication result is a product of the convolution result and (1-sigmoid(content-aware information)).

[0095] In some embodiments, the first image segmentation network can be trained based on a first loss function, the first loss function including: a first loss L s , used to indicate cross entropy between the image segmentation result and a real segmentation result; and a second loss L c , used to indicate cross entropy between the content-aware information and real-aware information. In an embodiment of the present disclosure, the first loss can be referred to as a segmentation loss, and the second loss can be referred to as a cross-correlation loss. The first loss function can be represented by formula (1):

[0096] L=λ s *L s +λ c *L c (1)

[0097] wherein λ s is a weight of the segmentation loss, λ c is a weight of the cross-correlation loss, λ s and λ c may be, for example, 1.

[0098] In some embodiments, in a case where the first image segmentation network that has completed training, i.e., a second image segmentation network, is obtained, the second image segmentation network can also be fine-tuned.

[0099] In an example, the second image segmentation network can be fine-tuned based on a second loss function, the second loss function including, in addition to the first loss and the second loss, a third loss L p , used to indicate mean square error between the image segmentation result and the real segmentation result. Here, the third loss can also be referred to as a mean square error loss. The second loss function can be represented by formula (2):

[0100] L=λ s *L s +λ c *L c +λ p *L p (2)

[0101] wherein λs The weights of the segmentation loss are λ. c It is the weight of the cross-correlation loss, λ p This is the weight of the mean squared error loss. In one example, λ s and λ c For example, we can take 1, λ p For example, 0.01.

[0102] In some embodiments, the sample image and the label image can be reconstructed in dimensions, resampled into 512*512 sub-blocks, and the sub-blocks can be horizontally and vertically flipped with a probability of 0.01. The results are then standardized. Through the above operations, the image samples can be enriched.

[0103] Figure 2 A system architecture diagram illustrating the application of an image segmentation network training method according to an embodiment of the present disclosure is shown schematically.

[0104] like Figure 2 As shown, the sample image 20 with labeled information is input into the first image segmentation network 200. In this example, the grass and trees are the foreground, and the sky is the background; the sample image 20 can be in JPG format.

[0105] First, feature extraction is performed on the sample image 20 based on the backbone network 201 in the first image segmentation network 200 to obtain feature information.

[0106] Next, the feature information is convolved based on the convolutional network 21 to obtain the convolution result, and the convolution result is reconstructed in dimension based on the reconstruction network 202 to obtain content-aware information 22.

[0107] On one hand, the scale of the real segmentation result 25 is adjusted so that the adjusted result is the same size as the content-aware information; and the adjusted result is hot-coded according to the hot coding network 204 to obtain the coding result; the cross-correlation matrix obtained by multiplying the coding result with the transpose of the coding result is used as the real-aware information 23. The input data format of the real segmentation result is PNG format.

[0108] like Figure 2 As shown, when the height of the sample image is H and the width is W, the size of the content-aware information is (H*W)*(H*W), and the size of the actual perceived information is also (H*W)*(H*W).

[0109] On the other hand, the content-aware information 22 is subjected to an activation operation to obtain a first activation result, the value of the first activation result being between 0 and 1, as shown in the figure, the activation function can be denoted as s, then the first activation result is s(content-aware information); the difference between 1 and the first activation result is taken as a second activation result, then the second activation result is (1-s(content-aware information)). The convolution result output by the convolution network 21 is multiplied by the first activation result and the second activation result respectively to obtain a first multiplication result and a second multiplication result; the first multiplication result, the second multiplication result and the features output by the last convolution layer in the backbone network 201 are connected through the connection network 203 to obtain a merged result. The merged result is subjected to a convolution operation through the convolution network 24, and the convolution result is subjected to an activation operation to obtain the image segmentation result.

[0110] The training loss of the first image segmentation network can include a segmentation loss 26 between the image segmentation result and the real segmentation result 25, and a cross-correlation loss 27 between the content-aware information and the real-aware information.

[0111] Figure 3 The contrast between the image segmentation result obtained by the image segmentation network of the embodiment of the present disclosure and the image segmentation result obtained by the method of the related art is schematically shown. Among them, the first column is the image to be processed, the second column is the image segmentation result obtained by the method of the related art, and the third column is the image segmentation result obtained by the image segmentation network of the embodiment of the present disclosure. As can be seen from the figure, the related art method appears to have mis-segmentation of the foreground in the background, such as the left area of the dog in the first group and the area inside the circle in the second group, while the image segmentation result obtained by the image segmentation network trained by the method of the embodiment of the present disclosure does not appear to have mis-segmentation of the foreground and the background. As can be seen from the contrast of the results of the third group, the foreground accuracy segmented by the method of the related art is somewhat defective, while the foreground information in the image segmentation result obtained by the image segmentation network trained by the method of the embodiment of the present disclosure is not lost. It can be seen that the image segmentation result obtained by the image segmentation network trained by the method of the embodiment of the present disclosure reduces the mis-segmentation of the foreground and the background, and has good segmentation accuracy.

[0112] The embodiment of the present disclosure also provides an image segmentation method, which can be executed by an electronic device or a server, the electronic device can be a terminal device such as a mobile phone, a tablet computer, a game console, a desktop computer, an advertising machine, an all-in-one machine, a vehicle-mounted terminal, etc., and the server includes a local server or a cloud server, etc., and the method can also be realized by a processor calling computer readable instructions stored in a memory. The method can include the following processing:

[0113] input the image to be processed into an image segmentation network to obtain feature information of the image to be processed, the image segmentation network being trained according to the training method of the image segmentation network of any embodiment of the present disclosure;

[0114] obtain an image segmentation result of the image to be processed according to the feature information.

[0115] To implement the training method of the image segmentation network of any embodiment of the present disclosure, the present embodiment further provides a training device of an image segmentation network. Figure 4 The structure of the training device of the image segmentation network according to an embodiment of the present disclosure is schematically shown, for example, the device can be applied to a computing device. In the following description, the functions of each module of the device will be simply described, and the detailed processing can be combined with the description of the training method of the image segmentation network of any embodiment of the present disclosure described above. As shown in the following figure, the device can include: Figure 4

[0116] The first obtaining unit 401 is configured to input a sample image into a first image segmentation network to obtain feature information of the sample image, wherein the first sample image has annotation information.

[0117] The second obtaining unit 402 is configured to obtain content-aware information of the sample image according to the feature information based on the first image segmentation network, the content-aware information indicating the correlation between a first pixel and pixels other than the first pixel in the sample image, the first pixel being any pixel in the sample image.

[0118] The third obtaining unit 403 is configured to obtain an image segmentation result of the sample image according to the feature information and the content-aware information.

[0119] The training unit 404 is configured to train the first image segmentation network according to the image segmentation result, the content-aware information, and the annotation information, and take the trained first image segmentation network as a second image segmentation network.

[0120] In some embodiments, the training unit is specifically configured to adjust network parameters of the first image segmentation network according to the difference between the image segmentation result and a true segmentation result indicated by the annotation information, and the difference between the content-aware information and true-aware information, wherein the true-aware information is obtained according to the annotation information. In some embodiments, the device further includes a true determination unit configured to:

[0121] ​The real segmentation result indicated by the annotation information is scaled, and the adjusted result is the same size as the content perception information; the adjusted result is hot coded to obtain a coded result;

[0122] A cross-correlation matrix obtained by multiplying the coded result and a transpose matrix of the coded result is taken as the real perception information.

[0123] In some embodiments, the second obtaining unit is specifically configured to:

[0124] The feature information is subjected to a convolution operation to obtain a convolution result, wherein the feature information is obtained by a backbone network in the first image segmentation network;

[0125] The content perception information is obtained by dimension reconstruction on the convolution result.

[0126] In some embodiments, the third obtaining unit is specifically configured to:

[0127] The content perception information is subjected to an activation operation to obtain a first activation result, and a value of the first activation result is between 0 and 1;

[0128] A difference between 1 and the first activation result is taken as a second activation result;

[0129] The convolution result is multiplied by the first activation result and the second activation result respectively to obtain a first multiplication result and a second multiplication result;

[0130] The first multiplication result, the second multiplication result and features output by a last convolution layer in the backbone network are connected to obtain a merged result;

[0131] The image segmentation result is obtained by subjecting the merged result to a convolution operation and an activation operation.

[0132] In some embodiments, the training unit is specifically configured to:

[0133] The first image segmentation network is trained based on a first loss function, and the first loss function includes:

[0134] A first loss for indicating a cross entropy between the image segmentation result and a real segmentation result;

[0135] A second loss for indicating a cross entropy between the content perception information and a real perception information.

[0136] In some embodiments, the apparatus further comprises an adjusting unit configured to train the second image segmentation network based on a second loss function, the second loss function comprising the first loss, the second loss, and a third loss, wherein the third loss is configured to indicate a mean square error between the image segmentation result and a ground truth segmentation result.

[0137] To implement the image segmentation method according to any of the embodiments of the present disclosure, the embodiments of the present disclosure further provide an image segmentation apparatus, which can be applied to a computing device. In the following description, the functions of each module of the apparatus will be simply described, and the detailed processing can be referred to the description of the image segmentation method according to any of the embodiments of the present disclosure. The image segmentation apparatus can comprise:

[0138] a feature extraction unit configured to input a to-be-processed image into the image segmentation network trained according to the training method of the image segmentation network according to any of the embodiments described above, to obtain feature information of the to-be-processed image;

[0139] an image segmentation unit configured to obtain an image segmentation result of the to-be-processed image according to the feature information.

[0140] It should be noted that, although several units / modules or sub-units / modules of the training apparatus of the image segmentation network and the image segmentation apparatus are mentioned in the foregoing detailed description, such division is merely exemplary and not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided into multiple units / modules.

[0141] The embodiments of the present disclosure further provide a computer-readable storage medium. The storage medium stores a computer program, which, when executed by a processor, can execute the training method of the image segmentation network according to any of the embodiments of the present disclosure.

[0142] The embodiments of the present disclosure further provide an electronic device, which can comprise a memory and a processor, the memory being configured to store computer instructions executable on the processor, and the processor being configured to implement the training method of the image segmentation network according to any of the embodiments of the present disclosure when executing the computer instructions.

[0143] Figure 5 An electronic device according to an exemplary embodiment of the present disclosure is shown in a block diagram.

[0144] Please refer to the accompanying drawings Figure 5FIG. 5 shows a block diagram of an example of a device. The device 500 can be a mobile phone, a computer, a digital broadcast terminal, a message receiver, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0145] Referring to Figure 5 The device 500 can include one or more of the following components: a processing component 502, a memory 504, a power supply component 506, a multimedia component 508, an audio component 510, an input / output (I / O) interface 512, a sensor component 514, and a communication component 516.

[0146] The processing component 502 usually controls overall operations of the device 500, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 502 can include one or more processors 520 to execute instructions to complete all or part of steps of the above-described methods. In addition, the processing component 502 can include one or more modules to facilitate interaction between the processing component 502 and other components. For example, the processing component 502 can include a multimedia module to facilitate the interaction between the multimedia component 508 and the processing component 502.

[0147] The memory 504 is configured to store various types of data to support operations of the device 500. Examples of these data include instructions for any application or method operating on the device 500, contact data, phonebook data, messages, pictures, videos, etc. The memory 504 can be implemented by any type of volatile or non-volatile storage devices or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0148] The power supply component 506 supplies electrical power for the various components of the device 500. The power supply component 506 can include a power management system, one or more power supplies, and other components associated with generating, managing and distributing electrical power for the device 500.

[0149] The multimedia component 508 includes a screen providing an output interface between the device 500 and a user. In some embodiments, the screen can include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, swiping, and gestures on the touch panel. The touch sensors can not only sense a boundary of a touching or swiping action, but also detect duration and pressure related to the touching or swiping action. In some embodiments, the multimedia component 508 includes a front camera and / or a rear camera. When the device 500 is in an operation mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each of the front and rear camera can be a fixed optical lens system or have a focal length and optical zooming capability.

[0150] The audio component 510 is configured to output and / or input audio signals. For example, the audio component 510 includes a microphone (MIC) to receive an external audio signal when the device 500 is in an operation mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory 504 or transmitted via the communication component 516. In some embodiments, the audio component 510 includes a speaker for outputting audio signals.

[0151] The I / O interface 512 provides an interface between the processing component 502 and peripheral interface modules, which can be a keypad, a click wheel, buttons, and the like. The buttons can include, but are not limited to, a home button, a volume button, a start button, and a lock button.

[0152] The sensor component 514 includes one or more sensors to provide various state assessments for the device 500. For example, the sensor component 514 can detect an open / closed position of the device 500, relative positioning of components, such as a display and a keypad of the device 500, a change in position of the device 500 or a component of the device 500, presence or absence of user contact with the device 500, a change in orientation of the device 500 or acceleration / deceleration of the device 500, and a temperature change of the device 500. The sensor component 514 can include a proximity sensor configured to detect presence of a nearby object without any physical touch. The sensor component 514 can include a light sensor, such as a CMOS or CCD image sensor, for use in an imaging application. In some embodiments, the sensor component 514 can include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0153] The communication component 516 is configured to facilitate wired or wireless communication between the device 500 and other devices. The device 500 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, 4G or 5G, or a combination thereof. In an exemplary embodiment, the communication component 516 receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 516 includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technology.

[0154] In an exemplary embodiment, the image segmentation network device can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors, or other electronic elements for performing the power supply method of the electronic device.

[0155] In an exemplary embodiment, the disclosure provides a non-transitory computer-readable storage medium including instructions, such as the memory 504 including instructions, which can be executed by the processor 520 of the device 500 to complete the power supply method of the electronic device. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0156] Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. It is intended that the specification and examples be considered as exemplary only, with the true scope and spirit of the disclosure being indicated by the following claims. Other variations of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. It is intended that the specification and examples be considered as exemplary only, with the true scope and spirit of the disclosure being indicated by the following claims.

[0157] It should be understood that the present disclosure is not limited to the precise structures as set forth above and as shown in the attached drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the claims appended hereto.

Claims

1. A method for training an image segmentation network, characterized in that, The method comprises: inputting a sample image into a first image segmentation network to obtain feature information of the sample image, wherein the sample image has annotation information; obtaining content perception information of the sample image according to the feature information based on the first image segmentation network, the content perception information indicating the correlation between a first pixel and pixels other than the first pixel in the sample image, the first pixel being any pixel in the sample image, the content perception information being represented in a matrix form, the size of the matrix being consistent with the pixel matrix of the sample image, and each element in the matrix representing the correlation between the corresponding pixel and all other pixels; obtaining an image segmentation result of the sample image according to the feature information and the content perception information; training the first image segmentation network according to the image segmentation result, the content perception information and the annotation information, and taking the trained first image segmentation network as a second image segmentation network.

2. The method of claim 1, wherein, The training of the first image segmentation network according to the segmentation result, the content perception information and the annotation information comprises: adjusting network parameters of the first image segmentation network according to the difference between the image segmentation result and a true segmentation result indicated by the annotation information and the difference between the content perception information and true perception information, wherein the true perception information is obtained according to the annotation information.

3. The method of claim 2, wherein, The method further comprises: scaling the true segmentation result indicated by the annotation information, the size of the adjusted result being the same as that of the content perception information; heat-encoding the adjusted result to obtain an encoded result; taking a cross-correlation matrix obtained by multiplying the encoded result and a transpose matrix of the encoded result as the true perception information.

4. The method of claim 1, wherein, The obtaining of the content perception information of the sample image according to the feature information comprises: performing convolution operation on the feature information to obtain a convolution result, wherein the feature information is obtained through a backbone network in the first image segmentation network; performing dimension reconstruction on the convolution result to obtain the content perception information.

5. The method of claim 4, wherein, The obtaining of the image segmentation result of the sample image according to the feature information and the content perception information comprises: performing activation operation on the content perception information to obtain a first activation result, the value of the first activation result being between 0 and 1; taking the difference between 1 and the first activation result as a second activation result; multiplying the convolution result with the first activation result and the second activation result respectively to obtain a first multiplication result and a second multiplication result; connecting the first multiplication result, the second multiplication result and the feature output by the last convolution layer in the backbone network to obtain a merged result; performing convolution operation and activation operation on the merged result to obtain the image segmentation result.

6. The method according to any one of claims 1 to 5, characterized in that, The training of the first image segmentation network according to the image segmentation result, the content perception information and the annotation information comprises: training the first image segmentation network based on a first loss function, the first loss function comprising: a first loss indicating a cross entropy between the image segmentation result and a ground truth segmentation result; a second loss indicating a cross entropy between the content-aware information and a ground truth content-aware information.

7. The method of claim 6, wherein, The method further includes training the second image segmentation network based on a second loss function, the second loss function including the first loss, the second loss, and a third loss, wherein the third loss indicates a mean square error between the image segmentation result and the ground truth segmentation result.

8. An image segmentation method characterized by, The method includes: inputting a to-be-processed image into the image segmentation network trained by the method according to any one of claims 1 to 7, to obtain feature information of the to-be-processed image; obtaining an image segmentation result of the to-be-processed image according to the feature information.

9. A training device for an image segmentation network, characterized in that, The apparatus includes: a first obtaining unit configured to input a sample image into a first image segmentation network to obtain feature information of the sample image, wherein the sample image has annotation information; a second obtaining unit configured to obtain content-aware information of the sample image based on the first image segmentation network and according to the feature information, the content-aware information indicating a correlation between a first pixel and pixels other than the first pixel in the sample image, the first pixel being any pixel in the sample image, the content-aware information being represented in a matrix form, a size of the matrix being consistent with a pixel matrix of the sample image, and each element in the matrix representing a correlation between a corresponding pixel and all other pixels; a third obtaining unit configured to obtain an image segmentation result of the sample image according to the feature information and the content-aware information; a training unit configured to train the first image segmentation network according to the image segmentation result, the content-aware information, and the annotation information, and use the trained first image segmentation network as a second image segmentation network.

10. The apparatus of claim 9, wherein, The training unit is specifically configured to adjust network parameters of the first image segmentation network according to a difference between the image segmentation result and a ground truth segmentation result indicated by the annotation information, and a difference between the content-aware information and a ground truth content-aware information, wherein the ground truth content-aware information is obtained according to the annotation information.

11. The apparatus of claim 10, wherein, The apparatus further includes a ground truth determining unit configured to: perform a scale adjustment on the ground truth segmentation result indicated by the annotation information, the adjusted result having a same size as the content-aware information; perform a hot encoding on the adjusted result to obtain an encoded result; use a cross-correlation matrix obtained by multiplying the encoded result and a transpose matrix of the encoded result as the ground truth content-aware information.

12. The apparatus of claim 9, wherein, The second obtaining unit is specifically configured to: perform a convolution operation on the feature information to obtain a convolution result, wherein the feature information is obtained by a backbone network in the first image segmentation network; perform a dimension reconstruction on the convolution result to obtain the content-aware information.

13. The apparatus of claim 12, wherein, The third obtaining unit is specifically configured to: perform an activation operation on the content-aware information to obtain a first activation result, a value of the first activation result being between 0 and 1; use a difference between 1 and the first activation result as a second activation result. The convolution result is multiplied with the first activation result and the second activation result respectively to obtain a first multiplication result and a second multiplication result; The first multiplication result and the second multiplication result are connected with the features output by the last convolution layer in the backbone network to obtain a merging result; The merging result is subjected to a convolution operation and an activation operation to obtain the image segmentation result.

14. The apparatus of any one of claims 9 to 13, wherein, The training unit is specifically configured to: train the first image segmentation network based on a first loss function, the first loss function comprising: a first loss, used to indicate a cross entropy between the image segmentation result and a real segmentation result; a second loss, used to indicate a cross entropy between the content-aware information and real-aware information.

15. The apparatus of claim 14, wherein, The device further comprises an adjusting unit configured to train the second image segmentation network based on a second loss function, the second loss function comprising the first loss, the second loss, and a third loss, wherein the third loss is used to indicate a mean square error between the image segmentation result and the real segmentation result.

16. An image segmentation apparatus characterized by comprising: The device comprises: a feature extraction unit configured to input a to-be-processed image into an image segmentation network trained according to the method of any one of claims 1 to 7 to obtain feature information of the to-be-processed image; an image segmentation unit configured to obtain an image segmentation result of the to-be-processed image according to the feature information.

17. An electronic device, comprising: The electronic device comprises: a memory configured to store processor-executable instructions; a processor configured to execute the executable instructions in the memory to implement the method of any one of claims 1 to 7 or claim 8.

18. A computer readable storage medium having stored thereon a computer program, characterized in that, The program, when executed by a processor, implements the method of any one of claims 1 to 7 or claim 8. The program, when executed by a processor, implements the method of any one of claims 1 to 7 or claim 8.

Images