Image segmentation method and device

Abstract

The present disclosure provides an image segmentation method and device. The method comprises: inputting an image to be segmented into a pre-trained image segmentation network to obtain a segmented image; the image segmentation network is trained in the following manner: inputting small sample data of a real image and target synthetic image data into a corresponding image segmentation network respectively to obtain first predicted image data and second predicted image data; the small sample data of the real image and the target synthetic image data contain the same target object; obtaining a first loss value based on the second predicted image data and the target synthetic image annotation data; obtaining a second loss value based on the first predicted image data and the second predicted image data; after adjusting the parameters of the image segmentation network using the first loss value and the second loss value, returning to the step of inputting small sample data of the real image and the target synthetic image data into the corresponding image segmentation network respectively, until the first loss value and the second loss value meet the specified conditions, thereby improving the image segmentation effect.

Description

Background Art

[0002] Deep learning technology plays an important role in the field of smart agriculture, especially fruit and vegetable image recognition based on convolutional neural networks, which provides a theoretical basis for the development of agricultural harvesting robots. However, in actual field or greenhouse environments, the background is complex and the occlusion phenomenon is serious. It is challenging to quickly and accurately identify fruits under the occlusion of stems, leaves, buds and other objects.

[0003] At present, the method of image segmentation through neural networks requires a large number of labeled image training samples during the training process of the neural network. However, in the actual environment of fruits and vegetables, data sample resources are very scarce, and the image segmentation model cannot be fully trained, resulting in poor image segmentation effect of the image segmentation model after inspection. Summary of the invention

[0004] In an exemplary embodiment of the present disclosure, an image segmentation method, an apparatus, an electronic device, and a computer storage medium are provided to improve the effect of image segmentation.

[0005] A first aspect of the present disclosure provides an image segmentation method, the method comprising:

[0006] Input the image to be segmented into a pre-trained image segmentation network to obtain the segmented image;

[0007] The image segmentation network is trained in the following way:

[0008] Inputting the small sample data of the real image and the target synthetic image data into the corresponding image segmentation network respectively, to obtain the first predicted image data and the second predicted image data, wherein the first predicted image data is obtained based on the small sample data of the real image, the second predicted image data is obtained based on the target synthetic image data, and the image data in the small sample data of the real image and the target synthetic image data contain the same target object;

[0009] Obtaining a first loss value based on the second predicted image data and preset target synthetic image annotation data;

[0010] Obtaining a second loss value according to the first predicted image data and the second predicted image data;

[0011] After adjusting the parameters of each image segmentation network using the first loss value and the second loss value, return to the step of inputting the small sample data of the real image and the target synthetic image data into the corresponding image segmentation network respectively until the first loss value and the second loss value meet the specified conditions.

[0012] In this embodiment, the image segmentation network is adjusted based on a first loss value obtained based on the second predicted image data and the preset target synthetic image annotation data to narrow the gap between the predicted image of the target synthetic image data and the real annotation data, and the image segmentation network is adjusted based on a second loss value obtained based on the first predicted image data and the second predicted image data to narrow the gap between the predicted image of the real image and the predicted image of the target synthetic image data, thereby further narrowing the gap between the predicted image data of the real image and the annotation data, thereby improving the image segmentation effect.

[0013] In one embodiment, obtaining the first loss value based on the second predicted image data and preset target synthetic image annotation data includes:

[0014] For any pixel in the second predicted image data, based on a first predicted object of the pixel in the second predicted image data and a labeled object of the pixel in the target synthetic image labeled data, obtaining an intermediate loss value of the pixel;

[0015] The first loss value is obtained according to the intermediate loss value corresponding to each pixel point in the second predicted image data.

[0016] This embodiment obtains the intermediate loss value of each pixel point through the first predicted object of each pixel point in the second predicted image data and the labeled object of each pixel point in the target synthetic image labeled data, and obtains the first loss value based on each intermediate loss, thereby ensuring that the obtained first loss value is more accurate.

[0017] In one embodiment, the first loss value is obtained by the following formula:

[0018]

[0019] Among them, L seg is the first loss value, H is the height of the second predicted image data, W is the width of the second predicted image data, is the first prediction object of the pixel point with the ordinate h and the abscissa w in the second prediction data, is the annotated object of the pixel point with the ordinate h and the abscissa w in the target synthetic image annotation data, h∈[0,H], w∈[0,W].

[0020] In one embodiment, obtaining a second loss value according to the first predicted image data and the second predicted image data includes:

[0021] Inputting the first predicted image data into a fully convolutional neural network to obtain a first predicted image feature;

[0022] Inputting the second predicted image data into a fully convolutional neural network to obtain second predicted image features;

[0023] The second loss value is obtained based on the first predicted image feature and the second predicted image feature.

[0024] This embodiment obtains the first predicted image feature and the second predicted image feature through a fully convolutional neural network, and obtains the second loss value based on the first predicted image feature and the second predicted image feature. Thus, more detailed features can be obtained through the fully convolutional neural network, making it more accurate to determine the second loss value.

[0025] In one embodiment, the second loss value is obtained by:

[0026] Obtaining a first prediction value corresponding to the first prediction image data by using a second prediction object of each pixel point in the first prediction image feature corresponding to the first prediction image data;

[0027] Obtaining a second prediction value corresponding to the second prediction image data according to a third prediction object of each pixel point in the second prediction image feature corresponding to the second prediction image data;

[0028] Based on the first prediction value and the second prediction value, the second loss value is obtained.

[0029] In one embodiment, before inputting the small sample data of the real image and the target synthetic image data into the corresponding image segmentation network respectively, the method further includes:

[0030] The small sample data of the real image and the synthetic image data are input into a pre-trained cycle-consistent adversarial network to obtain the target synthetic image data.

[0031] This embodiment obtains the target synthetic image data by inputting the small sample data of the real image and the synthetic image data into a pre-trained cycle-consistent adversarial network, so that the synthetic image data is closer to the real image in color distribution, thereby further improving the accuracy of image segmentation.

[0032] A second aspect of the present disclosure provides an image segmentation device, the device comprising:

[0033] The segmentation module is used to input the image to be segmented into a pre-trained image segmentation network to obtain the segmented image;

[0034] The image segmentation network is trained in the following way:

[0035] A prediction module, used to input the small sample data of the real image and the target synthetic image data into the corresponding image segmentation network respectively, to obtain first predicted image data and second predicted image data, wherein the first predicted image data is obtained based on the small sample data of the real image, the second predicted image data is obtained based on the target synthetic image data, and the image data in the small sample data of the real image and the target synthetic image data contain the same target object;

[0036] A first loss value determination module, configured to obtain a first loss value based on the second predicted image data and preset target synthetic image annotation data;

[0037] A second loss value determining module, configured to obtain a second loss value according to the first predicted image data and the second predicted image data;

[0038] An adjustment module is used to adjust the parameters of each image segmentation network using the first loss value and the second loss value, and then return to the step of inputting the small sample data of the real image and the target synthetic image data into the corresponding image segmentation network respectively until the first loss value and the second loss value meet the specified conditions.

[0039] In one embodiment, the first loss value determination module is specifically used to:

[0040] For any pixel in the second predicted image data, based on a first predicted object of the pixel in the second predicted image data and a labeled object of the pixel in the target synthetic image labeled data, obtaining an intermediate loss value of the pixel;

[0041] The first loss value is obtained according to the intermediate loss value corresponding to each pixel point in the second predicted image data.

[0042] In one embodiment, the first loss value determination module is specifically used to:

[0043] The first loss value is obtained by the following formula:

[0044]

[0045] Among them, L seg is the first loss value, H is the height of the second predicted image data, W is the width of the second predicted image data, is the first prediction object of the pixel point with the ordinate h and the abscissa w in the second prediction data, is the annotated object of the pixel point with the ordinate h and the abscissa w in the target synthetic image annotation data, h∈[0,H], w∈[0,W].

[0046] In one embodiment, the second loss value determination module is specifically used to:

[0047] Inputting the first predicted image data into a fully convolutional neural network to obtain a first predicted image feature;

[0048] Inputting the second predicted image data into a fully convolutional neural network to obtain second predicted image features;

[0049] The second loss value is obtained based on the first predicted image feature and the second predicted image feature.

[0050] In one embodiment, the second loss value determination module is specifically used to:

[0051] The second loss value is obtained by:

[0052] Obtaining a first prediction value corresponding to the first prediction image data by using a second prediction object of each pixel point in the first prediction image feature corresponding to the first prediction image data;

[0053] Obtaining a second prediction value corresponding to the second prediction image data according to a third prediction object of each pixel point in the second prediction image feature corresponding to the second prediction image data;

[0054] Based on the first prediction value and the second prediction value, the second loss value is obtained.

[0055] In one embodiment, the apparatus further comprises:

[0056] The target synthetic image data determination module is used to input the small sample data of the real image and the synthetic image data into a pre-trained cycle-consistent adversarial network before respectively inputting the small sample data of the real image and the target synthetic image data into the corresponding image segmentation network to obtain the target synthetic image data.

[0057] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0058] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executed by the at least one processor; the instructions are executed by the at least one processor so that the at least one processor can execute the method as described in the first aspect.

[0059] According to a fourth aspect provided by an embodiment of the present disclosure, a computer storage medium is provided, wherein the computer storage medium stores a computer program, and the computer program is used to execute the method as described in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0061] Figure 1 is one of the flowcharts of an image segmentation method according to an embodiment of the present disclosure;

[0062] Figure 2 The second flowchart of the image segmentation method according to one embodiment of the present disclosure;

[0063] Figure 3 A schematic diagram of a process for determining a second loss value according to an embodiment of the present disclosure;

[0064] Figure 4 A schematic diagram of the structure of a cycle consistent adversarial network according to an embodiment of the present disclosure;

[0065] Figure 5 is a color distribution correlation comparison diagram according to an embodiment of the present disclosure;

[0066] Figure 6 The third flowchart of the image segmentation method according to one embodiment of the present disclosure;

[0067] Figure 7 An image segmentation device according to an embodiment of the present disclosure;

[0068] Figure 8 The figure is a schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0070] In the embodiments of the present disclosure, the term "and / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent three situations: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0071] The application scenarios described in the embodiments of the present disclosure are intended to more clearly illustrate the technical solutions of the embodiments of the present disclosure, and do not constitute a limitation on the technical solutions provided by the embodiments of the present disclosure. It is known to those skilled in the art that with the emergence of new application scenarios, the technical solutions provided by the embodiments of the present disclosure are also applicable to similar technical problems. In the description of the present disclosure, unless otherwise specified, the meaning of "multiple" is two or more.

[0072] In the prior art, the method of image segmentation through neural networks requires a large number of labeled image training samples during the training process of the neural network. However, in the actual environment of fruits and vegetables, data sample resources are very scarce, and the image segmentation model cannot be fully trained, resulting in poor image segmentation effect of the image segmentation model after inspection.

[0073] Therefore, the present disclosure provides an image segmentation method, which adjusts the image segmentation network based on a first loss value obtained based on the second predicted image data and pre-set target synthetic image annotation data to narrow the gap between the predicted image of the target synthetic image data and the real annotation data, and adjusts the image segmentation network based on a second loss value obtained based on the first predicted image data and the second predicted image data to narrow the gap between the predicted image of the real image and the predicted image of the target synthetic image data, thereby further narrowing the gap between the predicted image data of the real image and the annotation data, thereby improving the image segmentation effect.

[0074] Next, we first introduce the training method of the image segmentation network in detail. Figure 1 As shown, it is a schematic diagram of the process of training an image segmentation network, which may include the following steps:

[0075] Step 101: inputting small sample data of a real image and target synthetic image data into corresponding image segmentation networks respectively to obtain first predicted image data and second predicted image data, wherein the first predicted image data is obtained based on the small sample data of the real image, the second predicted image data is obtained based on the target synthetic image data, and the image data in the small sample data of the real image and the target synthetic image data contain the same target object;

[0076] Among them, if the application scenario is a fruit and vegetable scene, the target objects include leaves, fruits, peduncles, stems, buds, etc. The specific target objects can be set according to the actual situation of the actual scene. The target object in this embodiment is only used for distance description and does not limit the target object in this application.

[0077] It should be noted that: the image segmentation network used in this embodiment is FRRN (Full-resolution Residual Networks). The use of the image segmentation network can be set according to actual conditions, and this embodiment does not limit it here.

[0078] Step 102: Obtain a first loss value based on the second predicted image data and preset target synthetic image annotation data;

[0079] In one embodiment, the first loss value is obtained by:

[0080] For any pixel in the second predicted image data, based on the first predicted object of the pixel in the second predicted image data and the annotated object of the pixel in the target synthetic image annotated data, the intermediate loss value of the pixel is obtained; and the first loss value is obtained according to the intermediate loss values ​​corresponding to each pixel in the second predicted image data. The first loss value can be determined by formula (1):

[0081]

[0082] Among them, L seg is the first loss value, H is the height of the second predicted image data, W is the width of the second predicted image data, is the first prediction object of the pixel point with the ordinate h and the abscissa w in the second prediction data, is the annotated object of the pixel point with the ordinate h and the abscissa w in the target synthetic image annotation data, h∈[0,H], w∈[0,W].

[0083] Step 103: Obtain a second loss value according to the first predicted image data and the second predicted image data;

[0084] It should be noted that the execution order of step 102 and step 103 is not limited in this embodiment, and step 102 may be executed first, and then step 103. Step 103 may also be executed first, and then step 102. Step 102 and step 103 may also be executed simultaneously.

[0085] In one embodiment, the second loss value can be obtained in the following manner: inputting the first predicted image data into a fully convolutional neural network to obtain a first predicted image feature; and inputting the second predicted image data into a fully convolutional neural network to obtain a second predicted image feature; based on the first predicted image feature and the second predicted image feature, obtaining the second loss value.

[0086] Step 104: determine whether the first loss value and the second loss value meet a specified condition. If so, end the process. If not, execute step 105.

[0087] The specified condition in this embodiment is that the first loss value is less than a first preset threshold, and the second loss value is less than a second preset threshold.

[0088] Step 105: After adjusting the parameters of each image segmentation network using the first loss value and the second loss value, return to step 101.

[0089] like Figure 2 As shown in, this is the training flow chart of the image segmentation network, such as Figure 2 It can be seen that the target synthetic image data is input into the second image segmentation network, and the small sample data of the real image is input into the first image segmentation network to obtain the first predicted image and the second predicted image. Then, based on the first predicted image data and the preset target synthetic image annotation data, a first loss value is obtained, and the parameters of the first image segmentation network are adjusted by the first loss value. And the first predicted image data and the second predicted image data are input into the fully convolutional neural network to obtain a second loss value, and the parameters of the second image segmentation network are adjusted based on the second loss value. However, since the parameters of the first image segmentation network and the second image segmentation network are shared, adjusting the parameters of the first image segmentation network will also adjust the parameters of the second image segmentation network. Adjusting the parameters of the second image segmentation network will also adjust the parameters of the first image segmentation network.

[0090] In one embodiment, if Figure 3 As shown, it is a schematic diagram of a process for determining a second loss value, which may include the following steps:

[0091] Step 301: using the second prediction object of each pixel point in the first prediction image feature corresponding to the first prediction image data, obtain the first prediction value corresponding to the first prediction image data; wherein the first prediction value can be determined according to formula (2):

[0092]

[0093] Among them, L D (I t ) is the first predicted value, The second prediction object is the pixel point with vertical coordinate h and horizontal coordinate w in the first prediction image feature corresponding to the first prediction image data, M is the height of the first prediction image data, N is the width of the first prediction image data, h∈[0,M], w∈[0,N].

[0094] Step 302: Obtain a second prediction value corresponding to the second prediction image data according to a third prediction object of each pixel point in the second prediction image feature corresponding to the second prediction image data; wherein the second prediction value can be determined by formula (3):

[0095]

[0096] Among them, L D (I s ) is the second predicted value, The third prediction object is the pixel point with ordinate h and abscissa w in the second prediction image feature corresponding to the second prediction image data, H is the height of the second prediction image data, W is the width of the second prediction image data, h∈[0,H], w∈[0,W].

[0097] It should be noted that the execution order of step 301 and step 302 is not limited in this embodiment, and step 301 may be executed first, and then step 302. Step 302 may also be executed first, and then step 301. Step 301 and step 302 may also be executed simultaneously.

[0098] Step 303: Based on the first predicted value and the second predicted value, obtain the second loss value. The second loss value L can be obtained by formula (4): p :

[0099] L p =L D (I t )+L D (I s )…(4).

[0100] In order to improve the accuracy of image segmentation, in one implementation, before executing step 101, the small sample data of the real image and the synthetic image data are input into a pre-trained cycle-consistent adversarial network to obtain the target synthetic image data.

[0101] like Figure 4 As shown, it is a schematic diagram of the structure of a cycle-consistent adversarial network, which includes a generator G1, a generator G2, a generator F1, a generator F2, a discriminator D1 and a discriminator D2.

[0102] Among them, generator G1 and generator G2 can convert the input image data to another domain (that is, the domain of the image data input by the corresponding generator) after training. The corresponding discriminator D1 and discriminator D2 can distinguish the generated image data from the image data of the original domain after training, so as to obtain the intermediate loss to adjust the parameters of each generator. Generator F1 and generator F2 convert the first group of images obtained by generator G1 and generator G2 back to the original domain to generate a second group of images. Then, the cycle consistency loss is calculated by comparing the second group of images with the initial input image to adjust the parameters of each generator. Each generator converts the image to the relative domain through training while maintaining the geometric consistency of the image.

[0103] like Figure 4 As shown in the figure, since the network is symmetrical, the inputs on both sides perform the same operation. The following takes the process of inputting the target synthetic image data as an example to illustrate the workflow of the network:

[0104] The target synthetic image data is input into the generator G1, and the small sample data of the real image is input into the generator F1, and the generator G1 generates an image A similar to the small sample data of the real image. Image A and the small sample data of the real image are input into the judgement device D1 to obtain a first intermediate loss, and the parameters of the generator G1 and the generator F2 are adjusted using the first intermediate loss, and then image A is input into the generator F2 to obtain an image C similar to the target synthetic image, and the image C and the target synthetic image data are used to obtain a second intermediate loss, and then the parameters of the generator G1 and the generator F1 are adjusted using the second intermediate loss, and then the step of inputting the target synthetic image data into the generator G1 and the small sample data of the real image into the generator F1 is returned to execute until the first intermediate loss and the second intermediate loss meet the specified conditions, and the obtained image C is determined as the target synthetic image data.

[0105] Therefore, in this embodiment, the target synthetic image data is made closer to the real image in color distribution through the above method, such as Figure 5 As shown in FIG. 1 , a list of color distribution correlations between small sample data of real images of fruit and vegetable images, synthetic image data, and target synthetic image data is shown. Figure 5 As shown in the figure, the color distribution correlation of each target object in the fruit and vegetable image is shown. Among them, the average color distribution correlation between the small sample data of the real image and the synthetic image data is 0.62. The average color distribution correlation between the small sample data of the real image and the target synthetic image data is 0.79. It is determined that the color distribution of the target synthetic image data is closer to the real image.

[0106] In one embodiment, after the image segmentation network is trained, the image to be segmented is input into a pre-trained image segmentation network to obtain a segmented image.

[0107] The image segmented in this embodiment can be compared with the image segmented in the prior art by pixel accuracy and average intersection-over-union ratio. The pixel accuracy can be determined by formula (5):

[0108]

[0109] Among them, P A is the pixel accuracy, p is the number of pixels of the segmented image whose second predicted object is the same as the labeled object, and n is the total number of pixels of the segmented image.

[0110] And the average intersection-over-union ratio MIOU can be determined by formula (6):

[0111]

[0112] Among them, k is the number of target objects, p ii is the number of pixels in the segmented image where both the second predicted object and the labeled object are i, p ij is the number of pixels in the segmented image whose labeled object is i but whose second predicted object is j, p ji is the number of pixels in the segmented image whose labeled object is j but whose second predicted object is i.

[0113] Therefore, the quality of the segmented image can be evaluated by the above-mentioned pixel accuracy and average intersection-union ratio.

[0114] In order to further understand the technical solution of the present disclosure, Figure 6 A detailed description may include the following steps:

[0115] Step 601: inputting the small sample data of the real image and the synthetic image data into a pre-trained cycle-consistent adversarial network to obtain target synthetic image data;

[0116] Step 602: inputting the small sample data of the real image and the target synthetic image data into corresponding image segmentation networks respectively, to obtain first predicted image data and second predicted image data, wherein the first predicted image data is obtained based on the small sample data of the real image, the second predicted image data is obtained based on the target synthetic image data, and the image data in the small sample data of the real image and the target synthetic image data contain the same target object;

[0117] Step 603: Obtaining a first loss value based on the second predicted image data and preset target synthetic image annotation data;

[0118] Step 604: Obtain a second loss value according to the first predicted image data and the second predicted image data;

[0119] Step 605: Determine whether the first loss value and the second loss value meet a specified condition, if yes, execute step 607, if no, execute step 606;

[0120] Step 606: After adjusting the parameters of each image segmentation network using the first loss value and the second loss value, return to step 602;

[0121] Step 607: input the image to be segmented into a pre-trained image segmentation network to obtain a segmented image.

[0122] Based on the same disclosed concept, the image segmentation method described above in the present disclosure can also be implemented by an image segmentation device. The effect of the image segmentation device is similar to that of the above method, and will not be described in detail here.

[0123] Figure 7 Schematic diagram of the structure of an image segmentation device according to an embodiment of the present disclosure.

[0124] like Figure 7 As shown, the image segmentation device 700 of the present disclosure may include a segmentation module 710 , a prediction module 720 , a first loss value determination module 730 , a second loss value determination module 740 and an adjustment module 750 .

[0125] The segmentation module 710 is used to input the image to be segmented into a pre-trained image segmentation network to obtain a segmented image;

[0126] The image segmentation network is trained in the following way:

[0127] A prediction module 720 is used to input the small sample data of the real image and the target synthetic image data into corresponding image segmentation networks respectively to obtain first predicted image data and second predicted image data, wherein the first predicted image data is obtained based on the small sample data of the real image, the second predicted image data is obtained based on the target synthetic image data, and the image data in the small sample data of the real image and the target synthetic image data contain the same target object;

[0128] A first loss value determination module 730, configured to obtain a first loss value based on the first predicted image data and preset target synthetic image annotation data;

[0129] A second loss value determining module 740, configured to obtain a second loss value according to the first predicted image data and the second predicted image data;

[0130] The adjustment module 750 is used to adjust the parameters of each image segmentation network using the first loss value and the second loss value, and then return to the step of inputting the small sample data of the real image and the target synthetic image data into the corresponding image segmentation network respectively until the first loss value and the second loss value meet the specified conditions.

[0131] In one embodiment, the first loss value determination module 730 is specifically configured to:

[0132] For any pixel in the second predicted image data, based on a first predicted object of the pixel in the second predicted image data and a labeled object of the pixel in the target synthetic image labeled data, obtaining an intermediate loss value of the pixel;

[0133] The first loss value is obtained according to the intermediate loss value corresponding to each pixel point in the second predicted image data.

[0134] In one embodiment, the first loss value determination module 730 is specifically configured to:

[0135] The first loss value is obtained by the following formula:

[0136]

[0137] Among them, L seg is the first loss value, H is the height of the second predicted image data, W is the width of the second predicted image data, is the first prediction object of the pixel point with the ordinate h and the abscissa w in the second prediction data, is the annotated object of the pixel point with the ordinate h and the abscissa w in the target synthetic image annotation data, h∈[0,H], w∈[0,W].

[0138] In one embodiment, the second loss value determination module 740 is specifically configured to:

[0139] Inputting the first predicted image data into a fully convolutional neural network to obtain a first predicted image feature;

[0140] Inputting the second predicted image data into a fully convolutional neural network to obtain second predicted image features;

[0141] The second loss value is obtained based on the first predicted image feature and the second predicted image feature.

[0142] In one embodiment, the second loss value determination module 740 is specifically configured to:

[0143] The second loss value is obtained by:

[0144] Obtaining a first prediction value corresponding to the first prediction image data by using a second prediction object of each pixel point in the first prediction image feature corresponding to the first prediction image data;

[0145] Obtaining a second prediction value corresponding to the second prediction image data according to a third prediction object of each pixel point in the second prediction image feature corresponding to the second prediction image data;

[0146] Based on the first prediction value and the second prediction value, the second loss value is obtained.

[0147] In one embodiment, the apparatus further comprises:

[0148] The target synthetic image data determination module 760 is used to input the small sample data of the real image and the synthetic image data into a pre-trained cycle-consistent adversarial network before inputting the small sample data of the real image and the target synthetic image data into the corresponding image segmentation network respectively, so as to obtain the target synthetic image data.

[0149] After introducing an image segmentation method and apparatus according to an exemplary embodiment of the present disclosure, next, an electronic device according to another exemplary embodiment of the present disclosure is introduced.

[0150] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0151] In some possible implementations, the electronic device according to the present disclosure may include at least one processor and at least one computer storage medium. The computer storage medium stores program code, and when the program code is executed by the processor, the processor executes the steps of the image segmentation method according to various exemplary implementations of the present disclosure described above in this specification. For example, the processor may execute the following steps: Figure 1 Steps 101-105 shown in FIG.

[0152] Refer to the following Figure 8 The electronic device 800 according to this embodiment of the present disclosure is described. Figure 8The electronic device 800 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0153] like Figure 8 As shown, the electronic device 800 is in the form of a general electronic device. The components of the electronic device 800 may include but are not limited to: at least one processor 801, at least one computer storage medium 802, and a bus 803 connecting different system components (including the computer storage medium 802 and the processor 801).

[0154] Bus 803 represents one or more of several types of bus structures, including a computer storage media bus or computer storage media controller, a peripheral bus, a processor, or a local bus using any of a variety of bus architectures.

[0155] The computer storage medium 802 may include readable media in the form of volatile computer storage media, such as random access computer storage media (RAM) 821 and / or cache storage media 822 , and may further include read-only computer storage media (ROM) 823 .

[0156] The computer storage medium 802 may also include a program / utility 825 having a set (at least one) of program modules 824, such program modules 824 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0157] The electronic device 800 may also communicate with one or more external devices 804 (e.g., keyboards, pointing devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 800, and / or communicate with any device that enables the electronic device 800 to communicate with one or more other electronic devices (e.g., routers, modems, etc.). This communication may be performed through an input / output (I / O) interface 805. In addition, the electronic device 800 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) through a network adapter 806. As shown, the network adapter 806 communicates with other modules for the electronic device 800 through a bus 803. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 800, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0158] In some possible implementations, various aspects of an image segmentation method provided by the present disclosure may also be implemented in the form of a program product, which includes program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of the image segmentation method according to various exemplary embodiments of the present disclosure described above in this specification.

[0159] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access computer storage medium (RAM), a read-only computer storage medium (ROM), an erasable programmable read-only computer storage medium (EPROM or flash memory), an optical fiber, a portable compact disk read-only computer storage medium (CD-ROM), an optical computer storage medium, a magnetic computer storage medium, or any suitable combination of the above.

[0160] The program product of the image segmentation of the embodiment of the present disclosure can adopt a portable compact disk read-only computer storage medium (CD-ROM) and include program code, and can be run on an electronic device. However, the program product of the present disclosure is not limited to this. In this document, the readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0161] The readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein the readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0162] The program code contained on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0163] Program code for performing the disclosed operations may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user electronic device, partially on the user device, as a separate software package, partially on the user electronic device and partially on a remote electronic device, or entirely on a remote electronic device or server. In the case of a remote electronic device, the remote electronic device may be connected to the user electronic device via any type of network including a local area network (LAN) or a wide area network (WAN), or may be connected to an external electronic device (e.g., via the Internet using an Internet service provider).

[0164] It should be noted that although several modules of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules described above can be embodied in one module. Conversely, the features and functions of one module described above can be further divided into multiple modules to be embodied.

[0165] In addition, although the operations of the disclosed method are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0166] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk computer storage media, CD-ROM, optical computer storage media, etc.) containing computer-usable program codes.

[0167] The present disclosure is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the present disclosure. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0168] These computer program instructions may also be stored in a computer-readable computer storage medium that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable computer storage medium produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0169] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0170] Obviously, those skilled in the art can make various changes and modifications to the present disclosure without departing from the spirit and scope of the present disclosure. Thus, if these modifications and variations of the present disclosure fall within the scope of the claims of the present disclosure and their equivalents, the present disclosure is also intended to include these modifications and variations.

Claims

1. An image segmentation method, characterized in that: The method comprises: Input the image to be segmented into a pre-trained image segmentation network to obtain the segmented image; The image segmentation network is trained in the following way: Inputting small sample data of real images and synthetic image data into a pre-trained cycle-consistent adversarial network to obtain target synthetic image data, wherein the image data in the small sample data of real images and the target synthetic image data contain the same target object; Inputting the small sample data of the real image and the target synthetic image data into corresponding image segmentation networks respectively, to obtain first predicted image data and second predicted image data, wherein the first predicted image data is obtained based on the small sample data of the real image, and the second predicted image data is obtained based on the target synthetic image data; Obtaining a first loss value based on the second predicted image data and preset target synthetic image annotation data; and Obtaining a second loss value according to the first predicted image data and the second predicted image data; and, After adjusting the parameters of each image segmentation network using the first loss value and the second loss value, return to the step of inputting the small sample data of the real image and the target synthetic image data into the corresponding image segmentation network respectively until the first loss value and the second loss value meet the specified conditions.

2. The method according to claim 1, characterized in that The obtaining a first loss value based on the second predicted image data and the preset target synthetic image annotation data includes: For any pixel in the second predicted image data, based on a first predicted object of the pixel in the second predicted image data and a labeled object of the pixel in the target synthetic image labeled data, obtaining an intermediate loss value of the pixel; The first loss value is obtained according to the intermediate loss value corresponding to each pixel point in the second predicted image data.

3. The method according to claim 2, characterized in that The first loss value is obtained by the following formula: Among them, L seg is the first loss value, H is the height of the second predicted image data, W is the width of the second predicted image data, is the first prediction object of the pixel point with the ordinate h and the abscissa w in the second prediction image data, is the annotated object of the pixel point with the ordinate h and the abscissa w in the target synthetic image annotation data, h∈[0,H], w∈[0,W].

4. The method according to claim 1, characterized in that: The obtaining a second loss value according to the first predicted image data and the second predicted image data comprises: Inputting the first predicted image data into a fully convolutional neural network to obtain a first predicted image feature; and, Inputting the second predicted image data into a fully convolutional neural network to obtain second predicted image features; The second loss value is obtained based on the first predicted image feature and the second predicted image feature.

5. The method according to claim 4, characterized in that The second loss value is obtained by: Obtaining a first prediction value corresponding to the first prediction image data by using a second prediction object of each pixel point in the first prediction image feature corresponding to the first prediction image data; and, Obtaining a second prediction value corresponding to the second prediction image data according to a third prediction object of each pixel point in the second prediction image feature corresponding to the second prediction image data; Based on the first prediction value and the second prediction value, the second loss value is obtained.

6. An image segmentation device, characterized in that: The device comprises: The segmentation module is used to input the image to be segmented into a pre-trained image segmentation network to obtain the segmented image; The image segmentation network is trained in the following way: Inputting small sample data of real images and synthetic image data into a pre-trained cycle-consistent adversarial network to obtain target synthetic image data, wherein the image data in the small sample data of real images and the target synthetic image data contain the same target object; A prediction module, used to input the small sample data of the real image and the target synthetic image data into corresponding image segmentation networks respectively, to obtain first predicted image data and second predicted image data, wherein the first predicted image data is obtained based on the small sample data of the real image, and the second predicted image data is obtained based on the target synthetic image data; a first loss value determining module, configured to obtain a first loss value based on the second predicted image data and preset target synthetic image annotation data; and a second loss value determining module, configured to obtain a second loss value according to the first predicted image data and the second predicted image data; and An adjustment module is used to adjust the parameters of each image segmentation network using the first loss value and the second loss value, and then return to the step of inputting the small sample data of the real image and the target synthetic image data into the corresponding image segmentation network respectively until the first loss value and the second loss value meet the specified conditions.

7. The device according to claim 6, characterized in that The first loss value determination module is specifically used to: For any pixel in the second predicted image data, based on a first predicted object of the pixel in the second predicted image data and a labeled object of the pixel in the target synthetic image labeled data, obtaining an intermediate loss value of the pixel; The first loss value is obtained according to the intermediate loss value corresponding to each pixel point in the second predicted image data.

8. The device according to claim 7, characterized in that The first loss value determination module is specifically used to: The first loss value is obtained by the following formula: Among them, L seg is the first loss value, H is the height of the second predicted image data, W is the width of the second predicted image data, is the first prediction object of the pixel point with the ordinate h and the abscissa w in the second prediction image data, is the annotated object of the pixel point with the ordinate h and the abscissa w in the target synthetic image annotation data, h∈[0,H], w∈[0,W].

9. The device according to claim 6, characterized in that The second loss value determination module is specifically used to: Inputting the first predicted image data into a fully convolutional neural network to obtain a first predicted image feature; and, Inputting the second predicted image data into a fully convolutional neural network to obtain second predicted image features; The second loss value is obtained based on the first predicted image feature and the second predicted image feature.

10. The device according to claim 9, characterized in that The second loss value determination module is specifically used to: The second loss value is obtained by: Obtaining a first prediction value corresponding to the first prediction image data by using a second prediction object of each pixel point in the first prediction image feature corresponding to the first prediction image data; and, Obtaining a second prediction value corresponding to the second prediction image data according to a third prediction object of each pixel point in the second prediction image feature corresponding to the second prediction image data; Based on the first prediction value and the second prediction value, the second loss value is obtained.

11. An electronic device, characterized in that: It comprises at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executed by the at least one processor; and the instructions are executed by the at least one processor so that the at least one processor can execute the method according to any one of claims 1-5.

12. A computer storage medium, characterized in that: The computer storage medium stores a computer program, and the computer program is used to execute the method according to any one of claims 1-5.

Images