Image processing method and device

Abstract

The present invention discloses an image processing method and device, which relate to the field of computer vision technology. A specific implementation of the method includes: obtaining an image to be processed; processing the image to be processed using a pre-trained semantic segmentation model, wherein the semantic segmentation model is based on an iteratively updated set of labeled pictures and a set of unlabeled pictures, and is obtained by iteratively training a fully convolutional neural network, and the iterative updating of the labeled picture set and the unlabeled picture set is synchronized with the iterative training of the fully convolutional neural network; based on the processing results, the pixel points included in the image to be processed are marked; based on the marking results, the target included in the image to be processed is determined. This implementation greatly reduces the labor cost of training an image semantic segmentation model with relatively few labeled samples, while effectively improving the accuracy of image semantic segmentation.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to an image processing method and device. Background Art

[0002] In many scenarios, such as autonomous driving, medical image analysis, and traffic violations, image semantic segmentation technology is needed to process images in order to mark or identify the targets of interest in the scene, such as road obstacles in autonomous driving scenarios and lesions in medical images.

[0003] Currently, image processing based on semantic image segmentation technology relies on training deep learning networks with large numbers of pixel-labeled image samples. However, pixel labels for image samples typically require manual annotation, which consumes significant human resources. Furthermore, labeling errors are unavoidable when labeling large numbers of samples. Due to the large number of labels, these errors are difficult to detect. Consequently, existing semantic image segmentation technology not only consumes a significant amount of labor in the early stages but also suffers from low accuracy. Summary of the Invention

[0004] In view of this, an embodiment of the present invention provides an image processing method and device, which can greatly reduce the manpower cost of training an image semantic segmentation model with relatively few labeled samples, while effectively improving the accuracy of image semantic segmentation and image processing.

[0005] To achieve the above object, according to one aspect of an embodiment of the present invention, there is provided an image processing method, comprising:

[0006] Get the image to be processed;

[0007] Processing the image to be processed using a pre-trained semantic segmentation model, wherein the semantic segmentation model is obtained by iteratively training a fully convolutional neural network based on an iteratively updated set of labeled images and a set of unlabeled images, wherein the iterative updating of the set of labeled images and the set of unlabeled images is synchronized with the iterative training of the fully convolutional neural network;

[0008] Marking the pixels of the image to be processed according to the processing result;

[0009] According to the marking result, the target included in the image to be processed is determined.

[0010] Optionally, processing the image to be processed by using a pre-trained semantic segmentation model includes:

[0011] Flipping the image to be processed horizontally, vertically, and in half respectively to obtain three flipped images to be processed;

[0012] The three flipped images to be processed are input into the trained fully convolutional neural network included in the semantic segmentation model.

[0013] Optionally, the processing of the image to be processed using a pre-trained semantic segmentation model further includes:

[0014] Performing sharpening conversion on the output results of the trained fully convolutional neural network for the three flipped images to be processed to obtain three feature maps to be processed;

[0015] Combining the three feature maps to be processed into a new feature map to be processed;

[0016] Performing horizontal flipping, vertical flipping, and half flipping on the new feature map to be processed, respectively, to obtain three new flipped images to be processed;

[0017] Based on the three flipped images to be processed and three new flipped images to be processed of the image to be processed, the trained fully convolutional neural network is used to calculate the probability that the pixel points in the image to be processed belong to the preset type.

[0018] Optionally, marking the pixels of the image to be processed includes:

[0019] The pixel point is marked according to the calculated probability that the pixel point belongs to a preset type.

[0020] Optionally, combining the three feature maps to be processed into a new feature map to be processed includes:

[0021] Perform weighted averaging on the pixels at corresponding positions of the three feature maps to be processed;

[0022] According to the weighted average result and the position of the pixel points, a new feature map to be processed is generated.

[0023] Optionally, determining the target included in the image to be processed includes:

[0024] According to the correspondence between the preset target and the marked label, the target corresponding to the marked result of the image to be processed is determined.

[0025] Optionally, the above image processing method further includes:

[0026] Iteratively train the fully convolutional neural network to obtain a trained semantic segmentation model, where

[0027] The following operations are performed for each iteration cycle of training the fully convolutional neural network until the iteration stopping condition is met:

[0028] Determining a marked picture set and an unmarked picture set used in a current iteration cycle, wherein the marked picture set includes a plurality of marked pictures, and the unmarked picture set includes a plurality of unmarked pictures;

[0029] Training a fully convolutional neural network using the labeled image set and the unlabeled image set;

[0030] Calculate the model loss based on the training results;

[0031] Adjusting the parameters of the fully convolutional neural network according to the model loss;

[0032] The marked picture set and the unmarked picture set are updated, and the updated marked picture set and the updated unmarked picture set are used as the marked picture set and the unmarked picture set of the next iteration cycle, and the adjusted fully convolutional neural network is used as the fully convolutional neural network of the next iteration cycle.

[0033] Optionally, the training a fully convolutional neural network using the labeled image set and the unlabeled image set includes:

[0034] For each unlabeled picture included in the unlabeled picture set, performing horizontal flipping, vertical flipping, and half flipping on the unlabeled picture to obtain three flipped pictures;

[0035] The three flipped images of each of the unlabeled images and each labeled image included in the labeled image set are input into the fully convolutional neural network.

[0036] Optionally, the calculating model loss includes:

[0037] Calculating a first loss for the labeled picture set and a second loss for the unlabeled picture set respectively;

[0038] The model loss is calculated using the first loss and the second loss.

[0039] Optionally, calculating a first loss for the set of labeled pictures includes:

[0040] A first loss is calculated using labels of labeled images in the labeled image set and outputs of the fully convolutional neural network.

[0041] Optionally, the second loss for the unlabeled picture set includes:

[0042] For each of the unlabeled images, performing a sharpening conversion on an output result of the fully convolutional neural network for the unlabeled image to obtain three feature maps; combining the three feature maps into a new feature map; and horizontally flipping, vertically flipping, and folding the new feature map to obtain three new flipped images;

[0043] A second loss for the set of unlabeled pictures is calculated based on the three flipped pictures of each of the unlabeled pictures and three new flipped pictures.

[0044] Optionally, combining the three feature maps into a new feature map includes:

[0045] Perform weighted averaging on the pixels at corresponding positions of the three feature maps;

[0046] A new feature map is generated based on the weighted average result and the position of the pixel points.

[0047] Optionally, after performing sharpening conversion on the output result of the fully convolutional neural network for the unlabeled image to obtain three feature maps, the method further includes:

[0048] For each of the feature maps, calculating the channel entropy of each channel included in the feature map, according to the channel entropy of each channel;

[0049] Calculate the score of the corresponding unlabeled image according to the channel entropy of the channel;

[0050] The step of selecting a set number of unlabeled pictures is performed according to the scores of the unlabeled pictures.

[0051] Optionally, performing sharpening conversion on an output result of the fully convolutional neural network for the unlabeled image includes:

[0052] Reverse-flipping the three initial feature maps included in the output result of the unlabeled image to obtain three reverse-flipped feature maps;

[0053] For each of the inverse flipped feature maps, channel-level multi-classification is performed on the inverse flipped feature map, and according to the classification result, the flipped feature map is sharpened to obtain a corresponding feature map.

[0054] Optionally, calculating the score of the corresponding unlabeled image includes:

[0055] For each of the feature maps, performing channel entropy analysis based on the channels included in the feature map to generate a two-dimensional map corresponding to the feature map;

[0056] For each unlabeled image, execute:

[0057] Adding corresponding pixels of the two-dimensional images corresponding to the three feature maps of the unlabeled image to obtain a two-dimensional feature map corresponding to the unlabeled image, and determining a probability value included in the two-dimensional feature map of the unlabeled image;

[0058] Calculate the score of the unlabeled image according to the probability value included in the two-dimensional feature map.

[0059] Optionally, updating the marked picture set and the unmarked picture set includes:

[0060] Selecting a set number of unmarked pictures from the unmarked picture set, and marking the selected set number of unmarked pictures;

[0061] The set number of newly marked pictures are added to the marked picture set, and the set number of unmarked pictures are removed from the unmarked picture set.

[0062] In a second aspect, an embodiment of the present invention provides an image processing device, comprising: an acquisition module and a semantic segmentation module, wherein:

[0063] The acquisition module is used to acquire the image to be processed;

[0064] The semantic segmentation module is used to process the image to be processed using a pre-trained semantic segmentation model, wherein the semantic segmentation model is obtained by iteratively training a fully convolutional neural network based on an iteratively updated set of labeled pictures and a set of unlabeled pictures, and the iterative updating of the labeled picture set and the unlabeled picture set is synchronized with the iterative training of the fully convolutional neural network; based on the processing results, the pixels included in the image to be processed are marked; and based on the marking results, the target included in the image to be processed is determined.

[0065] One embodiment of the above invention has the following advantages or beneficial effects: since the semantic segmentation model for processing the image to be processed is trained based on the iteratively updated set of labeled pictures and the set of unlabeled pictures, that is, the samples relied on for training the full convolutional neural network are not all labeled, the labeled picture set and the unlabeled picture set can be synchronously iteratively updated during the iterative training process to improve the accuracy of the semantic segmentation model and reduce the amount of manual labeling. While reducing labor costs, it can effectively improve the accuracy of image semantic segmentation and image processing.

[0066] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] The accompanying drawings are provided for a better understanding of the present invention and are not intended to limit the present invention.

[0068] Figure 1 is a schematic diagram of the main process of the image processing method according to an embodiment of the present invention;

[0069] Figure 2 1 is a schematic diagram of the main flow of the training process of each iterative cycle in the model training process according to an embodiment of the present invention;

[0070] Figure 3 is a schematic diagram of the main process of updating the marked picture set and the unmarked picture set according to an embodiment of the present invention;

[0071] Figure 4 2 is a schematic diagram of the main process of training a fully convolutional neural network using a labeled image set and an unlabeled image set according to an embodiment of the present invention;

[0072] Figure 5 is a schematic diagram of the main process of calculating the second loss according to an embodiment of the present invention;

[0073] Figure 6 is a schematic diagram of the main process of performing sharpening conversion on the output result of an unlabeled image according to an embodiment of the present invention;

[0074] Figure 7 is a schematic diagram of a main process of selecting a set number of unlabeled graphs according to an embodiment of the present invention;

[0075] Figure 8 is a schematic diagram of the main process of calculating the score of an unlabeled image according to an embodiment of the present invention;

[0076] Figure 9 is a schematic diagram of the main flow of an image processing method according to another embodiment of the present invention;

[0077] Figure 10 is a schematic diagram of a model training framework according to an embodiment of the present invention;

[0078] Figure 11 is a schematic diagram of main modules of an image processing apparatus according to an embodiment of the present invention;

[0079] Figure 12 is an exemplary system architecture diagram in which embodiments of the present invention may be applied;

[0080] Figure 13 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0081] The following description of exemplary embodiments of the present invention is made in conjunction with the accompanying drawings, in which various details of the embodiments of the present invention are included to facilitate understanding. These details should be considered as merely exemplary. Therefore, it should be appreciated by those skilled in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0082] Figure 1 FIG. 1 is a schematic diagram of the main flow of an image processing method according to an embodiment of the present invention. Figure 1 As shown, the image processing method may include the following steps:

[0083] Step S101: obtaining an image to be processed;

[0084] Among them, the image to be processed can be an image of the road captured by a camera, which is used to identify pedestrians, obstacles, etc. on the road for autonomous driving; the image to be processed can also be a medical image, etc.

[0085] Step S102: Processing the image to be processed using a pre-trained semantic segmentation model, wherein the semantic segmentation model is obtained by iteratively training a fully convolutional neural network based on an iteratively updated set of labeled images and a set of unlabeled images, wherein the iterative updating of the set of labeled images and the set of unlabeled images is synchronized with the iterative training of the fully convolutional neural network;

[0086] The labeled images in the labeled image set and the unlabeled images in the unlabeled image set generally match the type of image to be processed. For example, if the image to be processed is a road image, the labeled and unlabeled images are generally also collected road images. For another example, if the image to be processed is a medical image, the labeled and unlabeled images are also historical medical images.

[0087] The iterative updating of the labeled image set and the unlabeled image set and the iterative training of the full convolutional neural network in synchronization generally refers to updating the labeled image set and the unlabeled image set during each iterative training of the full convolutional neural network.

[0088] Step S103: marking the pixels included in the image to be processed according to the processing result;

[0089] The marking mainly marks the pixels with labels according to the correspondence between the preset pixel types and labels. For example, the pixel types are pedestrians, vehicles, curbs, trees, etc., and the corresponding labels are: pedestrian label 1, vehicle label 2, curb label 3, tree label 4, etc., then the pixels can be marked by pedestrian label 1, vehicle label 2, curb label 3, tree label 4, etc.

[0090] Step S104: Determine the target included in the image to be processed according to the marking result.

[0091] The specific implementation of step S104 may include: determining the target corresponding to the labeled result of the image to be processed based on the correspondence between the preset target and the labeled label. For example, if the label is 1, the target is a pedestrian; if the label is 2, the target is a vehicle, etc.

[0092] exist Figure 1 In the embodiment shown, since the semantic segmentation model for processing the image to be processed is trained based on the iteratively updated set of labeled pictures and the set of unlabeled pictures, that is, the samples relied on for training the fully convolutional neural network are not all labeled, the labeled picture set and the unlabeled picture set can be synchronously iteratively updated during the iterative training process to improve the accuracy of the semantic segmentation model and reduce the amount of manual labeling. While reducing labor costs, it can effectively improve the accuracy of image semantic segmentation and image processing.

[0093] In an embodiment of the present invention, the core of the image processing method is a pre-trained semantic segmentation model. Accordingly, the image processing method may include iteratively training a fully convolutional neural network to obtain a semantic segmentation model. Figure 2 As shown, the following steps S201 to S205 are performed for each iteration cycle until the iteration stop condition is met:

[0094] Step S201: determining a marked picture set and an unmarked picture set used in the current iteration cycle, wherein the marked picture set includes a plurality of marked pictures, and the unmarked picture set includes a plurality of unmarked pictures;

[0095] In the case where the current iteration cycle is the first iteration cycle, the marked picture set and the unmarked picture set are the initially set marked picture set L t and the unlabeled image collection U t .

[0096] In the case that the current iteration cycle is not the first iteration cycle, the marked picture set and the unmarked picture set are the marked picture set and the unmarked picture set updated in the previous iteration cycle of the current iteration cycle.

[0097] Step S202: training a fully convolutional neural network using the labeled image set and the unlabeled image set;

[0098] Step S203: Calculate the model loss based on the training results;

[0099] The model loss can reflect the training bias of the trained model to adjust the fully convolutional neural network through the model loss.

[0100] Step S204: Adjust the parameters of the fully convolutional neural network according to the model loss;

[0101] Step S205: Update the marked picture set and the unmarked picture set, and use the updated marked picture set and the updated unmarked picture set as the marked picture set and the unmarked picture set of the next iteration cycle, and use the adjusted full convolutional neural network as the full convolutional neural network of the next iteration cycle.

[0102] The specific implementation of step S203 may include: separately calculating a first loss for the labeled image set and a second loss for the unlabeled image set; and calculating the model loss using the first loss and the second loss. Because the training processes and training results for the labeled image set and the unlabeled image set differ to some extent, separately calculating the first loss for the labeled image set and the second loss for the unlabeled image set can make the calculation of the model loss more accurate.

[0103] The specific implementation of calculating the first loss for the labeled image set may include: calculating the first loss using the labels of the labeled images in the labeled image set and the output of the fully convolutional neural network.

[0104] Specifically, the first loss can be calculated using the following calculation formula (1).

[0105]

[0106] Among them, L ce Characterize the first loss; Represents the label of the labeled image in the labeled image set; logM(L)(h,w,c) represents the output of the model; h,w,c represent the three channels of the labeled image.

[0107] Channels are the length, width, and height of the labeled image features. h and w are related to the convolution size and can be set. c represents the number of image categories. Generally, there are as many c's as there are categories in a training task. For example, if the training targets are cats, dogs, and pedestrians, then c is set to 3.

[0108] After step S204, the process may further include determining whether the adjusted fully convolutional neural network meets an iteration stopping condition. If so, the iteration is terminated; otherwise, step S205 is executed. The iteration stopping condition may be that the number of iterations reaches a preset iteration number threshold, or that the model loss is not greater than a preset loss threshold.

[0109] In the embodiment of the present invention, Figure 3 As shown, the specific implementation of updating the marked picture set and the unmarked picture set may include:

[0110] Step S301: selecting a set number of unmarked pictures from an unmarked picture set, and marking the selected set number of unmarked pictures;

[0111] Step S302: adding a set number of newly marked pictures to the marked picture set, and removing a set number of unmarked pictures from the unmarked picture set.

[0112] For example, from the unlabeled image collection U t Select a set number of unlabeled pictures B to be labeled, and the updated unlabeled picture set is the unlabeled picture set U of the next iteration cycle. t+1 =U t -B, accordingly, the updated set of labeled images is the set of labeled images L in the next iteration cycle t+1 =L t +B.

[0113] By adding new labeled images in new iteration cycles, the accuracy of the trained model can be further improved.

[0114] In the embodiment of the present invention, Figure 4 As shown, the specific implementation of training a fully convolutional neural network using a set of labeled images and a set of unlabeled images may include the following steps:

[0115] Step S401: for each unlabeled image included in the unlabeled image set, perform horizontal flipping, vertical flipping, and half flipping on the unlabeled image to obtain three flipped images;

[0116] A horizontal flip is a 180-degree flip of the image from left to right or right to left; a vertical flip is a 180-degree flip from top to bottom or bottom to top; and a half-turn flip is a 180-degree rotation around the image's center. For example, the flipped image of an unlabeled image T after a horizontal flip is Ts, the flipped image of an unlabeled image T after a vertical flip is Tc, and the flipped image of an unlabeled image T after a half-turn is Td.

[0117] Step S402: Input the three flipped images of each unlabeled image and each labeled image included in the labeled image set into a fully convolutional neural network.

[0118] By inputting the flipped image into the fully convolutional neural network, the training samples can be increased, and the fully convolutional neural network can be trained from different angles of the image to further effectively improve the accuracy of the training results.

[0119] In the embodiment of the present invention, Figure 5 As shown, the specific implementation of calculating the second loss may further include:

[0120] For each unlabeled image, execute steps S501 to S503:

[0121] S501: performing a sharpening conversion on the output of the full convolutional neural network for the unlabeled image to obtain three feature maps;

[0122] For unlabeled images, since the input to the fully convolutional neural network is three flipped images, the output result for the unlabeled image is the output result for each flipped image. Then, the output result of each flipped image is sharpened to obtain a corresponding feature map. The process of sharpening the output result of each flipped image can be implemented by the following formula (2):

[0123]

[0124] in, Represents the sharpened result corresponding to the i-th channel; p i represents the features corresponding to the i-th channel; T represents the sharpening hyperparameter, the smaller T is, the higher the degree of sharpening; p j Represents the features corresponding to the j-th channel; i is any value from 1 to the number of image categories c; the value range of j is 1 to the number of image categories c.

[0125] S502: Combining the three feature maps into a new feature map;

[0126] S503: Flip the new feature map horizontally, vertically, and in half to obtain three new flipped images;

[0127] S504: Calculate a second loss for the unlabeled picture set based on the three flipped pictures of each unlabeled picture and the three new flipped pictures.

[0128] For example, by flipping the unlabeled image horizontally, vertically, and in half, we get three flipped images: and The above step S503 obtains three new flipped pictures and In order to make full use of the self-supervisory information, this step calculates the second loss by calculating formula (3):

[0129]

[0130] Among them, L mse Characterize the second loss; and Respectively represent flipped images and the new flip image The probability that the position (i,j) of (k=1,2,3) is of type c.

[0131] Accordingly, the model loss can be calculated by the following formula 4:

[0132] L total =L ce +L mse (4)

[0133] Among them, L total Characterize the model loss; L ce Characterizes the first loss; L mse Characterize the second loss.

[0134] Among them, the specific implementation method of combining three feature maps into a new feature map may include: performing weighted averaging of pixel points at corresponding positions of the three feature maps; and generating a new feature map based on the result of the weighted averaging and the positions of the pixel points.

[0135] In the embodiment of the present invention, Figure 6 As shown, a specific implementation method of sharpening the output result of the fully convolutional neural network for the unlabeled image may include the following steps:

[0136] Step S601: reverse-flipping the three initial feature maps included in the output result of the unlabeled image to obtain three reverse-flipped feature maps;

[0137] The reverse flipping refers to reverse flipping for horizontal flipping, vertical flipping and folding flipping. For example, horizontal flipping is a 180-degree rotation from left to right, and correspondingly, reverse flipping is a 180-degree rotation from right to left. The output results of the unlabeled image include three initial feature maps corresponding to the horizontally flipped image, the vertically flipped image and the folded flipped image. Accordingly, the initial feature image of the horizontally flipped image is reverse flipped for the horizontal flip; the initial feature image of the vertically flipped image is reverse flipped for the vertical flip; and the initial feature image of the folded flipped image is reverse flipped for the folded flip.

[0138] Step S602: For each inverse flipped feature map, perform channel-level multi-classification on the inverse flipped feature map, and sharpen the inverse flipped feature map according to the classification result to obtain the corresponding feature map.

[0139] By reverse flipping, different pixels in the feature map can be better classified, effectively improving the accuracy of classification.

[0140] In the embodiment of the present invention, Figure 7 As shown, in order to select a set number of unlabeled images, after the above step S501, the following steps may be further included:

[0141] Step S701: for each feature map, calculating the channel entropy of each channel included in the feature map, according to the channel entropy of each channel;

[0142] This step can be calculated using the following formula (5).

[0143]

[0144] in, It is the probability value of class c at the (i, j) position of the feature map. It is the probability value of class c at position (i, j) of the feature map after inverse flipping and sharpening; C represents the total number of types.

[0145] Step S702: Calculate the score of the corresponding unlabeled image according to the channel entropy of the channel;

[0146] This step mainly involves: adding all channels of each feature map into a two-dimensional map, then adding the three two-dimensional maps according to the corresponding pixels to obtain a two-dimensional feature map, and adding all the probability values ​​on the obtained two-dimensional feature map to obtain a score, which is the score of the type in the corresponding unlabeled image.

[0147] Step S703: selecting a set number of unlabeled pictures according to the scores of the unlabeled pictures.

[0148] By selecting unlabeled images with scores not lower than the preset score threshold or unlabeled images ranked in the top N scores, since unlabeled images with higher scores have a greater impact on training, unlabeled images are selected based on the score to increase the number of expanded labeled images, while ensuring that the labeled images can effectively improve the accuracy of the trained model.

[0149] Among them, such as Figure 8 As shown, the specific implementation of the above step S702 may include the following steps:

[0150] Step S801: for each feature map, executing channel entropy according to the channels included in the feature map to generate a two-dimensional map corresponding to the feature map;

[0151] Step S802: For each unlabeled image, add the corresponding pixels of the two-dimensional images corresponding to the three feature maps of the unlabeled image to obtain a two-dimensional feature map corresponding to the unlabeled image, and determine the probability value included in the two-dimensional feature map of the unlabeled image; calculate the score of the unlabeled image based on the probability value included in the two-dimensional feature map.

[0152] In order to obtain the target included in the image to be processed more accurately, the specific implementation method of processing the image to be processed based on the semantic segmentation model obtained in the above training process may include: flipping the image to be processed horizontally, vertically and in half to obtain three flipped images to be processed; and inputting the three flipped images to be processed into the trained fully convolutional neural network included in the semantic segmentation model.

[0153] More specifically, the specific implementation of the semantic segmentation model processing the image to be processed may further include: performing a sharpening conversion on the output results of the trained full convolutional neural network for the three flipped images to be processed to obtain three feature maps to be processed; combining the three feature maps to be processed into a new feature map to be processed; performing horizontal flipping, vertical flipping, and half flipping on the new feature map to be processed respectively to obtain three new flipped images to be processed; based on the three flipped images to be processed and the three new flipped images to be processed of the image to be processed, calculating the probability that the pixels in the image to be processed belong to a preset type; accordingly, marking the pixels included in the image to be processed may include: marking the pixels according to the calculated probability that the pixels belong to the preset type. The probability that the pixels in the image to be processed belong to the preset type may be calculated by the above calculation formula (5).

[0154] Specifically, the above-mentioned combining the three feature maps to be processed into a new feature map to be processed may include: performing weighted averaging on the pixels at corresponding positions of the three feature maps to be processed; and generating a new feature map to be processed according to the result of the weighted averaging and the positions of the pixels.

[0155] The following describes the implementation of the image processing method in detail using a specific embodiment that includes both the training process and the image semantic segmentation process. Figure 9 As shown, the image processing method may include the following steps:

[0156] The following steps S901 to S920 are performed for each iteration cycle until the iteration stop condition is met:

[0157] Step S901: determining a marked picture set and an unmarked picture set used in the current iteration cycle, wherein the marked picture set includes a plurality of marked pictures, and the unmarked picture set includes a plurality of unmarked pictures;

[0158] Step S902: for each unlabeled picture included in the unlabeled picture set, perform horizontal flipping, vertical flipping, and half flipping on the unlabeled picture to obtain three flipped pictures;

[0159] The three flipped images obtained are horizontal flipped Vertically flipped Fold and flip

[0160] Step S903: inputting the three flipped images of each unlabeled image and each labeled image included in the labeled image set into a fully convolutional neural network;

[0161] For each unlabeled image, execute steps S904 to S908:

[0162] Step S904: reverse-flipping the three initial feature maps included in the output result of the unlabeled image to obtain three reverse-flipped feature maps;

[0163] Step S905: For each inverse flipped feature map, perform channel-level multi-classification on the inverse flipped feature map, and sharpen the flipped feature map according to the classification results to obtain the corresponding feature map. After obtaining three feature maps, execute steps S906 and S919 respectively.

[0164] Step S906: performing weighted averaging on the pixels at corresponding positions of the three feature maps;

[0165] Step S907: Generate a new feature map based on the weighted average result and the positions of the pixels;

[0166] Step S908: Flip the new feature map horizontally, vertically, and in half to obtain three new flipped images;

[0167] The three reverse flipped feature maps obtained

[0168] Step S909: Calculate a first loss using the labels of the labeled images in the labeled image set and the output of the fully convolutional neural network;

[0169] Step S910: Calculate a second loss for the unlabeled image set based on the three flipped images of each unlabeled image and the three new flipped images;

[0170] Step S911: Calculate the model loss using the first loss and the second loss;

[0171] Step S912: Adjust the parameters of the fully convolutional neural network according to the model loss;

[0172] Step S913: Determine whether the adjusted fully convolutional neural network meets the iteration stop condition. If yes, end the iteration; otherwise, execute step S920;

[0173] Step S914: for each feature map obtained in step S905, calculating the channel entropy of each channel included in the feature map, according to the channel entropy of each channel;

[0174] Step S915: for each feature map, executing channel entropy according to the channels included in the feature map to generate a two-dimensional map corresponding to the feature map;

[0175] Step S916: For each unlabeled image, add the corresponding pixels of the two-dimensional images corresponding to the three feature maps of the unlabeled image to obtain a two-dimensional feature map corresponding to the unlabeled image, and determine the probability value included in the two-dimensional feature map of the unlabeled image;

[0176] Step S917: Calculate the score of the unlabeled image based on the probability value included in the two-dimensional feature map;

[0177] Step S918: selecting a set number of unlabeled pictures according to the scores of the unlabeled pictures, and labeling the selected set number of unlabeled pictures;

[0178] Step S919: adding a set number of newly marked pictures to the marked picture set, and removing a set number of unmarked pictures from the unmarked picture set;

[0179] Step S920: Using the updated labeled image set and the updated unlabeled image set as the labeled image set and the unlabeled image set for the next iteration cycle, and using the adjusted fully convolutional neural network as the fully convolutional neural network for the next iteration cycle, and executing step S901;

[0180] like Figure 10 As shown in , the above training process can be divided into a training phase and an unlabeled image selection phase.

[0181] Training phase: unlabeled images U t Perform horizontal flip, vertical flip, and fold flip to get the flipped image The image after flipping the unlabeled image And the labeled images are input to the model M t , calculate the first loss L for the labeled image set ce ; Model M for flipped images of unlabeled images t The output of Perform a series of transformations such as sharpening transformation, pixel weighted averaging and inverse flipping to obtain three new flipped images corresponding to each unlabeled image. and By Model M t The output of And three new flip images and Calculate the second loss L mse .

[0182] Selection stage (selecting the pictures to be labeled from the unlabeled pictures): Unlabeled pictures U t Perform horizontal flip, vertical flip, and fold flip to obtain the flipped image, and input the flipped image of the unlabeled image and the labeled image into the model M t , for model M t The output results of are reversed and three reversed feature maps are obtained. Perform sharpening and channel-level multi-classification on the inverse flipped feature map to obtain the corresponding feature map Calculate the channel entropy of the sharpened feature map to generate a two-dimensional map corresponding to the feature map; add the corresponding pixels of the two-dimensional maps corresponding to the three feature maps of the unlabeled image to obtain a two-dimensional feature map corresponding to the unlabeled image. All probability values ​​on the resulting two-dimensional feature map are summed to obtain a score, which is the score of the corresponding type in the unlabeled image. Based on the score, the image B to be labeled is selected, and the labeled image set and the unlabeled image set are updated based on the labeled image B.

[0183] Step S921: Acquire an image to be processed;

[0184] Step S922: flipping the image to be processed horizontally, vertically, and in half, respectively, to obtain three flipped images to be processed;

[0185] Step S923: inputting the three flipped images to be processed into the trained fully convolutional neural network included in the semantic segmentation model;

[0186] Step S924: performing a sharpening conversion on the output results of the trained fully convolutional neural network for the three flipped images to be processed, to obtain three feature maps to be processed;

[0187] Step S925: performing weighted averaging on the pixels at corresponding positions of the three feature maps to be processed;

[0188] Step S926: Generate a new feature map to be processed based on the weighted average result and the position of the pixel points;

[0189] Step S927: performing horizontal flipping, vertical flipping, and half flipping on the new feature map to be processed, respectively, to obtain three new flipped images to be processed;

[0190] Step S928: Calculating the probability that a pixel point in the image to be processed belongs to a preset type based on the three flipped images to be processed and the three new flipped images to be processed;

[0191] Step S929: Marking the pixel points according to the calculated probability that the pixel points belong to the preset type;

[0192] Step S930: Determine the target corresponding to the marking result of the image to be processed according to the correspondence between the preset target and the marked label.

[0193] like Figure 11 As shown, an embodiment of the present invention provides an image processing device 1100, which may include: an acquisition module 1101 and a semantic segmentation module 1102, wherein:

[0194] An acquisition module 1101 is used to acquire an image to be processed;

[0195] The semantic segmentation module 1102 is used to process the image to be processed using a pre-trained semantic segmentation model, wherein the semantic segmentation model is based on an iteratively updated set of labeled images and a set of unlabeled images, and is obtained by iteratively training a fully convolutional neural network, and the iterative updating of the labeled image set and the unlabeled image set is synchronized with the iterative training of the fully convolutional neural network; based on the processing results, the pixels included in the image to be processed are marked; based on the marking results, the target included in the image to be processed is determined.

[0196] In the embodiment of the present invention, the semantic segmentation module 1102 is further configured to determine the target corresponding to the labeling result of the image to be processed according to the correspondence between the preset target and the label.

[0197] In an embodiment of the present invention, the semantic segmentation module 1102 is further used to flip the image to be processed horizontally, vertically, and in half, respectively, to obtain three flipped images to be processed; and the three flipped images to be processed are input into the trained fully convolutional neural network included in the semantic segmentation model.

[0198] In an embodiment of the present invention, the semantic segmentation module 1102 is further used to perform a sharpening conversion on the output results of the trained fully convolutional neural network for the three flipped images to be processed to obtain three feature maps to be processed; combine the three feature maps to be processed into a new feature map to be processed; perform horizontal flipping, vertical flipping and half flipping on the new feature map to be processed respectively to obtain three new flipped images to be processed; based on the three flipped images to be processed and the three new flipped images to be processed, calculate the probability that the pixel points in the image to be processed belong to a preset type; and mark the pixel points according to the calculated probability that the pixel points belong to the preset type.

[0199] In an embodiment of the present invention, the semantic segmentation module 1102 is further configured to perform weighted averaging of pixels at corresponding positions of the three feature maps to be processed; and generate a new feature map to be processed based on the result of the weighted averaging and the positions of the pixels.

[0200] In the embodiment of the present invention, Figure 11 As shown, the image processing device 1100 may further include: a model training module 1103, wherein:

[0201] The model training module 1103 is used to iteratively train the fully convolutional neural network to obtain a trained semantic segmentation model, wherein the following operations are performed for each iteration cycle until the iteration stopping condition is met:

[0202] Determine a marked picture set and an unmarked picture set used in a current iteration cycle, wherein the marked picture set includes a plurality of marked pictures, and the unmarked picture set includes a plurality of unmarked pictures;

[0203] Train a fully convolutional neural network using a collection of labeled and unlabeled images;

[0204] Calculate the model loss based on the training results;

[0205] Adjust the parameters of the fully convolutional neural network based on the model loss;

[0206] Update the labeled image set and the unlabeled image set, and use the updated labeled image set and the updated unlabeled image set as the labeled image set and the unlabeled image set of the next iteration cycle, and use the adjusted fully convolutional neural network as the fully convolutional neural network of the next iteration cycle.

[0207] In an embodiment of the present invention, the model training module 1103 is further used to perform horizontal flipping, vertical flipping, and half flipping on each unlabeled image included in the unlabeled image set to obtain three flipped images; and input the three flipped images of each unlabeled image and each labeled image included in the labeled image set into a fully convolutional neural network.

[0208] In an embodiment of the present invention, the model training module 1103 is further configured to respectively calculate a first loss for the labeled image set and a second loss for the unlabeled image set; and calculate the model loss using the first loss and the second loss.

[0209] In an embodiment of the present invention, the model training module 1103 is further used to select a set number of unlabeled pictures from the unlabeled picture set to label the selected set number of unlabeled pictures; add the newly labeled set number of pictures to the labeled picture set, and remove the set number of unlabeled pictures from the unlabeled picture set.

[0210] In an embodiment of the present invention, the model training module 1103 is further configured to calculate a first loss using the labels of the labeled images in the labeled image set and the output of the fully convolutional neural network.

[0211] In an embodiment of the present invention, the model training module 1103 is further configured to perform a sharpening conversion on the output of the fully convolutional neural network for each unlabeled image to obtain three feature maps; combine the three feature maps into a new feature map; and perform horizontal flipping, vertical flipping, and half flipping on the new feature map to obtain three new flipped images.

[0212] The second loss for the set of unlabeled images is calculated based on the three flipped images of each unlabeled image and the three new flipped images.

[0213] In an embodiment of the present invention, the model training module 1103 is further used to perform weighted averaging of the pixels at corresponding positions of the three feature maps; and generate a new feature map based on the result of the weighted averaging and the positions of the pixels.

[0214] In an embodiment of the present invention, the model training module 1103 is further used to reversely flip the three initial feature maps included in the output results of the unlabeled image to obtain three reversed feature maps; for each reversed feature map, perform channel-level multi-classification on the reversed feature map, and sharpen the flipped feature map according to the classification results to obtain the corresponding feature map.

[0215] In an embodiment of the present invention, the model training module 1103 is further used to calculate the channel entropy of each channel included in the feature map for each feature map; based on the channel entropy of each channel, calculate the score of the corresponding unlabeled image; and based on the score of the unlabeled image, perform the step of selecting a set number of unlabeled images.

[0216] In an embodiment of the present invention, the model training module 1103 is further used to execute, for each feature map, channel entropy of the channels included in the feature map to generate a two-dimensional map corresponding to the feature map; for each unlabeled image, add the corresponding pixel points of the two-dimensional maps corresponding to the three feature maps of the unlabeled image to obtain a two-dimensional feature map corresponding to the unlabeled image, and determine the probability value included in the two-dimensional feature map of the unlabeled image; and calculate the score of the unlabeled image based on the probability value included in the two-dimensional feature map.

[0217] Figure 12 An exemplary system architecture 1200 is shown to which the image processing method or image processing apparatus according to the embodiment of the present invention can be applied.

[0218] like Figure 12 As shown, system architecture 1200 may include terminal devices 1201, 1202, 1203, a camera 1204, a network 1205, and a server 1206. Network 1205 is used as a medium for providing communication links between terminal devices 1201, 1202, 1203, the camera 1204, and the server 1206. Network 1205 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0219] Terminal devices 1201, 1202, 1203 and camera 1204 interact with server 1206 via network 1205 to receive or send messages, etc. Various client applications may be installed on terminal devices 1201, 1202, 1203, such as image display applications, web browser applications, etc. (only for example).

[0220] The terminal devices 1201 , 1202 , and 1203 may be various electronic devices with display screens and supporting web browsing, including but not limited to medical imaging equipment, vehicle-mounted displays, smart phones, tablet computers, laptop computers, and desktop computers, etc.

[0221] Server 1206 may be a server that provides various services, such as a background management server (for example only) that provides support for images sent by terminal devices 1201, 1202, 1203, and camera 1204. The background management server may analyze and process the received images and feed back the processing results (for example, images marked with objects - for example only) to the terminal devices.

[0222] It should be noted that the image processing method provided in the embodiment of the present invention is generally executed by the server 1206 , and accordingly, the image processing device is generally set in the server 1206 .

[0223] It should be understood that Figure 12 The number of terminal devices, cameras, networks, and servers in the embodiment is merely illustrative. Any number of terminal devices, cameras, networks, and servers may be used as needed.

[0224] Reference below Figure 13 , which shows a schematic structural diagram of a computer system 1300 of a terminal device suitable for implementing an embodiment of the present invention. Figure 13 The terminal device or server shown is only an example and should not limit the functions and scope of use of the embodiments of the present invention.

[0225] like Figure 13 As shown, the computer system 1300 includes a central processing unit (CPU) 1301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1302 or a program loaded from a storage unit 1308 into a random access memory (RAM) 1303. Various programs and data required for the operation of the system 1300 are also stored in the RAM 1303. The CPU 1301, the ROM 1302, and the RAM 1303 are connected to each other via a bus 1304. An input / output (I / O) interface 1305 is also connected to the bus 1304.

[0226] The following components are connected to the I / O interface 1305: an input section 1306 including a keyboard, a mouse, and the like; an output section 1307 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and speakers; a storage section 1308 including a hard disk; and a communication section 1309 including a network interface card such as a LAN card or a modem. The communication section 1309 performs communication processing via a network such as the Internet. A drive 1310 is also connected to the I / O interface 1305 as needed. Removable media 1311, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 1310 as needed, so that computer programs read therefrom can be installed into the storage section 1308 as needed.

[0227] In particular, according to the embodiments disclosed in the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 1309, and / or installed from a removable medium 1311. When the computer program is executed by the central processing unit (CPU) 1301, the above-mentioned functions defined in the system of the present invention are performed.

[0228] It should be noted that the computer-readable medium described in the present invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media can include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. This propagated data signal can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wireline, optical fiber cable, RF, or any suitable combination thereof.

[0229] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0230] The modules involved in the embodiments of the present invention may be implemented in software or hardware. The modules described may also be provided in a processor. For example, they may be described as: a processor including an acquisition module and a semantic segmentation module. The names of these modules do not, in some cases, limit the modules themselves. For example, the acquisition module may also be described as a "module for acquiring an image to be processed."

[0231] As another aspect, the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently and not be assembled into the device. The computer-readable medium carries one or more programs, and when the one or more programs are executed by a device, the device includes the following steps: obtaining an image to be processed; processing the image to be processed using a pre-trained semantic segmentation model, wherein the semantic segmentation model is obtained by iteratively training a fully convolutional neural network based on an iteratively updated set of labeled images and a set of unlabeled images, and the iterative updating of the labeled image set and the unlabeled image set is synchronized with the iterative training of the fully convolutional neural network; labeling the pixels included in the image to be processed based on the processing results; and determining the target included in the image to be processed based on the labeling results.

[0232] According to the technical solution of the embodiments of the present invention, since the semantic segmentation model for processing the image to be processed is trained based on the iteratively updated set of labeled pictures and the set of unlabeled pictures, that is, the samples relied on for training the fully convolutional neural network are not all labeled, the labeled picture set and the unlabeled picture set can be synchronously iteratively updated during the iterative training process to improve the accuracy of the semantic segmentation model and reduce the amount of manual labeling. This can effectively improve the accuracy of image semantic segmentation while reducing labor costs.

[0233] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. An image processing method, characterized in that: include: Get the image to be processed; Processing the image to be processed using a pre-trained semantic segmentation model, wherein the semantic segmentation model is obtained by iteratively training a fully convolutional neural network based on an iteratively updated set of labeled images and a set of unlabeled images, wherein the iterative updating of the set of labeled images and the set of unlabeled images is synchronized with the iterative training of the fully convolutional neural network; Marking the pixels of the image to be processed according to the processing result; Determining the target included in the image to be processed according to the marking result; Also includes: Iteratively train the fully convolutional neural network to obtain a trained semantic segmentation model, where The following operations are performed for each iteration cycle of training the fully convolutional neural network until the iteration stopping condition is met: Determining a marked picture set and an unmarked picture set used in a current iteration cycle, wherein the marked picture set includes a plurality of marked pictures, and the unmarked picture set includes a plurality of unmarked pictures; Training a fully convolutional neural network using the labeled image set and the unlabeled image set; According to the training results, respectively calculating a first loss for the set of labeled pictures and a second loss for the set of unlabeled pictures, and calculating a model loss using the first loss and the second loss; Adjusting the parameters of the fully convolutional neural network according to the model loss; Updating the labeled picture set and the unlabeled picture set, and using the updated labeled picture set and the updated unlabeled picture set as the labeled picture set and the unlabeled picture set of the next iteration cycle, and using the adjusted fully convolutional neural network as the fully convolutional neural network of the next iteration cycle; Calculating the second loss for the unlabeled image set includes: For each of the unlabeled images, performing a sharpening conversion on an output result of the fully convolutional neural network for the unlabeled image to obtain three feature maps; combining the three feature maps into a new feature map; and horizontally flipping, vertically flipping, and folding the new feature map to obtain three new flipped images; Based on three flipped images of each of the unlabeled images and three new flipped images, a second loss for the unlabeled image set is calculated; the three flipped images of the unlabeled images are obtained by flipping the unlabeled images horizontally, vertically, and in half, respectively.

2. The image processing method according to claim 1, wherein: The processing of the image to be processed by using a pre-trained semantic segmentation model includes: Flipping the image to be processed horizontally, vertically, and in half respectively to obtain three flipped images to be processed; The three flipped images to be processed are input into the trained fully convolutional neural network included in the semantic segmentation model.

3. The image processing method according to claim 2, wherein: The method further includes: processing the image to be processed by using a pre-trained semantic segmentation model; Performing sharpening conversion on the output results of the trained fully convolutional neural network for the three flipped images to be processed to obtain three feature maps to be processed; Combining the three feature maps to be processed into a new feature map to be processed; Performing horizontal flipping, vertical flipping, and half flipping on the new feature map to be processed, respectively, to obtain three new flipped images to be processed; Based on the three flipped images to be processed and three new flipped images to be processed of the image to be processed, the trained fully convolutional neural network is used to calculate the probability that the pixel points in the image to be processed belong to the preset type.

4. The image processing method according to claim 3, wherein: The marking of the pixels included in the image to be processed includes: The pixel point is marked according to the calculated probability that the pixel point belongs to a preset type.

5. The image processing method according to claim 3, wherein: Combining the three feature maps to be processed into a new feature map to be processed includes: Perform weighted averaging on the pixels at corresponding positions of the three feature maps to be processed; According to the weighted average result and the position of the pixel points, a new feature map to be processed is generated.

6. The image processing method according to claim 1, wherein: The determining of the target included in the image to be processed includes: According to the correspondence between the preset target and the marked label, the target corresponding to the marked result of the image to be processed is determined.

7. The image processing method according to claim 1, wherein: The training of a fully convolutional neural network using the labeled image set and the unlabeled image set includes: For each unlabeled picture included in the unlabeled picture set, performing horizontal flipping, vertical flipping, and half flipping on the unlabeled picture to obtain three flipped pictures; The three flipped images of each of the unlabeled images and each labeled image included in the labeled image set are input into the fully convolutional neural network.

8. The image processing method according to claim 1, wherein: The calculating a first loss for the set of labeled pictures includes: A first loss is calculated using labels of labeled images in the labeled image set and outputs of the fully convolutional neural network.

9. The image processing method according to claim 1, wherein: Combining the three feature maps into a new feature map includes: Perform weighted averaging on the pixels at corresponding positions of the three feature maps; A new feature map is generated based on the weighted average result and the position of the pixel points.

10. The image processing method according to claim 1, wherein: After performing sharpening conversion on the output result of the fully convolutional neural network for the unlabeled image to obtain three feature maps, the method further includes: For each of the feature maps, calculating the channel entropy of each channel included in the feature map, according to the channel entropy of each channel; Calculate the score of the corresponding unlabeled image according to the channel entropy of the channel; A set number of unlabeled images are selected according to the scores of the unlabeled images.

11. The image processing method according to claim 1 or 10, characterized in that: The sharpening conversion of the output result of the fully convolutional neural network for the unlabeled image includes: Reverse-flipping the three initial feature maps included in the output result of the unlabeled image to obtain three reverse-flipped feature maps; For each of the inverse flipped feature maps, channel-level multi-classification is performed on the inverse flipped feature map, and according to the classification result, the flipped feature map is sharpened to obtain a corresponding feature map.

12. The image processing method according to claim 10, wherein: The calculating the score of the corresponding unlabeled image includes: For each of the feature maps, performing channel entropy analysis based on the channels included in the feature map to generate a two-dimensional map corresponding to the feature map; For each unlabeled image, execute: Adding corresponding pixels of the two-dimensional images corresponding to the three feature maps of the unlabeled image to obtain a two-dimensional feature map corresponding to the unlabeled image, and determining a probability value included in the two-dimensional feature map of the unlabeled image; Calculate the score of the unlabeled image according to the probability value included in the two-dimensional feature map.

13. The image processing method according to claim 1, wherein: The updating of the marked picture set and the unmarked picture set includes: Selecting a set number of unmarked pictures from the unmarked picture set, and marking the selected set number of unmarked pictures; The set number of newly marked pictures are added to the marked picture set, and the set number of unmarked pictures are removed from the unmarked picture set.

14. An image processing device, characterized in that: include: Acquisition module, semantic segmentation module and model training module, among which, The acquisition module is used to acquire the image to be processed; The semantic segmentation module is configured to process the image to be processed using a pre-trained semantic segmentation model, wherein the semantic segmentation model is obtained by iteratively training a fully convolutional neural network based on an iteratively updated set of labeled images and a set of unlabeled images, wherein the iterative updating of the labeled image set and the unlabeled image set is synchronized with the iterative training of the fully convolutional neural network; based on the processing results, the pixels included in the image to be processed are labeled; and based on the labeling results, the objects included in the image to be processed are determined; The model training module is used to iteratively train the fully convolutional neural network to obtain a trained semantic segmentation model, wherein the following operations are performed for each iteration cycle until the iteration stopping condition is met: Determine a marked picture set and an unmarked picture set used in a current iteration cycle, wherein the marked picture set includes a plurality of marked pictures, and the unmarked picture set includes a plurality of unmarked pictures; Train a fully convolutional neural network using a collection of labeled and unlabeled images; According to the training results, respectively calculating a first loss for the set of labeled pictures and a second loss for the set of unlabeled pictures, and calculating a model loss using the first loss and the second loss; Adjust the parameters of the fully convolutional neural network based on the model loss; Update the labeled image set and the unlabeled image set, and use the updated labeled image set and the updated unlabeled image set as the labeled image set and the unlabeled image set of the next iteration cycle, and use the adjusted full convolutional neural network as the full convolutional neural network of the next iteration cycle; The model training module is further used to perform a sharpening conversion on the output result of the fully convolutional neural network for each of the unlabeled images to obtain three feature maps; combine the three feature maps into a new feature map; flip the new feature map horizontally, vertically, and in half to obtain three new flipped images; based on the three flipped images of each of the unlabeled images and the three new flipped images, calculate the second loss for the unlabeled image set; the three flipped images of the unlabeled images are obtained by flipping the unlabeled images horizontally, vertically, and in half.

15. An image processing electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 13.

16. A computer-readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 13 is implemented.

Images