Method, device and computer readable storage medium for image processing

Abstract

Embodiments of the present disclosure relate to a method, device and computer readable storage medium for image processing. A method of image processing includes obtaining a plurality of images, each of which has initial semantic segmentation labels indicating semantic categories of pixels in the image; obtaining a plurality of image masks corresponding to the plurality of images, each of which is used to select a target region in a corresponding image of the plurality of images; regenerating respective semantic segmentation labels of the plurality of images based on the plurality of image masks and the initial semantic segmentation labels of the plurality of images; and generating a mixed image and a semantic segmentation label of the mixed image based on the plurality of images and the regenerated respective semantic segmentation labels. By training an image semantic segmentation model with the generated mixed image and its semantic segmentation label as training data, it helps to reduce the redundant learning of the model on simple training samples and improve the problem of overfitting of the model, thereby improving the performance of the model.

Description

Technical Field

[0001] Embodiments of this disclosure relate to the field of image processing, and more specifically, to methods, apparatus, and computer-readable storage media for image processing. Background Technology

[0002] Image semantic segmentation is a crucial technique in image processing and computer vision, forming the foundation of image understanding. It involves classifying each pixel in an image, determining its semantic category (e.g., belonging to background, person, or vehicle), and then dividing the image into regions according to these categories.

[0003] Typically, image semantic segmentation models (e.g., deep neural networks) are trained on training datasets to achieve image semantic segmentation. Deep neural networks have powerful non-linear fitting capabilities and can be trained to handle complex tasks. However, this characteristic of deep neural networks easily leads to overfitting, meaning that the trained model overfits the training dataset and performs poorly on other datasets. Summary of the Invention

[0004] Embodiments of this disclosure provide methods, apparatus, and computer-readable storage media for image processing.

[0005] In a first aspect of this disclosure, an image processing method is provided. The method includes: acquiring a plurality of images, each image having an initial semantic segmentation label indicating a semantic category of pixels in that image; acquiring a plurality of image masks corresponding to the plurality of images, wherein each image mask is used to select a target region in a corresponding image among the plurality of images; regenerating corresponding semantic segmentation labels for the plurality of images based on the plurality of image masks and the initial semantic segmentation labels for the plurality of images; and generating a blended image and a semantic segmentation label for the blended image based on the plurality of images and the regenerated corresponding semantic segmentation labels.

[0006] In a second aspect of this disclosure, a model training method is provided. The method includes training an image semantic segmentation model by using a blended image generated according to a first aspect of this disclosure and semantic segmentation labels of the blended image as training data, to generate semantic segmentation labels for a target image.

[0007] In a third aspect of this disclosure, an image processing method is provided. The method includes: acquiring an input image; and generating semantic segmentation labels for the input image using an image semantic segmentation model trained according to a second aspect of this disclosure.

[0008] In a fourth aspect of this disclosure, a computing device is provided. The computing device includes at least one processor configured to: acquire a plurality of images, wherein each image has an initial semantic segmentation label indicating a semantic category of pixels in the image; acquire a plurality of image masks corresponding to the plurality of images, wherein each image mask is used to select a target region in a corresponding image among the plurality of images; regenerate corresponding semantic segmentation labels for the plurality of images based on the plurality of image masks and the initial semantic segmentation labels for the plurality of images; and generate a blended image and a semantic segmentation label for the blended image based on the plurality of images and the regenerated corresponding semantic segmentation labels.

[0009] In a fifth aspect of this disclosure, a computing device is provided. The computing device includes at least one processor configured to train an image semantic segmentation model by using a blended image generated according to a first aspect of this disclosure and semantic segmentation labels of the blended image as training data, for generating semantic segmentation labels for a target image.

[0010] In a sixth aspect of this disclosure, a computing device is provided. The computing device includes at least one processor configured to: acquire an input image; and generate semantic segmentation labels for the input image using an image semantic segmentation model trained according to a second aspect of this disclosure.

[0011] In a seventh aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium has machine-executable instructions stored thereon, which, when executed by a device, cause the device to perform any step of the method described in the first, second, or third aspect of this disclosure.

[0012] In an eighth aspect of this disclosure, a computer program product is provided. The computer program product is tangibly stored in a computer-readable storage medium and includes machine-executable instructions. When executed by a device, the machine-executable instructions cause the device to perform any step of the method described according to the first, second, or third aspect of this disclosure.

[0013] The summary section is provided to introduce a series of concepts in a simplified form, which will be further described in the detailed description below. The summary section is not intended to identify key or essential features of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0014] The objects, advantages, and other features of the invention will become more apparent from the following disclosure and claims. A non-limiting description of preferred embodiments is given herein with reference to the accompanying drawings, in which:

[0015] Figure 1 A schematic block diagram of an image processing system according to an embodiment of the present disclosure is shown;

[0016] Figure 2 A schematic diagram of image processing according to an embodiment of the present disclosure is shown;

[0017] Figure 3 A flowchart illustrating an example method for image processing according to an embodiment of the present disclosure is shown;

[0018] Figure 4 A flowchart illustrating an example model training method according to an embodiment of the present disclosure is shown;

[0019] Figure 5 A flowchart illustrating an example method for image processing according to embodiments of the present disclosure is shown; and

[0020] Figure 6 A schematic block diagram of an example computing device that can be used to implement embodiments of the present disclosure is shown.

[0021] In the various figures, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0022] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0023] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0024] As used herein, the term "circuit" can refer to hardware circuitry and / or a combination of hardware circuitry and software. For example, the circuitry can be a combination of analog and / or digital hardware circuitry with software / firmware. As another example, a circuit can be any part of a hardware processor (including a digital signal processor) with software, software, and memory, which work together to enable a device to perform various functions. In yet another example, a circuit can be hardware circuitry and / or a processor, such as a microprocessor or a portion thereof, which requires software / firmware to operate but may be absent when the software is not required to operate. As used herein, the term "circuit" also encompasses hardware circuitry or a processor alone, or a portion thereof and its accompanying software and / or firmware implementation.

[0025] As mentioned above, image semantic segmentation models (e.g., deep neural networks) are typically trained on training datasets to achieve image semantic segmentation. Deep neural networks have powerful non-linear fitting capabilities and can be trained to handle complex tasks. However, this characteristic of deep neural networks easily leads to overfitting. That is, the trained model overfits the training dataset and performs poorly on other datasets.

[0026] Regularization operations typically prevent models from overfitting the training dataset or becoming overconfident in their predictions. For example, mixup is a common image augmentation method in computer vision. It combines different images to expand the training dataset. Mixup can be viewed as a form of regularization, adding regularization terms or pruning the model through data augmentation. However, for pixel-by-pixel prediction problems like image semantic segmentation, standard regularization operations do not provide significant improvement. This is because pixel-by-pixel prediction requires a complex feature space, which reduces model training efficiency and may even hinder convergence.

[0027] Embodiments of this disclosure propose a scheme for image processing to address the aforementioned problems and / or other potential problems. The scheme acquires multiple images, each image having a semantic segmentation label indicating the semantic category of pixels in that image. The scheme acquires multiple image masks corresponding to the multiple images, where each image mask is used to select a target region in a corresponding image among the multiple images. Based on the multiple image masks and the initial semantic segmentation labels of the multiple images, the scheme regenerates corresponding semantic segmentation labels for the multiple images. The scheme then generates a blended image and its semantic segmentation label based on the multiple images and the regenerated corresponding semantic segmentation labels.

[0028] In this way, the scheme can select pixels with high contribution from multiple images for blending. Training the image semantic segmentation model using the generated blended image and its semantic segmentation labels as training data helps reduce redundant learning on simple training samples. Introducing perturbations into the training samples helps improve the model's overfitting problem, thereby enhancing its performance.

[0029] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings. In the embodiments of this disclosure, the term "model" is capable of processing inputs and providing corresponding outputs. Taking a neural network model as an example, it typically includes an input layer, an output layer, and one or more hidden layers between the input and output layers. Models used in deep learning applications (also called "deep learning models") typically include many hidden layers, thereby extending the depth of the network. The layers of a neural network model are sequentially connected such that the output of the previous layer is used as the input of the next layer, where the input layer receives the inputs of the neural network model, and the output of the output layer serves as the final output of the neural network model. Each layer of a neural network model includes one or more nodes (also called processing nodes or neurons), each node processing the input from the previous layer. In this document, the terms "neural network," "model," "network," and "neural network model" are used interchangeably.

[0030] Figure 1 A schematic block diagram of an image processing system 100 according to an embodiment of the present disclosure is shown. Figure 1 As shown, system 100 may include an image collection device 110, an image annotation device 120, an image processing device 130, a model training device 140, and a model application device 150. In some embodiments, the above-mentioned devices may be implemented in different physical devices. Alternatively, at least some of the above-mentioned devices may be implemented in the same physical device. It should be understood that... Figure 1 The system 100 shown is merely one example of an embodiment that can be implemented according to this disclosure and is not intended to limit the scope of this disclosure. The embodiments of this disclosure are equally applicable to other systems or architectures.

[0031] The image collection device 110 can collect an image set 101 for training the model. It should be understood that the images in the image set 101 can be of any format. Furthermore, the image collection device 110 can utilize any known or future developed method to collect the image set 101. The scope of this disclosure is not limited in this respect.

[0032] The image annotation device 120 can generate semantic segmentation labels for each image 101 to indicate the semantic category of each pixel in the image. The semantic category of a pixel refers to the type of object corresponding to that pixel in the image; objects in the image can include, but are not limited to, background, people, animals, vehicles, etc. For example, the generated semantic segmentation labels can be a vector or array indicating the true semantic category of each pixel, or they can be... Figure 1 The visualized image shows pixels of different semantic categories that can be identified using different colors. It should be understood that image annotation can be done manually. In this case, the image annotation device 120 can be a device or apparatus that is manually controlled to annotate each pixel in each image 101.

[0033] Image set 101 and its corresponding semantic segmentation labels 102 can be provided to image processing device 130. Image processing device 130 can acquire multiple images 101 from image set 101, where each image 101 is labeled with its corresponding semantic segmentation label 102. In this document, "semantic segmentation label 102" is also referred to as "initial semantic label". Image processing device 130 can acquire multiple image masks corresponding to the multiple images 101, each image mask being used to select a target region in the corresponding image 101. The target region may include multiple pixels in the image 101 whose semantic category is difficult to predict, that is, those pixels that contribute more to the training of the image semantic segmentation model. Image processing device 130 can regenerate semantic segmentation labels for the multiple images 101 based on the multiple image masks and the initial semantic segmentation labels 102 of the multiple images. For example, for each image 101 corresponding to an initial semantic segmentation label 102, the image processing device 130 can select the pixels in the target region corresponding to the initial semantic segmentation label 102 by performing a masking operation (i.e., dot product) between the image mask and the initial semantic segmentation label 102, while ignoring the pixels in the non-target region. Then, the image processing device 130 can generate a blended image and its semantic segmentation label based on the plurality of images 101 and the regenerated corresponding semantic segmentation labels, as a training sample in the training dataset 103. Alternatively, the image processing device 130 can also use each original image 101 and its corresponding initial semantic segmentation label 102 as a training sample in the training dataset 103. In this way, the image processing device 130 can generate a training dataset 103 including multiple training samples. It should be understood that the training dataset 103 may include only training samples consisting of the blended image and its semantic segmentation label, or only training samples consisting of the original images and their semantic segmentation labels, or both.

[0034] Training dataset 103 can be provided to model training device 140 for training image semantic segmentation model 104. Image semantic segmentation model 104 can be trained to predict the semantic category of each pixel in an input image. For example, model training device 140 can train the model based on training dataset 103 to determine the model parameters of image semantic segmentation model 104. The trained image semantic segmentation model 104 (e.g., the determined model parameters) can be provided to model application device 150. Model application device 150 can acquire input image 105 and use the trained image semantic segmentation model 104 to predict the semantic category of each pixel in input image 105. Model application device 150 can generate semantic segmentation labels 106 indicating the input image 105 based on the prediction results. For example, the generated semantic segmentation labels 106 can be a vector or array indicating the semantic category of each pixel in input image 105, or can be... Figure 1 The visualization shown illustrates how pixels of different semantic categories can be filled with different colors.

[0035] Image semantic segmentation model 104 can be applied to various fields. In the medical field, the image semantic segmentation model can perform semantic segmentation on various input medical images to identify and understand lesions, thereby helping doctors diagnose diseases. For example, a medical image processing method may include: acquiring an input medical image; using a trained image semantic segmentation model to generate a semantic segmentation label for the medical image, wherein the semantic segmentation label indicates the semantic category of pixels in the medical image; and identifying lesions in the medical image based on the semantic segmentation label. In the field of autonomous driving, the image semantic segmentation model can be used to perform semantic segmentation on images captured by a vehicle during driving, thereby identifying and understanding street scenes to help autonomous driving systems make driving decisions, and so on. For example, an autonomous driving method may include: acquiring an image captured by a vehicle during driving; using a trained image semantic segmentation model to generate a semantic segmentation label for the image, wherein the semantic segmentation label indicates the semantic category of pixels in the image; identifying street scenes in the image based on the semantic segmentation label; and generating autonomous driving decisions based on the identification results.

[0036] The following will be further combined Figure 2 To describe in detail, such as Figure 1 The operation performed by the image processing device 130 shown. Figure 2 Multiple images 101-1 and 101-2 obtained from image set 101 are shown, wherein image 101-1 has an annotated initial semantic segmentation label 102-1 and image 101-2 has an annotated initial semantic segmentation label 102-2. Although Figure 2This illustration only shows the acquisition of a pair of images from image set 101 for processing, and it should be understood that this is for illustrative purposes only and does not imply any limitation on the scope of this disclosure. In some embodiments, image processing apparatus 130 may acquire more than two images from image set 101 for processing.

[0037] In some embodiments, such as Figure 2 As shown, for each of the multiple images 101-1 and 101-2, the image processing device 130 can use the image semantic segmentation model 210 to predict the corresponding semantic category of multiple pixels in that image. The image semantic segmentation model 210 can be the same model as the image semantic segmentation model 104 or a different model. For example, the image semantic segmentation model 210 can be a pre-trained image semantic segmentation model, a pre-trained image semantic segmentation model, or an untrained image semantic segmentation model. That is, the image semantic segmentation model 210 can be implemented using a semantic segmentation model with lower prediction accuracy, or other algorithms or devices capable of semantic segmentation.

[0038] The image processing device 130 can further determine the corresponding scores of multiple pixels in the image based on the semantic prediction results for each image, wherein the score of each pixel can indicate the degree of association between the pixel and the target region in the image. In other words, the score of each pixel can indicate the degree of contribution of the pixel to the training of the image semantic segmentation model. Figure 2 As shown, the corresponding scores of multiple pixels in image 101-1 can be represented by heatmap 220-1, and the corresponding scores of multiple pixels in image 101-2 can be represented by heatmap 220-2.

[0039] In some embodiments, for each pixel in each image, the image processing apparatus 130 may determine a score for that pixel based on the difference between the pixel's predicted semantic category and its true semantic category, wherein the true semantic category of the pixel is indicated by the semantic segmentation label 102 of the image. For example, the image semantic segmentation model 210 may predict the probability that each pixel in each image belongs to a different semantic category, while the labeled semantic segmentation label of the image may indicate the true semantic category to which each pixel belongs. The image processing apparatus 130 may calculate the loss between the probability predicted by the image semantic segmentation model 210 that each pixel belongs to its true semantic category and a predetermined probability value (e.g., 100%) as the score for each pixel. That is, the higher the score of a pixel, the greater the difference between the pixel's predicted semantic category and its true semantic category.

[0040] Alternatively, in some embodiments, for each pixel in each image, the image processing apparatus 130 may determine a score for that pixel based on the degree of uncertainty or confusion (e.g., entropy) of the pixel's predicted semantic category. For example, the image semantic segmentation model 210 may predict the probability that each pixel in each image belongs to different semantic categories. The image processing apparatus 130 may utilize any known or suitable algorithm to determine the degree of uncertainty or confusion of the pixel's predicted semantic category based on the predicted probability of belonging to different semantic categories for each pixel, as the pixel's score. That is, the higher the pixel's score, the higher the degree of uncertainty or confusion in the pixel's predicted semantic category.

[0041] In some embodiments, such as Figure 2 As shown, the image processing apparatus 130 can determine an image mask 230-1 corresponding to image 101-1 based on a heatmap 220-1 indicating the corresponding scores of multiple pixels in image 101-1. Similarly, the image processing apparatus 130 can determine an image mask 230-2 corresponding to image 101-2 based on a heatmap 220-2 indicating the corresponding scores of multiple pixels in image 101-2.

[0042] In some embodiments, the image processing device 130 may sort the scores of multiple pixels in each image from largest to smallest, and then select the top N% of pixels or pixels with scores exceeding a threshold score n. For example, the value of N% or n can be determined based on a verification experiment. The image processing device 130 can generate an image mask for the image by setting the mask value of the selected pixel's location to 1 and the mask value of other locations to 0. Alternatively, in some embodiments, the image processing device 130 may sort the scores of all pixels in a batch of images from largest to smallest, and then select the top N% of pixels or pixels with scores exceeding a threshold score n. For example, the value of N% or n can be determined based on a verification experiment. The image processing device 130 can generate an image mask for each image in the batch of images by setting the mask value of the selected pixel's location to 1 and the mask value of other locations to 0. Alternatively, in other embodiments, the image processing device 130 may also directly use a heatmap indicating the corresponding scores of multiple pixels in each image as the image mask for that image.

[0043] In some embodiments, such as Figure 2As shown, the image processing apparatus 130 can perform a masking operation (i.e., dot product) between the image mask 230-1 and the initial semantic segmentation label 102-1 of image 101-1 to regenerate the semantic segmentation label 240-1 of image 101-1. Similarly, the image processing apparatus 130 can perform a masking operation (i.e., dot product) between the image mask 230-2 and the initial semantic segmentation label 102-2 of image 101-2 to regenerate the semantic segmentation label 240-2 of image 101-2. From Figure 2 As can be seen, the regenerated semantic segmentation labels 240-1 and 240-2 retain the semantic segmentation labels of the pixels in the target region corresponding to the initial semantic segmentation labels 102-1 and 102-2, respectively, while ignoring the semantic segmentation labels of the pixels in the non-target region.

[0044] In some embodiments, such as Figure 2 As shown, the image processing apparatus 130 can further generate a hybrid image 250 and its semantic segmentation label 260 based on multiple images 101-1 and 101-2 and the regenerated semantic segmentation labels 240-1 and 240-2. In some embodiments, the image processing apparatus 130 can perform a weighted summation of multiple images 101-1 and 101-2 based on their respective weights to obtain the hybrid image 250. The image processing apparatus 130 can also perform a weighted summation of semantic segmentation labels 240-1 and 240-2 based on their respective weights to obtain the semantic segmentation label 260 of the hybrid image 250. It should be understood that the image processing apparatus 130 can also utilize other methods to generate the hybrid image 250 and its semantic segmentation label 260. The hybrid image 250 and its semantic segmentation label 260 can serve as such as Figure 1 A training sample from the training dataset 103 shown is used to train the image semantic segmentation model 104.

[0045] Figure 3 A flowchart of an example method 300 for image processing according to an embodiment of the present disclosure is shown. For example, method 300 may be performed by, for example... Figure 1 The image processing device 130 shown is used to perform this operation. The following is in conjunction with... Figure 1 and Figure 2 Method 300 is described herein. It should be understood that method 300 may also include additional boxes not shown and / or certain boxes shown may be omitted. The scope of this disclosure is not limited in this respect.

[0046] At box 310, acquire multiple images (e.g., such as...). Figure 2 Images 101-1 and 101-2 are shown, each image having an initial semantic segmentation label (e.g., as indicated by the semantic category of the pixels in that image). Figure 2 The semantic segmentation labels shown are 102-1 and 102-2.

[0047] At box 320, obtain multiple image masks corresponding to multiple images (e.g., such as...). Figure 2 Image masks 230-1 and 230-2 are shown, where each image mask is used to select a target region in a corresponding image among multiple images.

[0048] In some embodiments, obtaining multiple image masks corresponding to multiple images includes: for a first image among the multiple images (e.g., such as...) Figure 2 Using the image 101-1 or 101-2 shown, determine the corresponding predicted semantic category of multiple pixels in the first image; based on the corresponding predicted semantic category of the multiple pixels, determine the corresponding score of the multiple pixels (e.g., as shown in the image 101-1 or 101-2). Figure 2 The heatmap shown (220-1 or 220-2) indicates that the score of each pixel indicates the degree of association between that pixel and the target region in the first image; and based on the corresponding scores of multiple pixels, a first image mask of the first image is determined (e.g., as shown in the heatmap 220-1 or 220-2). Figure 2 The image mask shown is 230-1 or 230-2), wherein the first image mask is used to select at least a portion of the pixels with higher scores among a plurality of pixels.

[0049] In some embodiments, determining the corresponding predicted semantic category of multiple pixels in a first image includes: utilizing an image semantic segmentation model (e.g., ... Figure 2 The image semantic segmentation model 210 shown determines the corresponding predicted semantic category of multiple pixels in the first image.

[0050] In some embodiments, determining the corresponding scores of a plurality of pixels in a first image includes: for a first pixel among the plurality of pixels, determining the score of the first pixel based on the difference between the predicted semantic category of the first pixel and the semantic category of the first pixel, wherein the semantic category of the first pixel is indicated by the semantic segmentation label of the first image.

[0051] In some embodiments, determining the corresponding scores of a plurality of pixels in a first image includes: for a second pixel among the plurality of pixels, determining the score of the second pixel based on the uncertainty of the predicted semantic category of the second pixel.

[0052] In some embodiments, determining the corresponding scores of a plurality of pixels in a first image includes: for a third pixel among the plurality of pixels, determining the score of the third pixel based on the degree of confusion of the predicted semantic category of the third pixel.

[0053] At box 330, based on multiple image masks and the initial semantic segmentation labels of multiple images, the corresponding semantic segmentation labels of multiple images are regenerated (e.g., such as...). Figure 2The semantic segmentation labels shown are 240-1 and 240-2.

[0054] In some embodiments, regenerating the corresponding semantic segmentation labels for multiple images includes: performing a masking operation on each of the multiple image masks and the initial semantic segmentation label of the corresponding image in the multiple images to regenerate the semantic segmentation label of the corresponding image.

[0055] At box 340, a hybrid image is generated based on multiple images and the regenerated corresponding semantic segmentation labels (e.g., ...). Figure 2 The hybrid image 250 shown) and the semantic segmentation label of the hybrid image (e.g., such as Figure 2 The semantic segmentation label shown is 260.

[0056] In some embodiments, generating a hybrid image and a semantic segmentation label for the hybrid image includes: weighting and summing multiple images based on their respective weights to generate a hybrid image; and weighting and summing regenerated semantic segmentation labels based on their respective weights to generate a semantic segmentation label for the hybrid image.

[0057] Figure 4 A flowchart of an example model training method 400 according to an embodiment of the present disclosure is shown. For example, method 400 can be performed by, for example... Figure 1 The model training device 140 shown is used to perform this. The following is in conjunction with... Figure 1 and Figure 2 Method 400 is described herein. It should be understood that method 400 may also include additional boxes not shown and / or certain boxes shown may be omitted. The scope of this disclosure is not limited in this respect.

[0058] At box 410, training data is acquired. For example, model training device 140 can obtain training data from, for example, Figure 1 The training dataset 103 shown is used to obtain training data. In some embodiments, the obtained training data may include the hybrid image and its semantic labels (e.g., as shown in the example) generated by method 300. Figure 2 The hybrid image 250 shown and its semantic label 260. Additionally or alternatively, in some embodiments, the acquired training data may also include the original image and its corresponding initial semantic segmentation label (e.g., as shown in the image). Figure 1 Image 101 and its corresponding initial semantic segmentation label 102 are shown.

[0059] At box 420, an image semantic segmentation model is trained based on the training data (e.g., ...). Figure 1 and Figure 2The image semantic segmentation model 104 shown is used to generate semantic segmentation labels for target images. In some embodiments, the model training device 140 can train the model based on the acquired training data to determine the model parameters of the image semantic segmentation model 104.

[0060] Figure 5 A flowchart of an example method 500 for image processing according to an embodiment of the present disclosure is shown. For example, method 500 may be performed by, for example... Figure 1 The model shown is executed using device 150. The following is in conjunction with... Figure 1 and Figure 2 Method 500 is described herein. It should be understood that method 500 may also include additional boxes not shown and / or certain boxes shown may be omitted. The scope of this disclosure is not limited in this respect.

[0061] At box 510, obtain the input image (e.g., as shown in box 510). Figure 1 The input image 105 is shown. At box 520, a semantic segmentation label for the input image is generated using an image semantic segmentation model (e.g., as shown). Figure 1 The semantic segmentation label 106 shown indicates the semantic category of a pixel in the input image. For example, the generated semantic segmentation label 106 may be a vector or array indicating the semantic category of each pixel in the input image 105, or it may be as follows: Figure 1 The visualization shown illustrates how pixels of different semantic categories can be filled with different colors.

[0062] Table 1 shows a performance comparison of image semantic segmentation models trained using different processing schemes. Table 1 also shows the accuracy of image semantic segmentation models trained using schemes 1 through 4 on the same training dataset.

[0063] Table 1

[0064] Option 1 77.56％ Option 2 77.98％ Option 3 78.98％ Option 4 79.06％

[0065] In Table 1, "Scheme 1" represents the baseline scheme, which uses the original image and its standard semantic segmentation labels to train the image semantic segmentation model. "Scheme 2" represents the traditional mixup operation scheme. "Scheme 3" and "Scheme 4" are schemes according to embodiments of the present disclosure, wherein "Scheme 3" generates an image mask for selecting target regions based on the loss between the predicted semantic category and the true semantic category of a pixel, while "Scheme 4" generates an image mask for selecting target regions based on the degree of confusion (i.e., "entropy") of the predicted semantic category of a pixel. Table 1 shows the average accuracy of each scheme for image segmentation for 20 predetermined semantic categories. The experimental data shown in Table 1 demonstrates that the schemes according to embodiments of the present disclosure can effectively improve the performance of the trained image semantic segmentation model. For example, accuracy can be improved by 1% to 2% for each of the 20 predetermined semantic categories.

[0066] In addition to being applicable to semantic segmentation tasks, embodiments of this disclosure are also applicable to instance segmentation tasks. As described above, the purpose of a semantic segmentation task is to identify the semantic category to which each pixel in an image belongs. Unlike semantic segmentation tasks, the purpose of instance segmentation tasks is to identify the instance category to which each pixel in an image belongs. For example, a semantic segmentation task can group pixels belonging to people in an image into the same category, while an instance segmentation task can group pixels belonging to different people in an image into different categories.

[0067] To apply the embodiments of this disclosure to an instance segmentation task, the initial semantic label 102 annotated for each image 101 needs to indicate the instance category to which each pixel in the image belongs. For example, if two pixels are semantically categorized as "person" but correspond to different people, then these two pixels will be labeled as belonging to different instance categories. Subsequent processing is similar to the processing described above for the semantic segmentation task and will not be repeated here.

[0068] Figure 6 A schematic block diagram of an example computing device 600 that can be used to implement embodiments of the present disclosure is shown. For example, as Figure 1 One or more devices in the system 100 shown may be implemented by device 600. Examples of device 600 may include any general-purpose computer, special-purpose computer, server, virtual machine, and any other computing-capable electronic device (such as user equipment, mobile phone, cellular phone, smartphone, personal digital assistant, image capture device (such as digital camera), gaming device, music storage and playback device, or Internet device that enables wireless or wired Internet access and browsing, etc.).

[0069] As shown in the figure, device 600 includes a central processing unit (CPU) 601 (also referred to as "processor 601"), which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) 602 or loaded from storage unit 608 into random access memory (RAM) 603. RAM 603 may also store various programs and data required for the operation of device 600. CPU 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0070] Multiple components in device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of monitors, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0071] Processor 601 may be configured to perform the various processes and procedures described above, such as methods 300, 400, and / or 500. For example, in some embodiments, methods 300, 400, and / or 500 may be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by CPU 601, one or more steps of methods 300, 400, and / or 500 described above may be performed.

[0072] In some embodiments, a computing device includes at least one circuit configured to: acquire a plurality of images, wherein each image has an initial semantic segmentation label indicating a semantic category of pixels in each image; acquire a plurality of image masks corresponding to the plurality of images, wherein each image mask is used to select a target region in a corresponding image among the plurality of images; regenerate corresponding semantic segmentation labels for the plurality of images based on the plurality of image masks and the initial semantic segmentation labels for the plurality of images; and generate a hybrid image and a semantic segmentation label for the hybrid image based on the plurality of images and the regenerated corresponding semantic segmentation labels.

[0073] In some embodiments, the at least one circuit is configured to: for a first image among the plurality of images, determine corresponding predicted semantic categories of a plurality of pixels in the first image; determine corresponding scores of the plurality of pixels based on the corresponding predicted semantic categories of the plurality of pixels, wherein the score of each pixel indicates the degree of association between each pixel and a target region in the first image; and determine a first image mask of the first image based on the corresponding scores of the plurality of pixels, wherein the first image mask is used to select at least a subset of pixels among the plurality of pixels that have higher scores.

[0074] In some embodiments, the at least one circuit is configured to: determine the corresponding predicted semantic category of the plurality of pixels in the first image using an image semantic segmentation model.

[0075] In some embodiments, the at least one circuit is configured to: for a first pixel among the plurality of pixels, determine a score for the first pixel based on the difference between the predicted semantic category of the first pixel and the semantic category of the first pixel, wherein the semantic category of the first pixel is indicated by the semantic segmentation label of the first image.

[0076] In some embodiments, the at least one circuit is configured to: for a second pixel among the plurality of pixels, determine a score for the second pixel based on the uncertainty of the predicted semantic category of the second pixel.

[0077] In some embodiments, the at least one circuit is configured to: determine a score for a third pixel among the plurality of pixels based on the degree of confusion in the predicted semantic category of the third pixel.

[0078] In some embodiments, the at least one circuit is configured to: perform a masking operation on each of the plurality of image masks and the initial semantic segmentation label of the corresponding image in the plurality of images to regenerate the semantic segmentation label of the corresponding image.

[0079] In some embodiments, the at least one circuit is configured to: perform a weighted summation on the plurality of images based on their respective weights to generate the hybrid image; and perform a weighted summation on the regenerated corresponding semantic segmentation labels based on their respective weights to generate semantic segmentation labels for the hybrid image.

[0080] In some embodiments, a computing device includes at least one circuit configured to: acquire training data, the training data including a blended image and semantic segmentation labels of the blended image; and train an image semantic segmentation model based on the training data to generate semantic segmentation labels for a target image.

[0081] In some embodiments, a computing device includes at least one circuit configured to: acquire an input image; and generate semantic segmentation labels for the input image using a trained image semantic segmentation model.

[0082] This disclosure can be implemented as a system, method, and / or computer program product. When this disclosure is implemented as a system, in addition to being able to be implemented on a single device, the components described herein can also be implemented in the form of a cloud computing architecture. In a cloud computing environment, these components can be remotely deployed and can work together to achieve the functions described herein. Cloud computing can provide computing, software, data access, and storage services without requiring end users to know the physical location or configuration of the systems or hardware providing these services. Cloud computing can provide services over a wide area network (such as the Internet) using appropriate protocols. For example, cloud computing providers offer applications over a wide area network, and they can be accessed through a browser or any other computing component. The components of cloud computing, along with the corresponding data, can be stored on remote servers. Computing resources in a cloud computing environment can be consolidated at a remote data center location, or these computing resources can be distributed. Cloud computing infrastructure can provide services through a shared data center, even if they appear as a single access point for users. Therefore, the various functions described herein can be provided from remote service providers using a cloud computing architecture. Alternatively, they can be provided from conventional servers, or they can be installed directly or otherwise on client devices. Furthermore, this disclosure can also be implemented as a computer program product, which may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of this disclosure.

[0083] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0084] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0085] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0086] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0087] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0088] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0089] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0090] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. An image processing method, comprising: Acquire multiple images, each image having an initial semantic segmentation label indicating the semantic category of pixels in each image; Obtain multiple image masks corresponding to the multiple images, wherein each image mask is used to select a target region in the corresponding image among the multiple images; Based on the multiple image masks and the initial semantic segmentation labels of the multiple images, the corresponding semantic segmentation labels of the multiple images are regenerated, wherein regenerating the corresponding semantic segmentation labels of the multiple images includes: Each of the plurality of image masks is masked with the initial semantic segmentation label of the corresponding image in the plurality of images to regenerate the semantic segmentation label of the corresponding image, wherein the regenerated semantic segmentation label includes the semantic segmentation label of the initial semantic segmentation label corresponding to the pixel in the target region; and Based on the multiple images and the regenerated corresponding semantic segmentation labels, a hybrid image and a semantic segmentation label for the hybrid image are generated, wherein generating the hybrid image and the semantic segmentation label for the hybrid image includes: Based on the corresponding weights of the multiple images, a weighted sum is performed on the multiple images to generate the hybrid image; and Based on the corresponding weights of the multiple images, the regenerated semantic segmentation labels are weighted and summed to generate the semantic segmentation labels of the hybrid image.

2. The method according to claim 1, wherein obtaining the plurality of image masks corresponding to the plurality of images comprises: For the first image among the plurality of images, determine the corresponding predicted semantic category of multiple pixels in the first image; Based on the corresponding predicted semantic categories of the plurality of pixels, the corresponding scores of the plurality of pixels are determined, wherein the score of each pixel indicates the degree of association between each pixel and the target region in the first image; as well as Based on the corresponding scores of the plurality of pixels, a first image mask is determined for the first image, wherein the first image mask is used to select at least a portion of the pixels with higher scores among the plurality of pixels.

3. The method of claim 2, wherein determining the corresponding predicted semantic category of a plurality of pixels in the first image comprises: Using an image semantic segmentation model, the corresponding predicted semantic category of the plurality of pixels in the first image is determined.

4. The method of claim 2, wherein determining the corresponding scores of a plurality of pixels in the first image comprises: For a first pixel among the plurality of pixels, a score for the first pixel is determined based on the difference between the predicted semantic category of the first pixel and the semantic category of the first pixel, wherein the semantic category of the first pixel is indicated by the semantic segmentation label of the first image.

5. The method of claim 2, wherein determining the corresponding scores of a plurality of pixels in the first image comprises: For the second pixel among the plurality of pixels, the score of the second pixel is determined based on the uncertainty of the predicted semantic category of the second pixel.

6. The method of claim 2, wherein determining the corresponding scores of a plurality of pixels in the first image comprises: For the third pixel among the plurality of pixels, the score of the third pixel is determined based on the degree of confusion of the predicted semantic category of the third pixel.

7. A model training method, comprising: Acquire training data, the training data including a hybrid image generated by the method according to any one of claims 1 to 6 and semantic segmentation labels of the hybrid image; as well as The image semantic segmentation model is trained based on the training data to generate semantic segmentation labels for the target image.

8. An image processing method, comprising: Obtain the input image; as well as Using the image semantic segmentation model trained according to claim 7, semantic segmentation labels for the input image are generated.

9. A computing device, comprising: At least one processor is configured as follows: Acquire multiple images, each image having an initial semantic segmentation label indicating the semantic category of pixels in each image; Obtain multiple image masks corresponding to the multiple images, wherein each image mask is used to select a target region in the corresponding image among the multiple images; Based on the multiple image masks and the initial semantic segmentation labels of the multiple images, the corresponding semantic segmentation labels of the multiple images are regenerated, wherein regenerating the corresponding semantic segmentation labels of the multiple images includes: Each of the plurality of image masks is masked with the initial semantic segmentation label of the corresponding image in the plurality of images to regenerate the semantic segmentation label of the corresponding image, wherein the regenerated semantic segmentation label includes the semantic segmentation label of the initial semantic segmentation label corresponding to the pixel in the target region; and Based on the multiple images and the regenerated corresponding semantic segmentation labels, a hybrid image and a semantic segmentation label for the hybrid image are generated, wherein the processor is configured to generate the hybrid image and the semantic segmentation label for the hybrid image by: Based on the corresponding weights of the multiple images, a weighted sum is performed on the multiple images to generate the hybrid image; and Based on the corresponding weights of the multiple images, the regenerated semantic segmentation labels are weighted and summed to generate the semantic segmentation labels of the hybrid image.

10. The computing device of claim 9, wherein the at least one processor is configured to: For the first image among the plurality of images, determine the corresponding predicted semantic category of multiple pixels in the first image; Based on the corresponding predicted semantic categories of the plurality of pixels, a corresponding score is determined for each pixel, wherein the score of each pixel indicates the degree of association between each pixel and the target region in the first image; and Based on the corresponding scores of the plurality of pixels, a first image mask is determined for the first image, wherein the first image mask is used to select at least a portion of the pixels with higher scores among the plurality of pixels.

11. The computing device of claim 10, wherein the at least one processor is configured to: Using an image semantic segmentation model, the corresponding predicted semantic category of the plurality of pixels in the first image is determined.

12. The computing device of claim 10, wherein the at least one processor is configured to: For a first pixel among the plurality of pixels, a score for the first pixel is determined based on the difference between the predicted semantic category of the first pixel and the semantic category of the first pixel, wherein the semantic category of the first pixel is indicated by the semantic segmentation label of the first image.

13. The computing device of claim 10, wherein the at least one processor is configured to: For the second pixel among the plurality of pixels, the score of the second pixel is determined based on the uncertainty of the predicted semantic category of the second pixel.

14. The computing device of claim 10, wherein the at least one processor is configured to: For the third pixel among the plurality of pixels, the score of the third pixel is determined based on the degree of confusion of the predicted semantic category of the third pixel.

15. A computing device, comprising: At least one processor is configured as follows: Acquire training data, said training data including a hybrid image generated by the method according to any one of claims 1 to 6 and semantic segmentation labels of said hybrid image; and The image semantic segmentation model is trained based on the training data to generate semantic segmentation labels for the target image.

16. A computing device, comprising: At least one processor is configured as follows: Obtain the input image; as well as Using the image semantic segmentation model trained according to claim 8, semantic segmentation labels for the input image are generated.

Images