Image processing method and device, equipment, storage medium and program product

Abstract

The embodiment of the invention discloses an image processing method and device, equipment, a storage medium and a program product. The method comprises the following steps: acquiring a to-be-processed image, wherein the to-be-processed image comprises a missing region; carrying out weight distribution on pixel points in the to-be-processed image to obtain a weight distribution diagram; inputting the to-be-processed image and the weight distribution diagram into a target generator; performing feature extraction through a target generator according to the weight distribution map and the to-be-processed image, and performing image completion on a missing region of the to-be-processed image according to a feature extraction result to obtain a completed image; inputting the complemented image into an image quality discriminator, so that the image quality discriminator discriminates the image quality of the complemented image; and under the condition that the image quality of the complemented image passes identification, determining the complemented image as a target image. According to the embodiment of the invention, the image has high quality while the image is complemented.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and in particular, relates to an image processing method, apparatus, device, storage medium and program product. Background Art

[0002] Image completion refers to the method of restoring the missing areas in the image due to occlusion or damage. In the process of collecting images by unmanned boats on the sea surface, due to the occlusion of the external equipment and operators on the unmanned boats, as well as the interference caused by the sea surface light and water vapor, the collected image information is incomplete, which in turn affects the judgment of the unmanned boat on the sea conditions and reduces the utilization rate of the unmanned boat images. Therefore, it is necessary to process the images collected by the unmanned boats to improve the utilization rate of the images.

[0003] With the widespread application of deep learning, a number of image completion models have demonstrated their superior performance. However, most of these models focus on experimental environments or relatively simple practical scenarios. For images collected by unmanned boats on the more complex sea surface, the current completion technology cannot accurately complete them. In addition, in the images collected by unmanned boats, the degradation of image quality caused by light and water vapor will also affect the usability of the image. The current completion technology usually only focuses on the completion of missing areas, which cannot meet the quality requirements of unmanned boat images. Summary of the invention

[0004] The embodiments of the present application provide an image processing method, apparatus, device, computer storage medium, and computer program product, which can complete an image while making the image have higher quality.

[0005] In a first aspect, an embodiment of the present application provides an image processing method, comprising:

[0006] Acquire an image to be processed, where the image to be processed includes a missing area;

[0007] Weights are assigned to the pixels in the image to be processed to obtain a weight distribution map;

[0008] Input the image to be processed and the weight distribution map into the target generator;

[0009] The target generator performs feature extraction according to the weight distribution map and the image to be processed, and completes the missing area of ​​the image to be processed according to the feature extraction result to obtain a completed image;

[0010] Inputting the completed image into the image quality discriminator so that the image quality discriminator discriminates the image quality of the completed image;

[0011] When the image quality of the complement image passes the identification, the complement image is determined to be the target image.

[0012] In an optional implementation, the target generator is trained by a generative adversarial network model, and the generative adversarial network model is constructed by a generator and a joint discriminator, and the joint discriminator includes an image quality discriminator.

[0013] In an optional embodiment, the joint discriminator further includes a completion completion discriminator;

[0014] After extracting features according to the weight distribution map and the image to be processed by the target generator and completing the missing area of ​​the image to be processed according to the feature extraction result and obtaining the completed image, the method further includes:

[0015] The completed image is input to the completion completion discriminator, so that the completion completion discriminator discriminates the completion completion of the missing area.

[0016] In an optional embodiment, the joint identifier further includes a completion content identifier;

[0017] After extracting features according to the weight distribution map and the image to be processed by the target generator and completing the missing area of ​​the image to be processed according to the feature extraction result and obtaining the completed image, the method further includes:

[0018] The completed image is input to the completed content identifier, so that the completed content identifier identifies the completed content of the missing area.

[0019] In an optional implementation, weights are assigned to pixels in the image to be processed to obtain a weight distribution map, including:

[0020] Determine a first pixel point by a region growing algorithm, the first pixel point is a target pixel point that is similar to and connected to the second pixel point, and the second pixel point is a target pixel point adjacent to the missing region in the image to be processed;

[0021] Classifying the pixels in the image to be processed to obtain first target pixels, second target pixels and non-target pixels, wherein the first target pixels include the first pixel and the second pixel, the second target pixels include target pixels other than the first target pixel, and the non-target pixels include pixels in the missing area of ​​the image to be processed;

[0022] According to a preset weight allocation rule, determine a first weight corresponding to the first target pixel, a second weight corresponding to the second target pixel, and a third weight corresponding to the non-target pixel, wherein the first weight, the second weight, and the third weight decrease in descending order;

[0023] A weight distribution map is generated based on a first weight corresponding to the first target pixel, a second weight corresponding to the second target pixel, and a third weight corresponding to the non-target pixel.

[0024] In an optional implementation, the target generator includes an encoder; generating a weight distribution map based on a first weight corresponding to a first target pixel, a second weight corresponding to a second target pixel, and a third weight corresponding to a non-target pixel, including:

[0025] Generate an initial weight distribution map based on a first weight corresponding to the first target pixel, a second weight corresponding to the second target pixel, and a third weight corresponding to the non-target pixel;

[0026] Input the initial weight distribution map into the attention mechanism module, whose structure and parameters are the same as those of the encoder;

[0027] The initial weight distribution map is processed by each network layer of the attention mechanism module respectively to obtain the weight distribution map corresponding to each network layer.

[0028] In an optional implementation, inputting the completed image into an image quality discriminator so that the image quality discriminator discriminates the image quality of the completed image includes:

[0029] Input the completed image into the image quality discriminator;

[0030] The pixel difference between each pixel and its adjacent pixels in the completed image is extracted by an image quality discriminator;

[0031] For each pixel point, according to the corresponding relationship between the membership degree of each classification and the pixel difference, the membership degree of the pixel point in each classification is calculated; according to the membership degree of the pixel point in each classification, the classification of the pixel point is determined;

[0032] Calculate the ratio of the number of pixels in the target classification to the total number of pixels in the completed image;

[0033] An image quality identification result corresponding to the completed image is generated based on the ratio.

[0034] In an optional implementation, for each pixel point, according to the preset corresponding relationship between the membership of each category and the pixel difference, the membership of the pixel point in each category is calculated, including:

[0035] For each pixel, the degree of membership of each pixel in each category is calculated based on the pixel difference between the pixel and its adjacent pixels, the expected value of the pixel in each category, and the Gaussian random number of the pixel in each category.

[0036] In a second aspect, the present application provides an image processing device, comprising:

[0037] An acquisition module is used to acquire an image to be processed, where the image to be processed includes a missing area;

[0038] An allocation module is used to allocate weights to the pixels in the image to be processed to obtain a weight distribution map;

[0039] An input module, used for inputting the image to be processed and the weight distribution map into a target generator, wherein the target generator is trained by a generative adversarial network model, the generative adversarial network model is constructed by a generator and a joint discriminator, and the joint discriminator includes an image quality discriminator;

[0040] A processing module is used to extract features according to the weight distribution map and the image to be processed through a target generator, and to complete the missing area of ​​the image to be processed according to the feature extraction result to obtain a completed image;

[0041] The input module is further used to input the completed image into the image quality discriminator, so that the image quality discriminator discriminates the image quality of the completed image;

[0042] The determination module is used to determine that the completed image is the target image when the image quality of the completed image passes the identification.

[0043] In a third aspect, an embodiment of the present application provides an electronic device, the device comprising: a processor and a memory storing computer program instructions;

[0044] When the processor executes the computer program instructions, it implements the image processing method as described in any optional embodiment of the first aspect of the present application.

[0045] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which computer program instructions are stored. When the computer program instructions are executed by a processor, an image processing method as in any optional implementation of the first aspect of the present application is implemented.

[0046] In a fifth aspect, an embodiment of the present application provides a computer program product. When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes an image processing method as described in any optional implementation of the first aspect of the present application.

[0047] The image processing method, device, equipment, computer storage medium and computer program product of the embodiment of the present application can obtain an image to be processed, which includes a missing area. Then, the pixel points in the image to be processed are weighted to obtain a weight distribution map. Then the image to be processed and the weight distribution map are input into the target generator. Then, the target generator performs feature extraction according to the weight distribution map and the image to be processed, and performs image completion on the missing area of ​​the image to be processed according to the feature extraction result to obtain a completed image. Subsequently, the completed image is input into the image quality discriminator, so that the image quality discriminator can identify the image quality of the completed image. When the image quality of the completed image passes the identification, the completed image is determined to be the target image. In this way, not only can the missing area of ​​the image to be processed be completed, but also the target image can have a higher image quality. In this way, it is conducive to improving the utilization rate of the unmanned boat image. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solution of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0049] Figure 1 is a schematic diagram of the structure of a generator provided by an embodiment of the present application;

[0050] Figure 2 is a structural diagram of a completion degree identifier provided by another embodiment of the present application;

[0051] Figure 3 is a structural diagram of a complementary content identifier provided by another embodiment of the present application;

[0052] Figure 4 is a flowchart of an image processing method provided by yet another embodiment of the present application;

[0053] Figure 5 is a structural schematic diagram of an image processing device provided by yet another embodiment of the present application;

[0054] Figure 6 It is a structural diagram of an electronic device provided in yet another embodiment of the present application. DETAILED DESCRIPTION

[0055] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.

[0056] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0057] As described in the background technology, with the widespread application of deep learning, a number of image completion models have demonstrated their superior performance. However, most of these models focus on experimental environments or relatively simple actual scenes. For images collected by unmanned boats on more complex sea surfaces, the current completion technology cannot accurately complete them. In addition, in the images collected by unmanned boats, the degradation of image quality caused by light and water vapor will also affect the usability of the image. The current completion technology usually only focuses on the completion of missing areas and cannot meet the quality requirements of unmanned boat images.

[0058] As an example, the related technology relates to a method of image restoration combining multiple granularities. This method completes the image of the occluded area in the image by analyzing the features of different granularity layers of the image. Specifically, the method enhances the image through the Multi-Scale Retinex (MSR) algorithm, and then uses a generative adversarial network to extract features of the image from three granular layers of hue, saturation and brightness, and uses traditional content discriminators and pixel discriminators to constrain the model from two aspects: completion content and completion quality. After repeated iterative training, the optimal image completion model is obtained. However, since this method only focuses on the constraints of image completion, the improvement of the overall image quality cannot meet the needs of the sea surface unmanned boat scene.

[0059] As another example, the related technology also relates to a multi-task completion method for time series images based on local specific deepening. For the image, this method extracts the local specific information at each moment through a local information extraction network, then extracts the global information through a global information extraction network, and finally fuses the global information and the local specific information through an image completion network to obtain fused information, and finally outputs the completed image. This method is mainly used to complete sea temperature images with relatively simple scenes, and time series data is used as a reference. However, in the scenario of unmanned boats on the sea surface, time series data will increase the time complexity of the model and cannot help the unmanned boat to respond in a timely manner. Therefore, this method cannot meet the completion requirements in specific unmanned boat application scenarios on the sea surface.

[0060] As another example, the related technology also involves an image completion method based on an attention mechanism. In the preprocessing stage, this method uses a binary mask to distinguish between damaged areas and known areas in the image, and uses this as a weight to input it and the damaged image into the image completion model, and then uses a partial convolutional layer to extract features only for the known area and generate the completed result. This method mainly completes conventional images, and the model used is relatively simple. It has not been optimized for actual application scenarios, and it is impossible to improve image quality while completing content.

[0061] In view of this, the inventor, after in-depth thinking, cleverly proposed an image processing method, device, equipment, computer storage medium and computer program product.

[0062] The image processing method provided in the embodiment of the present application is introduced below through specific embodiments and their application scenarios in conjunction with the accompanying drawings. In the image processing method provided in the embodiment of the present application, the device that executes the sending may be an image processing device, or a partial module in the image processing device for executing the image processing method. In the embodiment of the present application, the image processing method provided in the embodiment of the present application is described in detail by taking the image processing device executing the image processing method as an example.

[0063] In addition, it should be noted that the image processing method provided in the embodiment of the present application, after weighting the pixels in the image to be processed and obtaining the weight distribution map, needs to be completed by a target generator, and the target generator is trained by a generative adversarial network model. Therefore, before completing the image by the target generator, it is necessary to first train the generative adversarial network model to obtain the target generator. The following is a description of the specific implementation method of the training method of the corresponding generative adversarial network model for the target generator used in the image processing method provided in the embodiment of the present application.

[0064] In one embodiment, the training method of the target generator used in image processing and its corresponding generative adversarial network model may include the following steps S101 to S111.

[0065] S101, constructing a generative adversarial network model with the generator and the joint discriminator.

[0066] In step S101, the generator may include an encoder and a decoder. Exemplarily, the encoder may include a convolutional layer and a dilated convolutional layer, and the decoder may include a convolutional layer and a deconvolutional layer. The convolutional layer may be used to extract image features, the dilated convolutional layer may use receptive fields of different sizes to extract features under different receptive fields, and the deconvolutional layer may be used to restore the feature map with changed size back to the size consistent with the input image. As an example, the structure of the generator may be as follows: Figure 1 As shown in . The joint discriminator may include an image quality discriminator.

[0067] S102, obtaining a first training sample set and a second training sample set, wherein the first training sample set includes a plurality of first images, the first images include missing areas, and the second training sample set includes a plurality of second images corresponding to the plurality of first images, the second images are complete images.

[0068] In step S102, the first training sample set and the second training sample set may be composed of images collected by an actual sea surface unmanned boat. The ratio of the number of first images in the first training sample set to the number of second images in the second training sample set may be 1:1. The second training sample set may include second images that correspond one-to-one to the first images in the first training sample set.

[0069] S103 , for each first image, weights are assigned to pixels in the first image to obtain a weight distribution map corresponding to each first image.

[0070] S104, for each first image, respectively execute the following steps S10 to S20.

[0071] S10, inputting the first image and the weight distribution map corresponding to the first image into the generator.

[0072] S20, performing feature extraction according to the weight distribution map and the first image by the generator, and completing the missing area of ​​the first image according to the feature extraction result to obtain a third image.

[0073] In step S20, the encoder of the generator can extract features according to the weight distribution map and the first image. In this way, feature extraction can be performed in combination with the weight distribution map, and feature extraction of known areas in the missing image is more accurate and sufficient.

[0074] S105: Create a third training sample set using third images corresponding to each of the plurality of first images.

[0075] S106, respectively inputting the second training sample set and the third training sample set into a joint discriminator to obtain a first identification result.

[0076] S107, calculating a first loss function value of the joint discriminator according to the first identification result.

[0077] In step S107, the first loss function value can be determined in a variety of ways, which are not limited here. As an example, the identification result corresponding to the image may include the category of the image, which may be, for example, whether it is a complete image, whether it is a high-quality image, etc. According to the category of each second image, the first probability of passing the identification of the second training set can be determined. According to the category of each third image, the second probability of passing the identification of the third training set can be determined. According to the first probability, the second probability and the joint discriminator loss function, the above-mentioned first loss function value can be calculated.

[0078] S108 , updating parameters of the joint discriminator according to the first loss function value to obtain an updated joint discriminator.

[0079] In step S108 , the method for updating the parameters of the joint discriminator may include but is not limited to error correction methods such as gradient descent method and back propagation algorithm.

[0080] S109, inputting the third training sample set into the updated joint discriminator to obtain a second identification result.

[0081] S110, calculating a second loss function value of the generator according to the second identification result.

[0082] In step S110, a third probability that the third training set passes the identification can be determined according to the second identification result, and the second loss function value can be calculated according to the third probability and the generator loss function.

[0083] S111, update the parameters of the generator according to the value of the second loss function to obtain an updated generator, and return to each first image, respectively executing the following steps S10 to S20 until the iteration end condition is met, and use the updated generator as the target generator.

[0084] In step S111, the method for updating the parameters of the generator may include but is not limited to error correction methods such as gradient descent method and back propagation algorithm. The iteration end condition may include reaching a certain number of iterations, or the performance of the generative adversarial network model reaches a preset requirement.

[0085] In this way, during the training of the generative adversarial network model, the generator and the union can be iteratively optimized. In this way, the target generator obtained based on the trained generative adversarial network model can complete the image while making the completed image have higher image quality.

[0086] In one embodiment, the joint discriminator may include a completion completeness discriminator, a completion content discriminator, and an image quality discriminator.

[0087] Inputting the second training sample set and the third training sample set into the joint discriminator respectively to obtain a first identification result may specifically include:

[0088] The second training sample set and the third training sample set are respectively input into the completion degree identifier to obtain the completion degree identification result corresponding to the second training sample set and the completion degree identification result corresponding to the third training sample set.

[0089] The second training sample set and the third training sample set are respectively input into the complementary content identifier to obtain complementary content identification results corresponding to the second training sample set and complementary content identification results corresponding to the third training sample set.

[0090] The second training sample set and the third training sample set are respectively input into the image quality discriminator to obtain the image quality identification result corresponding to the second training sample set and the image quality identification result corresponding to the third training sample set.

[0091] In the above embodiment, the completion completion discriminator can be used to determine the completion degree of the second image for the image completion task. The completion content discriminator can be used to determine the degree of consistency between the completed area in the second image and the corresponding area in the complete image. The image quality discriminator can be used to determine the image quality of the second image.

[0092] According to the above implementation, in the process of constraining the generative adversarial network generator to complete the result, the joint discriminator can constrain the generator from three aspects: the overall content of the generated result, the content of the completed area, and the image quality, respectively, to ensure that the model after iterative adversarial training can make the overall content of the generated result consistent with the input image, improve the accuracy of the completed missing area, and make the image quality of the completed result higher than the input image. In this way, the overall quality and utilization rate of the sea surface unmanned boat image can be improved.

[0093] In one embodiment, calculating the first loss function value of the joint discriminator according to the first identification result may include:

[0094] The first sub-loss function value is calculated according to the completion completion identification result.

[0095] The second sub-loss function value is calculated based on the completion content identification result.

[0096] The third sub-loss function value is calculated based on the image quality identification result.

[0097] Updating the parameters of the joint discriminator according to the first loss function value may include:

[0098] Update the parameters of the completion completeness discriminator according to the value of the first sub-loss function.

[0099] Update the parameters of the completion content discriminator according to the value of the second sub-loss function.

[0100] The parameters of the image quality discriminator are updated according to the value of the third sub-loss function.

[0101] In one embodiment, the completion completeness discriminator can be constructed based on a traditional residual network. The input of the completion completeness discriminator can be an image, and the output can be a real number [0,1], which represents the consistency between the image input to the completion completeness discriminator and the complete image. As an example, the network structure of the completion completeness discriminator can be as follows: Figure 2 As shown in the figure, the convolution layer can be used to extract the features of the entire image, the pooling layer can be used to reduce the number of parameters, and the fully connected layer can be used for classification.

[0102] In one embodiment, the complement content discriminator can be constructed based on a partial convolutional layer. The input of the complement content discriminator can be an image, and the output can be a real number [0,1], which represents the degree of consistency between the complement region in the image input to the complement content discriminator and the corresponding region in the complete image. The network structure of the complement content discriminator can be as follows: Figure 3 As shown in the figure, some convolutional layers can be used to extract only the features of the completed missing areas without extracting the features of the known areas, the pooling layer can be used to reduce the number of parameters, and the fully connected layer can be used for classification.

[0103] In one embodiment, the image quality discriminator may be constructed based on a heuristic Gaussian cloud model.

[0104] In one embodiment, the second training sample set is input into the image quality discriminator to obtain the image quality identification result corresponding to the second training sample set, including:

[0105] inputting the second training sample set into the image quality discriminator;

[0106] For each second image in the second training sample set, perform the following steps respectively:

[0107] The pixel difference between each pixel point and adjacent pixel points in the second image is extracted by the image quality identifier.

[0108] For each pixel point, the membership of the pixel point in each category is calculated according to the corresponding relationship between the membership of each category and the pixel difference; and the category of the pixel point is determined according to the membership of the pixel point in each category.

[0109] The ratio of the number of pixels of the target classification to the total number of pixels of the second image is calculated.

[0110] An image quality identification result corresponding to the second image is generated based on the ratio.

[0111] Correspondingly, the third training sample set is input into the image quality discriminator to obtain the image quality identification result corresponding to the third training sample set. The steps may be the same as those in the above implementation manner and will not be described in detail herein.

[0112] According to the above implementation, the image quality discriminator can classify pixels into three categories: large gap pixels, small gap pixels and no gap pixels according to the gap between each pixel in the input image and its surrounding pixels, and each type of pixel is represented by a triplet (expected Ex, entropy En, super entropy He). After motion blur occurs, the number of small gap pixels will increase. It can be seen that the more small gap pixels there are, the worse the image quality. Therefore, pixels can be classified with small gap pixels as the target, and the quality of the image can be measured by the proportion of small gap pixels in the image.

[0113] In one embodiment, for each pixel point, according to the preset corresponding relationship between the membership degree of each category and the pixel difference, the membership degree of the pixel point in each category is calculated, which may specifically include:

[0114] For each pixel, the degree of membership of each pixel in each category is calculated based on the pixel difference between the pixel and its adjacent pixels, the expected value of the pixel in each category, and the Gaussian random number of the pixel in each category.

[0115] In the above implementation, the corresponding relationship between the preset membership degree of each classification and the pixel difference can be shown as the following formula 1.

[0116]

[0117] In formula 1, k can represent the classification of pixels. For example, when k is 1, it can represent pixels with large gaps, when k is 2, it can represent pixels with small gaps, and when k is 3, it can represent pixels with no gaps. C(x) k It can represent the membership of the pixel in the category k. x can represent the value of the pixel difference. k It can represent the expected value of classification k. k It can represent the entropy of classification k, He k It can represent the super entropy of category k. En' can represent a Gaussian random number with mean En and variance He.

[0118] After determining the classification of each pixel in the second image according to Formula 1, the ratio α of the number of pixels in the target classification to the total number of pixels in the second image can be calculated. 1-α is used as the image quality identification result corresponding to the second image, indicating the probability that the second image is a high-quality image.

[0119] According to the above implementation, an image quality discriminator that conforms to human logic can be constructed based on the cloud model, thereby improving the accuracy of the discriminator in judging image quality. In addition, the above image quality discriminator only needs to process nine parameters, which greatly reduces the spatial complexity of the model and is more conducive to the deployment of the model in actual application scenarios.

[0120] In one embodiment, the iteration termination condition may include that the loss function value of the generative adversarial network model is less than a preset threshold. The loss function value of the generative adversarial network model can be specifically calculated by the following formula

[0121] Calculated using formula 2.

[0122]

[0123] In formula 2, P r It can represent the distribution of complete images in the training set, P θ can represent the distribution of images generated by the generator. sup can represent the supremum. y can represent the input image of the joint discriminator. f g () can represent the completion completeness discriminator, f g (y) can represent the output result of the completion completeness discriminator. p () can represent the completion content identifier, f p (y) can represent the output result of the completion content identifier. q () can represent the image quality discriminator, f q (y) may represent the output result of the image quality discriminator. It can be expressed in two different image distributions P r and P θ Next, the function f g The expected value difference, ‖f g ‖ L ≤1 g It can represent the constraints of the adversarial loss function of the completion discriminator in the generative adversarial network model. It can be expressed in two different image distributions P r and P θ Next, the function f p The expected value difference, ‖f p ‖ L ≤1 pIt can represent the constraints of the adversarial loss function of the completion content discriminator in the generative adversarial network model. It can be expressed in two different image distributions P r and P θ Next, the function f q The expected value difference, ‖f q ‖ L ≤1 q It can represent the constraints of the adversarial loss function of the image quality discriminator in the generative adversarial network model.

[0124] In one embodiment, the first image may include target pixels and non-target pixels. The target pixels may include pixels in a known area of ​​the first image, and the non-target pixels may include pixels in a missing area of ​​the first image.

[0125] For each first image, weights are assigned to the pixels in the first image to obtain a weight distribution map corresponding to each first image, which may specifically include performing the following steps on each first image.

[0126] The first pixel point is determined by a region growing algorithm, the first pixel point is a target pixel point that is similar to and connected to the second pixel point, and the second pixel point is a target pixel point in the first image that is adjacent to the missing region.

[0127] The pixels in the first image are classified to obtain first target pixels, second target pixels and non-target pixels, wherein the first target pixels include first pixels and second pixels, the second target pixels include target pixels other than the first target pixels, and the non-target pixels include pixels in the missing area of ​​the first image.

[0128] According to the preset weight distribution rule, the first weight corresponding to the first target pixel point, the second weight corresponding to the second target pixel point and the third weight corresponding to the non-target pixel point are determined, wherein the first weight, the second weight and the third weight decrease in sequence.

[0129] A weight distribution map corresponding to the first image is generated based on a first weight corresponding to the first target pixel, a second weight corresponding to the second target pixel, and a third weight corresponding to the non-target pixel.

[0130] In one embodiment, generating a second weight distribution map based on a first weight corresponding to a first target pixel, a second weight corresponding to a second target pixel, and a third weight corresponding to a non-target pixel may specifically include:

[0131] Based on the first weight corresponding to the first target pixel, the second weight corresponding to the second target pixel, and the third weight corresponding to the non-target pixel, a weight distribution map corresponding to the first image is directly generated.

[0132] In the weight distribution map corresponding to the first image, the weight of the first target pixel is the first weight, the weight of the second target pixel is the second weight, and the weight of the non-target pixel is the third weight.

[0133] In one embodiment, the generator includes an encoder. Based on the first weight corresponding to the first target pixel, the second weight corresponding to the second target pixel, and the third weight corresponding to the non-target pixel, generating a weight distribution map corresponding to the first image may specifically include:

[0134] An initial weight distribution map is generated based on a first weight corresponding to the first target pixel, a second weight corresponding to the second target pixel, and a third weight corresponding to the non-target pixel.

[0135] The initial weight distribution map is input into the attention mechanism module, and the structure and parameters of the attention mechanism module are the same as those of the encoder.

[0136] The initial weight distribution map is processed by each network layer of the attention mechanism module respectively to obtain the attention weight distribution map corresponding to each network layer.

[0137] For ease of understanding, the above implementation is illustrated here with a specific example. In one example, for an input image with a size of image width w×image height h×image channel number 3, a region growing algorithm is used to locate known pixels that are similar to and connected to known pixels around the missing area, and a weight of 1 is assigned to them, a weight of 0.5 is assigned to other known pixels, and a weight of 0 is assigned to the pixels in the missing area, thereby obtaining an initial weight distribution map of the input image with a size of w×h×3. The initial weight distribution map is input into the attention mechanism module, and the attention mechanism module built based on the partial convolutional layer is used to calculate the attention weight distribution map for each network layer of the generator.

[0138] It is understandable that the structure and parameters of the attention mechanism module are the same as those of the encoder. During the training process of the generative adversarial network model, the parameters of the attention mechanism module can be adjusted according to the adjustment of the encoder parameters in the generator. Accordingly, after adjusting the parameters of the attention mechanism module, the weight distribution map corresponding to the first image can be regenerated, and the model training can be performed through the newly generated weight distribution map corresponding to the first image.

[0139] According to the above implementation, the region growing algorithm is used to obtain the initial image weight distribution map, and then the attention weight distribution of each network layer in the feature extraction process is calculated by the attention mechanism module based on the encoder structure in the generator. In this way, the attention weight distribution map can be calculated for each network layer of the generator using the attention mechanism module constructed based on the partial convolution layer. The calculation result shows that the initial attention weight distribution map changes after each layer of the network in the encoder is calculated, for example, the weight of a certain area is 1 in the input stage, and the weight becomes 0.7 after several layers of network calculation. In this way, the changes in weights during feature extraction are discovered, and the changes are fed back to the generator, so that feature extraction can be performed more accurately. In this way, the generator can focus on extracting known information that is strongly related to the missing area during the feature extraction process, while reducing the interference of the missing area pixel value on the final completion result. In this way, the robustness of the generator can be improved, which is conducive to the generator to perform reasonable and accurate completion in various missing situations.

[0140] The following is combined with Figure 4 The image processing method provided in the embodiment of the present application is described in detail.

[0141] Figure 4 FIG. 1 is a flow chart of an image processing method provided by an embodiment of the present application. Figure 4 As shown, the image processing method may include the following steps S210 to S260.

[0142] S210, obtaining an image to be processed, where the image to be processed includes a missing area.

[0143] S220, weighting the pixels in the image to be processed to obtain a weight distribution map.

[0144] S230, inputting the image to be processed and the weight distribution map into the target generator.

[0145] S240, extracting features according to the weight distribution map and the image to be processed by the target generator, and completing the missing area of ​​the image to be processed according to the feature extraction result to obtain a completed image.

[0146] S250, inputting the completed image into an image quality identifier, so that the image quality identifier identifies the image quality of the completed image.

[0147] S260: When the image quality of the complemented image passes the identification, determine the complemented image as the target image.

[0148] In step S260, the image quality of the completed image passes the identification, which may indicate that the image quality identification result of the completed image meets the preset high-quality image determination condition. The high-quality image determination condition may be, for example, that the image quality score of the completed image reaches a preset threshold, or that the probability of the completed image being a high-quality image reaches a preset threshold, etc.

[0149] The image processing method of the embodiment of the present application can obtain an image to be processed, which includes a missing area. Then, the pixel points in the image to be processed are weighted to obtain a weight distribution map. Then, the image to be processed and the weight distribution map are input into the target generator. Next, the target generator performs feature extraction according to the weight distribution map and the image to be processed, and according to the feature extraction result, the missing area of ​​the image to be processed is completed to obtain a completed image. Subsequently, the completed image is input into the image quality discriminator, so that the image quality discriminator can identify the image quality of the completed image. When the image quality of the completed image passes the identification, the completed image is determined to be the target image. In this way, not only can the missing area of ​​the image to be processed be completed, but also the target image can have a higher image quality. In this way, it is beneficial to improve the utilization rate of the unmanned boat image.

[0150] In one embodiment, the target generator is trained by a generative adversarial network model, and the generative adversarial network model is constructed by a generator and a joint discriminator, and the joint discriminator includes an image quality discriminator.

[0151] In the above embodiments, the joint discriminator may include an image quality discriminator and other discriminators that can be used to identify image completion situations.

[0152] According to the above embodiment, the target generator is trained by a generative adversarial network model. The generative adversarial network model is constructed by a generator and a joint discriminator, and the joint discriminator includes an image quality discriminator. In the generative adversarial network model, the generator and the joint discriminator confront each other and learn continuously. In this way, the trained target generator can generate a completed image with higher image quality.

[0153] In one embodiment, the joint discriminator may also include a completion completeness discriminator.

[0154] After extracting features according to the weight distribution map and the image to be processed by the target generator and completing the missing area of ​​the image to be processed according to the feature extraction result and obtaining the completed image, the method may further include:

[0155] The completed image is input to the completion completion discriminator, so that the completion completion discriminator discriminates the completion completion of the missing area.

[0156] In the above embodiment, the completion completion discriminator can be used to determine the completion of the completed image for the image completion task. In this way, the completed image can be evaluated from the overall content of the generated result.

[0157] In one embodiment, the joint discriminator may also include a complementary content discriminator.

[0158] After extracting features according to the weight distribution map and the image to be processed by the target generator and completing the missing area of ​​the image to be processed according to the feature extraction result and obtaining the completed image, the method may further include:

[0159] The completed image is input to the completed content identifier, so that the completed content identifier identifies the completed content of the missing area.

[0160] In the above embodiment, the complement content identifier can be used to determine the degree of consistency between the complement region in the complement image and the corresponding region in the complete image. In this way, the complement image can be evaluated based on the content of the complement region.

[0161] In one embodiment, weight allocation is performed on the pixels in the image to be processed, which may specifically include:

[0162] The first pixel point is determined by a region growing algorithm, the first pixel point is a target pixel point that is similar to and connected to the second pixel point, and the second pixel point is a target pixel point adjacent to the missing region in the image to be processed.

[0163] The pixel points in the image to be processed are classified to obtain first target pixel points, second target pixel points and non-target pixel points, wherein the first target pixel points include the first pixel points and the second pixel points, the second target pixel points include the target pixel points other than the first target pixel points, and the non-target pixel points include the pixel points of the missing area of ​​the image to be processed.

[0164] According to the preset weight distribution rule, the first weight corresponding to the first target pixel point, the second weight corresponding to the second target pixel point and the third weight corresponding to the non-target pixel point are determined, wherein the first weight, the second weight and the third weight decrease in sequence.

[0165] A weight distribution map is generated based on a first weight corresponding to the first target pixel, a second weight corresponding to the second target pixel, and a third weight corresponding to the non-target pixel.

[0166] According to the above implementation, different weights are assigned to different categories of pixels, which is conducive to the target generator to perform targeted feature extraction on the known area of ​​the image to be processed according to the weight distribution map. In this way, the accuracy of the completion result can be effectively improved, and the robustness of the generator in the sea surface unmanned boat scene can be improved.

[0167] In one embodiment, the target generator may include an encoder. Based on a first weight corresponding to a first target pixel, a second weight corresponding to a second target pixel, and a third weight corresponding to a non-target pixel, a weight distribution map is generated, which may specifically include:

[0168] An initial weight distribution map is generated based on a first weight corresponding to the first target pixel, a second weight corresponding to the second target pixel, and a third weight corresponding to the non-target pixel.

[0169] The initial weight distribution map is input into the attention mechanism module, and the structure and parameters of the attention mechanism module are the same as those of the encoder.

[0170] The initial weight distribution map is processed by each network layer of the attention mechanism module respectively to obtain the attention weight distribution map corresponding to each network layer.

[0171] According to the above implementation, the structure and parameters of the attention mechanism module are the same as those of the encoder. The attention mechanism module provides a weight distribution map for each network layer of the encoder, which is conducive to the target generator to accurately extract image features. In this way, the accuracy of the completion result can be further improved, and the robustness of the generator in the sea surface unmanned boat scene can be further improved.

[0172] In one embodiment, the completed image is input into an image quality discriminator so that the image quality discriminator discriminates the image quality of the completed image, which may specifically include:

[0173] The completed image is fed into the image quality discriminator.

[0174] The pixel difference between each pixel and its adjacent pixels in the completed image is extracted through the image quality discriminator.

[0175] For each pixel point, the membership of the pixel point in each category is calculated according to the corresponding relationship between the membership of each category and the pixel difference; and the category of the pixel point is determined according to the membership of the pixel point in each category.

[0176] Calculate the ratio of the number of pixels in the target classification to the total number of pixels in the completed image.

[0177] An image quality identification result corresponding to the completed image is generated based on the ratio.

[0178] According to the above implementation, the image quality discriminator can classify pixels into three categories: large gap pixels, small gap pixels and no gap pixels according to the gap between each pixel in the input image and its surrounding pixels, and each type of pixel is represented by a triplet (Ex, En, He). After motion blur occurs, the number of small gap pixels will increase. It can be seen that the more small gap pixels there are, the worse the image quality. Therefore, pixels can be classified with small gap pixels as the target, and the quality of the image can be measured by the proportion of small gap pixels in the image.

[0179] In one embodiment, for each pixel point, according to the preset corresponding relationship between the membership degree of each category and the pixel difference, the membership degree of the pixel point in each category is calculated, which may specifically include:

[0180] For each pixel, the degree of membership of each pixel in each category is calculated based on the pixel difference between the pixel and its adjacent pixels, the expected value of the pixel in each category, and the Gaussian random number of the pixel in each category.

[0181] In the above implementation, the corresponding relationship between the preset membership degree of each classification and the pixel difference can be expressed by the above-mentioned formula 1, which will not be elaborated here.

[0182] According to the above implementation, an image quality discriminator that conforms to human logic can be constructed based on the cloud model, thereby improving the accuracy of the discriminator in judging image quality. In addition, the above image quality discriminator only needs to process nine parameters, which greatly reduces the space complexity of the image quality discriminator and is more conducive to the deployment of the model in actual application scenarios.

[0183] Based on the same inventive concept as the image processing method, an embodiment of the present application also provides an image processing device.

[0184] like Figure 5 As shown, the image processing device 300 may include an acquisition module 301 , an allocation module 302 , a first input module 303 , a processing module 304 and a determination module 305 .

[0185] The acquisition module 301 is used to acquire the image to be processed, where the image to be processed includes a missing area.

[0186] The allocation module 302 is used to allocate weights to the pixels in the image to be processed to obtain a weight distribution map.

[0187] The first input module 303 is used to input the image to be processed and the weight distribution map into the target generator.

[0188] The processing module 304 is used to perform feature extraction according to the weight distribution map and the image to be processed through the target generator, and to complete the missing area of ​​the image to be processed according to the feature extraction result to obtain a completed image.

[0189] The first input module 303 is further used to input the completed image to the image quality identifier, so that the image quality identifier can identify the image quality of the completed image.

[0190] The determination module 305 is used to determine that the complemented image is the target image when the image quality of the complemented image passes the identification.

[0191] The image processing device of the embodiment of the present application can obtain an image to be processed, which includes a missing area. Then, weights are assigned to the pixels in the image to be processed to obtain a weight distribution map. Then, the image to be processed and the weight distribution map are input into the target generator. Next, the target generator performs feature extraction based on the weight distribution map and the image to be processed, and performs image completion on the missing area of ​​the image to be processed based on the feature extraction result to obtain a completed image. Subsequently, the completed image is input into the image quality discriminator, so that the image quality discriminator can identify the image quality of the completed image. When the image quality of the completed image passes the identification, the completed image is determined to be the target image. In this way, not only can the missing area of ​​the image to be processed be completed, but also the target image can have a higher image quality. In this way, it is beneficial to improve the utilization rate of the unmanned boat image.

[0192] In one embodiment, the target generator is trained by a generative adversarial network model, and the generative adversarial network model is constructed by a generator and a joint discriminator, and the joint discriminator includes an image quality discriminator.

[0193] In one embodiment, the joint discriminator may also include a completion completeness discriminator.

[0194] The device may also include:

[0195] The second input module is used to extract features according to the weight distribution map and the image to be processed through the target generator, and complete the missing area of ​​the image to be processed according to the feature extraction result. After the completed image is obtained, the completed image is input to the completion completion discriminator so that the completion completion discriminator can identify the completion completion of the missing area.

[0196] In one embodiment, the joint discriminator may also include a complementary content discriminator.

[0197] The device may also include:

[0198] The third input module is used to perform feature extraction according to the weight distribution map and the image to be processed through the target generator, and complete the missing area of ​​the image to be processed according to the feature extraction result. After the completed image is obtained, the completed image is input to the completion content identifier so that the completion content identifier can identify the completion content of the missing area.

[0199] In one embodiment, the allocation module is used to allocate weights to pixels in the image to be processed to obtain a weight distribution map, which may specifically include:

[0200] The determination submodule is used to determine the first pixel point through a region growing algorithm, where the first pixel point is a target pixel point similar to and connected to the second pixel point, and the second pixel point is a target pixel point adjacent to the missing area in the image to be processed.

[0201] The classification submodule is used to classify the pixel points in the image to be processed to obtain the first target pixel point, the second target pixel point and the non-target pixel point, wherein the first target pixel point includes the first pixel point and the second pixel point, the second target pixel point includes the target pixel point other than the first target pixel point, and the non-target pixel point includes the pixels in the missing area of ​​the image to be processed.

[0202] The determination submodule is also used to determine the first weight corresponding to the first target pixel point, the second weight corresponding to the second target pixel point, and the third weight corresponding to the non-target pixel point according to a preset weight allocation rule, wherein the first weight, the second weight, and the third weight decrease in sequence.

[0203] The generating submodule is used to generate a weight distribution map based on the first weight corresponding to the first target pixel point, the second weight corresponding to the second target pixel point, and the third weight corresponding to the non-target pixel point.

[0204] In one embodiment, the target generator includes an encoder. The generation submodule can be specifically used for:

[0205] An initial weight distribution map is generated based on a first weight corresponding to the first target pixel, a second weight corresponding to the second target pixel, and a third weight corresponding to the non-target pixel.

[0206] The initial weight distribution map is input into the attention mechanism module, and the structure and parameters of the attention mechanism module are the same as those of the encoder.

[0207] The initial weight distribution map is processed by each network layer of the attention mechanism module respectively to obtain the attention weight distribution map corresponding to each network layer.

[0208] In one embodiment, the input module is further used to input the completed image into the image quality discriminator, so that the image quality discriminator discriminates the image quality of the completed image, which may specifically include:

[0209] The input submodule is used to input the completed image into the image quality discriminator.

[0210] The extraction submodule is used to extract the pixel difference between each pixel and adjacent pixels in the completed image through an image quality discriminator.

[0211] The processing submodule is used to calculate the membership of each pixel point in each category according to the corresponding relationship between the membership of each category and the pixel difference; and determine the category of the pixel point according to the membership of the pixel point in each category.

[0212] The calculation submodule is used to calculate the ratio of the number of pixels of the target classification to the total number of pixels of the completed image.

[0213] An image quality identification result corresponding to the completed image is generated based on the ratio.

[0214] In one embodiment, the processing submodule is used to calculate the membership of each pixel point in each category according to the corresponding relationship between the membership of each category and the pixel difference, which may specifically include:

[0215] The calculation subunit is used to calculate the membership of each pixel point in each category according to the pixel difference between the pixel point and the adjacent pixel points, the expected value of the pixel point in each category and the Gaussian random number of the pixel point in each category.

[0216] The image processing device provided in the embodiment of the present application can achieve Figure 4 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0217] Figure 6 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application is shown.

[0218] The electronic device may include a processor 401 and a memory 402 storing computer program instructions.

[0219] Specifically, the processor 401 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0220] Memory 402 may include a large capacity memory for data or instructions. By way of example and not limitation, memory 402 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, memory 402 may include a removable or non-removable (or fixed) medium. In appropriate cases, memory 402 may be inside or outside of an integrated gateway disaster recovery device. In a specific embodiment, memory 402 is a non-volatile solid-state memory.

[0221] The memory may include read-only memory (ROM), random access memory (RAM), magnetic disk storage media devices, optical storage media devices, flash memory devices, electrical, optical or other physical / tangible memory storage devices. Thus, typically, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to an aspect of the present disclosure.

[0222] The processor 401 implements any one of the image processing methods in the above embodiments by reading and executing computer program instructions stored in the memory 402 .

[0223] As an example, the electronic device may further include a communication interface 403 and a bus 410. Figure 6 As shown, the processor 401, the memory 402, and the communication interface 403 are connected via a bus 410 and communicate with each other.

[0224] The communication interface 403 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0225] Bus 410 includes hardware, software or both, and the parts of image processing equipment are coupled to each other. For example, but not limitation, bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front side bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 410 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the present application considers any suitable bus or interconnection.

[0226] The electronic device can execute the image processing method in the embodiment of the present application, thereby realizing the combination Figure 4 and Figure 5 The invention describes an image processing method and apparatus.

[0227] In addition, in combination with the image processing method in the above embodiment, the embodiment of the present application can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any one of the image processing methods in the above embodiment is implemented.

[0228] An embodiment of the present application also provides a computer program product, including a computer program, which implements any one of the image processing methods in the above embodiments when the computer program is processed and executed.

[0229] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0230] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0231] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0232] Aspects of the present disclosure are described above with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It can also be understood that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs a specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0233] The above is only a specific implementation of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.

Claims

1. An image processing method, characterized in that: include: Acquire an image to be processed, wherein the image to be processed includes a missing area; Performing weight distribution on pixel points in the image to be processed to obtain a weight distribution map; Inputting the image to be processed and the weight distribution map into a target generator; The target generator performs feature extraction according to the weight distribution map and the image to be processed, and performs image completion on the missing area of ​​the image to be processed according to the feature extraction result to obtain a completed image; Inputting the completed image into an image quality discriminator, so that the image quality discriminator discriminates the image quality of the completed image; When the image quality of the complemented image passes the identification, the complemented image is determined to be the target image.

2. The method according to claim 1, characterized in that The target generator is trained by a generative adversarial network model, and the generative adversarial network model is constructed by a generator and a joint discriminator, and the joint discriminator includes the image quality discriminator.

3. The method according to claim 2, characterized in that The joint discriminator also includes a completion completion discriminator; After extracting features according to the weight distribution map and the image to be processed by the target generator and completing the missing area of ​​the image to be processed according to the feature extraction result to obtain the completed image, the method further includes: The completed image is input to the completion degree identifier, so that the completion degree identifier identifies the completion degree of the missing region.

4. The method according to claim 2, characterized in that: The joint discriminator also includes a completion content discriminator; After extracting features according to the weight distribution map and the image to be processed by the target generator and completing the missing area of ​​the image to be processed according to the feature extraction result to obtain the completed image, the method further includes: The complement image is input to the complement content identifier, so that the complement content identifier identifies the complement content of the missing area.

5. The method according to claim 1, characterized in that The step of performing weight distribution on the pixels in the image to be processed to obtain a weight distribution map includes: Determine a first pixel point by a region growing algorithm, wherein the first pixel point is a target pixel point that is similar to and connected to a second pixel point, and the second pixel point is a target pixel point that is adjacent to a missing region in the image to be processed; Classifying the pixels in the image to be processed to obtain first target pixels, second target pixels and non-target pixels, wherein the first target pixels include the first pixels and the second pixels, the second target pixels include target pixels other than the first target pixels, and the non-target pixels include pixels in a missing area of ​​the image to be processed; According to a preset weight allocation rule, determine a first weight corresponding to the first target pixel, a second weight corresponding to the second target pixel, and a third weight corresponding to the non-target pixel, wherein the first weight, the second weight, and the third weight decrease in sequence; The weight distribution map is generated based on a first weight corresponding to the first target pixel, a second weight corresponding to the second target pixel, and a third weight corresponding to a non-target pixel.

6. The method according to claim 5, characterized in that The target generator includes an encoder; the generating the weight distribution map based on the first weight corresponding to the first target pixel, the second weight corresponding to the second target pixel, and the third weight corresponding to the non-target pixel includes: generating an initial weight distribution map based on a first weight corresponding to the first target pixel, a second weight corresponding to the second target pixel, and a third weight corresponding to the non-target pixel; Inputting the initial weight distribution map into an attention mechanism module, wherein the structure and parameters of the attention mechanism module are the same as those of the encoder; The initial weight distribution map is processed by each network layer of the attention mechanism module respectively to obtain the attention weight distribution map corresponding to each network layer.

7. The method according to claim 1, characterized in that Inputting the completed image into an image quality discriminator so that the image quality discriminator discriminates the image quality of the completed image, comprising: inputting the completed image into the image quality discriminator; Extracting the pixel difference between each pixel and adjacent pixels in the completed image by the image quality discriminator; For each pixel point, according to the corresponding relationship between the membership of each classification and the pixel difference, the membership of the pixel point in each classification is calculated; according to the membership of the pixel point in each classification, the classification of the pixel point is determined; Calculating the ratio of the number of pixels of the target classification to the total number of pixels of the completed image; An image quality identification result corresponding to the complemented image is generated based on the ratio.

8. The method according to claim 7, characterized in that The step of calculating the membership of each pixel point in each category according to the corresponding relationship between the membership of each category and the pixel difference is as follows: For each pixel, the degree of membership of each pixel in each category is calculated based on the pixel difference between the pixel and its adjacent pixels, the expected value of the pixel in each category, and the Gaussian random number of the pixel in each category.

9. An image processing device, characterized in that: include: An acquisition module, used for acquiring an image to be processed, wherein the image to be processed includes a missing area; An allocation module, used for performing weight allocation on pixel points in the image to be processed to obtain a weight distribution map; An input module, used for inputting the image to be processed and the weight distribution map into a target generator; A processing module, configured to perform feature extraction according to the weight distribution map and the image to be processed through the target generator, and to complete the missing area of ​​the image to be processed according to the feature extraction result to obtain a completed image; The input module is further used to input the completed image into the image quality discriminator, so that the image quality discriminator discriminates the image quality of the completed image; The determination module is used to determine that the completed image is a target image when the image quality of the completed image passes the identification.

10. An image processing device, characterized in that: The device comprises: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the image processing method according to any one of claims 1 to 8 is implemented.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, the image processing method according to any one of claims 1 to 8 is implemented.

12. A computer program product, characterized in that When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device is enabled to perform the image processing method as described in any one of claims 1 to 8.

Images