Image generation method and device and storage medium

Abstract

The invention relates to an image generation method and device and a storage medium, and the method comprises the steps: obtaining an original image, and determining a to-be-processed region in the original image; preprocessing the to-be-processed area to obtain an image mask; processing the image mask to obtain a target prior feature of the to-be-processed region; and generating a target image according to the image mask and the target prior feature. In this way, the target prior features of the to-be-processed area in the original image are generated through the image mask, the target image is generated accordingly, the problem that reference pixels are insufficient under the condition of the large-area to-be-processed area is solved, and the image processing effect is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an image generation method, device and storage medium. Background Art

[0002] With the rapid development and widespread application of digital technology, people's lives are filled with a large number of digital images. However, these images may suffer various forms of damage during the generation, storage and transmission process, such as pixel loss, blur, damage, etc. This damage will affect the quality and usability of the image, so there is a need to repair the image, or these images may carry various forms of unnecessary content during the generation process, such as background characters in the image, so there is a need to erase the image.

[0003] In the related art, deep learning networks are usually used to repair or erase images. However, in the process of repairing or erasing images, there are scenes with large-scale erasure areas or large-scale repair areas. For this reason, the receptive field of the deep learning network is expanded to better refer to global information. For scenes with large-scale erasure areas or large-scale repair areas, although the receptive field of the deep learning network is expanded, large areas of pixels are erased or damaged, resulting in limited pixels that can be referenced, which in turn leads to poor image repair or erasure effects. Summary of the invention

[0004] In order to solve the above-mentioned scenario of large-area erasure or large-area repair, although the receptive field of the deep learning network is expanded, large-area pixels are erased or damaged, resulting in limited reference pixels, which in turn leads to poor image repair or erasure effects. The embodiments of the present application provide an image generation method, device, electronic device and storage medium. The specific technical solutions are as follows:

[0005] In a first aspect of an embodiment of the present application, a method for generating an image is first provided, the method comprising:

[0006] Acquire an original image, and determine a region to be processed in the original image;

[0007] Preprocessing the area to be processed to obtain an image mask;

[0008] Processing the image mask to obtain target prior features of the area to be processed;

[0009] A target image is generated according to the image mask and the target priori features.

[0010] In an optional implementation, the processing of the image mask to obtain the target prior features of the area to be processed includes:

[0011] Performing feature extraction processing on the image mask to obtain initial prior features of the area to be processed;

[0012] A preset noise is obtained, and a target priori feature of the area to be processed is generated according to the preset noise and the initial priori feature.

[0013] In an optional implementation, the performing feature extraction processing on the image mask to obtain the initial prior features of the area to be processed includes:

[0014] The image mask is input into a pre-trained prior feature extraction network to obtain initial prior features of the area to be processed.

[0015] In an optional embodiment, the pre-trained prior feature extraction network includes N pre-trained prior feature extraction sub-networks, where N is a positive integer greater than or equal to 1;

[0016] The step of inputting the image mask into a pre-trained prior feature extraction network to obtain initial prior features of the area to be processed includes:

[0017] Iterate and perform the following operations until the Nth intermediate prior feature is obtained and then stop the following operations:

[0018] Inputting the i-th feature to be extracted into the i-th pre-trained prior feature extraction subnetwork to obtain the i-th intermediate prior feature;

[0019] Among them, the i-th intermediate prior feature is the i+1-th feature to be extracted, the 1st feature to be extracted is the image mask, the N-th intermediate prior feature is the initial prior feature, and i is 1, 2, 3, ..., N in sequence.

[0020] In an optional implementation, generating the target priori feature of the area to be processed according to the preset noise and the initial priori feature includes:

[0021] Iterate the following operations until the number of iterations reaches the preset first number threshold and then stops the following operations:

[0022] Input the features to be denoised in the current round into the pre-trained denoising network to obtain denoising prior features;

[0023] Among them, the features to be denoised in the first round are the concatenation results of the preset noise and the initial prior features, the denoising prior features are the features to be denoised in the next round, and the denoising prior features in the last round are the target prior features.

[0024] In an optional implementation, generating a target image according to the image mask and the target prior features includes:

[0025] Inputting the image mask and the target prior features into a pre-trained image generation network to obtain a processed image;

[0026] The processed image is fused with the non-to-be-processed area in the original image to obtain a target image.

[0027] In an optional implementation, the step of fusing the processed image with the non-to-be-processed region in the original image to obtain a target image includes:

[0028] Fusing the processed image with the non-to-be-processed area in the original image to obtain a fused image;

[0029] Searching the processed area corresponding to the area to be processed from the fused image, and detecting whether the processed area is a pure color area;

[0030] In the case where the processed area is a pure color area, determining a target pixel value, and filling the processed area according to the target pixel value to obtain a target image;

[0031] Alternatively, when the processed area is not a pure color area, the fused image is determined as the target image.

[0032] In an optional implementation, the detecting whether the processed area is a pure color area includes:

[0033] Detecting pixel values ​​in the processed area, and determining that the processed area is a pure color area when the pixel values ​​in the processed area are close to each other; comprising:

[0034] Acquire a connected domain of the processed area, and perform expansion processing on the connected domain to obtain a target connected domain;

[0035] Subtract the target connected domain from the connected domain to obtain a difference region;

[0036] When the pixel values ​​in the difference area are close, the processed area is determined to be a pure color area; including: obtaining the maximum pixel value and the minimum pixel value in the difference area, and subtracting the maximum pixel value from the minimum pixel value to obtain a pixel value difference; when the pixel value difference is less than or equal to a preset pixel threshold, the processed area is determined to be a pure color area.

[0037] In an optional implementation, determining the target pixel value includes:

[0038] A pixel value is selected from the difference area, and the pixel value is determined as a target pixel value.

[0039] In an optional embodiment, the pre-trained image generation network is obtained by:

[0040] Acquire a first sample mask image and a first sample processed image;

[0041] Searching, from the first sample processed image or the first sample original image, a first sample processed area corresponding to the mask in the first sample mask image;

[0042] Preprocessing the first sample processing area to obtain a first sample image mask;

[0043] Processing the first sample image mask to obtain a priori features of a predicted target corresponding to the first sample processing area;

[0044] Inputting the first sample image mask and the predicted target prior features into an initially trained image generation network to obtain a predicted processed image;

[0045] Fusing the predicted processed image with the first sample processed image or a non-first sample processed region in the first sample original image to obtain a predicted target image;

[0046] Determine an image loss between the predicted target image and the first sample processed image, and train the initially trained image generation network according to the image loss;

[0047] When the image loss converges, the training is stopped to obtain the pre-trained image generation network.

[0048] In an optional implementation, the acquiring the first sample mask image and the first sample processed image includes:

[0049] Get the original slide, and clear the elements in the slide to obtain the target slide;

[0050] Converting the original slide into a first image and converting the target slide into a second image;

[0051] Generate a first sample mask image according to the first image and the second image;

[0052] The first image is determined as a first sample original image, and the second image is determined as a first sample processed image.

[0053] In an optional implementation, generating a first sample mask image according to the first image and the second image includes:

[0054] Subtract the first image from the second image to obtain a difference image;

[0055] Detecting the circumscribed rectangular frame of each element in the difference image, and obtaining the vertex coordinates of the circumscribed rectangular frame;

[0056] From the vertex coordinates, searching for the vertex coordinates with the same abscissa;

[0057] The circumscribed rectangular frames of the elements corresponding to the vertex coordinates with the same abscissa are merged to obtain a first sample mask image.

[0058] In an optional implementation, the processing of the first sample image mask to obtain a priori features of a predicted target corresponding to the first sample processing area includes:

[0059] Performing feature extraction processing on the first sample image mask to obtain predicted initial prior features corresponding to the first sample processing area;

[0060] Performing feature extraction processing on the first sample image mask and the first sample processed image to obtain a compressed priori feature;

[0061] Performing noise adding processing on the compressed priori feature to obtain a noisy priori feature;

[0062] A predicted target priori feature corresponding to the first sample processing area is generated according to the noisy priori feature and the predicted initial priori feature.

[0063] In an optional implementation, the performing feature extraction processing on the first sample image mask to obtain the predicted initial prior features corresponding to the first sample processing area includes:

[0064] The first sample image mask is input into a pre-trained prior feature extraction network to obtain a predicted initial prior feature corresponding to the first sample processing area.

[0065] In an optional implementation, the performing feature extraction processing on the first sample image mask and the first sample processed image to obtain a compressed priori feature includes:

[0066] The first sample image mask and the first sample processed image are input into a pre-trained prior feature extraction network to obtain a compressed prior feature.

[0067] In an optional implementation, the step of adding noise to the compressed priori feature to obtain the noisy priori feature includes:

[0068] Iterate the following steps until the number of iterations reaches the preset second number threshold and then stop the following operations:

[0069] Input the features to be denoised in the current round into the denoising network to obtain the denoising prior features;

[0070] The features to be noised in the first round are the compressed prior features, the noised prior features are the features to be noised in the next round, and the noised prior features in the last round are the noisy prior features.

[0071] In an optional implementation, generating the predicted target priori feature corresponding to the first sample processing area according to the noisy priori feature and the predicted initial priori feature includes:

[0072] Iterate the following operations until the number of iterations reaches the preset first number threshold and then stops the following operations:

[0073] Input the features of the samples to be denoised in the current round into the denoising network to obtain the predicted denoising prior features;

[0074] Among them, the first round of sample denoising features are the concatenation results of the noisy prior features and the predicted initial prior features, the predicted denoising prior features are the next round of sample denoising features, and the last round of predicted denoising prior features are the predicted target prior features.

[0075] In an optional embodiment, the initially trained image generation network is obtained by:

[0076] Acquire a second sample mask image and a second sample processed image;

[0077] Searching, from the second sample processed image or the second sample original image, a second sample processed region corresponding to the mask in the second sample mask image;

[0078] Preprocessing the second sample processing area to obtain a second sample image mask;

[0079] Performing feature extraction on the second sample image mask and the second sample processed image to obtain initial compression prior features;

[0080] Inputting the second sample image mask and the initial compressed prior features into an image generation network to obtain an initial predicted processed image;

[0081] Fusing the initially predicted processed image with the second sample processed image or a non-sample processed area in the second sample original image to obtain an initially predicted image;

[0082] Determine an initial image loss between the initial predicted image and the second sample processed image, and train the image generation network according to the initial image loss;

[0083] When the initial image loss converges, the training is stopped to obtain the initially trained image generation network.

[0084] In an optional implementation, the extracting features from the second sample image mask and the second sample processed image to obtain initial compression prior features includes:

[0085] The second sample image mask and the second sample processed image are input into a priori feature extraction network to obtain initial compression prior features.

[0086] In a second aspect of the embodiments of the present application, an image generating device is further provided, the device comprising:

[0087] An image acquisition module, used to acquire an original image and determine a region to be processed in the original image;

[0088] A region processing module, used for preprocessing the region to be processed to obtain an image mask;

[0089] A mask processing module, used for processing the image mask to obtain a priori target features of the area to be processed;

[0090] The image generation module is used to generate a target image according to the image mask and the target prior features.

[0091] In a third aspect of the embodiments of the present application, there is further provided an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0092] Memory, used to store computer programs;

[0093] The processor is used to implement any image generation method described in the first aspect when executing the program stored in the memory.

[0094] In a fourth aspect of the embodiments of the present application, a storage medium is further provided, wherein instructions are stored in the storage medium, and when the storage medium is run on a computer, the computer executes any image generation method described in the first aspect above.

[0095] In a fifth aspect of the embodiments of the present application, a computer program product comprising instructions is also provided, which, when executed on a computer, enables the computer to execute any of the above-mentioned image generation methods.

[0096] The technical solution provided in the embodiment of the present application obtains the original image, determines the area to be processed in the original image, preprocesses the area to be processed to obtain an image mask, processes the image mask to obtain target prior features of the area to be processed, and generates a target image based on the image mask and the target prior features.

[0097] By preprocessing the area to be processed in the original image, an image mask is obtained, and the image mask is processed to obtain the target prior features of the area to be processed. According to the image mask and the target prior features, the target image is generated. In this way, the target prior features of the area to be processed in the original image are generated through the image mask, and the target image is generated accordingly, which alleviates the problem of insufficient reference pixels in the case of a large area to be processed and improves the image processing effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0098] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0099] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0100] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0101] Figure 1 A schematic diagram of an implementation flow of an image generation method shown in an embodiment of the present application;

[0102] Figure 2 A schematic diagram of an original image shown in an embodiment of the present application;

[0103] Figure 3 A schematic diagram of an image mask shown in an embodiment of the present application;

[0104] Figure 4 A schematic diagram of an erased image shown in an embodiment of the present application;

[0105] Figure 5 It is a schematic diagram of an implementation process of another image generation method shown in an embodiment of the present application;

[0106] Figure 6It is a schematic diagram of an implementation process of another image generation method shown in an embodiment of the present application;

[0107] Figure 7 A schematic diagram of an implementation flow of a first-stage network training method shown in an embodiment of the present application;

[0108] Figure 8 A schematic diagram of an implementation flow of a second-stage network training method shown in an embodiment of the present application;

[0109] Fig. 9 is a schematic structural diagram of an image generating device shown in an embodiment of the present application;

[0110] Fig.10 It is a schematic diagram of the structure of an electronic device shown in an embodiment of the present application. DETAILED DESCRIPTION

[0111] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0112] The disclosure below provides many different embodiments or examples to realize the different structures of the present application. In order to simplify the disclosure of the present application, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present application. In addition, the present application can repeat reference numbers and / or letters in different examples. This repetition is for the purpose of simplification and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed.

[0113] like Figure 1 FIG. 1 is a schematic diagram of an implementation flow of an image generation method provided in an embodiment of the present application. The method is applied to an electronic device and may specifically include the following steps:

[0114] S101, obtaining an original image and determining a region to be processed in the original image.

[0115] In an embodiment of the present application, an original image is obtained, wherein the original image may be an image to be repaired or an image to be erased, and thus determining the area to be processed in the original image is essentially determining the area to be repaired or the area to be erased in the original image.

[0116] For example, the original image Figure 2As shown, the original image can be acquired and the area to be erased in the original image can be determined, which means that the user wants to erase some content in the original image.

[0117] It should be noted that the area to be processed in the original image, that is, the area to be repaired or the area to be erased, can be specified by the user or automatically identified, and the embodiments of the present application do not limit this.

[0118] S102, pre-processing the area to be processed to obtain an image mask.

[0119] In the embodiment of the present application, for the area to be processed in the original image determined in the above steps, the area to be processed can be preprocessed to obtain an image mask. In addition, the non-area to be processed in the original image needs to remain unchanged.

[0120] For example, for Figure 2 The original image shown in FIG. 1 can be preprocessed in the area to be erased in the original image. The preprocessing here can be understood as resetting the area to be erased to black. In addition, the non-erased area in the original image remains unchanged, so that the image mask is obtained, as shown in FIG. Figure 3 shown.

[0121] It should be noted that for the area to be processed in the original image, the area to be processed is preprocessed. The preprocessing here can be understood as for concealing the content in the area to be processed, for example, resetting the area to be erased to black to facilitate concealing the content in the area to be erased. The embodiment of the present application is not limited to this.

[0122] S103, processing the image mask to obtain target prior features of the area to be processed.

[0123] In the embodiment of the present application, the image mask obtained in the above steps can be processed to obtain the target prior features of the area to be processed in the original image.

[0124] For example, Figure 3 By processing the image mask shown, the target prior features of the area to be erased in the original image can be obtained.

[0125] S104, generating a target image according to the image mask and the target prior features.

[0126] In an embodiment of the present application, for the target prior features of the area to be processed in the original image obtained in the above steps, and the image mask corresponding to the original image, a target image can be generated based on the image mask and the target prior features.

[0127] For example, get Figure 3After obtaining the image mask shown in FIG. 1 and the target prior features of the area to be erased in the original image, the target image can be generated according to the image mask and the target prior features. The target image is the image after erasure, as shown in FIG. Figure 4 As shown, the content in the original image is erased in this way.

[0128] Through the above description of the technical solution provided in the embodiment of the present application, the area to be processed in the original image is preprocessed to obtain an image mask, and the image mask is processed to obtain target prior features of the area to be processed. According to the image mask and the target prior features, a target image is generated. In this way, the target prior features of the area to be processed in the original image are generated through the image mask, and the target image is generated accordingly, which alleviates the problem of insufficient reference pixels in the case of a large area to be processed and improves the image processing effect.

[0129] In addition, the embodiment of the present application generates a priori target features of the area to be processed through the diffusion network, and then guides the image generation network to restore the processed image. In this way, the diffusion network generates rich a priori target features of the area to be processed, which alleviates the problem of insufficient reference pixels in the case of a large area to be processed and improves the image processing effect.

[0130] Based on this, Figure 5 FIG. 1 is a schematic diagram of an implementation flow of another image generation method provided in an embodiment of the present application. The method is applied to an electronic device and may specifically include the following steps:

[0131] S501, acquiring an original image and determining a region to be processed in the original image.

[0132] In the embodiment of the present application, this step is similar to the above-mentioned step S101, and the embodiment of the present application will not be described one by one here.

[0133] S502, pre-processing the area to be processed to obtain an image mask.

[0134] In the embodiment of the present application, this step is similar to the above-mentioned step S102, and the embodiment of the present application will not be described one by one here.

[0135] S503, performing feature extraction processing on the image mask to obtain initial prior features of the area to be processed.

[0136] In an embodiment of the present application, for the image mask obtained in the above steps, feature extraction processing can be performed on the image mask to obtain initial prior features of the area to be processed. In particular, feature extraction processing can be performed on the image mask with the help of a pre-trained prior feature extraction network.

[0137] Specifically, the image mask is input into a pre-trained prior feature extraction network to obtain the initial prior features of the area to be processed. The pre-trained prior feature extraction network includes N pre-trained prior feature extraction sub-networks, where N is a positive integer greater than or equal to 1.

[0138] To this end, the image mask is input into the pre-trained prior feature extraction network to obtain the initial prior features of the area to be processed. Specifically, the following operations can be performed iteratively until the Nth intermediate prior feature is obtained, and then the following operations are stopped: the i-th feature to be extracted is input into the i-th pre-trained prior feature extraction subnetwork to obtain the i-th intermediate prior feature; wherein the i-th intermediate prior feature is the i+1th feature to be extracted, the 1st feature to be extracted is the image mask, the Nth intermediate prior feature is the initial prior feature, and i is 1, 2, 3, ..., N respectively.

[0139] It should be noted that the pre-trained prior feature extraction subnetwork is usually composed of residual blocks and linear layers, which means that for the pre-trained prior feature extraction network, its input image mask passes through multiple residual blocks and linear layers, and outputs a C-dimensional vector. The C-dimensional vector is the initial prior feature, and C here can be, for example, 128, 256 or 512, etc., and the embodiments of the present application are not limited to this.

[0140] S504, obtaining a preset noise, and generating a target priori feature of the area to be processed according to the preset noise and the initial priori feature.

[0141] In the embodiment of the present application, a preset noise is obtained, and a target prior feature of the area to be processed is generated according to the preset noise and the initial prior feature. The preset noise may be, for example, a random noise such as Gaussian noise, which is not limited in the embodiment of the present application.

[0142] Among them, in the embodiment of the present application, the target prior features of the area to be processed can be generated with the help of a diffusion network. For the diffusion network, it is mainly composed of two parts, namely the forward diffusion process and the reverse denoising process. The forward diffusion process is to continuously add noise to the input until it completely becomes random noise, and the reverse denoising process is to continuously denoise the random noise generated by the forward diffusion process until the original input is obtained. Among them, the reverse denoising process also needs to provide a reference, so as to continuously denoise the random noise generated by the forward diffusion process until the original input is obtained.

[0143] Based on this, in the network inference stage, the preset noise (such as Gaussian noise) and the initial prior features can be obtained through the above steps. The preset noise here can be regarded as the output result of the noise adding network in the diffusion network (corresponding to the above forward diffusion process), so as to use the denoising network in the diffusion network (corresponding to the above reverse denoising process) and refer to the initial prior features to continuously denoise the input preset noise until the original prior features in the area to be processed are obtained.

[0144] To this end, the target prior features of the area to be processed are generated according to the preset noise and the initial prior features. Specifically, the following operations are iteratively performed until the number of iterations reaches the preset first number threshold and then the following operations are stopped: the features to be denoised in the current round are input into the pre-trained denoising network to obtain denoising prior features; wherein the features to be denoised in the first round are the concatenation results of the preset noise and the initial prior features, the denoising prior features are the features to be denoised in the next round, and the denoising prior features in the last round are the target prior features.

[0145] S505, inputting the image mask and the target prior features into a pre-trained image generation network to obtain a processed image.

[0146] In the embodiment of the present application, the target prior features of the area to be processed obtained in the above steps can be used to guide the image generation network to restore the processed image. To this end, the image mask and the target prior features are input into the pre-trained image generation network to obtain the processed image.

[0147] For example, for the target prior features of the area to be erased and the image mask obtained in the above steps, the image mask and the target prior features of the area to be erased can be input into a pre-trained image generation network to obtain an erased image.

[0148] It should be noted that, in one embodiment of the present application, the pre-trained image generation network can be composed of a network structure such as Unet, and a transformer-block can be added thereto to increase the expression ability and receptive field of the network, which is not limited in this embodiment of the present application.

[0149] S506: Fusing the processed image with the non-to-be-processed area in the original image to obtain a target image.

[0150] In an embodiment of the present application, the processed image corresponding to the original image can be obtained through the above steps, and the processed image can be fused with the non-processed area in the original image to obtain the target image (i.e. the final processed image).

[0151] For example, for the erased image corresponding to the original image, the erased image can be fused with the non-erased area in the original image to obtain the target image, that is, the final erased image, such as Figure 4 shown.

[0152] In this way, the diffusion network generates the target prior features of the area to be processed, and then guides the image generation network to restore the processed image. In this way, the diffusion network generates rich target prior features of the area to be processed, which alleviates the problem of insufficient reference pixels in the case of a large area to be processed and improves the image processing effect.

[0153] In addition, a deep learning network is usually used to repair or erase an image, that is, in the process of repairing or erasing an image, a deep learning network is used to predict the pixel values ​​of the area to be repaired or erased in the image, and then the pixel values ​​are used to fill the area to be repaired or erased in the image, and the non-to-be-repaired area or non-to-be-erased area in the image remains unchanged. This method often produces serious color difference in the area to be repaired or erased, with obvious synthetic traces. For this reason, a pure color area post-processing judgment is added in the embodiment of the present application to solve the color difference problem.

[0154] Based on this, Figure 6 FIG. 1 is a schematic diagram of an implementation flow of another image generation method provided in an embodiment of the present application. The method is applied to an electronic device and may specifically include the following steps:

[0155] S601, acquiring an original image and determining a region to be processed in the original image.

[0156] In the embodiment of the present application, this step is similar to the above-mentioned step S501, and the embodiment of the present application will not be described one by one here.

[0157] S602, pre-processing the area to be processed to obtain an image mask.

[0158] In the embodiment of the present application, this step is similar to the above-mentioned step S502, and the embodiment of the present application will not be described one by one here.

[0159] S603, performing feature extraction processing on the image mask to obtain initial prior features of the area to be processed.

[0160] In the embodiment of the present application, this step is similar to the above-mentioned step S503, and the embodiment of the present application will not be described one by one here.

[0161] S604, obtaining a preset noise, and generating a target priori feature of the area to be processed according to the preset noise and the initial priori feature.

[0162] In the embodiment of the present application, this step is similar to the above-mentioned step S504, and the embodiment of the present application will not be described one by one here.

[0163] S605, inputting the image mask and the target prior features into a pre-trained image generation network to obtain a processed image.

[0164] In the embodiment of the present application, this step is similar to the above-mentioned step S505, and the embodiment of the present application will not be described one by one here.

[0165] S606: Fusing the processed image with the non-to-be-processed area in the original image to obtain a fused image.

[0166] In an embodiment of the present application, a processed image corresponding to the original image can be obtained through the above steps, and the processed image can be fused with the non-processed area in the original image to obtain a fused image. The fused image needs to be post-processed and judged for the pure color area to solve the color difference problem.

[0167] S607, searching the fused image for a processed region corresponding to the region to be processed, and detecting whether the processed region is a pure color region.

[0168] In the embodiment of the present application, for the fused image, the processed area corresponding to the area to be processed is found from the fused image, and whether the processed area is a pure color area is detected. For example, for the fused image, the erased area corresponding to the area to be erased is found from the fused image, and whether the erased area is a pure color area is detected.

[0169] It should be noted that for a pure color area, it can be understood that the pixel values ​​in the area are close (close means that the difference between the pixel values ​​is less than a certain threshold), and the colors displayed are close, which can be regarded as one color without other colors. The embodiments of the present application are not limited to this.

[0170] Based on this, whether the processed area is a pure color area can be detected by specifically detecting the pixel values ​​in the processed area. When the pixel values ​​in the processed area are close, it is determined that the processed area is a pure color area.

[0171] Among them, for the case where the pixel values ​​in the processed area are close, specifically, the connected domain of the processed area can be obtained, the connected domain is expanded to obtain the target connected domain, and the target connected domain is subtracted from the connected domain to obtain the difference area. When the pixel values ​​in the difference area are close, it means that the pixel values ​​in the processed area are close. At this time, the processed area is determined to be a pure color area.

[0172] In the case where the pixel values ​​in the difference area are close, the maximum pixel value and the minimum pixel value in the difference area are obtained, and the maximum pixel value and the minimum pixel value are subtracted to obtain the pixel value difference. When the pixel value difference is less than or equal to the preset pixel threshold, it means that the pixel values ​​in the difference area are close, and the processed area is determined to be a pure color area. In addition, when the pixel value difference is greater than the preset pixel threshold, it means that the pixel values ​​in the difference area are not close, and the processed area is determined not to be a pure color area.

[0173] For example, for the processed area, the connected domain of the processed area is obtained, the connected domain is expanded to obtain the target connected domain, the target connected domain is subtracted from the connected domain to obtain the difference area (the difference area is a narrow strip area), the maximum pixel value and the minimum pixel value in the difference area are obtained, and the maximum pixel value is subtracted from the minimum pixel value to obtain the pixel value difference. When the pixel value difference is less than or equal to 3, the processed area is determined to be a pure color area. When the pixel value difference is greater than 3, the processed area is determined not to be a pure color area.

[0174] S608, when the processed area is a pure color area, determine the target pixel value, and fill the processed area according to the target pixel value to obtain a target image.

[0175] In an embodiment of the present application, for the processed area, when the processed area is a pure color area, a target pixel value is determined, and the processed area is filled according to the target pixel value to obtain a target image.

[0176] Among them, for the target pixel value, that is, the pixel value in the above-mentioned difference area (the difference area is a narrow strip area), it means that the processed area can be filled with the pixel value in the difference area, while other areas in the fused image remain unchanged, so as to obtain the target image.

[0177] Based on this, a pixel value is obtained from the difference area (the difference area is a narrow strip area), the pixel value is determined as the target pixel value, and the processed area is filled according to the target pixel value to obtain the target image.

[0178] It should be noted that the pixel value obtained from the difference area may be an average pixel value in the difference area or any pixel value in the difference area, and this embodiment of the present application does not limit this.

[0179] S609: When the processed area is not a pure color area, the fused image is determined as a target image.

[0180] In the embodiment of the present application, for the processed area, when the processed area is not a pure color area, the fused image is determined as the target image, which means that the image result output by the network is retained.

[0181] In this way, in the embodiment of the present application, post-processing judgment of the pure color area is added. When the processed area is a pure color area, the processed area is filled with the pixel values ​​in the difference area, which can solve the problem of color difference and eliminate obvious traces of image synthesis.

[0182] In addition, in the embodiment of the present application, the training of the aforementioned network is divided into two stages. The first stage is to connect the prior feature extraction network and the image generation network in series for joint training, so that the image generation network can restore the processed image when the prior features are input. In the actual reasoning process, there will be no sample processed image, so the first stage is only to enable the image generation network to have the function of restoring the image.

[0183] In the second stage of the training process, the prior features output by the prior feature extraction network are used to supervise the training of the diffusion network, so that when the network has input random noise and prior features, it can predict the output prior features. At the same time, in order to reduce the impact of the error of the prior features output by the diffusion network on the image generation network, the second stage of the training process connects the diffusion network and the image generation network in series for joint training. Generally speaking, the training of the diffusion network occupies a high amount of video memory, is time-consuming to train, and has a long step length in the reverse denoising process, so it is not suitable for joint training with other networks. However, the embodiment of the present application directly uses the prior features output by the prior feature extraction network, rather than the entire image, which effectively alleviates this problem.

[0184] Based on this, Figure 7 FIG. 1 is a schematic diagram of an implementation flow of a first-stage network training method provided in an embodiment of the present application. The method is applied to an electronic device and may specifically include the following steps:

[0185] S701, obtaining a second sample mask image and a second sample processed image.

[0186] In the embodiment of the present application, a second sample mask image (for example, the second sample mask image is ILQ, i.e., the smeared image) and a second sample processed image (for example, the second sample processed image IGt, i.e., an image obtained by fusing the processed image with the non-to-be-processed area in the original image) are obtained. The second sample mask image refers to the to-be-processed area mask corresponding to the processed area in the second sample processed image.

[0187] Among them, in order to make the generated training data distribution closer to the actual erasure scene, the embodiment of the present application uses PPT software to parse PPT data, construct training data pairs before and after element erasure and corresponding erasure area masks, and simulate the actual element erasure scene.

[0188] Based on this, the original slide is obtained, and the elements in the slide (such as text, pictures, tables) are cleared (for example, randomly cleared) to obtain the target slide, the original slide is converted into a first image, the target slide is converted into a second image, and a second sample mask image (that is, the above-mentioned erased area mask) is generated according to the first image and the second image, the first image is determined as the second sample original image, and the second image is determined as the second sample processed image.

[0189] Among them, the first image is subtracted from the second image to obtain a difference image, the circumscribed rectangular frame of each element in the difference image is detected, and the vertex coordinates of the circumscribed rectangular frame (for example, the upper left vertex coordinates) are obtained, and from these vertex coordinates, the vertex coordinates with the same horizontal coordinates are found, and the circumscribed rectangular frames of the elements corresponding to the vertex coordinates with the same horizontal coordinates are merged to obtain a second sample mask image (that is, the above-mentioned erased area mask).

[0190] S702: Searching for a second sample processing region corresponding to the mask in the second sample mask image from the second sample processed image or the second sample original image.

[0191] In the embodiment of the present application, for the second sample processed image or the second sample original image, the second sample processed area corresponding to the mask in the second sample mask image can be searched from the second sample processed image or the second sample original image.

[0192] The mask in the second sample mask image refers to the result of merging the circumscribed rectangular frames of the elements corresponding to the vertex coordinates with the same horizontal coordinates, that is, the erased area mask, which is not limited in the embodiment of the present application.

[0193] S703: Preprocess the second sample processing area to obtain a second sample image mask.

[0194] In the embodiment of the present application, for the second sample processed image or the second sample original image determined in the above steps, the second sample processed area is preprocessed to obtain a second sample image mask. In addition, the non-second sample processed area in the second sample processed image or the second sample original image needs to remain unchanged.

[0195] S704, extracting features from the second sample image mask and the second sample processed image to obtain initial compression prior features.

[0196] In the embodiment of the present application, for the second sample image mask ILQ and the second sample processed image IGt, feature extraction is performed on the second sample image mask ILQ and the second sample processed image IGt to obtain initial compression prior features.

[0197] The second sample image mask ILQ and the second sample processed image IGt can be input into the prior feature extraction network to obtain the initial compressed prior feature Z. The second sample image mask ILQ and the second sample processed image IGt can be spliced ​​in the RGB channel dimension and then input into the prior feature extraction network, which means that the prior feature extraction network inputs the splicing image of the second sample image mask ILQ and the second sample processed image IGt in the RGB channel dimension, passes through multiple residual blocks and linear layers, and outputs a C-dimensional vector, i.e., the initial compressed prior feature Z, where C can be, for example, 128, 256, or 512, etc., which is not limited in the embodiment of the present application.

[0198] S705, inputting the second sample image mask and the initial compression prior features into the image generation network to obtain an initial predicted processed image.

[0199] In an embodiment of the present application, the second sample image mask ILQ and the initial compression prior feature Z obtained in the above steps can be input into the image generation network to obtain an initial predicted processed image.

[0200] S706: Fusing the initially predicted processed image with the second sample processed image or the non-sample processed region in the second sample original image to obtain an initially predicted image.

[0201] In an embodiment of the present application, for the initial predicted processed image obtained in the above steps, the initial predicted processed image is fused with the second sample processed image or the non-sample processed area in the second sample original image to obtain the initial predicted image.

[0202] S707, determining an initial image loss between the initial predicted image and the second sample processed image, and training the image generation network according to the initial image loss.

[0203] S708, when the initial image loss converges, stop training to obtain an initially trained image generation network.

[0204] In the embodiment of the present application, for the initial prediction image obtained in the above steps, the initial image loss between the initial prediction image and the second sample processed image IGt can be determined. The initial image loss can be calculated by, for example, image RGB-L1loss, which is not limited in the embodiment of the present application.

[0205] For the determined initial image loss, the image generation network can be trained according to the initial image loss. In addition, the prior feature extraction network is also included, which means that the image generation network and the prior feature extraction network are jointly trained according to the initial image loss. When the initial image loss converges (for example, the initial image loss is less than a certain loss threshold), the training is stopped to obtain the initially trained image generation network and the pre-trained prior feature extraction network. The subsequently initially trained image generation network must be jointly trained with the diffusion network to obtain the final pre-trained image generation network, and the pre-trained prior feature extraction network is trained at this point.

[0206] In addition, if Figure 8 FIG. 1 is a schematic diagram of an implementation flow of a second-stage network training method provided in an embodiment of the present application. The method is applied to an electronic device and may specifically include the following steps:

[0207] S801, obtaining a first sample mask image and a first sample processed image.

[0208] In the embodiment of the present application, a first sample mask image ILQ and a first sample processed image IGt are obtained, wherein the first sample mask image refers to a to-be-processed area mask corresponding to a processed area in the first sample processed image.

[0209] Among them, in order to make the generated training data distribution closer to the actual erasure scene, the embodiment of the present application uses PPT software to parse PPT data, construct training data pairs before and after element erasure and corresponding erasure area masks, and simulate the actual element erasure scene.

[0210] Based on this, the original slide is obtained, and the elements in the slide (such as text, pictures, tables) are cleared (for example, randomly cleared) to obtain the target slide, the original slide is converted into a first image, the target slide is converted into a second image, and a first sample mask image (that is, the above-mentioned erased area mask) is generated according to the first image and the second image, the first image is determined as the first sample original image, and the second image is determined as the first sample processed image.

[0211] Among them, the first image is subtracted from the second image to obtain a difference image, the circumscribed rectangular frame of each element in the difference image is detected, and the vertex coordinates of the circumscribed rectangular frame (for example, the upper left vertex coordinates) are obtained, and from these vertex coordinates, the vertex coordinates with the same horizontal coordinates are found, and the circumscribed rectangular frames of the elements corresponding to the vertex coordinates with the same horizontal coordinates are merged to obtain the first sample mask image (that is, the above-mentioned erased area mask).

[0212] S802: Searching for a first sample processing region corresponding to the mask in a first sample mask image from a first sample processed image or a first sample original image.

[0213] In the embodiment of the present application, for the first sample processed image or the first sample original image, the first sample processed area corresponding to the mask in the first sample mask image can be searched from the first sample processed image or the first sample original image.

[0214] The mask in the first sample mask image refers to the result of merging the circumscribed rectangular frames of the elements corresponding to the vertex coordinates with the same horizontal coordinates, that is, the erasure area mask, which is not limited in the embodiment of the present application.

[0215] S803: Preprocess the first sample processing area to obtain a first sample image mask.

[0216] In the embodiment of the present application, for the first sample processed image or the first sample original image determined in the above steps, the first sample processed area is preprocessed to obtain a first sample image mask. In addition, the non-first sample processed area in the first sample processed image or the first sample original image needs to remain unchanged.

[0217] S804, processing the first sample image mask to obtain a priori features of the predicted target corresponding to the first sample processing area.

[0218] In the embodiment of the present application, the first sample image mask obtained in the above steps can be processed to obtain a priori features of the predicted target corresponding to the first sample processing area.

[0219] Among them, for the first sample image mask, feature extraction processing can be performed on the first sample image mask to obtain the predicted initial prior features corresponding to the first sample processing area, feature extraction processing can be performed on the first sample image mask and the first sample processed image to obtain compressed prior features, noise addition processing can be performed on the compressed prior features to obtain noisy prior features, and based on the noisy prior features and the predicted initial prior features, predicted target prior features corresponding to the first sample processing area are generated.

[0220] Among them, the first sample image mask can be subjected to feature extraction processing with the aid of a pre-trained prior feature extraction network. Based on this, the first sample image mask is input into the pre-trained prior feature extraction network to obtain the predicted initial prior features corresponding to the first sample processing area. The pre-trained prior feature extraction network is composed of a plurality of residual blocks and linear layers. The pre-trained prior feature extraction network inputs the first sample image mask, passes through a plurality of residual blocks and linear layers, and outputs a C-dimensional vector. The C-dimensional vector is the predicted initial prior feature, where C can be, for example, 128, 256 or 512, etc., and the embodiments of the present application do not limit this.

[0221] In addition, the first sample image mask and the first sample processed image can be feature extracted with the help of a pre-trained prior feature extraction network to obtain compressed prior features to supervise the training of the diffusion network. Based on this, the first sample image mask and the first sample processed image are input into the pre-trained prior feature extraction network to obtain compressed prior features. Among them, the first sample image mask and the first sample processed image can be spliced ​​in the RGB channel dimension, and then input into the pre-trained prior feature extraction network, which means that the pre-trained prior feature extraction network inputs the splicing image of the first sample image mask ILQ and the first sample processed image IGt in the RGB channel dimension, and after multiple residual blocks and linear layers, outputs a C-dimensional vector, i.e., the compressed prior feature Z, where C can be, for example, 128, 256 or 512, etc., and the embodiments of the present application are not limited to this.

[0222] In addition, the diffusion network can be used to continuously add noise to the compressed prior features to obtain the noisy prior features, and then refer to the predicted initial prior features and continuously remove noise to obtain the predicted target prior features. Based on this, the diffusion network can be regarded as including two networks, namely the noise adding network and the noise removal network.

[0223] In the noisy network, the following steps are iteratively performed until the number of iterations reaches the preset second number threshold and the following operation is stopped: the features to be noisy in the current round are input into the noisy network to obtain noisy prior features; wherein the features to be noisy in the first round are compressed prior features, the noisy prior features are the features to be noisy in the next round, and the noisy prior features in the last round are noisy prior features.

[0224] In the denoising network, the following operations are iteratively performed until the number of iterations reaches the preset first number threshold and the following operations are stopped: the features of the samples to be denoised in the current round are input into the denoising network to obtain the predicted denoising prior features; among them, the features of the samples to be denoised in the first round are the concatenation results of the noisy prior features and the predicted initial prior features, the predicted denoising prior features are the features of the samples to be denoised in the next round, and the predicted denoising prior features in the last round are the predicted target prior features.

[0225] S805, inputting the first sample image mask and the prediction target prior features into the initially trained image generation network to obtain a predicted processed image.

[0226] In an embodiment of the present application, the first sample image mask and the predicted target prior features obtained in the above steps can be input into an initially trained image generation network to obtain a predicted processed image.

[0227] S806: Fusing the predicted processed image with the first sample processed image or the non-first sample processed region in the first sample original image to obtain a predicted target image.

[0228] In an embodiment of the present application, for the predicted processed image obtained in the above steps, the predicted processed image is fused with the first sample processed image or the non-first sample processed area in the first sample original image to obtain a predicted target image.

[0229] S807, determining an image loss between the predicted target image and the first sample processed image, and training the initially trained image generation network according to the image loss.

[0230] S808, when the image loss converges, stop training to obtain a pre-trained image generation network.

[0231] In the embodiment of the present application, for the predicted target image obtained in the above steps, the image loss between the predicted target image and the image after the first sample is processed can be determined. The image loss can be calculated by, for example, image RGB-L1loss, which is not limited in the embodiment of the present application.

[0232] For a determined image loss, the initially trained image generation network can be trained according to the image loss. In addition, a diffusion network (i.e., the denoising network and the denoising network mentioned above) is also included, which means that according to the image loss, the initially trained image generation network and the diffusion network are jointly trained. When the image loss converges (for example, the image loss is less than a certain loss threshold), the training is stopped to obtain a pre-trained image generation network and a pre-trained diffusion network. After two stages of network training, the final pre-trained image generation network can be obtained. In the first stage, a pre-trained prior feature extraction network can be obtained, and in the second stage, a pre-trained diffusion network can be obtained.

[0233] Corresponding to the above method embodiment, the present application embodiment also provides an image generating device, such as Fig. 9 As shown, the device may include: an image acquisition module 910 , a region processing module 920 , a mask processing module 930 , and an image generation module 940 .

[0234] The image acquisition module 910 is used to acquire an original image and determine a region to be processed in the original image;

[0235] The region processing module 920 is used to pre-process the region to be processed to obtain an image mask;

[0236] The mask processing module 930 is used to process the image mask to obtain the target prior features of the area to be processed;

[0237] The image generation module 940 is used to generate a target image according to the image mask and the target prior features.

[0238] In an optional implementation, the mask processing module specifically includes:

[0239] A priori feature extraction submodule, used to perform feature extraction processing on the image mask to obtain initial priori features of the area to be processed;

[0240] The a priori feature generation submodule is used to obtain preset noise, and generate target a priori features of the area to be processed according to the preset noise and the initial a priori features.

[0241] In an optional implementation, the prior feature extraction submodule is specifically used for:

[0242] The image mask is input into a pre-trained prior feature extraction network to obtain initial prior features of the area to be processed.

[0243] In an optional embodiment, the pre-trained prior feature extraction network includes N pre-trained prior feature extraction sub-networks, where N is a positive integer greater than or equal to 1; the prior feature extraction sub-module is specifically used for:

[0244] Iterate and perform the following operations until the Nth intermediate prior feature is obtained and then stop the following operations:

[0245] Inputting the i-th feature to be extracted into the i-th pre-trained prior feature extraction subnetwork to obtain the i-th intermediate prior feature;

[0246] Among them, the i-th intermediate prior feature is the i+1-th feature to be extracted, the 1st feature to be extracted is the image mask, the N-th intermediate prior feature is the initial prior feature, and i is 1, 2, 3, ..., N in sequence.

[0247] In an optional implementation, the prior feature generation submodule is specifically used for:

[0248] Iterate the following operations until the number of iterations reaches the preset first number threshold and then stops the following operations:

[0249] Input the features to be denoised in the current round into the pre-trained denoising network to obtain denoising prior features;

[0250] Among them, the features to be denoised in the first round are the concatenation results of the preset noise and the initial prior features, the denoising prior features are the features to be denoised in the next round, and the denoising prior features in the last round are the target prior features.

[0251] In an optional implementation, the image generation module specifically includes:

[0252] A processed image generation submodule, used for inputting the image mask and the target prior features into a pre-trained image generation network to obtain a processed image;

[0253] The target image generation submodule is used to fuse the processed image with the non-to-be-processed area in the original image to obtain a target image.

[0254] In an optional implementation, the target image generation submodule specifically includes:

[0255] A fused image generating unit, used for fusing the processed image with the non-to-be-processed area in the original image to obtain a fused image;

[0256] A processed region detection unit, used to search the processed region corresponding to the to-be-processed region from the fused image, and detect whether the processed region is a pure color region;

[0257] a target pixel value determining unit, configured to determine a target pixel value when the processed area is a pure color area;

[0258] A processed area filling unit, used to fill the processed area according to the target pixel value to obtain a target image;

[0259] Alternatively, the target image determining unit is configured to determine the fused image as the target image when the processed area is not a pure color area.

[0260] In an optional implementation, the post-processing region detection unit is specifically used to:

[0261] Detecting pixel values ​​in the processed area, and determining that the processed area is a pure color area when the pixel values ​​in the processed area are close to each other; comprising:

[0262] Acquire a connected domain of the processed area, and perform expansion processing on the connected domain to obtain a target connected domain;

[0263] Subtract the target connected domain from the connected domain to obtain a difference region;

[0264] When the pixel values ​​in the difference area are close, the processed area is determined to be a pure color area; including: obtaining the maximum pixel value and the minimum pixel value in the difference area, and subtracting the maximum pixel value from the minimum pixel value to obtain a pixel value difference; when the pixel value difference is less than or equal to a preset pixel threshold, the processed area is determined to be a pure color area.

[0265] In an optional implementation, the target pixel value determination unit is specifically configured to:

[0266] A pixel value is selected from the difference area, and the pixel value is determined as a target pixel value.

[0267] In an optional embodiment, the device further comprises:

[0268] A first sample mask image and a first sample processed image acquisition module, used for acquiring the first sample mask image and the first sample processed image;

[0269] A first sample processing region searching module, configured to search the first sample processing region corresponding to the mask in the first sample mask image from the first sample processed image or the first sample original image;

[0270] A first sample processing area preprocessing module, used for preprocessing the first sample processing area to obtain a first sample image mask;

[0271] A first sample image mask processing module, used for processing the first sample image mask to obtain a priori features of a predicted target corresponding to the first sample processing area;

[0272] A predicted processed image generation module, used for inputting the first sample image mask and the predicted target prior features into the initially trained image generation network to obtain a predicted processed image;

[0273] A prediction target image generation module, used for fusing the prediction processed image with the first sample processed image or a non-first sample processed area in the first sample original image to obtain a prediction target image;

[0274] The initially trained image generation network training module is used to determine the image loss between the predicted target image and the first sample processed image, and train the initially trained image generation network according to the image loss; when the image loss converges, the training is stopped to obtain the pre-trained image generation network.

[0275] In an optional implementation, the first sample mask image and the first sample processed image acquisition module specifically include:

[0276] The slide acquisition submodule is used to acquire the original slide and clear the elements in the slide to obtain the target slide;

[0277] A slide conversion submodule, used to convert the original slide into a first image and convert the target slide into a second image;

[0278] A first sample mask image generating submodule, configured to generate a first sample mask image according to the first image and the second image;

[0279] The first sample processed image determining submodule is used to determine the first image as the first sample original image, and determine the second image as the first sample processed image.

[0280] In an optional implementation, the first sample mask image generating submodule is specifically used for:

[0281] Subtract the first image from the second image to obtain a difference image;

[0282] Detecting the circumscribed rectangular frame of each element in the difference image, and obtaining the vertex coordinates of the circumscribed rectangular frame;

[0283] From the vertex coordinates, searching for the vertex coordinates with the same abscissa;

[0284] The circumscribed rectangular frames of the elements corresponding to the vertex coordinates with the same abscissa are merged to obtain a first sample mask image.

[0285] In an optional implementation, the first sample image mask processing module specifically includes:

[0286] A prediction initial priori feature generation submodule, used for performing feature extraction processing on the first sample image mask to obtain a prediction initial priori feature corresponding to the first sample processing area;

[0287] A compressed priori feature generation submodule, used for performing feature extraction processing on the first sample image mask and the first sample processed image to obtain a compressed priori feature;

[0288] A noisy prior feature generation submodule, used for performing noise addition processing on the compressed prior feature to obtain a noisy prior feature;

[0289] The prediction target prior feature generation submodule is used to generate the prediction target prior feature corresponding to the first sample processing area according to the noisy prior feature and the prediction initial prior feature.

[0290] In an optional embodiment, the prediction initial priori feature generation submodule is specifically used for:

[0291] The first sample image mask is input into a pre-trained prior feature extraction network to obtain a predicted initial prior feature corresponding to the first sample processing area.

[0292] In an optional implementation, the compressed priori feature generation submodule is specifically used for:

[0293] The first sample image mask and the first sample processed image are input into a pre-trained prior feature extraction network to obtain a compressed prior feature.

[0294] In an optional implementation, the noisy prior feature generation submodule is specifically used for:

[0295] Iterate the following steps until the number of iterations reaches the preset second number threshold and then stop the following operations:

[0296] Input the features to be denoised in the current round into the denoising network to obtain the denoising prior features;

[0297] The features to be noised in the first round are the compressed prior features, the noised prior features are the features to be noised in the next round, and the noised prior features in the last round are the noisy prior features.

[0298] In an optional implementation, the prediction target prior feature generation submodule is specifically used for:

[0299] Iterate the following operations until the number of iterations reaches the preset first number threshold and then stops the following operations:

[0300] Input the features of the samples to be denoised in the current round into the denoising network to obtain the predicted denoising prior features;

[0301] Among them, the first round of sample denoising features are the concatenation results of the noisy prior features and the predicted initial prior features, the predicted denoising prior features are the next round of sample denoising features, and the last round of predicted denoising prior features are the predicted target prior features.

[0302] In an optional embodiment, the device further comprises:

[0303] A second sample mask image and a second sample processed image acquisition module, used for acquiring a second sample mask image and a second sample processed image;

[0304] A second sample processing region searching module, configured to search the second sample processing region corresponding to the mask in the second sample mask image from the second sample processed image or the second sample original image;

[0305] A second sample image mask generating module, used for preprocessing the second sample processing area to obtain a second sample image mask;

[0306] An initial compression priori feature generation module, used for extracting features from the second sample image mask and the second sample processed image to obtain an initial compression priori feature;

[0307] An initial prediction processed image generation module, used for inputting the second sample image mask and the initial compression prior feature into an image generation network to obtain an initial prediction processed image;

[0308] An initial prediction image generation module, used for fusing the initial prediction processed image with the second sample processed image or the non-sample processed area in the second sample original image to obtain an initial prediction image;

[0309] The image generation network training module is used to determine the initial image loss between the initial predicted image and the second sample processed image, and train the image generation network according to the initial image loss; when the initial image loss converges, the training is stopped to obtain the initially trained image generation network.

[0310] In an optional implementation, the initial compression priori feature generation module is specifically used to:

[0311] The second sample image mask and the second sample processed image are input into a priori feature extraction network to obtain initial compression prior features.

[0312] The present application also provides an electronic device, such as Fig.10 As shown, it includes a processor 101, a communication interface 102, a memory 103 and a communication bus 104, wherein the processor 101, the communication interface 102, and the memory 103 communicate with each other through the communication bus 104.

[0313] Memory 103, used for storing computer programs;

[0314] The processor 101 is used to execute the program stored in the memory 103, and implements the following steps:

[0315] Acquire an original image and determine a region to be processed in the original image; preprocess the region to be processed to obtain an image mask; process the image mask to obtain a target priori feature of the region to be processed; generate a target image based on the image mask and the target priori feature.

[0316] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0317] The communication interface is used for communication between the above electronic device and other devices.

[0318] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0319] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0320] In another embodiment provided in the present application, a storage medium is provided, in which instructions are stored. When the instructions are executed on a computer, the computer executes any one of the image generation methods in the above embodiments.

[0321] In another embodiment provided by the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any one of the image generation methods in the above embodiments.

[0322] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a storage medium, or transmitted from one storage medium to another storage medium, for example, the computer instructions may be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)), etc.

[0323] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0324] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0325] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. An image generation method, characterized in that: The method comprises: Acquire an original image, and determine a region to be processed in the original image; Preprocessing the area to be processed to obtain an image mask; Processing the image mask to obtain target prior features of the area to be processed; A target image is generated according to the image mask and the target priori features.

2. The method according to claim 1, characterized in that The processing of the image mask to obtain a priori target features of the area to be processed includes: Performing feature extraction processing on the image mask to obtain initial prior features of the area to be processed; A preset noise is obtained, and a target priori feature of the area to be processed is generated according to the preset noise and the initial priori feature.

3. The method according to claim 2, characterized in that The step of performing feature extraction processing on the image mask to obtain initial prior features of the area to be processed includes: The image mask is input into a pre-trained prior feature extraction network to obtain initial prior features of the area to be processed.

4. The method according to claim 3, characterized in that The pre-trained prior feature extraction network includes N pre-trained prior feature extraction sub-networks, where N is a positive integer greater than or equal to 1; The step of inputting the image mask into a pre-trained prior feature extraction network to obtain initial prior features of the area to be processed includes: Iterate and perform the following operations until the Nth intermediate prior feature is obtained and then stop the following operations: Inputting the i-th feature to be extracted into the i-th pre-trained prior feature extraction subnetwork to obtain the i-th intermediate prior feature; Among them, the i-th intermediate prior feature is the i+1-th feature to be extracted, the 1st feature to be extracted is the image mask, the N-th intermediate prior feature is the initial prior feature, and i is 1, 2, 3, ..., N in sequence.

5. The method according to claim 2, characterized in that: The step of generating a target a priori feature of the area to be processed according to the preset noise and the initial a priori feature includes: Iterate the following operations until the number of iterations reaches the preset first number threshold and then stops the following operations: Input the features to be denoised in the current round into the pre-trained denoising network to obtain denoising prior features; Among them, the features to be denoised in the first round are the concatenation results of the preset noise and the initial prior features, the denoising prior features are the features to be denoised in the next round, and the denoising prior features in the last round are the target prior features.

6. The method according to claim 1, characterized in that The step of generating a target image according to the image mask and the target prior features includes: Inputting the image mask and the target prior features into a pre-trained image generation network to obtain a processed image; The processed image is fused with the non-to-be-processed area in the original image to obtain a target image.

7. The method according to claim 6, characterized in that The step of fusing the processed image with the non-to-be-processed area in the original image to obtain a target image includes: Fusing the processed image with the non-to-be-processed area in the original image to obtain a fused image; Searching the processed area corresponding to the area to be processed from the fused image, and detecting whether the processed area is a pure color area; In the case where the processed area is a pure color area, determining a target pixel value, and filling the processed area according to the target pixel value to obtain a target image; Alternatively, when the processed area is not a pure color area, the fused image is determined as the target image.

8. The method according to claim 7, characterized in that The detecting whether the processed area is a pure color area comprises: Detecting pixel values ​​in the processed area, and determining that the processed area is a pure color area when the pixel values ​​in the processed area are close to each other; comprising: Acquire a connected domain of the processed area, and perform expansion processing on the connected domain to obtain a target connected domain; Subtract the target connected domain from the connected domain to obtain a difference region; When the pixel values ​​in the difference area are close, the processed area is determined to be a pure color area; including: obtaining the maximum pixel value and the minimum pixel value in the difference area, and subtracting the maximum pixel value from the minimum pixel value to obtain a pixel value difference; when the pixel value difference is less than or equal to a preset pixel threshold, the processed area is determined to be a pure color area.

9. The method according to claim 8, characterized in that The determining of the target pixel value comprises: A pixel value is selected from the difference area, and the pixel value is determined as a target pixel value.

10. The method according to claim 6, characterized in that The pre-trained image generation network is obtained by: Acquire a first sample mask image and a first sample processed image; Searching, from the first sample processed image or the first sample original image, a first sample processed area corresponding to the mask in the first sample mask image; Preprocessing the first sample processing area to obtain a first sample image mask; Processing the first sample image mask to obtain a priori features of a predicted target corresponding to the first sample processing area; Inputting the first sample image mask and the predicted target prior features into an initially trained image generation network to obtain a predicted processed image; Fusing the predicted processed image with the first sample processed image or a non-first sample processed region in the first sample original image to obtain a predicted target image; Determine an image loss between the predicted target image and the first sample processed image, and train the initially trained image generation network according to the image loss; When the image loss converges, the training is stopped to obtain the pre-trained image generation network.

11. The method according to claim 10, characterized in that The step of obtaining a first sample mask image and a first sample processed image includes: Get the original slide, and clear the elements in the slide to obtain the target slide; Converting the original slide into a first image and converting the target slide into a second image; Generate a first sample mask image according to the first image and the second image; The first image is determined as a first sample original image, and the second image is determined as a first sample processed image.

12. The method according to claim 11, characterized in that Generating a first sample mask image according to the first image and the second image includes: Subtract the first image from the second image to obtain a difference image; Detecting the circumscribed rectangular frame of each element in the difference image, and obtaining the vertex coordinates of the circumscribed rectangular frame; From the vertex coordinates, searching for the vertex coordinates with the same abscissa; The circumscribed rectangular frames of the elements corresponding to the vertex coordinates with the same abscissa are merged to obtain a first sample mask image.

13. The method according to claim 10, characterized in that The step of processing the first sample image mask to obtain a priori features of a predicted target corresponding to the first sample processing area includes: Performing feature extraction processing on the first sample image mask to obtain predicted initial prior features corresponding to the first sample processing area; Performing feature extraction processing on the first sample image mask and the first sample processed image to obtain a compressed priori feature; Performing noise adding processing on the compressed priori feature to obtain a noisy priori feature; A predicted target priori feature corresponding to the first sample processing area is generated according to the noisy priori feature and the predicted initial priori feature.

14. The method according to claim 10, characterized in that The initially trained image generation network is obtained in the following way: Acquire a second sample mask image and a second sample processed image; Searching, from the second sample processed image or the second sample original image, a second sample processed region corresponding to the mask in the second sample mask image; Preprocessing the second sample processing area to obtain a second sample image mask; Performing feature extraction on the second sample image mask and the second sample processed image to obtain initial compression prior features; Inputting the second sample image mask and the initial compressed prior features into an image generation network to obtain an initial predicted processed image; Fusing the initially predicted processed image with the second sample processed image or a non-sample processed area in the second sample original image to obtain an initially predicted image; Determine an initial image loss between the initial predicted image and the second sample processed image, and train the image generation network according to the initial image loss; When the initial image loss converges, the training is stopped to obtain the initially trained image generation network.

15. The method according to claim 14, characterized in that The step of extracting features from the second sample image mask and the second sample processed image to obtain initial compression prior features includes: The second sample image mask and the second sample processed image are input into a priori feature extraction network to obtain initial compression prior features.

16. An image generating device, characterized in that: The device comprises: An image acquisition module, used to acquire an original image and determine a region to be processed in the original image; A region processing module, used for preprocessing the region to be processed to obtain an image mask; A mask processing module, used for processing the image mask to obtain a priori target features of the area to be processed; The image generation module is used to generate a target image according to the image mask and the target prior features.

17. A storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 15 is implemented.

Images