Image processing method and device, electronic equipment and storage medium

Abstract

The invention relates to an image processing method and device, equipment and a storage medium, and relates to the technical field of image processing. The method comprises the steps that in response to an image modification instruction input by a user, an image modification area and image modification types are determined, and the image modification types comprise a first image modification type and a second image modification type; performing style feature extraction on the to-be-processed image to obtain image style feature data; based on the image modification area and the image modification type, performing noise addition processing on the to-be-processed image to obtain a first image; and performing image generation according to the to-be-processed image, the image style feature data, the first image and the image modification type to obtain a target generated image. The image style of the to-be-processed image is combined, so that the generated image is kept consistent in style, the image processing quality of the target generated image is improved, user requirements are better met, and a more intelligent image processing service is provided for a user.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to an image processing method, apparatus, electronic device and storage medium. Background Technology

[0002] AI image content removal technology refers to using artificial intelligence algorithms to remove specified content or objects from an image, making the image appear clearer and cleaner. Currently, AI image content removal technology has been widely researched and applied. AI image content local redrawing technology is an image editing method based on artificial intelligence, mainly used to redraw a specific part of an image. This technology can be used in many applications, such as removing image noise, repairing damaged images, and image cutout.

[0003] Currently, AI image content removal and AI image content local redrawing have become popular research directions in the fields of computer vision and image processing. While related technologies such as deep learning can be used to perform local content removal or local redrawing on realistic images, the increasing demands from users, such as the desire to perform local content removal or local redrawing on non-realistic images like comics, have placed higher requirements on image processing technologies for local content removal and local redrawing.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] This disclosure provides an image processing method, apparatus, electronic device, and storage medium.

[0006] According to a first aspect of the present disclosure, an image processing method is provided, comprising:

[0007] In response to a user's input image modification command, the image modification area and the image modification type are determined; the image modification type includes a first image modification type and a second image modification type.

[0008] Style features are extracted from the image to be processed to obtain image style feature data;

[0009] Based on the image modification region and the image modification type, noise is added to the image to be processed to obtain a first image;

[0010] Based on the image to be processed, the image style feature data, the first image, and the image modification type, an image is generated to obtain the target generated image.

[0011] In some embodiments, style feature extraction is performed on the image to be processed to obtain image style feature data, including:

[0012] Style detection is performed on the image to be processed to obtain the image style classification of the image to be processed;

[0013] Based on the image style classification, style features are extracted from the image to be processed to obtain the image style feature data.

[0014] In some embodiments, the provided image processing method further includes:

[0015] The image style feature data is mapped to obtain mapped image style feature data that conforms to the output dimension of the low-rank adaptive model.

[0016] In some embodiments, image generation is performed based on the image to be processed, the image style feature data, the first image, and the image modification type to obtain a target generated image, including:

[0017] The mapped image style feature data is concatenated along the channel dimension through the input of the self-attention layer corresponding to the image generation model; wherein, the image generation module is obtained by adjusting and training the latent diffusion model.

[0018] In some embodiments, noise addition processing is performed on the image to be processed based on the image modification region and the image modification type to obtain a first image, including:

[0019] In response to the image modification type being a first image modification type, the boundary range of the image modification region is extracted from the image to be processed based on the image modification region;

[0020] The first image is obtained by adding noise to the image within the boundary of the image modification region and the image within the image modification region.

[0021] In some embodiments, the boundary range of the image modification region includes a first range and a second range, and the image modification region includes the first range;

[0022] The first image is obtained by adding noise to the image within the boundary of the image modification region and the image modification region.

[0023] Random noise is added to the image within the boundary of the image modification region and to the image modification region itself to obtain a second image;

[0024] The noise in the second image is randomized to obtain the third image;

[0025] The image noise features within the first range of the third image are retained, and the image noise outside the first range of the third image is set to zero to obtain the fourth image;

[0026] The first image is obtained by combining the fourth image and the image to be processed.

[0027] In some embodiments, noise addition processing is performed on the image to be processed based on the image modification region and the image modification type to obtain a first image, including:

[0028] In response to the image modification type being the second image modification type, the image to be processed is subjected to image removal to obtain the fifth image;

[0029] Based on the image modification region, extract the boundary range of the image modification region from the fifth image;

[0030] The first image is obtained by adding noise to the image within the boundary of the image modification region and the image within the image modification region.

[0031] In some embodiments, noise is added to the image within the boundary of the image modification region and the image modification region to obtain the first image, including:

[0032] Random noise is added to the image within the boundary of the image modification region and the image modification region in the fifth image to obtain the sixth image;

[0033] Based on the image noise features within the boundary range of the image modification region in the sixth image, noise filling is performed on the image of the image modification region to obtain the seventh image;

[0034] The first image is obtained by combining the seventh image and the image to be processed.

[0035] In some embodiments, image generation is performed based on the image to be processed, the image style feature data, the first image, and the image modification type to obtain a target generated image, including:

[0036] Text encoding is performed based on the acquired image processing information to obtain image processing text encoding features;

[0037] The image style feature data, the first image, and the image processing text encoding features are input into the image generation model to generate an output image.

[0038] Determine the first portion of the image within the image modification region in the output image;

[0039] Identify a second portion of the image to be processed, excluding the image modification area.

[0040] The first part of the image and the second part of the image are stitched together to obtain the target generated image.

[0041] In some embodiments, image generation is performed based on the image to be processed, the image style feature data, the first image, and the image modification type to obtain a target generated image, which further includes:

[0042] The following steps are executed iteratively:

[0043] The target generated image, the image style feature data, and the image processing text encoding features are input into the image generation model to regenerate the output image;

[0044] Redetermine the first portion of the image within the image modification region in the output image;

[0045] The first part of the image and the second part of the image are stitched together to obtain the updated target generated image.

[0046] According to a second aspect of the present disclosure, an image processing apparatus is provided, comprising:

[0047] An information determination unit is used to determine the image modification area and the image modification type in response to a user input image modification command; the image modification type includes a first image modification type and a second image modification type.

[0048] The style feature extraction unit is used to extract style features from the image to be processed, and obtain image style feature data.

[0049] A noise addition unit is used to add noise to the image to be processed based on the image modification region and the image modification type to obtain a first image;

[0050] The image generation unit is used to generate an image based on the image to be processed, the image style feature data, the first image, and the image modification type to obtain a target generated image.

[0051] According to a third aspect of the present disclosure, an image processing apparatus is provided, comprising:

[0052] processor;

[0053] Memory used to store processor-executable instructions;

[0054] The processor is configured to implement the image processing method described in the first aspect above.

[0055] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, which, when the instructions in the storage medium are executed by a processor of a mobile terminal, enables the mobile terminal to perform the image processing method described in the first aspect.

[0056] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0057] In response to user-inputted image modification commands, the system determines the image modification area and the image modification type, including a first image modification type and a second image modification type. Style features are extracted from the image to be processed to obtain image style feature data. Based on the image modification area and image modification type, noise is added to the image to be processed to obtain a first image. The system then generates a target generated image based on the image to be processed, the image style feature data, the first image, and the image modification type. By extracting the obtained image style feature data and generating the target generated image based on it, the system can combine the image style of the image to be processed with the image style of the target generated image, ensuring that the target generated image maintains a consistent image style after either the first or second image modification. This improves the image processing quality of the generated target image, better meets user needs, and provides users with a more intelligent image processing service.

[0058] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0059] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0060] Figure 1 This is a flowchart of an image processing method shown according to some embodiments of the present disclosure. Figure 1 .

[0061] Figure 2 This is a flowchart illustrating one implementation of step S104 according to some embodiments of the present disclosure.

[0062] Figure 3 This is a flowchart of an image processing method shown according to some embodiments of the present disclosure. Figure 2 .

[0063] Figure 4 This is an implementation process flow of step S106 shown in some embodiments of this disclosure. Figure 1 .

[0064] Figure 5 This is a flowchart illustrating one implementation of step S404 according to some embodiments of the present disclosure.

[0065] Figure 6 This is an implementation process flow of step S106 shown in some embodiments of this disclosure. Figure 2 .

[0066] Figure 7 This is a flowchart illustrating one implementation of step S606 according to some embodiments of the present disclosure.

[0067] Figure 8 This is a flowchart illustrating one implementation of step S108 according to some embodiments of the present disclosure.

[0068] Figure 9 This is a schematic diagram of an image processing process according to some embodiments of the present disclosure, illustrating a specific example.

[0069] Figure 10 This is a block diagram of an image processing apparatus according to some embodiments of the present disclosure.

[0070] Figure 11 This is a block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0071] Some embodiments of this disclosure will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. Various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent upon understanding this disclosure. For example, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein, but can be changed as will become apparent upon understanding this disclosure, except for operations that must be performed in a particular order. Furthermore, for clarity and brevity, descriptions of features known in the art may be omitted.

[0072] The embodiments described in the following examples of this disclosure are not representative of all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0073] The specific implementation methods of the embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0074] The applicant discovered that the image local redrawing or image local removal techniques provided in the relevant technologies, after processing stylized images such as comics, watercolors, line drawings, and sketches, produce new images with significant stylistic differences from the original images. Furthermore, the integration between the newly generated content and the original image content in the new image is poor, with obvious boundary discontinuities, leading to low user satisfaction with the generated images. This is particularly problematic in emerging AI-generated image applications, resulting in users not receiving the expected images and a poor user experience.

[0075] Figure 1 This is a flowchart of an image processing method shown according to some embodiments of the present disclosure. Figure 1 ,like Figure 1 As shown, the image processing method can be applied to electronic devices, including but not limited to terminal devices such as smartphones, wearable devices, and smart tablets, and can also include server-side devices such as local servers and cloud servers, which can be deployed in a computer or a computer cluster composed of multiple computers.

[0076] Figure 1 The image processing method shown includes the following steps.

[0077] In step S102, in response to the user's input image modification command, the image modification area and the image modification type are determined.

[0078] It should be noted that when an electronic device receives an image modification instruction from a user—for example, an AI-powered image editing application—this instruction contains information such as the image to be processed, the requirements for image modification, and the area to be modified. Based on the user's input instruction, the image to be processed, the area to be modified, and the type of image modification can be determined. The image to be processed can be of any type, including real images such as portraits or landscapes taken by the user using an electronic device, and stylized images such as watercolor paintings created by the user using drawing software, downloaded comics, or ink paintings generated using scanning techniques.

[0079] It should be noted that the image modification region is the area in the image to be processed that the user wishes to modify. This could be a region selected by the user on the electronic device screen, or a region obtained through natural language processing based on the user's modification request. It can also be obtained by further combining image analysis of the image to be processed. For example, the user selects the image region to be modified by using a circle or smear method on the device. The electronic device then selects the corresponding region and maps it to a mask area with RGB values ​​of 0 or 1 to obtain a mask for the image modification region.

[0080] It should be noted that image modification types include a first image modification type and a second image modification type. The first image modification type can be partial redrawing, and the second image modification type can be partial content removal. Partial image redrawing, also known as image completion, refers to removing a region from an image and then redrawing new image content in that region. Partial content removal refers to removing content from a specific area of ​​an image and filling or repairing that area as naturally as possible, making the final image appear as if that area never existed, retaining only the background scene. Furthermore, image modification instructions can also include image processing information, such as user-defined modifications to the generated image.

[0081] In an exemplary embodiment of this disclosure, for example, when a user uses an AI-powered image generation application, they upload a cartoon image and input a request to modify the tree area in the cartoon image into a rose bush. Upon receiving the cartoon image, it is identified as the image to be processed. Natural language processing (NLP) of the user's input request determines that the image modification area is a tree area. Combined with image analysis of the cartoon image to be processed, the coordinate range of the image modification area in the image is determined. Furthermore, semantic analysis of the user's input request determines the image modification type to be a first image modification type, namely local redrawing, and the request information is modified to be drawn as a rose bush.

[0082] In step S104, style features are extracted from the image to be processed to obtain image style feature data.

[0083] It should be noted that style features can be extracted by inputting the image to be processed into a style feature extraction model. This model is pre-trained and used to determine the image style feature data of the input image. The style feature extraction model involves capturing specific visual elements from the image, such as color, texture, and shape, which collectively define the image's "style." Image style feature data refers to quantitative information that can represent a specific visual style, including but not limited to color distribution, texture patterns, and shape structure.

[0084] In exemplary embodiments of this disclosure, the style feature extraction model can be a deep learning-based convolutional neural network to determine image style feature data of the input image. The style feature extraction model can also extract image style feature data by determining the image style type of the input image and then determining the corresponding style feature extraction sub-model based on the image style type. Decomposing style feature extraction into two parts—style type detection and image style feature data extraction for a specific style type—avoids complex, computationally intensive deep learning models, reduces the difficulty of model training, and improves the accuracy of model training.

[0085] In step S106, noise is added to the image to be processed based on the image modification area and the image modification type to obtain the first image.

[0086] It should be noted that when performing image rendering or noise removal, noise needs to be added to the image before a new image is generated by using an appropriate algorithm to remove the noise. Understandably, since the image processing is localized, the area corresponding to the noise addition process needs to be determined based on the image modification area to precisely control the processing range. Furthermore, different image modification types employ different noise addition methods. Therefore, based on the image modification area and type, noise addition is performed on the image to obtain the first image after noise initialization.

[0087] In step S108, an image is generated based on the image to be processed, image style feature data, first image, and image modification type to obtain the target generated image.

[0088] It should be noted that image generation can be performed using a trained image generation model. An image generation model is an artificial intelligence algorithm capable of generating new images based on given input (such as random noise, text descriptions, or other images). In some embodiments of this disclosure, the image generation model is pre-trained and used to generate new images based on input model images and data, ensuring that the generated new images meet the requirements of image processing information.

[0089] In exemplary embodiments of this disclosure, the image generation model can be a Latent Diffusion Model (LDM), a technique combining diffusion models and latent variable models, used to generate high-quality images. Specifically, the SD1.5 series models (a series based on the Stable Diffusion architecture, widely used in image generation, artistic creation, and various visual design tasks) can be employed. Those skilled in the art will understand that the model types used in the image generation model are merely examples and are not intended to limit the scope of protection of the embodiments of this disclosure. Other models capable of generating new images, such as generative adversarial networks, autoregressive models, flow-based models, and transformers for image generation, can also be used.

[0090] As can be seen from the above steps, the image processing method provided in this embodiment of the present disclosure, in response to a user's input image modification command, determines the image modification area and the image modification type, wherein the image modification type includes a first image modification type and a second image modification type; extracts style features from the image to be processed to obtain image style feature data; performs noise addition processing on the image to be processed based on the image modification area and the image modification type to obtain a first image; and generates an image based on the image to be processed, the image style feature data, the first image, and the image modification type to obtain a target generated image. By extracting the obtained image style feature data and generating the target generated image based on the image style feature data, the image style of the image to be processed can be combined to ensure that the target generated image after the first image modification or the second image modification maintains a consistent image style, thereby improving the image processing quality of the generated target image, better meeting user needs, and providing users with more intelligent image processing services.

[0091] like Figure 2 The diagram shown is a flowchart illustrating the implementation process of step S104 provided in some exemplary embodiments of this disclosure, including the following steps.

[0092] In step S202, style detection is performed on the image to be processed to obtain the image style classification of the image to be processed.

[0093] In step S204, based on image style classification, style features are extracted from the image to be processed to obtain image style feature data.

[0094] In some exemplary embodiments of this disclosure, the image to be processed can be input into a style detection sub-model to obtain image style feature data, the corresponding style feature extraction sub-model can be determined based on image style classification, and the image to be processed can be input into the style feature extraction sub-model to obtain image style feature data.

[0095] In an exemplary embodiment of this disclosure, the style feature extraction model includes a two-layer sub-model: a style detection sub-model and a style feature extraction sub-model. The style detection sub-model is used to detect the style type of the input image and obtain an image style classification, which can be a classification such as a real image, a cartoon image, or a watercolor image. In specific implementation, a neural network model can be constructed to detect the style type of the input image and obtain an image style classification. A training set consisting of multiple input images and style type identifiers corresponding to each input image is provided to train the constructed neural network model, resulting in a trained style detection sub-model.

[0096] Specifically, to improve the accuracy of image style classification, a wider range of stylized images can be collected in the training set mentioned above. For example, the style detection sub-model structure consists of a convolutional encoder model plus a linear layer classifier structure, which is obtained through classification training on a large amount of data.

[0097] In some exemplary embodiments of this disclosure, there may be multiple style feature extraction sub-models. Each style feature extraction sub-model is used to extract image style feature data of the input image for one image style classification. For example, the style feature extraction sub-model may include: a first style feature extraction sub-model for extracting style feature data of images classified as watercolor images; a second style feature extraction sub-model for extracting style feature data of images classified as oil painting images; and a third style feature extraction sub-model for extracting style feature data of images classified as real images.

[0098] Since image data of different styles have different feature distributions, models are built for image types with different styles. This allows for the classification of each image style, the targeted extraction of image style feature data, more accurate extraction, simplified model structure, and reduced model processing volume and training difficulty.

[0099] Those skilled in the art will understand that the aforementioned real image is a special type of stylized image. The image style feature data of the real image can be understood as the original image features, that is, the image mean feature data, in order to preserve the element style of the real image. The neural network architectures used by the multiple style feature extraction sub-models are similar. The style feature extraction sub-model used to extract the image mean feature data performs mean processing on the features extracted by each layer of the neural network.

[0100] For example, the style feature extraction sub-model can be a 4-layer Transformer-Encoder-Decoder model, trained using real images and images with various stylizations. It should be noted that the style feature extraction sub-model used to extract the image mean feature data only needs the Encoder output as the model output; a Decoder layer is not required.

[0101] In an exemplary embodiment of this disclosure, the style feature extraction model has self-learning capability. When the style classification of the input image is not within the pre-trained range, the model can determine the style classification of the input image through self-learning capability and construct a style feature extraction sub-model corresponding to this classification to extract image style features. The model retains the corresponding data to continuously train the newly constructed style feature extraction sub-model and continuously improve the extraction and processing of image style feature data.

[0102] It should be noted that the latent diffusion model cannot generate new images that are stylized and consistent with the original image style. This embodiment of the disclosure achieves image stylization injection by incorporating the concept of LoRA (Low-Rank Adaptation) with the latent diffusion model.

[0103] like Figure 3 The diagram shown is a flowchart of an image processing method according to some embodiments of the present disclosure. Figure 2 , Figure 3 In the image processing method shown, steps S302 to S304 and steps S308 to S310 are... Figure 1 Steps S102 to S108 in the image processing method shown correspond to each other and will not be repeated here.

[0104] In this embodiment of the disclosure, in Figure 1 Based on the image processing method shown, Figure 3 The image processing method shown may also include the following steps.

[0105] In step S306, the image style feature data is mapped to obtain mapped image style feature data that conforms to the output dimension of the low-rank adaptive model.

[0106] Accordingly, step S310 specifically includes: concatenating the mapped image style feature data along the channel dimension through the input of the self-attention layer corresponding to the image generation model. It should be noted that the image generation module is obtained by adjusting and training the latent diffusion model.

[0107] It should be noted that LoRA (Low-Rank Adaptation) is a technique for fine-tuning large pre-trained models. Its main idea is to adapt to specific tasks by adding a small number of low-rank parameters to the original model. The method provided in this disclosure, however, does not require LoRA training. It maps image style feature data to obtain the feature output dimension of a LoRA model with a structure similar to the basic latent diffusion model. The mapped image style feature data is then concatenated along the channel dimensions through the input of the self-attention layer corresponding to the basic latent diffusion model and introduced into the latent diffusion model. This allows the extracted image stylization-related content to be incorporated into the latent diffusion model, enabling the generation of new images with a unified style through an image generation model.

[0108] This embodiment of the present disclosure obtains mapped image style feature data that conforms to the output dimension of a low-rank adaptive model by mapping image style feature data, and then concatenates the mapped image style feature data through the channel dimension of the input of the self-attention layer corresponding to the image generation model. This can significantly reduce the computational resources and time required to fine-tune the potential diffusion model while maintaining good performance.

[0109] like Figure 4 The following is a flowchart illustrating the implementation process of step S106 provided for some exemplary embodiments of this disclosure. Figure 1 The process includes the following steps.

[0110] In step S402, in response to the image modification type being the first image modification type, the boundary range of the image modification region is extracted from the image to be processed based on the image modification region.

[0111] In an exemplary embodiment of this disclosure, the first image modification type may be a partial redraw. It is understood that, to avoid abrupt transitions at the boundaries of the final generated image, the boundary range of the image modification region can be extracted from the image to be processed based on the image modification region, and noise addition processing can be performed by combining the boundary range of the image modification region, so that the newly generated image portion in the final generated new image has a better boundary blending effect.

[0112] In exemplary embodiments of this disclosure, a region with a preset pixel range can be extracted from the image to be processed based on the image modification region as the boundary range of the image modification region. Exemplarily, in embodiments with proven effectiveness, a region extending 30 pixels outward and 20 pixels inward from the edge of the image modification region can be selected as the boundary range of the image modification region. That is, the edge of the image modification region is proportionally expanded so that the expanded range has 30 more pixels than the image modification region, and the edge of this expanded range is the outer edge of the image modification region boundary range. Conversely, the edge of the image modification region is proportionally reduced so that the reduced range has 20 fewer pixels than the image modification region, and the edge of this reduced range is the inner edge of the image modification region boundary range. Those skilled in the art will understand that the above selection range of the image modification region boundary range is merely an example, and the range can be selected based on the actual test results or the actual situation of image processing; this disclosure will not elaborate further on this aspect.

[0113] In step S404, noise is added to the image within the boundary of the image modification area and the image modification area to obtain the first image.

[0114] In an exemplary embodiment of this disclosure, the boundary range of the image modification region includes a first range and a second range, and the image modification region includes the first range. Using the above example, the first range is the range extending inwards by 20 pixels, and the second range is the range extending outwards by 30 pixels. Accordingly, as... Figure 5 The diagram shown is a flowchart illustrating the implementation process of step S404 provided in some exemplary embodiments of this disclosure, including the following steps.

[0115] In step S502, random noise is added to the image within the boundary of the image modification region and the image modification region to obtain a second image.

[0116] In an exemplary embodiment of this disclosure, the boundary range of the image modification region and the image modification region overlap, forming a first range. Random noise is added to the image within the boundary range of the image modification region and the image modification region; that is, random noise is added to the image within the union range formed by the boundary range of the image modification region and the image modification region, where the edge of the union range is the outer edge of the boundary range of the image modification region. For example, random Gaussian noise can be added to the image within the union range based on the original image to obtain a second image.

[0117] In step S504, the noise in the second image is randomized to obtain the third image.

[0118] It should be noted that, based on the addition of random noise in step S502, the added random noise is further randomized by randomizing it, for example, by superimposing Gaussian noise, mixing different types of noise together, amplifying the existing noise, and using noise distributions such as Poisson noise or Laplace noise, to further increase the randomness of the noise.

[0119] In step S506, the image noise features within the first range of the third image are retained, and the image noise outside the first range of the third image is set to zero to obtain the fourth image.

[0120] It should be noted that the images within the union range of the third image contain noise, and this union range includes the first range. The fourth image can be obtained by preserving the noise features of the images within the first range and setting the noise of the images outside the first range in the union range to zero.

[0121] In step S508, the first image is obtained by combining the fourth image and the image to be processed.

[0122] It is understandable that for the image to be processed, only the image modification area and its boundary range are desired to be changed; other parts of the image do not need to be modified. Therefore, the image portion of the image to be processed excluding the aforementioned union range is retained and combined with the image portion within the union range of the fourth image to obtain the first image.

[0123] In some exemplary embodiments of this disclosure, the noise initialization processing of the image modification region and the boundary range of the image modification region is completed through the above steps, and the obtained first image is sent as input to the noise initialization process of the latent diffusion model.

[0124] This embodiment of the disclosure fills the image modification region with pure zero noise, allowing the region to generate any new image content, making it easier to generate redrawn image content. Furthermore, randomizing the noise at the boundaries of the image modification region results in better boundary blending of the final generated image content.

[0125] like Figure 6 The following is a flowchart illustrating the implementation process of step S106 provided for some exemplary embodiments of this disclosure. Figure 2 The process includes the following steps.

[0126] In step S602, in response to the image modification type being the second image modification type, image removal is performed on the image to be processed to obtain the fifth image.

[0127] In an exemplary embodiment of this disclosure, the second image modification type can be local content removal. A coarse image removal process can be performed on the image to be processed using the LaMa image removal technique (also known as LaMa, Large Mask Inpainting with MultiscaleAttention) to obtain a fifth image.

[0128] It should be noted that after LAM removal, the removal effect deteriorates from the edge to the interior. That is, the removal effect of LAM is better and cleaner at the boundary, and the removal effect becomes worse as you go in. At the same time, if the content to be removed is complex, it is easy to fail and leave more residue. In order to further improve the image removal effect, this disclosure provides a noise addition step in steps S604 to S606 to achieve a good removal effect.

[0129] In step S604, the boundary range of the image modification region is extracted from the fifth image based on the image modification region.

[0130] Understandably, in order to avoid abrupt transitions in the boundaries of the final image after removal, the boundary range of the image modification region can be extracted from the fifth image based on the image modification region, and noise can be added by combining the boundary range of the image modification region, so that the image part of the removed content in the final new image has a better boundary fusion effect.

[0131] In an exemplary embodiment of this disclosure, a region with a preset pixel range can be extracted from the image to be processed based on the image modification region as the boundary range of the image modification region, so as to introduce more boundary content to better guide the image generation module. Since the region around the boundary of the image modification region is relatively clean after the initial elimination in step S602, it is beneficial for the subsequent image generation module to eliminate image content and complete the image background content.

[0132] For example, in embodiments where the actual verification has shown good results, the boundary range of the image modification region can be selected by extending 20 pixels outward and 10 pixels inward from the edge of the image modification region. That is, the edge of the image modification region is proportionally expanded so that the expanded range has 20 more pixels than the original image modification region, and the edge of this expanded range is the outer edge of the image modification region boundary range. Conversely, the edge of the image modification region is proportionally reduced so that the reduced range has 10 fewer pixels than the original image modification region, and the edge of this reduced range is the inner edge of the image modification region boundary range. Those skilled in the art will understand that the above selection range of the image modification region boundary range is merely an example, and the range can be selected based on the actual test results or the actual situation of image processing. This disclosure will not elaborate further on this aspect.

[0133] In step S606, noise is added to the image within the boundary of the image modification area and the image modification area to obtain the first image.

[0134] In exemplary embodiments of this disclosure, the boundary range of the image modification region partially overlaps with the image modification region itself. Using the above example, the boundary range of the image modification region is the sum of an inward expansion of 10 pixels and an outward expansion of 20 pixels. Accordingly, as... Figure 7 The diagram shown is a flowchart of the implementation process of step S606 provided in some exemplary embodiments of this disclosure, including the following steps.

[0135] In step S702, random noise is added to the image within the boundary of the image modification region and the image modification region in the fifth image to obtain the sixth image.

[0136] In an exemplary embodiment of this disclosure, random noise is added to the image within the boundary range of the image modification region and the image modification region itself. Specifically, random noise is added to the image within the union range formed by the boundary range of the image modification region and the image modification region, where the edge of the union range is the outer edge of the boundary range of the image modification region. For example, random Gaussian noise can be added to the image within the union range based on the original image of the fifth image to obtain the sixth image.

[0137] In step S704, noise is filled into the image of the modified region based on the image noise features within the boundary of the modified region in the sixth image to obtain the seventh image.

[0138] It should be noted that, based on the image noise features within the boundary range of the image modification region in the sixth image, the Fill method is used to replace the pixel values ​​in the image of the image modification region with the image noise within the boundary range of the image modification region.

[0139] In step S706, the first image is obtained by combining the seventh image and the image to be processed.

[0140] It is understandable that for the image to be processed, only the image modification area and its boundary range are desired to be changed; other parts of the image do not need to be modified. Therefore, the image portion of the image to be processed, excluding the union range formed by the boundary range of the image modification area and the image modification area, is retained and combined with the image portion within the union range of the seventh image to obtain the first image.

[0141] In some exemplary embodiments of this disclosure, the noise initialization processing of the image modification region and the boundary range of the image modification region is completed through the above steps, and the obtained first image is sent as input to the noise initialization process of the latent diffusion model.

[0142] This embodiment of the disclosure fills the image modification region with a stable distribution of noise, which helps the image generation model generate a cleaner background image when processing the image modification region, thus improving the effect of local content removal. Furthermore, randomizing the noise at the boundary of the image modification region allows the removed content in the final generated image to have a better boundary fusion effect with the surrounding image.

[0143] like Figure 8 The following is a flowchart illustrating the implementation process of step S108 provided for some exemplary embodiments of this disclosure. Figure 1 The process includes the following steps.

[0144] In step S802, text encoding is performed based on the acquired image processing information to obtain image processing text encoding features.

[0145] It should be noted that image processing information can be obtained from the image modification instructions input by the user. This image processing information represents the modification requirements for the image to be processed. For example, a user-inputted raw image request can include first image processing information corresponding to a first image modification type, or second image processing information corresponding to a second image modification type. The image processing is different depending on the type of image processing information; that is, the text encoding is different. The image processing text encoding features are used to guide the generation of new image content in the image generation module.

[0146] In some exemplary embodiments of this disclosure, preset prompts can be used to reduce ambiguity and describe the desired image characteristics, enabling the image generation model to more accurately understand the user's intent.

[0147] For example, in response to image processing information including first image processing information, the preset prompt may include image quality-related prompts, such as High Sharpness, Best Quality, Best Details, etc. It may also include negative prompts, also known as negative prompts or negation prompts, such as "No overexposure," to tell the image generation model what content or features should be avoided during the generation process. By combining positive and negative prompts, the resulting image can be more precisely controlled, ensuring that the generated image meets expectations while excluding unwanted elements.

[0148] For example, in response to image processing information including second image processing information, the preset prompt can include image quality-related prompts, such as: background Simple, clean (the background is clean and uncluttered), transparent, background, simple background, blank background (the background should be transparent or blank, meaning there are no complex patterns or color changes), 8k wallpaper (high resolution). This ensures that no additional content is generated and guarantees the quality of the regenerated content after removal, ensuring no residue remains after removal. It can also include negative prompts, also known as adverse prompts or negation prompts, such as no cluttered background, to tell the image generation model what content or features should be avoided during the generation process. By combining positive and negative prompts, the generated image can be more precisely controlled, ensuring that the generated image meets expectations while excluding unwanted elements.

[0149] In some exemplary embodiments of this disclosure, the values ​​of some parameters in the preset prompt can be determined based on image processing information, and personalized prompt words can be determined. The preset prompt and personalized prompt words are text-encoded by a T5 text encoder, and the image processing text encoding features are introduced into the cross-attention layer of the latent diffusion model.

[0150] In step S804, the image style feature data, the first image, and the image processing text encoding features are input into the image generation model to generate the output image.

[0151] It should be noted that by introducing the text encoding features of image processing into the cross-attention layer of the image generation model, the style feature data of the mapped image is concatenated along the channel dimension through the input of the self-attention layer of the image generation model, and the first image is fed into the noise initialization process of the image generation model, thus obtaining the generated output image.

[0152] In step S806, the first portion of the image within the image modification area in the output image is determined.

[0153] In some exemplary embodiments of this disclosure, pixels within the image modification region are selected in the output image to obtain a first portion of the image.

[0154] In step S808, a second part of the image to be processed, excluding the image modification area, is determined.

[0155] In some exemplary embodiments of this disclosure, pixels other than the image modification area are selected from the image to be processed to obtain a second part of the image.

[0156] In step S810, the first part of the image and the second part of the image are stitched together to obtain the target generated image.

[0157] Understandably, the first part of the image is new content generated by the image generation module based on input data such as image style feature data and guiding words. Combined with the unchanged second part of the image, the target generated image after local redrawing or local content removal can be obtained.

[0158] In some exemplary embodiments of this disclosure, multiple iterations of image generation can be performed to optimize the generated target image and better meet user needs. Accordingly, after step 810, the method further includes iteratively executing the following steps: inputting the target generated image, image style feature data, and image processing text encoding features into the image generation model to regenerate the output image; redetermining the first portion of the image within the image modification area of ​​the output image; and stitching the first portion and the second portion to obtain an updated target generated image. By continuously iterating and generating new first portion images, which are then stitched together with the unchanged second portion image, the target generated image is obtained, thereby improving the fit of the target generated image.

[0159] In some exemplary embodiments of this disclosure, during pre-training, the image generation module can fine-tune itself by constructing a training set containing real images and various stylized images to generate target generated images with the same image style as the image to be processed. For example, the ratio of real images to various stylized images in the training set can be set to 7:3, which can also improve the correlation between text prompts and the image to be processed. The generated target images can also be scored, and those that meet preset scoring requirements can be used as data for self-supervised training to optimize and self-learn the image generation module, thereby improving its performance.

[0160] It can be seen that the image processing method provided in this disclosure can be applied to (1) image restoration and enhancement, including restoration of old photos, red-eye removal, watermark removal, etc.; (2) image beautification function, background removal and replacement, object removal, local color adjustment, augmented reality (AR), painting and coloring, etc.; (3) photo editing and object removal in images: users can delete unwanted objects or people in photos, such as tourists, garbage, etc., and keep a clean background; background cleaning: remove messy background elements in photos to make the subject stand out more; blemish removal: automatically eliminate blemishes in images, such as stains, scratches and red-eye; (4) background replacement: in AR applications, users can remove real backgrounds and replace them with virtual backgrounds, such as using virtual conference room backgrounds in video calls.

[0161] To better illustrate the image processing method provided in the embodiments of this disclosure, a specific example is provided for further explanation, such as... Figure 9 The diagram shows the processing of local redrawing and local content removal for two different images in this specific example.

[0162] by Figure 9Using the image of the little boy waving from the left as an example, the local redrawing process is illustrated. First, the user selects a region to obtain the Mask region. Based on the user's input image modification instructions, the image processing method is determined to be local redrawing, and the boundary features of the Mask region are extracted. Potential noise initialization and zeroing of the noise in the Mask region are then performed to generate initial noise, which is input into the SD1.5 (Fine-tune) model.

[0163] The image is input into the style detection sub-model, and the style classification of the image is determined to be watercolor. The style feature extraction sub-model is then used to extract style feature data. After low-rank adaptive feature compression, the style feature data is input into the SD1.5 (Fine-tune) model.

[0164] By analyzing the image processing information input by the user, the redraw prompt and personalized prompt words are determined. These are then input into the Text-Encoder to obtain the text encoding features of the prompt words, which are then input into the SD1.5 (Fine-tune) model.

[0165] The SD1.5 (Fine-tune) model performs multiple generation iterations. Each iteration generates images along the same dimensions as the input image, retaining only the masked region and stitching it together with the original image's non-masked content to form a complete image. That is, if the image to be processed is R, the input for the nth step is X(n), and the image generated in the next step is X(n+1), meaning the current generated content is X(n)-T', where T' represents the non-masked region in the current generated result, i.e., X(n+1) = R - Mask + (X(n) - T'). After multiple iterations, for example, an upper limit can be set for the number of iterations. Once the upper limit is reached, the output image is as follows: Figure 9 As shown, the boy's clothing and body posture have been redrawn, and the redrawn content is similar in style to the original image.

[0166] by Figure 9 Using the image of a little girl with a backpack on the right as an example, the process of local content removal is illustrated. First, the user selects a region to obtain the Mask region. Based on the user's input image modification instructions, the image processing method is determined to be local removal. Lama pre-removal is performed on the original image, and the Mask region is dilated to obtain the boundary range of the image modification region. Then, a latent noise Fill operation is performed to generate initial noise, which is input into the SD1.5 (Fine-tune) model.

[0167] The image is input into the style detection sub-model, and the style classification of the image is determined to be watercolor. The style feature extraction sub-model is then used to extract style feature data. After low-rank adaptive feature compression, the style feature data is input into the SD1.5 (Fine-tune) model.

[0168] By analyzing the image processing information input by the user, the prompts to be eliminated and the personalized prompt words are determined. The information is then input into the Text-Encoder to obtain the text encoding features of the prompt words, which are then input into the SD1.5 (Fine-tune) model.

[0169] The SD1.5 (Fine-tune) model performs multiple generation iterations. Each iteration generates images along the same dimensions as the input image, retaining only the masked region and stitching it together with the original image's non-masked content to form a complete image. That is, if the image to be processed is R, the input for the nth step is X(n), and the image generated in the next step is X(n+1), meaning the current generated content is X(n)-T', where T' represents the non-masked region in the current generated result, i.e., X(n+1) = R - Mask + (X(n) - T'). After multiple iterations, for example, an upper limit can be set for the number of iterations. Once the upper limit is reached, the output image is as follows: Figure 9 As shown, the little girl in the image has been removed, and the style of the removed image is similar to that of the original image.

[0170] As can be seen from the above process, this specific example can handle various styles of images and rich personalized images generated by AI, and solves the problems of edge stitching non-fusion and mismatch between the style of the generated image and the original image. Through an independent style extraction module, information on image style features is collected and compressed into low-rank model parameters related to the scale of the image generation model. This decomposes the original high-dimensional or high-complexity model parameter matrix into the product of multiple low-rank matrices, preserving the key information of the model while significantly reducing the number of model parameters, ensuring better consistency between the style quality and color expression of the generated image and the original image.

[0171] This specific example also demonstrates the ability to use the same model for both local redrawing and local elimination, but optimizes the noise initialization and preprocessing methods differently for redrawing and elimination. This avoids interference that may occur in different application scenarios. For example, in redrawing scenarios, there may be issues with the failure or incomplete generation of new content, while in elimination scenarios, there may be issues with residual content. In redrawing scenarios, by introducing edge information of the redrawing area, the fusion difference between the new content and the original image at the edge positions can be reduced. In elimination scenarios, by preprocessing the content to be eliminated and performing initial elimination, the main content within the area can be eliminated (blurring may leave residual content). Furthermore, by expanding the edge method and referencing more edge content, it helps to complete the image within the elimination area, ensuring a better fusion effect.

[0172] It should be noted that the acquisition, storage, use, and processing of data in this disclosed technical solution all comply with the relevant provisions of national laws and regulations.

[0173] The following are embodiments of the apparatus disclosed herein, which can be used to execute embodiments of the method disclosed herein. For details not disclosed in the apparatus embodiments of this disclosure, please refer to the embodiments of the method disclosed herein.

[0174] Figure 10 This is a block diagram of an image processing apparatus according to some embodiments of the present disclosure. (Refer to...) Figure 10 The device includes: an information determination unit 1001, a style feature extraction unit 1002, a noise addition unit 1003, and an image generation unit 1004.

[0175] The information determination unit 1001 is used to determine the image modification area and the image modification type in response to a user input image modification command; the image modification type includes a first image modification type and a second image modification type.

[0176] The style feature extraction unit 1002 is used to extract style features from the image to be processed, and obtain image style feature data.

[0177] The noise addition unit 1003 is used to add noise to the image to be processed based on the image modification area and the image modification type to obtain a first image.

[0178] The image generation unit 1004 is used to generate an image based on the image to be processed, image style feature data, a first image, and image modification type to obtain a target generated image.

[0179] In some exemplary embodiments of this disclosure, the style feature extraction unit 1002 is configured to: perform style detection on the image to be processed to obtain an image style classification of the image to be processed;

[0180] Based on image style classification, style features are extracted from the image to be processed to obtain image style feature data.

[0181] In some exemplary embodiments of this disclosure, a feature data mapping unit is further included, which is used to: perform mapping processing on image style feature data to obtain mapped image style feature data that conforms to the output dimension of the low-rank adaptive model.

[0182] Accordingly, in some exemplary embodiments of this disclosure, the image generation unit 1004 is configured to: concatenate the mapped image style feature data along the channel dimension through the input of the self-attention layer corresponding to the image generation model. The image generation module is obtained by adjusting and training a latent diffusion model.

[0183] In some exemplary embodiments of this disclosure, the noise addition unit 1003 is configured to:

[0184] In response to the image modification type being the first image modification type, the boundary range of the image modification region is extracted from the image to be processed based on the image modification region;

[0185] Noise is added to the image within the boundary of the image modification region and the image modification region to obtain the first image.

[0186] It should be noted that the boundary range of the image modification region includes a first range and a second range, and the image modification region includes the first range. In some exemplary embodiments of this disclosure, the noise addition unit 1003 is configured to:

[0187] Random noise is added to the image within the boundary of the image modification region and the image modification region itself to obtain a second image;

[0188] The noise in the second image is randomized to obtain the third image;

[0189] The image noise features within the first range of the third image are retained, and the image noise outside the first range in the third image is set to zero to obtain the fourth image;

[0190] The first image is obtained by combining the fourth image and the image to be processed.

[0191] In some exemplary embodiments of this disclosure, the noise addition unit 1003 is configured to:

[0192] In response to the image modification type being the second image modification type, image removal is performed on the image to be processed to obtain the fifth image;

[0193] Extract the boundary range of the image modification region from the fifth image based on the image modification region;

[0194] Noise is added to the image within the boundary of the image modification region and the image modification region to obtain the first image.

[0195] In some exemplary embodiments of this disclosure, the noise addition unit 1003 is configured to:

[0196] Random noise is added to the image within the boundary of the image modification region and the image modification region in the fifth image to obtain the sixth image;

[0197] Based on the image noise features within the boundary range of the image modification region in the sixth image, noise filling is performed on the image of the image modification region to obtain the seventh image;

[0198] By combining the seventh image and the image to be processed, the first image is obtained.

[0199] In some exemplary embodiments of this disclosure, the image generation unit 1004 is configured to:

[0200] Text encoding is performed based on the acquired image processing information to obtain image processing text encoding features;

[0201] The image style feature data, the first image, and the image processing text encoding features are input into the image generation model to generate the output image.

[0202] Determine the first portion of the image within the modified region of the output image;

[0203] Identify the second portion of the image to be processed, excluding the image modification area;

[0204] By stitching together the first and second parts of the image, the target generated image is obtained.

[0205] In some exemplary embodiments of this disclosure, the image generation unit 1004 is configured to:

[0206] The following steps are executed iteratively:

[0207] Input the target generated image, image style feature data, and image processing text encoding features into the image generation model to regenerate the output image;

[0208] Re-determine the first portion of the image within the modified region of the output image;

[0209] By stitching together the first and second parts of the image, the updated target generated image is obtained.

[0210] Figure 11This is a block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure. For example, device 1100 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness device, personal digital assistant, etc.

[0211] Reference Figure 11 The device 1100 may include one or more of the following components: a processing component 1102, a memory 1104, a power supply component 1106, a multimedia component 1108, an audio component 1110, an input / output (I / O) interface 1112, a sensor component 1114, and a communication component 1116.

[0212] Processing component 1102 typically controls the overall operation of device 1100, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 1102 may include one or more processors 1120 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 1102 may include one or more modules to facilitate interaction between processing component 1102 and other components. For example, processing component 1102 may include a multimedia module to facilitate interaction between multimedia component 1108 and processing component 1102.

[0213] Memory 1104 is configured to store various types of data to support the operation of device 1100. Examples of such data include instructions for any application or method operating on device 1100, contact data, phonebook data, messages, pictures, videos, etc. Memory 1104 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0214] Power supply component 1106 provides power to various components of device 1100. Power supply component 1106 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to device 1100.

[0215] Multimedia component 1108 includes a screen that provides an output interface between the device 1100 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 1108 includes a front-facing camera and / or a rear-facing camera. When the device 1100 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0216] Audio component 1110 is configured to output and / or input audio signals. For example, audio component 1110 includes a microphone (MIC) configured to receive external audio signals when device 1100 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 1104 or transmitted via communication component 1116. In some embodiments, audio component 1110 also includes a speaker for outputting audio signals.

[0217] I / O interface 1112 provides an interface between processing component 1102 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0218] Sensor assembly 1114 includes one or more sensors for providing status assessments of various aspects of device 1100. For example, sensor assembly 1114 may detect the on / off state of device 1100, the relative positioning of components such as the display and keypad of device 1100, changes in the position of device 1100 or a component of device 1100, the presence or absence of user contact with device 1100, the orientation or acceleration / deceleration of device 1100, and temperature changes of device 1100. Sensor assembly 1114 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 1114 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 1114 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0219] Communication component 1116 is configured to facilitate wired or wireless communication between device 1100 and other devices. Device 1100 can access wireless networks based on communication standards, such as WiFi, 3G, 4G, 5G, other communication standards, or combinations thereof. In some embodiments of this disclosure, communication component 1116 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In some embodiments of this disclosure, communication component 1116 further includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0220] In some embodiments of this disclosure, the apparatus 1100 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0221] In some embodiments of this disclosure, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 1104 including instructions, which can be executed by a processor 1120 of device 1100 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0222] In some embodiments of this disclosure, a non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a processor of a mobile terminal, enables the mobile terminal to perform an image processing method, the method comprising:

[0223] In response to a user's input image modification command, the system determines the image modification area and the image modification type; it extracts style features from the image to be processed to obtain image style feature data; based on the image modification area and the image modification type, it adds noise to the image to be processed to obtain a first image; and it generates an image based on the image to be processed, the image style feature data, the first image, and the image modification type to obtain a target generated image.

[0224] In some embodiments of this disclosure, a computer program product is also provided, including a computer program / instructions that, when executed by a processor, implement an image processing method, the method comprising:

[0225] In response to a user's input image modification command, the system determines the image modification area and the image modification type; it extracts style features from the image to be processed to obtain image style feature data; based on the image modification area and the image modification type, it adds noise to the image to be processed to obtain a first image; and it generates an image based on the image to be processed, the image style feature data, the first image, and the image modification type to obtain a target generated image.

[0226] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0227] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that, include: In response to a user's input image modification command, the image modification area and the image modification type are determined; the image modification type includes a first image modification type and a second image modification type. Style features are extracted from the image to be processed to obtain image style feature data; Based on the image modification region and the image modification type, noise is added to the image to be processed to obtain a first image; Based on the image to be processed, the image style feature data, the first image, and the image modification type, an image is generated to obtain the target generated image.

2. The image processing method according to claim 1, characterized in that, Style features are extracted from the image to be processed to obtain image style feature data, including: Style detection is performed on the image to be processed to obtain the image style classification of the image to be processed; Based on the image style classification, style features are extracted from the image to be processed to obtain the image style feature data.

3. The image processing method according to claim 2, characterized in that, Also includes: The image style feature data is mapped to obtain mapped image style feature data that conforms to the output dimension of the low-rank adaptive model.

4. The image processing method according to claim 3, characterized in that, Image generation is performed based on the image to be processed, the image style feature data, the first image, and the image modification type to obtain the target generated image, including: The mapped image style feature data is concatenated along the channel dimension through the input of the self-attention layer corresponding to the image generation model; wherein, the image generation module is obtained by adjusting and training the latent diffusion model.

5. The image processing method according to claim 1, characterized in that, Based on the image modification region and the image modification type, noise addition processing is performed on the image to be processed to obtain a first image, including: In response to the image modification type being a first image modification type, the boundary range of the image modification region is extracted from the image to be processed based on the image modification region; The first image is obtained by adding noise to the image within the boundary of the image modification region and the image within the image modification region.

6. The image processing method according to claim 5, characterized in that, The boundary range of the image modification region includes a first range and a second range, and the image modification region includes the first range; The first image is obtained by adding noise to the image within the boundary of the image modification region and the image modification region. Random noise is added to the image within the boundary of the image modification region and to the image modification region itself to obtain a second image; The noise in the second image is randomized to obtain the third image; The image noise features within the first range of the third image are retained, and the image noise outside the first range of the third image is set to zero to obtain the fourth image; The first image is obtained by combining the fourth image and the image to be processed.

7. The image processing method according to claim 1, characterized in that, Based on the image modification region and the image modification type, noise addition processing is performed on the image to be processed to obtain a first image, including: In response to the image modification type being the second image modification type, the image to be processed is subjected to image removal to obtain the fifth image; Based on the image modification region, extract the boundary range of the image modification region from the fifth image; The first image is obtained by adding noise to the image within the boundary of the image modification region and the image within the image modification region.

8. The image processing method according to claim 7, characterized in that, The image within the boundary of the image modification region and the image of the image modification region are subjected to noise addition processing to obtain the first image, including: Random noise is added to the image within the boundary of the image modification region and the image modification region in the fifth image to obtain the sixth image; Based on the image noise features within the boundary range of the image modification region in the sixth image, noise filling is performed on the image of the image modification region to obtain the seventh image; The first image is obtained by combining the seventh image and the image to be processed.

9. The image processing method according to claim 1, characterized in that, Image generation is performed based on the image to be processed, the image style feature data, the first image, and the image modification type to obtain the target generated image, including: Text encoding is performed based on the acquired image processing information to obtain image processing text encoding features; The image style feature data, the first image, and the image processing text encoding features are input into the image generation model to generate an output image. Determine the first portion of the image within the image modification region in the output image; Identify a second portion of the image to be processed, excluding the image modification area. The first part of the image and the second part of the image are stitched together to obtain the target generated image.

10. The image processing method according to claim 9, characterized in that, The process of generating an image based on the image to be processed, the image style feature data, the first image, and the image modification type to obtain a target generated image further includes: The following steps are executed iteratively: The target generated image, the image style feature data, and the image processing text encoding features are input into the image generation model to regenerate the output image; Redetermine the first portion of the image within the image modification region in the output image; The first part of the image and the second part of the image are stitched together to obtain the updated target generated image.

11. An image processing apparatus, characterized in that, include: An information determination unit is used to determine the image modification area and the image modification type in response to a user input image modification command; the image modification type includes a first image modification type and a second image modification type. The style feature extraction unit is used to extract style features from the image to be processed, and obtain image style feature data. A noise addition unit is used to add noise to the image to be processed based on the image modification region and the image modification type to obtain a first image; The image generation unit is used to generate an image based on the image to be processed, the image style feature data, the first image, and the image modification type to obtain a target generated image.

12. An image processing device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to implement the image processing method according to any one of claims 1 to 10.

13. A non-transitory computer-readable storage medium, wherein when instructions in the storage medium are executed by a processor of a mobile terminal, the mobile terminal is enabled to perform an image processing method according to any one of claims 1 to 10.

Images