Image processing method and device, electronic equipment and readable storage medium

Abstract

The invention discloses an image processing method and device, electronic equipment and a readable storage medium, and belongs to the technical field of artificial intelligence. The image processing method comprises the following steps: receiving a first image and prompt information, wherein the prompt information comprises generation instruction information; determining a first mask image according to the first image and the generation instruction information; determining a second mask image according to the first mask image; generating a second image according to the first image, the second mask image and the prompt information; and performing fusion processing on the first image and the second image to obtain a third image.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and specifically relates to an image processing method and device, an electronic device, and a readable storage medium. Background Art

[0002] The photos taken by users may have problems such as bad expressions and bad postures. Since it is difficult for users to reproduce the scene at that time later and the cost of photo editing is relatively high, users hope to be able to edit their portraits through simple text input. For example, by directly inputting: bitter face to smile, the editing tool can realize the user's editing intention.

[0003] With the development of AIGC (AI-Generated Content) technology, there are now solutions for intelligently editing images with the help of generative AI (Artificial Intelligence) big models and image editing algorithms based on diffusion models.

[0004] However, existing image editing algorithms have defects such as poor generated image quality, poor generated content consistency, and insufficient instruction follow-up, which makes the intelligently generated images less effective. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide an image processing method and device, an electronic device and a readable storage medium, which can improve the image effect of intelligent generation.

[0006] In a first aspect, an embodiment of the present application provides an image processing method, the method comprising: receiving a first image and prompt information, the prompt information comprising generating instruction information; determining a first mask image based on the first image and the generating instruction information; determining a second mask image based on the first mask image; generating a second image based on the first image, the second mask image and the prompt information; and fusing the first image and the second image to obtain a third image.

[0007] In a second aspect, an embodiment of the present application provides an image processing device, which includes: a receiving unit, used to receive a first image and prompt information, the prompt information including generation instruction information; a processing unit, used to determine a first mask image based on the first image and the generation instruction information; the processing unit is also used to determine a second mask image based on the first mask image; the processing unit is also used to generate a second image based on the first image, the second mask image and the prompt information; the processing unit is also used to fuse the first image and the second image to obtain a third image.

[0008] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the image processing method of the first aspect are implemented.

[0009] In a fourth aspect, an embodiment of the present application provides a readable storage medium having a program or instruction stored thereon, which, when executed by a processor, implements the steps of the image processing method of the first aspect.

[0010] In a fifth aspect, an embodiment of the present application provides a chip, which includes a processor and a communication interface, wherein the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the steps of the image processing method of the first aspect.

[0011] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the steps of the image processing method of the first aspect.

[0012] In the image processing method provided in the embodiment of the present application, a first image and prompt information are received, and the prompt information includes generation instruction information; a first mask image is determined according to the first image and the generation instruction information; a second mask image is determined according to the first mask image; a second image is generated according to the first image, the second mask image and the prompt information; and the first image and the second image are fused to obtain a third image. Through the above-mentioned image processing method, based on the first image input by the user and the generation instruction information, the first mask image is determined, and then the second mask image to be edited is determined. Based on this, according to the first image input by the user, the prompt information and the second mask image to be edited, an initial image that meets the user's requirements, that is, a second image, is intelligently generated, and the original first image and the intelligently generated second image are fused to obtain the final third image. In this way, the image is intelligently edited according to the user prompt and the editable mask image, and more flexible image editing can be achieved while maintaining the original image information. Moreover, by fusing the original image and the generated image, the color difference of the generated image can be reduced, the followability of the user's instructions in the process of intelligently generating the image is ensured, the consistency of the generated content is ensured, the quality of the intelligently generated image is improved, and the image effect of the intelligently generated image is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 One of the flowcharts of the image processing method provided in the embodiment of the present application;

[0014] Figure 2 A schematic diagram of the operation interface of the image processing method provided in an embodiment of the present application;

[0015] Figure 3 The second flowchart of the image processing method provided in the embodiment of the present application;

[0016] Figure 4 The third flowchart of the image processing method provided in the embodiment of the present application;

[0017] Figure 5 A fourth flowchart of the image processing method provided in an embodiment of the present application;

[0018] Figure 6 One of the principle diagrams of the image processing method provided in the embodiment of the present application;

[0019] Figure 7 The second schematic diagram of the image processing method provided in the embodiment of the present application;

[0020] Figure 8 The third schematic diagram of the image processing method provided in the embodiment of the present application;

[0021] Fig. 9 The fourth schematic diagram of the image processing method provided in the embodiment of the present application;

[0022] Fig.10 A structural block diagram of an image processing device provided in an embodiment of the present application;

[0023] Fig.11 A structural block diagram of an electronic device provided in an embodiment of the present application;

[0024] Fig.12 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0026] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0027] The image processing method provided in the embodiment of the present application is described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0028] like Figure 1 As shown, the embodiment of the present application provides an image processing method, which may include the following S102 to S110:

[0029] S102: Receive a first image and prompt information, where the prompt information includes generating instruction information.

[0030] The image processing method proposed in the embodiment of the present application is executed by an electronic device, which may specifically be an intelligent electronic device such as a smart phone, a tablet computer, a laptop computer, and a smart watch, and is not specifically limited here.

[0031] The first image is an original image to be edited.

[0032] Furthermore, the first image may be a historical image or an image taken in real time by an electronic device, or an image downloaded by a user from the Internet, or an image obtained by a user from an external storage device such as a USB flash drive or a disk, without specific limitation herein.

[0033] Furthermore, the prompt information is information input by the user for indicating how to edit the first image.

[0034] Furthermore, the prompt information may include generation instruction information, which may be information input by the user through a keyboard or voice. The generation instruction information is used to indicate the user's editing requirements for the first image, such as replacing a crying face in the first image with a smiling face, adding a hat to the portrait in the first image, and other instruction information.

[0035] In actual application, the user can input the above-mentioned first image and prompt information through touch input of the electronic device. For example, the user can select any one or more portrait images in the album of the electronic device, and then input the generation instruction information and / or the expected image information by clicking the portrait editing button. Alternatively, in the camera interface of the electronic device, when the electronic device detects a portrait in the image preview state, the portrait editing button can be displayed, and the user can input the generation instruction information and / or the expected image information by clicking the portrait editing button.

[0036] For example, Figure 2 As shown, the display interface 202 of the electronic device displays a first image 204 and a portrait editing button 206. The user clicks the portrait editing button 206, and the electronic device responds to the single-click input and displays an information input box 208. The user can input generation instruction information and / or desired image information through touch input to the information input box 208.

[0037] S104: Determine a first mask image according to the first image and the generation instruction information.

[0038] The generated instruction information includes the editing object or editing area of ​​the first image by the user. For example, when the generated instruction information is to replace the crying face in the first image with a smiling face, the editing area is the face area; when the generated instruction information is to add a hat to the portrait in the first image, the editing area is the head area.

[0039] Furthermore, the first mask image is a mask image stored in a mask image library.

[0040] Specifically, after the generation instruction information input by the user is obtained, based on the editing object or editing area corresponding to the generation instruction information, a first mask image matching the editing object or editing area is selected from a pre-built mask image library.

[0041] S106: Determine a second mask image according to the first mask image.

[0042] The second mask image corresponds to the editing object or editing area included in the generation instruction information. The mask image is a MASK image, which uses 0 and 1 to represent the background area and the foreground area respectively. The MASK image can be obtained by binarizing the image content. The MASK image can also be obtained by manual painting, which is not specifically limited here.

[0043] Specifically, after selecting a first mask image from a mask image library, the second mask image is obtained by scaling, adjusting the angle, filling the content, etc. of the first mask image, so that the first image can be subsequently intelligently edited based on the second mask image.

[0044] In actual application, the second mask image can also be obtained by binarizing the image content of the first image. In the second mask image, for the editing object or editing area included in the generation instruction information, it is represented as a white area with a pixel value of 1, and the white area is the area in the first image that needs to be retained or edited later; and for the image area not included in the generation instruction information, it is represented as a black area with a pixel value of 0, and the black area is the area in the first image that needs to be deleted or kept later.

[0045] Specifically, after obtaining the first image input by the user and generating instruction information, the image content of the first image is binarized based on the editing object or editing area corresponding to the generating instruction information, and a second mask image corresponding to the first image is obtained, so as to distinguish the image area to be edited and the image area that does not need to be edited in the first image through the second mask image.

[0046] S108: Generate a second image according to the first image, the second mask image and the prompt information.

[0047] Specifically, in the image processing method provided in the embodiment of the present application, under the prompt information and the prompt of the second mask image, the first image is intelligently edited and generated to obtain the edited initial image, that is, the second image.

[0048] S110: Fusing the first image and the second image to obtain a third image.

[0049] The second image is an initial image obtained by intelligently editing and generating the first image.

[0050] Specifically, after obtaining the second image, the original image to be edited, i.e., the first image, and the intelligently generated second image are weightedly fused to obtain the final generated image, i.e., the third image. In this way, the color difference of the generated image content can be reduced, the consistency of the generated image content can be ensured, and the quality and effect of the generated image can be improved.

[0051] The above-mentioned image processing method provided by the embodiment of the present application receives a first image and prompt information, and the prompt information includes generation instruction information; determines a first mask image according to the first image and the generation instruction information; determines a second mask image according to the first mask image; generates a second image according to the first image, the second mask image and the prompt information; and fuses the first image and the second image to obtain a third image. Through the above-mentioned image processing method, based on the first image input by the user and the generation instruction information, the first mask image is determined, and then the second mask image to be edited is determined. Based on this, according to the first image input by the user, the prompt information and the second mask image to be edited, the initial image that meets the user's requirements, that is, the second image, is intelligently generated, and the original first image and the intelligently generated second image are fused to obtain the final third image. In this way, the image is intelligently edited according to the user prompt and the editable mask image, which can achieve more flexible image editing while maintaining the original image information, and by fusing the original image and the generated image, the color difference of the generated image can be reduced, the followability of the user's instructions in the process of intelligently generating the image is guaranteed, the consistency of the generated content is guaranteed, the quality of the intelligently generated image is improved, and the image effect of the intelligently generated image is improved.

[0052] In the embodiment of the present application, the above S104 may specifically include the following S104a to S104c:

[0053] S104a: Perform face detection and face segmentation on the first image to obtain a third mask image.

[0054] The third mask image corresponds to the portrait area in the first image.

[0055] Furthermore, different portrait areas correspond to different portrait attributes, and the portrait attributes may specifically include eyes, nose, mouth, cheeks, hair, and the like.

[0056] Specifically, after obtaining the first image input by the user, face detection and face segmentation are performed on the first image to obtain multiple portrait attributes of the first image, and then based on each portrait attribute, a MASK operation is performed on the first image to obtain a third mask image corresponding to each portrait attribute.

[0057] Among them, for the third mask image corresponding to each portrait attribute, the image area corresponding to the portrait attribute is represented as a white area with a pixel value of 1, and other image areas are represented as black areas with a pixel value of 0.

[0058] S104b: Obtain entity nouns in the generation instruction information.

[0059] The entity noun is used to indicate an object to be edited or added to the first image that the user desires.

[0060] Specifically, after obtaining the generation instruction information input by the user, the NER (Named Entity Recognition) technology is used to identify and extract entity nouns in the generation instruction information, such as hats, headscarves, earrings, necklaces, etc.

[0061] In the actual application process, after obtaining the entity nouns in the generation instruction information, a noun table to be edited may be generated, and the extracted entity nouns may be saved in the noun table to be edited.

[0062] S104c: Determine a first mask image according to the third mask image and the entity noun.

[0063] Specifically, after the entity noun in the generation instruction information is extracted, based on the entity noun, a first mask image matching the third mask image to be edited is selected from a pre-built mask image library.

[0064] In the above-mentioned embodiment provided by the present application, in the process of determining the first mask image according to the first image and the generation instruction information, face detection and face segmentation are performed on the first image to obtain a third mask image, wherein the third mask image corresponds to the portrait area in the first image; the entity noun in the generation instruction information is obtained; and the first mask image is determined according to the third mask image and the entity noun. In this way, based on the user's generation instruction information, the first mask image is determined in a targeted manner, which ensures the accuracy of the second mask image determined subsequently, thereby improving the accuracy of the intelligent editing image and the effect of the intelligent generation of the image.

[0065] In the embodiment of the present application, the above S104a may specifically include the following S112 to S118:

[0066] S112: Perform face detection on the first image to determine a face detection frame.

[0067] Specifically, after obtaining the first image input by the user, a face detection algorithm such as the RetinaFace algorithm is used to perform face detection on the first image to obtain a face detection frame.

[0068] Among them, the RetinaFace algorithm is a face detection network based on deep learning, which can detect a large number of face frames in dense face images.

[0069] Furthermore, taking the upper left corner vertex of the first image as the origin, the face detection box can be expressed as box:=[x1, y1, x2, y2], where the symbol “:=” means defining the character on the left as the operation on the right, (x1, y1) represents the coordinates of the upper left corner vertex of the face detection box, and (x2, y2) represents the coordinates of the lower right corner vertex of the face detection box.

[0070] In actual application, if multiple faces are detected in the first image, the electronic device may prompt the user to select one of the faces for editing.

[0071] S114: constructing a target face frame according to the coordinate information and size information of the face detection frame.

[0072] The coordinate information of the face detection frame may specifically include the coordinates of the center point of the face detection frame. Taking the upper left corner vertex of the first image as the origin, when the face detection frame is represented as box:=[x1, y1, x2, y2], the coordinates of the center point of the face detection frame may be represented as [xc, yc]=[(x1+x2) / 2, (y1+y2) / 2].

[0073] Furthermore, the size information of the face detection frame may specifically include the length of the long side of the face detection frame. Taking the upper left corner vertex of the first image as the origin, when the face detection frame is represented as box:=[x1, y1, x2, y2], the length of the long side of the face detection frame hw=max(x2-x1, y2-y1).

[0074] Furthermore, the target face frame is a square frame.

[0075] Specifically, after performing face detection on the first image to obtain a face detection frame, a new square frame, namely, a target face frame, is reconstructed based on the coordinates of the center point and the length of the long side of the face detection frame.

[0076] Among them, when the upper left corner vertex of the first image is taken as the origin, the coordinates of the center point of the face detection frame are [xc, yc] = [(x1+x2) / 2, (y1+y2) / 2], and the long side length of the face detection frame is hw = max(x2-x1, y2-y1), the target face frame can be expressed as box_new: = [xc-hw / 2, yc-hw / 2, xc+hw / 2, yc+hw / 2].

[0077] S116: When the size of the target face frame does not meet the first condition, adjust the size of the target face frame through the first process until the size of the target face frame meets the first condition.

[0078] The first condition is that the area of ​​the target face frame is within a first range, or the area ratio of the target face frame in the first image is within a second range.

[0079] Specifically, after constructing the target face frame, it is determined whether the size of the target face frame meets the set first condition. If the size of the target face frame does not meet the first condition, the size of the target face frame is dynamically adjusted through the first processing until the size of the target face frame meets the first condition, and then the image in the target face frame is segmented. In this way, the face area in the first image is scaled and then segmented, which can ensure the segmentation effect of the face.

[0080] In actual application, the first range may be [100×100, 400×400], and the second range may be [0.1, 0.6], which are not specifically limited herein.

[0081] S118: When the size of the target face frame meets the first condition, segment the image within the target face frame to obtain a third mask image.

[0082] Specifically, when the size of the target face frame meets the first condition, the image in the target face frame is segmented based on a face segmentation algorithm or model such as a BiSeNet model to obtain a third mask image corresponding to each portrait attribute in the first image.

[0083] Among them, the BiSeNet model is a segmentation framework, which is specifically a face attribute segmentation model based on convolutional neural network.

[0084] Furthermore, when the target face frame has been scaled by the first processing, after obtaining the third mask image, the third mask image needs to be reversely operated by the reverse processing of the first processing, that is, the second processing, so that the size of the third mask image is the same as that of the original image.

[0085] In actual application, Figure 6As shown, before performing face detection on the first image, the first image may be preprocessed, such as subtracting a fixed mean and variance from the first image. For the RetinaFace model, the size of the first image needs to be adjusted to 960×960, while for the input of the BiSeNet model, the original image is used.

[0086] In the above embodiment provided by the present application, in the process of performing face detection and face segmentation on the first image to obtain the third mask image, face detection is performed on the first image to determine the face detection frame; a target face frame is constructed according to the coordinate information and size information of the face detection frame; if the size of the target face frame does not meet the first condition, the size of the target face frame is adjusted through the first processing until the size of the target face frame meets the first condition; if the size of the target face frame meets the first condition, the image in the target face frame is segmented to obtain the third mask image. In this way, the face area in the first image is scaled and then segmented, which can ensure the face segmentation effect.

[0087] In the embodiment of the present application, when the size of the target face frame does not meet the first condition, the step of adjusting the size of the target face frame through the first processing may specifically include the following S120 to S124:

[0088] S120: When the area of ​​the target face frame is greater than the first area or the area ratio of the target face frame in the first image is greater than a first value, reduce the size information of the first image to reduce the area of ​​the target face frame until the area of ​​the target face frame meets the first condition, and adjust the pixel information in the target face frame according to the scaling factor of the first image.

[0089] The first area is the upper limit of the first range, and the first area may be specifically 400×400.

[0090] Furthermore, the first numerical value is an upper limit value of the second range, and the first numerical value may specifically be 0.6.

[0091] Specifically, after constructing the target face frame, when the area of ​​the target face frame is larger than the first area, or when the area ratio of the target face frame in the first image is larger than the first value, the size information of the first image is reduced to reduce the area of ​​the target face frame until the area of ​​the target face frame is smaller than the first area, or the area ratio of the target face frame in the first image is smaller than the first value. At this time, the scaling factor of the first image is recorded, and the pixel information in the target face frame is synchronously scaled according to the scaling factor of the first image. Specifically, each pixel value in the target face frame is multiplied by the scaling factor of the first image.

[0092] S122: When the area of ​​the target face frame is smaller than the second area or the area proportion of the target face frame in the first image is smaller than a second value, increase the size information of the face detection frame.

[0093] The second area is the lower limit of the first range, and the first area may be specifically 100×100.

[0094] Furthermore, the second numerical value is the lower limit of the second range, and the first numerical value may specifically be 0.1.

[0095] Specifically, after constructing the target face frame, when the area of ​​the target face frame is smaller than the second area, or when the area ratio of the target face frame in the first image is smaller than the second value, the size information of the face detection frame is increased to increase the area of ​​the target face frame until the area of ​​the target face frame is larger than the second area, or the area ratio of the target face frame in the first image is larger than the second value. In this way, before segmenting the face region in the first image, the face region in the first image is first scaled to a suitable size, so that the face segmentation effect can be ensured.

[0096] S124: When the boundary of the target face frame coincides with the boundary of the first image, adjust the coordinates of the center point of the face detection frame.

[0097] Specifically, in the image processing method provided in the embodiment of the present application, in the process of adjusting the size of the target face frame, when the boundary of the target face frame coincides with the boundary of the first image, the center point coordinates of the face detection frame are adjusted to adjust the position information of the target face frame, thereby avoiding the problem of the target face frame crossing the boundary while containing the boundary.

[0098] In summary, in the image processing method provided in the embodiment of the present application, the effect diagram of face detection and segmentation can be shown as follows: Figure 6 As shown in the figure, the process of face detection and segmentation is as follows: Figure 4 As shown, the following steps S402 to S412 may be specifically included:

[0099] S402: Preprocess the first image.

[0100] S404: Perform face detection on the first image and determine a face detection frame.

[0101] S406: scaling the size of the face detection frame.

[0102] S408: Segment the image within the face detection frame.

[0103] S410: The segmented image is interpolated and enlarged to its original size.

[0104] S412: Obtain a refined face segmentation result.

[0105] Among them, Figure 5 As shown, the above S406 may specifically include the following S414 to S432:

[0106] S414: constructing a target face frame according to the long side length and center point coordinates of the face detection frame.

[0107] S416: Calculate the area of ​​the target face frame and its area ratio in the first image.

[0108] S418: Determine: the area of ​​the target face frame is greater than the first area or the area ratio is greater than the first value. If so, execute S420; if not, execute S422.

[0109] S420: reducing the size of the first image and the target face frame.

[0110] S422: Fix the center point and draw a square with the length of the long side as the side length.

[0111] S424: Determine: the square size meets the first condition, if so, execute S426, if not, execute S428.

[0112] S426: Obtain an enlarged face detection frame.

[0113] S428: Determine: the square is located at the first image boundary, if so, execute S426, if not, execute S430.

[0114] S430: Moving the center point of the face detection frame.

[0115] S432: Increase the length of the long side of the face detection frame.

[0116] In the above-mentioned embodiment provided by the present application, in the process of adjusting the size of the target face frame by the first processing, when the area of ​​the target face frame is larger than the first area or the area ratio of the target face frame in the first image is larger than the first value, the size information of the first image is reduced to reduce the area of ​​the target face frame until the area of ​​the target face frame meets the first condition, and the pixel information in the target face frame is adjusted according to the scaling factor of the first image; when the area of ​​the target face frame is smaller than the second area or the area ratio of the target face frame in the first image is smaller than the second value, the size information of the face detection frame is increased to increase the area of ​​the target face frame until the area of ​​the target face frame meets the first condition; when the boundary of the target face frame coincides with the boundary of the first image, the coordinates of the center point of the face detection frame are adjusted to adjust the position information of the target face frame. In this way, the face area in the first image is first scaled to a suitable size before segmentation, which can ensure the segmentation effect of the face and avoid the problem of the target face frame crossing the boundary.

[0117] In the embodiment of the present application, the above S104c may specifically include the following S126 to S130:

[0118] S126: Select M fourth mask images from the mask image library according to the entity noun.

[0119] The mask image library is a database pre-built and stored in an electronic device, and the mask image library contains different mask images with different portrait attributes.

[0120] In the actual application process, in the process of constructing the above-mentioned mask image library, the portrait dataset can be constructed by using open source datasets, Internet data collection and other methods, and then the portrait segmentation algorithms such as SAM and BiSeNet are used to segment the portrait attributes of each portrait in the portrait dataset to obtain the portrait attributes of each portrait such as hat, headdress, hair, earrings, nose, eyes and necklace. Furthermore, for each portrait attribute of each portrait, a mask image corresponding to each portrait attribute of each portrait is obtained through binarization processing, thereby forming a mask image library.

[0121] Furthermore, M is a positive integer greater than 1.

[0122] Specifically, after the entity noun in the generation instruction information is extracted, a mask image in a pre-constructed mask image library is retrieved based on the entity noun to obtain M fourth mask images corresponding to the entity noun in the mask image library.

[0123] For example, if the entity noun is a hat, all mask images of hat attributes are retrieved and obtained from the mask image library.

[0124] S128: Determine an intersection-to-union ratio of the areas of each fourth mask image and the third mask image according to the area of ​​the third mask image, the area of ​​each fourth mask image, the intersection area of ​​each fourth mask image and the third mask image, and the union area of ​​each fourth mask image and the third mask image.

[0125] Specifically, after retrieving and obtaining M mask images corresponding to the entity noun from the mask image library, each fourth mask image is IOU matched with the third mask image. Specifically, the intersection-over-union ratio of the area of ​​each fourth mask image to the third mask image is determined according to the area of ​​the third mask image, the area of ​​each fourth mask image, the intersection area of ​​each fourth mask image and the third mask image, and the union area of ​​each fourth mask image and the third mask image.

[0126] In actual application, the area intersection-to-union ratio of each fourth mask image to the third mask image may be calculated according to the following formula:

[0127] IOU=max[(A∩B) / (A∪B), (A∩B) / A, (A∩B) / B],

[0128] Among them, A represents the area of ​​the mask image, B represents the area of ​​the third mask image, A∩B represents the intersection area of ​​the mask image and the third mask image, and A∪B represents the union area of ​​the mask image and the third mask image. IOU represents the intersection-over-union ratio of the mask image and the third mask image. The larger the IOU value, the more similar the mask image is to the third mask image. In this way, the boundary conditions can be taken into account, the accuracy of the IOU value calculation is guaranteed, and the accuracy of the subsequent determination of the first mask image is guaranteed.

[0129] S130: Determine a first mask image from the M fourth mask images according to the area intersection-over-union ratio.

[0130] Specifically, after calculating the area intersection and union ratio of each fourth mask image to the third mask image, the selected M fourth mask images are sorted in order of the area intersection and union ratio from small to large, and then according to a certain sampling probability, a fourth mask image is sampled from the M fourth mask images to obtain a first mask image.

[0131] In the above-mentioned embodiment provided by the present application, in the process of determining the first mask image according to the third mask image and the entity noun, M fourth mask images are selected from the mask image library according to the entity noun, where M is a positive integer greater than 1; the area intersection-and-union ratio of each fourth mask image to the third mask image is determined according to the area of ​​the third mask image, the area of ​​each fourth mask image, the intersection area of ​​each fourth mask image and the third mask image, and the union area of ​​each fourth mask image and the third mask image; and the first mask image is determined from the M fourth mask images according to the area intersection-and-union ratio. In this way, the area intersection-and-union ratio of each fourth mask image to the third mask image is calculated taking into account the boundary conditions, thereby ensuring the accuracy of the area intersection-and-union ratio calculation, and further ensuring the accuracy of the determined first mask image.

[0132] In the embodiment of the present application, the above S106 may specifically include the following S106a to S106c:

[0133] S106a: Adjust the size of the first mask image according to the size of the target face frame.

[0134] It is understandable that, among the M fourth mask images retrieved and acquired from the mask image library, it may happen that any mask image cannot match the third mask image.

[0135] Therefore, in the image processing method provided in the embodiment of the present application, after determining the first mask image, the size of the first mask image is scaled according to the size of the target face frame to improve the matching degree between the first mask image and the third mask image.

[0136] S106b: Displaying the adjusted first mask image.

[0137] In the image processing method provided in the embodiment of the present application, a function of manually editing the mask image can also be provided to the user.

[0138] Specifically, after determining the first mask image, the electronic device may display the first mask image on the display interface to show the first mask image to the user. At this time, if the user is not satisfied with the first mask image, the user may click and activate the layer of the first mask image and modify the first mask image.

[0139] S106c: According to the user's adjustment input on the first mask image, adjust the first mask image to obtain a second mask image.

[0140] Specifically, after showing the first mask image to the user, the electronic device may adjust the size, angle, filling content, etc. of the first mask image based on the user's adjustment input to the first mask image, thereby obtaining a second mask image that meets the user's requirements.

[0141] In the above embodiment provided by the present application, in the process of determining the second mask image according to the first mask image, the size of the first mask image is adjusted according to the size of the target face frame; the adjusted first mask image is displayed; and according to the user's adjustment input to the first mask image, the first mask image is adjusted to obtain the second mask image. In this way, on the one hand, the matching degree between the first mask image and the third mask image is improved, and the accuracy of the second mask image determined subsequently is guaranteed; on the other hand, the interactive generation of the second mask image is realized, and the second mask image is guaranteed to meet the user's requirements, and the accuracy of the subsequent intelligent generation and editing of the image based on the second mask image is guaranteed.

[0142] In the embodiment of the present application, the above S108 may specifically include the following S108a to S108c:

[0143] S108a: Determine a first image feature according to the prompt information.

[0144] The first image feature is used to indicate the user's desired feature for the generated image.

[0145] Specifically, after the prompt information input by the user is obtained, the user's desired feature for generating an image, that is, the first image feature, is determined according to the prompt information, so as to generate the image required by the user according to the first image feature.

[0146] S108b: Determine a second image feature according to the first image and the second mask image.

[0147] The second image feature is used to indicate the feature to be edited for the input image.

[0148] Specifically, after the second mask image is obtained, the feature to be edited of the input image, ie, the second image feature, is determined according to the first image and the second mask image.

[0149] S108c: Generate a second image according to the first image feature and the second image feature.

[0150] Specifically, after obtaining the first image feature and the second image feature, forward reasoning is performed based on the first image feature and the second image feature, so as to intelligently edit and generate the first image to obtain the second image.

[0151] In the above-mentioned embodiment provided by the present application, in the process of generating the second image according to the first image, the second mask image and the prompt information, the first image feature is determined according to the prompt information; the second image feature is determined according to the first image and the second mask image; and the second image is generated according to the first image feature and the second image feature. In this way, the image is intelligently edited and generated by combining the user's expected features for the generated image and the features to be edited of the input image, thereby improving the quality of the intelligently generated image and ensuring the followability of the user's instructions in the process of intelligently generating the image.

[0152] In the embodiment of the present application, the prompt information also includes expected image information, and the above S108a may specifically include the following S132 to S136:

[0153] S132: Extracting the text feature vector of the generation instruction information and the image feature vector of the expected image information.

[0154] The desired image information is the content that the user wants to add or edit in the first image, and the desired image information is input in the form of an image. For example, the desired image information may be a specific type of hat image, glasses image, etc. that the user wants to add in the first image.

[0155] Furthermore, the feature dimension of the text feature vector is 77×1024, and the feature dimension of the image feature vector is 256×1024.

[0156] Specifically, after the prompt information input by the user is acquired, the text feature vector of the generation instruction information in the prompt information is extracted, and the image feature vector of the expected image information in the prompt information is extracted.

[0157] S134: Perform linear mapping on the image feature vector.

[0158] Specifically, after the image feature vector of the desired image information is extracted, the image feature vector is linearly mapped through a linear layer to adjust the feature dimension of the image feature vector to 77×1024, so that the feature dimension of the image feature vector remains the same as that of the text feature vector.

[0159] S136: Determine a first image feature according to the text feature vector and the mapped image feature vector.

[0160] Furthermore, the first image feature may specifically include a query vector and a key vector.

[0161] Specifically, in the image processing method provided in the embodiment of the present application, a pre-trained model based on contrasting text-image pairs is used to determine the user's desired features for generating an image, i.e., the first image features, based on the text feature vector and the mapped image feature vector, so as to control the generation of the image required by the user based on the first image feature. In this way, image condition control is added on the basis of text condition control, which improves the controllability of condition control of image editing and facilitates the user to control the final effect of image generation by inputting an image.

[0162] Among them, the pre-training model based on contrastive text-image pairs can be specifically a CLIP (Contrastive Language-Image Pre-training) encoder. The CLIP encoder is a model based on contrastive learning training and is used to align images and corresponding text descriptions to the same feature space. Using this model, the input image content and text content can be uniformly encoded respectively, so as to obtain high-level semantic information of the image.

[0163] In actual application, the prompt information may also include only the generation instruction information or the expected image information. In the case where the prompt information includes only one of the generation instruction information or the expected image information, the missing feature dimension may be filled with a 77×1024 all-zero vector. That is, in the case where the prompt information includes only one of the generation instruction information or the expected image information, the first image feature is determined based on the text feature vector and the all-zero vector using a pre-trained model based on contrasting text-image pairs, or the first image feature is determined based on the mapped image feature vector and the all-zero vector.

[0164] In the above-mentioned embodiment provided by the present application, the prompt information also includes the expected image information. In the process of determining the first image feature according to the prompt information, the text feature vector of the generation instruction information and the image feature vector of the expected image information are extracted; the image feature vector is linearly mapped; and the first image feature is determined according to the text feature vector and the mapped image feature vector. In this way, the user's expected features for the generated image are determined based on the user's prompt information, which facilitates the control of generating the image required by the user and improves the effect of intelligently generating the image.

[0165] In the embodiment of the present application, the above S108b may specifically include the following S138 to S146:

[0166] S138: Determine a fourth image according to the first image and the second mask image.

[0167] The fourth image contains elements that the user wants to retain, that is, the fourth image is an image of the background area excluding the mask area in the first image.

[0168] Specifically, after obtaining the first image and the second mask image, a fourth image including elements that the user wishes to retain is generated based on the first image and the second mask image, so that image areas that the user does not need to edit are retained when subsequently generating images.

[0169] In actual application, the fourth image may be determined according to the following formula:

[0170] ImageBackground=Image×(1-Mask),

[0171] Among them, ImageBackground represents the fourth image, Image represents the first image, and Mask represents the second mask image.

[0172] S140: Encode the fourth image to obtain a first feature vector.

[0173] The feature dimension of the first feature vector is 128×128×4.

[0174] Specifically, after obtaining the fourth image, VAE (Variational Auto Encoder) is used to encode the fourth image to obtain a first eigenvector.

[0175] S142: Adjust the size of the second mask image, and determine a second feature vector according to mask information in the adjusted second mask image.

[0176] The feature dimension of the second feature vector is 128×128×1.

[0177] Specifically, in the image processing method provided in the embodiment of the present application, in the process of determining the second image features, the second mask image is resized, and a second feature vector with element values ​​of 0 or 1 is determined based on the mask information in the adjusted second mask image.

[0178] S144: Determine a third eigenvector according to the Gaussian noise data.

[0179] The feature dimension of the third feature vector is 128×128×4.

[0180] Specifically, in the image processing method provided in the embodiment of the present application, in the process of determining the second image feature, a third feature vector is also determined based on randomly initialized Gaussian noise data.

[0181] S146: Concatenate the target dimensions of the first eigenvector, the second eigenvector, and the third eigenvector to obtain a second image feature.

[0182] The target dimension is the last dimension of the first eigenvector, the second eigenvector, and the third eigenvector.

[0183] Furthermore, the feature dimension of the second image feature is 128×128×9, wherein 9=4 (Gaussian noise data)+1 (second mask image)+4 (fourth image).

[0184] Specifically, after obtaining the first eigenvector, the second eigenvector and the third eigenvector, the last dimension of the first eigenvector, the second eigenvector and the third eigenvector is concatenated to obtain the second image feature. In this way, the image input condition is determined based on Gaussian noise data, the mask image to be edited and the background area image that the user wants to keep in the original image, which is conducive to learning the reconstruction effect of the mask area, making the generated content more natural and reasonable.

[0185] In the above embodiment provided by the present application, in the process of determining the second image feature according to the first image and the second mask image, the fourth image is determined according to the first image and the second mask image; the fourth image is encoded to obtain the first feature vector; the size of the second mask image is adjusted, and the second feature vector is determined according to the mask information in the adjusted second mask image; the third feature vector is determined according to the Gaussian noise data; the target dimensions of the first feature vector, the second feature vector and the third feature vector are spliced ​​to obtain the second image feature. In this way, the accuracy of the determined second image feature is guaranteed, and the accuracy of the subsequent intelligent editing and generation of the image based on the second image feature is guaranteed.

[0186] In the embodiment of the present application, the above S138 may specifically include the following S138a to S138c:

[0187] S138a: Determine a rectangular area image corresponding to the mask information in the second mask image.

[0188] The rectangular area image is the minimum circumscribed rectangle of the mask information in the second mask image.

[0189] S138b: Increase the resolution of the rectangular area image to obtain a processed rectangular area image.

[0190] It is understandable that the resolution of the first image may be very high, in which case it is unnecessary to edit and generate the entire image. Therefore, in the image processing method provided in the embodiment of the present application, after obtaining the second mask image, the minimum bounding rectangle of the mask information in the second mask image is calculated to obtain the corresponding rectangular area image, and then the resolution of the rectangular area image is increased to 1024×1024 or other resolutions to obtain the processed rectangular area image.

[0191] S138c: Determine a fourth image based on the processed rectangular area image and the first image.

[0192] In the above embodiment provided by the present application, in the process of determining the fourth image according to the first image and the second mask image, the rectangular area image corresponding to the mask information in the second mask image is determined; the resolution of the rectangular area image is increased to obtain the processed rectangular area image; and the fourth image is determined according to the processed rectangular area image and the first image. In this way, it is easy to implement local editing of the area of ​​interest to the user in the first image without generating the entire image, which reduces the amount of calculation and processing and improves the efficiency of image processing.

[0193] In the embodiment of the present application, the image editing model includes multiple network modules, and the above S108c may specifically include the following S148 and S150:

[0194] S148: Input the first image feature and the second image feature into the trained image editing model.

[0195] Among them, the image editing model is a specific model independently designed by the embodiment of the present application. The image editing model is a structure based on the diffusion model and is designed using a deep neural network.

[0196] Furthermore, the image editing model can intelligently edit and generate the original image to be edited, i.e., the first image, based on prompt information input by the user, such as generation instructions and / or expected image information, and output the edited initial image, i.e., the second image.

[0197] Furthermore, the image editing model includes multiple network modules.

[0198] Furthermore, the first image feature is a control input feature of the image editing model. The first image feature may specifically be a latent space feature. The first image feature is used to indicate the user's desired features for the generated image of the image editing model.

[0199] Furthermore, the second image feature is an image input feature of the image editing model, and the second image feature is used to indicate a feature to be edited by the image editing model for the input image.

[0200] Specifically, after obtaining the first image feature and the second image feature, the first image feature and the second image feature are input into the trained image editing model, so that the trained image editing model performs forward reasoning based on the first image feature and the second image feature, thereby intelligently editing and generating the first image to obtain the second image.

[0201] S150: Using multiple network modules in sequence to process the first image features and the second image features to generate a second image.

[0202] The output information of the Nth network module is the input information of the N+1th network module, and N is a positive integer.

[0203] Specifically, after the first image feature and the second image feature are input into the trained image editing model, according to the principle that the output information of the Nth network module is the input information of the N+1th network module, multiple network modules are used in sequence to process the first image feature and the second image feature to generate a second image.

[0204] Among them, Figure 8 As shown, each network module includes a residual network (ResNet), a first attention module (the first M-Attention module), a second attention module (the second M-Attention module) and a sampling module (DownSample / Upsample module) connected in sequence.

[0205] Among them, the residual network is ResNet (Residual Network), which is composed of Skip-Connect connection and convolutional blocks. Skip-Connect is used to perform the operation of adding two vectors element by element.

[0206] Furthermore, the first attention module and the second attention module are specifically M-Attention modules.

[0207] Among them, the M-attention module is used to re-weight the value vector (V) input into it so that the image editing model can focus on more useful information, thereby improving the accuracy of the image editing model.

[0208] Furthermore, if Fig. 9 As shown in the figure, the M-attention module mainly includes Linear layer, Layernorm&Activation layer, BMM layer, Normalize layer and Reweight layer.

[0209] The Linear layer is a fully connected layer that accepts the input tensor and converts it into an output tensor. Its mathematical formula is y = x·A T +b, where x is the input, A is the Linear weight matrix, and A T is the transposed matrix of A, b is the bias matrix of Linear, and y is the output of Linear.

[0210] Furthermore, the Layernorm&Activation layer is a layer normalization and activation layer, which is used to normalize the Nd dimension of the input [batch, Nd, dim]. Specifically, in the Nd dimension, the mean of each pixel is subtracted and then divided by the variance, and then the activated function such as the Relu activation function is used to process the processed data, such as Relu(x)=max(0,x).

[0211] Furthermore, the BMM (Batch Matrix Multiplication) layer is used to implement BMM operations between matrices. BMM operations are three-dimensional matrix multiplication operations. For example, A b×n1×n2 ×B b×n2×n3 =C b×n1×n3 , which means that a new three-dimensional matrix C is obtained by multiplying a three-dimensional matrix A by a three-dimensional matrix B, where the third dimension of A and the second dimension of B must be the same.

[0212] Furthermore, the Normalize layer is used to normalize a three-dimensional matrix by row. For example, by normalizing the position of the i-th row and j-th column of the two-dimensional matrix A, we can get:

[0213] Furthermore, the Reweight layer is used to adjust the weight of the input feature matrix V according to a three-dimensional weight matrix W (the output of the Normalize layer). Specifically, the Reweight layer calls the BMM operation to implement the matrix multiplication between the weight matrix W and the feature matrix V.

[0214] Further, the sampling module is specifically a DownSample / Upsample module, which is used to upsample or downsample the resolution of the feature map corresponding to the third output information, and simultaneously increase or decrease the number of channels of the feature map. The sampling module uses a convolution operation to implement the upsampling or downsampling operation.

[0215] Specifically, in the image processing method provided in the embodiment of the present application, Figure 7As shown, the image editing model includes multiple network modules. In the process of generating the second image using the trained image editing model, for each network module, the second image feature is input into the residual network, the first output information of the residual network is input into the first attention module, the first image feature and the second output information of the first attention module are input into the second attention module, and the third output information of the second attention module is input into the sampling module. On this basis, according to the principle that the output information of the Nth network module is the input information of the N+1th network module, and according to the above-mentioned single network module processing the first image feature and the second image feature, multiple network modules are used in sequence to process the first image feature and the second image feature to generate the second image. In this way, the image editing model is used to intelligently edit and generate images, which improves the quality of intelligently generated images and ensures the followability of user instructions in the process of intelligently generating images.

[0216] The first output information includes a first query vector, a first key vector and a first value vector. The second output information includes a second value vector. The third output information may be specifically expressed in the form of a feature map.

[0217] In actual application, Figure 7 As shown, the image editing model may specifically include 6 network modules, wherein the number of channels of the 6 network modules are 320, 640, 1280, 1280, 640 and 320 respectively.

[0218] In the embodiment of the present application, each network module includes a residual network, a first attention module, a second attention module and a sampling module connected in sequence, and the above S150 may specifically include the following S150a to S150d:

[0219] S150a: For each network module, process the second image feature through the residual network to obtain a first query vector, a first key vector and a first value vector.

[0220] Specifically, in the process of generating the second image using the trained image editing model, for each network module in the image editing model, such as Figure 8 As shown, the second image feature is processed by the residual network, namely ResNet, to obtain a first query vector (Q), a first key vector (K) and a first value vector (V).

[0221] S150b: Process the first query vector, the first key vector, and the first value vector through the first attention module to obtain a second value vector.

[0222] Specifically, in the process of generating the second image using the trained image editing model, for each network module in the image editing model, such as Figure 8As shown, the first attention module, i.e., the first M-Attention module, receives and processes the first query vector (Q), the first key vector (K), and the first value vector (V), and outputs the second value vector (V).

[0223] S150c: Process the first image feature and the second value vector through a second attention module to obtain third output information.

[0224] Specifically, in the process of generating the second image using the trained image editing model, for each network module in the image editing model, such as Figure 8 As shown, the second value vector (V) is received through the second attention module, i.e., the second M-Attention module, and then the first image feature and the second value vector (V) are processed, and the third output information is output to the sampling module, i.e., the DownSample / Upsample module.

[0225] S150d: Sample the resolution of the feature map corresponding to the third output information through the sampling module, and adjust the number of channels of the feature map to obtain the output information of each network module, until the last network module completes processing the first image feature and the second image feature to obtain the second image.

[0226] Specifically, in the process of generating the second image using the trained image editing model, for each network module in the image editing model, such as Figure 8 As shown, through the sampling module, i.e., the DownSample / Upsample module, the resolution of the feature map corresponding to the third output information is sampled, and the number of channels of the feature map is adjusted to obtain the output information of each network module, until the last network module completes processing of the first image features and the second image features, and obtains the second image output by the last network module.

[0227] In the above-mentioned embodiment provided by the present application, in the process of sequentially processing the first image feature and the second image feature using multiple network modules to generate the second image, for each network module, the second image feature is processed by the residual network to obtain the first query vector, the first key vector and the first value vector; the first query vector, the first key vector and the first value vector are processed by the first attention module to obtain the second value vector; the first image feature and the second value vector are processed by the second attention module to obtain the third output information; the resolution of the feature map corresponding to the third output information is sampled by the sampling module, and the number of channels of the feature map is adjusted to obtain the output information of each network module, until the last network module completes the processing of the first image feature and the second image feature to obtain the second image. In this way, based on the structure of the diffusion model, the deep neural network is used to intelligently edit and generate images, the quality of intelligently generated images is improved, and the followability of user instructions in the process of intelligently generating images is ensured.

[0228] In the embodiment of the present application, the above S150b may specifically include the following S152 to S164:

[0229] S152: Perform linear processing on the first query vector, the first key vector and the first value vector respectively to obtain a second query vector, a second key vector and a third value vector.

[0230] Specifically, Fig. 9 As shown, the first attention module, namely the M-Attention module, includes a Linear layer. During the processing of the first attention module, the first query vector (Q), the first key vector (K), and the first value vector (V) input therein are linearly processed by the Linear layer to obtain a second query vector, a second key vector, and a third value vector.

[0231] S154: Activate and normalize the second query vector and the second key vector to obtain a third query vector and a third key vector.

[0232] Specifically, Fig. 9 As shown, the first attention module, namely the M-Attention module, includes a Layernorm&Activation layer. After obtaining the second query vector and the second key vector, the Layernorm&Activation layer is used to activate and normalize the second query vector and the second key vector to obtain the third query vector and the third key vector.

[0233] S156: Perform a multiplication operation on the third query vector and the third key vector to obtain a target vector.

[0234] Specifically, Fig. 9 As shown, the first attention module, namely the M-Attention module, includes a BMM layer. After obtaining the third query vector and the third key vector, the BMM layer continues to multiply the third query vector and the third key vector to obtain a target vector.

[0235] S158: Normalize the target vector in the row dimension to obtain a weight matrix.

[0236] Specifically, Fig. 9 As shown, the first attention module, namely the M-Attention module, includes a Normalize layer. After obtaining the target vector, the Normalize layer is continued to normalize the target vector in the row dimension to obtain a weight matrix.

[0237] S160: Adjust the weight of the third value vector according to the weight matrix to obtain a fourth value vector.

[0238] Specifically, Fig. 9 As shown, the first attention module, namely the M-Attention module, includes a Reweight layer. After obtaining the third value vector and the weight matrix, the weight of the third value vector is adjusted according to the weight matrix through the Reweight layer to obtain the fourth value vector. In this way, the image editing model can focus on more useful information and improve the generation accuracy of the image editing model.

[0239] S162: Perform linear processing on the fourth value vector to obtain a fifth value vector.

[0240] Specifically, Fig. 9 As shown, after obtaining the fourth value vector, the fourth value vector is linearly processed through the Linear layer to obtain the fifth value vector.

[0241] S164: Add the first value vector and the fifth value vector element by element to obtain a second value vector.

[0242] Specifically, Fig. 9 As shown, the first attention module, namely the M-Attention module, includes a Skip-Connect layer. After obtaining the fifth value vector, the first value vector and the fifth value vector are element-wise added through the Skip-Connect layer to obtain the second value vector.

[0243] In summary, in the working process of the M-Attention module, if the input is Q, K and V, the output is The entire calculation process of the M-Attention module can specifically include the following formula:

[0244]

[0245] Among them, Q, K and V are the vectors input into the M-Attention module. and are the vectors of Q and K after being processed by LayerNorm&Activation respectively, is the vector after Linear processing, K T is the transposed vector of vector K, For vector The transposed vector, W is a three-dimensional matrix, W b,i,j Represents the matrix element at the j-th row, i-th column, and b-th layer in the three-dimensional matrix W.

[0246] In addition, in the image processing method provided in the embodiment of the present application, Fig. 9As shown, the image editing model also includes an FFN module, i.e., a nonlinear spatial mapping module. When the dimension of the second value vector output by the M-Attention module meets the second condition, the second value vector is nonlinearly mapped by the FFN module to adjust the dimension of the second value vector. In this way, the complexity of the image processing by the image editing model can be improved, so that the image editing model can learn more complex image processing methods, and the accuracy of the image generated by the image editing model is improved.

[0247] The second condition is that the dimension of the second value vector is [batch, Nd, dim]. That is, the FFN module is used to perform nonlinear spatial mapping on the second value vector of dimension [batch, Nd, dim] output by the M-Attention module to adjust the dimension of the second value vector.

[0248] In actual application, the FFN module can work based on the following formula:

[0249] FFN(x)=max(0,x×W1+b1)×W2+b2,

[0250] Among them, x is the second value vector output by the M-Attention module, and W1, W2, b1 and b2 are the module parameters of the FFN module.

[0251] In the above embodiment provided by the present application, in the process of processing the first query vector, the first key vector and the first value vector by the first attention module to obtain the second value vector, the first query vector, the first key vector and the first value vector are linearly processed respectively to obtain the second query vector, the second key vector and the third value vector; the second query vector and the second key vector are activated and normalized to obtain the third query vector and the third key vector; the third query vector and the third key vector are multiplied to obtain the target vector; the target vector is normalized in the row dimension to obtain the weight matrix; the weight of the third value vector is adjusted according to the weight matrix to obtain the fourth value vector; the fourth value vector is linearly processed to obtain the fifth value vector; the first value vector and the fifth value vector are element-wise added to obtain the second value vector; wherein, when the dimension of the second value vector meets the second condition, the second value vector is nonlinearly mapped to adjust the dimension of the second value vector. In this way, by re-weighting the value vector input therein through the attention module, the image editing model can focus on more useful information, thereby improving the accuracy of the image editing model.

[0252] In the embodiment of the present application, the above S110 may specifically include the following S110a and S110b:

[0253] S110a: Performing Gaussian blur processing on the edge of the first region in the second image corresponding to the second mask image to obtain a processed first region.

[0254] In the image processing method provided in the embodiment of the present application, after obtaining the second image, Gaussian blur processing is performed on the edge of the first area corresponding to the second mask image in the second image to soften the edge of the first area and obtain the processed first area.

[0255] S110b: Perform weighted fusion on the processed first region and the second region in the first image corresponding to the second mask image to obtain a third image.

[0256] Specifically, in the image processing method provided in the embodiment of the present application, the processed first region and the second region of the first image corresponding to the second mask image are weightedly fused to obtain the final generated image, i.e., the third image. In this way, the color difference of the generated image content can be reduced, the consistency of the generated image content can be ensured, and the quality and effect of the generated image can be improved.

[0257] In the above embodiment provided by the present application, in the process of fusing the first image and the second image to obtain the third image, Gaussian blur processing is performed on the edge of the first area of ​​the second image corresponding to the second mask image to obtain the processed first area; and weighted fusion is performed on the processed first area and the second area of ​​the first image corresponding to the second mask image to obtain the third image. In this way, the edge of the mask area can be feathered while retaining the generated content, which can improve the color difference problem of the intelligently generated image.

[0258] In the embodiment of the present application, the above-mentioned image processing method may further specifically include the following S166 to S174:

[0259] S166: Acquire a sample data set, where the sample data set includes a plurality of sample images and description information corresponding to each sample image.

[0260] The sample images include portrait images and object images.

[0261] Specifically, in the image processing method provided in the embodiment of the present application, a plurality of training images are obtained, and then the people and objects in each training image are segmented through a detection and segmentation model, such as a Grounded-Segment-Anything model. Furthermore, a mask image and a corresponding label of the segmented object image are constructed, and a multimodal large model is used to mark the segmented object image to obtain description information of the object image. Furthermore, a face detection model and a human body detection model are used to cut out the detected faces and human bodies, and a multimodal large language model is used to obtain face description information and human body description information in the portrait image. On this basis, a sample data set is constructed based on the obtained object images and their description information, and portrait images and their description information.

[0262] S168: Encode each sample image and add noise information to obtain a desired processing result corresponding to each sample image.

[0263] Specifically, after constructing the sample data set, each sample image in the sample data set is encoded through VAE, and noise information is added to each sample image to obtain the expected processing result corresponding to each sample image.

[0264] S170: Process each sample image using the image editing model to obtain a model output result corresponding to each sample image.

[0265] Specifically, after constructing the sample data set, the image editing model is used to process each sample image in the sample data set to obtain the model output result corresponding to each sample image.

[0266] S172: Determine the loss value of the image editing model according to the model output result and the expected processing result corresponding to each sample image.

[0267] Specifically, in the image processing method provided in the embodiment of the present application, each time the image editing model processes a sample image in the sample data set, the loss value of the image editing model is determined based on the model output result and the expected processing result corresponding to the sample image.

[0268] In actual application, the loss value of the image editing model can be calculated according to the following formula:

[0269] loss=mean(target-predict),

[0270] Among them, loss represents the loss value of the image editing model, target represents the expected processing result corresponding to each sample image, predict represents the model output result corresponding to each sample image, and mean represents the average value.

[0271] S174: Using a back propagation algorithm, iteratively update the weights of the model parameters of the image editing model according to the loss value until the loss value of the image editing model converges, thereby obtaining a trained image editing model.

[0272] Specifically, after calculating the loss value of the image editing model, the weights of the model parameters of the image editing model are iteratively updated using the back propagation algorithm based on the loss value until the loss value of the image editing model converges, thereby obtaining a trained image editing model.

[0273] The above-mentioned embodiment provided by the present application obtains a sample data set, which includes multiple sample images and description information corresponding to each sample image; encodes each sample image and adds noise information to obtain the expected processing result corresponding to each sample image; uses the image editing model to process each sample image to obtain the model output result corresponding to each sample image; determines the loss value of the image editing model according to the model output result and the expected processing result corresponding to each sample image; uses the back propagation algorithm to iteratively update the weights of the model parameters of the image editing model according to the loss value until the loss value of the image editing model converges to obtain the trained image editing model. In this way, the image editing model is trained based on images of people and objects, which ensures the accuracy of the training of the image editing model and improves the image processing performance of the image editing model.

[0274] In summary, the image processing method provided in the embodiment of the present application is as follows: Figure 7 As shown, after the user uploads the first image, face detection is performed on the first image to obtain the face position, and the face area image is segmented to obtain the third mask image corresponding to different portrait attributes. Further, semantic analysis is performed on the generation instruction information input by the user, the entity noun in the generation instruction information is extracted, and the first mask image is determined from the mask image library based on the entity noun and the third mask image. Further, based on the user's interactive input of the first mask image, a second mask image that meets the user's requirements is determined. Further, combined with the generation instruction information input by the user and / or the prompt of the expected image information, the filling content of the mask area is generated through the independently designed image editing model, and the filling content is filled into the first image to generate the corresponding edited second image. Further, the original first image and the edited second image are fused to obtain the third image. In addition, the electronic device can issue a prompt to the user. When the user needs to continue editing the image, he can choose to continue inputting the prompt information or directly manually edit the first mask image until the generated image meets the user's requirements.

[0275] Specifically, Figure 3 As shown, the image processing method provided in the embodiment of the present application may specifically include the following S302 to S326:

[0276] S302: The user inputs a first image.

[0277] S304: Perform face detection on the first image.

[0278] S306: face enlargement and attribute segmentation.

[0279] S308: Determine a first mask image from the M fourth mask images according to the attribute segmentation result.

[0280] S310: interactively generate a second mask image.

[0281] S312: Start the image editing model.

[0282] S314: Generate content and add it to the mask area.

[0283] S316: Determine: the user continues editing, if yes, execute S320, if not, execute S318.

[0284] S318: Save the edited second image.

[0285] S320: Receive user input.

[0286] S322: Obtain the user's generation instruction information.

[0287] S324: Perform semantic analysis on the generation instruction information to extract entity nouns in the generation instruction information.

[0288] S326: Retrieve M fourth mask images corresponding to the entity noun from the mask image library.

[0289] The image processing method provided in the embodiment of the present application can be executed by an image processing device. In the embodiment of the present application, an image processing device executing the above-mentioned image processing method is taken as an example to illustrate the image processing device provided in the embodiment of the present application.

[0290] like Fig.10 As shown, an embodiment of the present application provides an image processing device 500, which may include the following receiving unit 502 and processing unit 504.

[0291] A receiving unit 502, configured to receive a first image and prompt information, wherein the prompt information includes generating instruction information;

[0292] The processing unit 504 is used to determine a first mask image according to the first image and the generation instruction information;

[0293] The processing unit 504 is further configured to determine a second mask image according to the first mask image;

[0294] The processing unit 504 is further configured to generate a second image according to the first image, the second mask image and the prompt information;

[0295] The processing unit 504 is further configured to perform fusion processing on the first image and the second image to obtain a third image.

[0296] Through the image processing device 500 provided in the embodiment of the present application, a first image and prompt information are received, and the prompt information includes generation instruction information; a first mask image is determined according to the first image and the generation instruction information; a second mask image is determined according to the first mask image; a second image is generated according to the first image, the second mask image and the prompt information; the first image and the second image are fused to obtain a third image. Through the above-mentioned image processing device 500, based on the first image input by the user and the generation instruction information, the first mask image is determined, and then the second mask image to be edited is determined. Based on this, according to the first image input by the user, the prompt information and the second mask image to be edited, an initial image that meets the user's requirements, that is, a second image, is intelligently generated, and the original first image and the intelligently generated second image are fused to obtain the final third image. In this way, the image is intelligently edited and generated in combination with the user's prompt information, the quality of the intelligently generated image is improved, the followability of the user's instructions in the process of intelligently generating the image is guaranteed, and the consistency of the generated content is guaranteed through image fusion, and the image effect of the intelligent generation is improved.

[0297] In an embodiment of the present application, the processing unit 504 is specifically used to: perform face detection and face segmentation on the first image to obtain a third mask image, wherein the third mask image corresponds to the portrait area in the first image; obtain the entity noun in the generation instruction information; and determine the first mask image based on the third mask image and the entity noun.

[0298] In the above-mentioned embodiment provided by the present application, in the process of determining the first mask image according to the first image and the generation instruction information, face detection and face segmentation are performed on the first image to obtain a third mask image, wherein the third mask image corresponds to the portrait area in the first image; the entity noun in the generation instruction information is obtained; and the first mask image is determined according to the third mask image and the entity noun. In this way, based on the user's generation instruction information, the first mask image is determined in a targeted manner, which ensures the accuracy of the second mask image determined subsequently, thereby improving the accuracy of the intelligent editing image and the effect of the intelligent generation of the image.

[0299] In the embodiment of the present application, the processing unit 504 is specifically used to: perform face detection on the first image to determine a face detection frame; construct a target face frame based on the coordinate information and size information of the face detection frame; when the size of the target face frame does not meet the first condition, adjust the size of the target face frame through a first processing until the size of the target face frame meets the first condition; when the size of the target face frame meets the first condition, segment the image within the target face frame to obtain a third mask image.

[0300] In the above embodiment provided by the present application, in the process of performing face detection and face segmentation on the first image to obtain the third mask image, face detection is performed on the first image to determine the face detection frame; a target face frame is constructed according to the coordinate information and size information of the face detection frame; if the size of the target face frame does not meet the first condition, the size of the target face frame is adjusted through the first processing until the size of the target face frame meets the first condition; if the size of the target face frame meets the first condition, the image in the target face frame is segmented to obtain the third mask image. In this way, the face area in the first image is scaled and then segmented, which can ensure the face segmentation effect.

[0301] In the embodiment of the present application, the processing unit 504 is specifically used to: when the area of ​​the target face frame is larger than the first area or the area ratio of the target face frame in the first image is larger than the first value, reduce the size information of the first image to reduce the area of ​​the target face frame until the area of ​​the target face frame meets the first condition, and adjust the pixel information in the target face frame according to the scaling factor of the first image; when the area of ​​the target face frame is smaller than the second area or the area ratio of the target face frame in the first image is smaller than the second value, increase the size information of the face detection frame to increase the area of ​​the target face frame until the area of ​​the target face frame meets the first condition; when the boundary of the target face frame coincides with the boundary of the first image, adjust the coordinates of the center point of the face detection frame to adjust the position information of the target face frame.

[0302] In the above-mentioned embodiment provided by the present application, in the process of adjusting the size of the target face frame by the first processing, when the area of ​​the target face frame is larger than the first area or the area ratio of the target face frame in the first image is larger than the first value, the size information of the first image is reduced to reduce the area of ​​the target face frame until the area of ​​the target face frame meets the first condition, and the pixel information in the target face frame is adjusted according to the scaling factor of the first image; when the area of ​​the target face frame is smaller than the second area or the area ratio of the target face frame in the first image is smaller than the second value, the size information of the face detection frame is increased to increase the area of ​​the target face frame until the area of ​​the target face frame meets the first condition; when the boundary of the target face frame coincides with the boundary of the first image, the coordinates of the center point of the face detection frame are adjusted to adjust the position information of the target face frame. In this way, the face area in the first image is first scaled to a suitable size before segmentation, which can ensure the segmentation effect of the face and avoid the problem of the target face frame crossing the boundary.

[0303] In the embodiment of the present application, the processing unit 504 is specifically used to: select M fourth mask images from the mask image library according to the entity noun, where M is a positive integer greater than 1; determine the area intersection-union ratio of each fourth mask image to the third mask image according to the area of ​​the third mask image, the area of ​​each fourth mask image, the intersection area of ​​each fourth mask image with the third mask image, and the union area of ​​each fourth mask image with the third mask image; and determine the first mask image from the M fourth mask images according to the area intersection-union ratio.

[0304] In the above-mentioned embodiment provided by the present application, in the process of determining the first mask image according to the third mask image and the entity noun, M fourth mask images are selected from the mask image library according to the entity noun, where M is a positive integer greater than 1; the area intersection-and-union ratio of each fourth mask image to the third mask image is determined according to the area of ​​the third mask image, the area of ​​each fourth mask image, the intersection area of ​​each fourth mask image and the third mask image, and the union area of ​​each fourth mask image and the third mask image; and the first mask image is determined from the M fourth mask images according to the area intersection-and-union ratio. In this way, the area intersection-and-union ratio of each fourth mask image to the third mask image is calculated taking into account the boundary conditions, thereby ensuring the accuracy of the area intersection-and-union ratio calculation, and further ensuring the accuracy of the determined first mask image.

[0305] In the embodiment of the present application, the processing unit 504 is specifically used to: adjust the size of the first mask image according to the size of the target face frame; display the adjusted first mask image; adjust the first mask image according to the user's adjustment input to the first mask image to obtain the second mask image.

[0306] In the above embodiment provided by the present application, in the process of determining the second mask image according to the first mask image, the size of the first mask image is adjusted according to the size of the target face frame; the adjusted first mask image is displayed; and according to the user's adjustment input to the first mask image, the first mask image is adjusted to obtain the second mask image. In this way, on the one hand, the matching degree between the first mask image and the third mask image is improved, and the accuracy of the second mask image determined subsequently is guaranteed; on the other hand, the interactive generation of the second mask image is realized, and the second mask image is guaranteed to meet the user's requirements, and the accuracy of the subsequent intelligent generation and editing of the image based on the second mask image is guaranteed.

[0307] In the embodiment of the present application, the processing unit 504 is specifically used to: determine the first image feature according to the prompt information; determine the second image feature according to the first image and the second mask image; and generate the second image according to the first image feature and the second image feature.

[0308] In the above-mentioned embodiment provided by the present application, in the process of generating the second image according to the first image, the second mask image and the prompt information, the first image feature is determined according to the prompt information; the second image feature is determined according to the first image and the second mask image; and the second image is generated according to the first image feature and the second image feature. In this way, the image is intelligently edited and generated by combining the user's expected features for the generated image and the features to be edited of the input image, thereby improving the quality of the intelligently generated image and ensuring the followability of the user's instructions in the process of intelligently generating the image.

[0309] In an embodiment of the present application, the prompt information also includes expected image information, and the processing unit 504 is specifically used to: extract the text feature vector of the generation instruction information and the image feature vector of the expected image information; perform linear mapping on the image feature vector; and determine the first image feature based on the text feature vector and the mapped image feature vector.

[0310] In the above-mentioned embodiment provided by the present application, the prompt information also includes the expected image information. In the process of determining the first image feature according to the prompt information, the text feature vector of the generation instruction information and the image feature vector of the expected image information are extracted; the image feature vector is linearly mapped; and the first image feature is determined according to the text feature vector and the mapped image feature vector. In this way, the user's expected features for the generated image are determined based on the user's prompt information, which facilitates the control of generating the image required by the user and improves the effect of intelligently generating the image.

[0311] In the embodiment of the present application, the processing unit 504 is specifically used to: determine the fourth image based on the first image and the second mask image; encode the fourth image to obtain a first eigenvector; adjust the size of the second mask image, and determine the second eigenvector based on the mask information in the adjusted second mask image; determine the third eigenvector based on Gaussian noise data; and splice the target dimensions of the first eigenvector, the second eigenvector, and the third eigenvector to obtain a second image feature.

[0312] In the above embodiment provided by the present application, in the process of determining the second image feature according to the first image and the second mask image, the fourth image is determined according to the first image and the second mask image; the fourth image is encoded to obtain the first feature vector; the size of the second mask image is adjusted, and the second feature vector is determined according to the mask information in the adjusted second mask image; the third feature vector is determined according to the Gaussian noise data; the target dimensions of the first feature vector, the second feature vector and the third feature vector are spliced ​​to obtain the second image feature. In this way, the accuracy of the determined second image feature is guaranteed, and the accuracy of the subsequent intelligent editing and generation of the image based on the second image feature is guaranteed.

[0313] In the embodiment of the present application, the processing unit 504 is specifically used to: determine the rectangular area image corresponding to the mask information in the second mask image; increase the resolution of the rectangular area image to obtain a processed rectangular area image; and determine the fourth image based on the processed rectangular area image and the first image.

[0314] In the above embodiment provided by the present application, in the process of determining the fourth image according to the first image and the second mask image, the rectangular area image corresponding to the mask information in the second mask image is determined; the resolution of the rectangular area image is increased to obtain the processed rectangular area image; and the fourth image is determined according to the processed rectangular area image and the first image. In this way, it is easy to implement local editing of the area of ​​interest to the user in the first image without generating the entire image, which reduces the amount of calculation and processing and improves the efficiency of image processing.

[0315] In an embodiment of the present application, the processing unit 504 is specifically used to: input the first image feature and the second image feature into a trained image editing model, the image editing model including multiple network modules; sequentially process the first image feature and the second image feature using multiple network modules to generate a second image; wherein the output information of the Nth network module is the input information of the N+1th network module, and N is a positive integer.

[0316] In the above-mentioned embodiment provided by the present application, in the process of generating the second image according to the first image feature and the second image feature, the first image feature and the second image feature are input into the trained image editing model, and the image editing model includes multiple network modules; the first image feature and the second image feature are processed in sequence by the multiple network modules to generate the second image; wherein the output information of the Nth network module is the input information of the N+1th network module, and N is a positive integer. In this way, the image editing model is used to intelligently edit and generate images, which improves the quality of the intelligently generated images and ensures the followability of user instructions in the process of intelligently generating images.

[0317] In an embodiment of the present application, each network module includes a residual network, a first attention module, a second attention module and a sampling module connected in sequence, and the processing unit 504 is specifically used to: for each network module, process the second image feature through the residual network to obtain a first query vector, a first key vector and a first value vector; process the first query vector, the first key vector and the first value vector through the first attention module to obtain a second value vector; process the first image feature and the second value vector through the second attention module to obtain third output information; sample the resolution of the feature map corresponding to the third output information through the sampling module, and adjust the number of channels of the feature map to obtain the output information of each network module, until the last network module completes processing the first image feature and the second image feature to obtain the second image.

[0318] In the above-mentioned embodiment provided by the present application, each network module includes a residual network, a first attention module, a second attention module and a sampling module connected in sequence. In the process of sequentially processing the first image feature and the second image feature using multiple network modules to generate the second image, for each network module, the second image feature is processed by the residual network to obtain the first query vector, the first key vector and the first value vector; the first query vector, the first key vector and the first value vector are processed by the first attention module to obtain the second value vector; the first image feature and the second value vector are processed by the second attention module to obtain the third output information; the resolution of the feature map corresponding to the third output information is sampled by the sampling module, and the number of channels of the feature map is adjusted to obtain the output information of each network module, until the last network module completes the processing of the first image feature and the second image feature to obtain the second image. In this way, the image editing model is used to intelligently edit and generate images, the quality of intelligently generated images is improved, and the followability of user instructions in the process of intelligently generating images is ensured.

[0319] In an embodiment of the present application, the processing unit 504 is specifically used to: perform linear processing on the first query vector, the first key vector and the first value vector respectively to obtain the second query vector, the second key vector and the third value vector; perform activation processing and normalization processing on the second query vector and the second key vector to obtain the third query vector and the third key vector; perform multiplication operation on the third query vector and the third key vector to obtain the target vector; perform normalization processing on the target vector in the row dimension to obtain the weight matrix; adjust the weight of the third value vector according to the weight matrix to obtain the fourth value vector; perform linear processing on the fourth value vector to obtain the fifth value vector; perform element-wise addition on the first value vector and the fifth value vector to obtain the second value vector; wherein, when the dimension of the second value vector meets the second condition, the second value vector is nonlinearly mapped to adjust the dimension of the second value vector.

[0320] In the above embodiment provided by the present application, in the process of processing the first query vector, the first key vector and the first value vector by the first attention module to obtain the second value vector, the first query vector, the first key vector and the first value vector are linearly processed respectively to obtain the second query vector, the second key vector and the third value vector; the second query vector and the second key vector are activated and normalized to obtain the third query vector and the third key vector; the third query vector and the third key vector are multiplied to obtain the target vector; the target vector is normalized in the row dimension to obtain the weight matrix; the weight of the third value vector is adjusted according to the weight matrix to obtain the fourth value vector; the fourth value vector is linearly processed to obtain the fifth value vector; the first value vector and the fifth value vector are element-wise added to obtain the second value vector; wherein, when the dimension of the second value vector meets the second condition, the second value vector is nonlinearly mapped to adjust the dimension of the second value vector. In this way, by re-weighting the value vector input therein through the attention module, the image editing model can focus on more useful information, thereby improving the accuracy of the image editing model.

[0321] In the embodiment of the present application, the processing unit 504 is specifically used to: perform Gaussian blur processing on the edge of the first area corresponding to the second mask image in the second image to obtain the processed first area; perform weighted fusion on the processed first area and the second area corresponding to the second mask image in the first image to obtain a third image.

[0322] In the above embodiment provided by the present application, in the process of fusing the first image and the second image to obtain the third image, Gaussian blur processing is performed on the edge of the first area of ​​the second image corresponding to the second mask image to obtain the processed first area; and weighted fusion is performed on the processed first area and the second area of ​​the first image corresponding to the second mask image to obtain the third image. In this way, the edge of the mask area can be feathered while retaining the generated content, which can improve the color difference problem of the intelligently generated image.

[0323] In an embodiment of the present application, the processing unit 504 is also used to: obtain a sample data set, the sample data set including multiple sample images and descriptive information corresponding to each sample image; encode each sample image and add noise information to obtain an expected processing result corresponding to each sample image; use the image editing model to process each sample image to obtain a model output result corresponding to each sample image; determine the loss value of the image editing model based on the model output result and the expected processing result corresponding to each sample image; use the back propagation algorithm to iteratively update the weights of the model parameters of the image editing model according to the loss value until the loss value of the image editing model converges to obtain the trained image editing model.

[0324] The above-mentioned embodiment provided by the present application obtains a sample data set, which includes multiple sample images and description information corresponding to each sample image; encodes each sample image and adds noise information to obtain the expected processing result corresponding to each sample image; uses the image editing model to process each sample image to obtain the model output result corresponding to each sample image; determines the loss value of the image editing model according to the model output result and the expected processing result corresponding to each sample image; uses the back propagation algorithm to iteratively update the weights of the model parameters of the image editing model according to the loss value until the loss value of the image editing model converges to obtain the trained image editing model. In this way, the image editing model is trained based on images of people and objects, which ensures the accuracy of the training of the image editing model and improves the image processing performance of the image editing model.

[0325] The image processing device 500 in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0326] The image processing device 500 in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0327] The image processing device 500 provided in the embodiment of the present application can realize Figure 1 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0328] Alternatively, if Fig.11 As shown, an embodiment of the present application further provides an electronic device 600, including a processor 602 and a memory 604, wherein the memory 604 stores programs or instructions that can be executed on the processor 602, and when the program or instructions are executed by the processor 602, the various steps of the above-mentioned image processing method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0329] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.

[0330] Fig.12 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0331] The electronic device 700 includes but is not limited to: a radio frequency unit 701, a network module 702, an audio output unit 703, an input unit 704, a sensor 705, a display unit 706, a user input unit 707, an interface unit 708, a memory 709, a processor 710 and other components.

[0332] Those skilled in the art will appreciate that the electronic device 700 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 710 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig.12 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0333] The processor 710 is used to receive a first image and prompt information, where the prompt information includes generating instruction information.

[0334] The processor 710 is further configured to determine a first mask image according to the first image and the generation instruction information;

[0335] The processor 710 is further configured to determine a second mask image according to the first mask image.

[0336] The processor 710 is further configured to generate a second image according to the first image, the second mask image and the prompt information.

[0337] The processor 710 is further configured to perform fusion processing on the first image and the second image to obtain a third image.

[0338] In an embodiment of the present application, a first image and prompt information are received, wherein the prompt information includes generation instruction information; a first mask image is determined according to the first image and the generation instruction information; a second mask image is determined according to the first mask image; a second image is generated according to the first image, the second mask image and the prompt information; and the first image and the second image are fused to obtain a third image. In an embodiment of the present application, the first mask image is determined based on the first image input by the user and the generation instruction information, and then the second mask image to be edited is determined. Based on this, an initial image that meets the user's requirements, i.e., a second image, is intelligently generated according to the first image input by the user, the prompt information and the second mask image to be edited, and the final third image is obtained by fusion of the original first image and the intelligently generated second image. In this way, the image is intelligently edited and generated in combination with the user's prompt information, thereby improving the quality of the intelligently generated image, ensuring the followability of the user's instructions in the process of intelligently generating the image, and ensuring the consistency of the generated content through image fusion, thereby improving the image effect of intelligent generation.

[0339] Optionally, the processor 710 is specifically used to: perform face detection and face segmentation on the first image to obtain a third mask image, wherein the third mask image corresponds to the portrait area in the first image; obtain the entity noun in the generation instruction information; and determine the first mask image based on the third mask image and the entity noun.

[0340] In the above-mentioned embodiment provided by the present application, in the process of determining the first mask image according to the first image and the generation instruction information, face detection and face segmentation are performed on the first image to obtain a third mask image, wherein the third mask image corresponds to the portrait area in the first image; the entity noun in the generation instruction information is obtained; and the first mask image is determined according to the third mask image and the entity noun. In this way, based on the user's generation instruction information, the first mask image is determined in a targeted manner, which ensures the accuracy of the second mask image determined subsequently, thereby improving the accuracy of the intelligent editing image and the effect of the intelligent generation of the image.

[0341] Optionally, the processor 710 is specifically used to: perform face detection on the first image to determine a face detection frame; construct a target face frame according to the coordinate information and size information of the face detection frame; when the size of the target face frame does not meet the first condition, adjust the size of the target face frame through a first processing until the size of the target face frame meets the first condition; when the size of the target face frame meets the first condition, segment the image within the target face frame to obtain a third mask image.

[0342] In the above embodiment provided by the present application, in the process of performing face detection and face segmentation on the first image to obtain the third mask image, face detection is performed on the first image to determine the face detection frame; a target face frame is constructed according to the coordinate information and size information of the face detection frame; if the size of the target face frame does not meet the first condition, the size of the target face frame is adjusted through the first processing until the size of the target face frame meets the first condition; if the size of the target face frame meets the first condition, the image in the target face frame is segmented to obtain the third mask image. In this way, the face area in the first image is scaled and then segmented, which can ensure the face segmentation effect.

[0343] Optionally, the processor 710 is specifically used to: when the area of ​​the target face frame is larger than the first area or the area ratio of the target face frame in the first image is larger than a first value, reduce the size information of the first image to reduce the area of ​​the target face frame until the area of ​​the target face frame meets the first condition, and adjust the pixel information in the target face frame according to the scaling factor of the first image; when the area of ​​the target face frame is smaller than the second area or the area ratio of the target face frame in the first image is smaller than the second value, increase the size information of the face detection frame to increase the area of ​​the target face frame until the area of ​​the target face frame meets the first condition; when the boundary of the target face frame coincides with the boundary of the first image, adjust the coordinates of the center point of the face detection frame to adjust the position information of the target face frame.

[0344] In the above-mentioned embodiment provided by the present application, in the process of adjusting the size of the target face frame by the first processing, when the area of ​​the target face frame is larger than the first area or the area ratio of the target face frame in the first image is larger than the first value, the size information of the first image is reduced to reduce the area of ​​the target face frame until the area of ​​the target face frame meets the first condition, and the pixel information in the target face frame is adjusted according to the scaling factor of the first image; when the area of ​​the target face frame is smaller than the second area or the area ratio of the target face frame in the first image is smaller than the second value, the size information of the face detection frame is increased to increase the area of ​​the target face frame until the area of ​​the target face frame meets the first condition; when the boundary of the target face frame coincides with the boundary of the first image, the coordinates of the center point of the face detection frame are adjusted to adjust the position information of the target face frame. In this way, the face area in the first image is first scaled to a suitable size before segmentation, which can ensure the segmentation effect of the face and avoid the problem of the target face frame crossing the boundary.

[0345] Optionally, the processor 710 is specifically used to: select M fourth mask images from a mask image library according to the entity noun, where M is a positive integer greater than 1; determine the area intersection-union ratio of each fourth mask image to the third mask image according to the area of ​​the third mask image, the area of ​​each fourth mask image, the intersection area of ​​each fourth mask image with the third mask image, and the union area of ​​each fourth mask image with the third mask image; and determine the first mask image from the M fourth mask images according to the area intersection-union ratio.

[0346] In the above-mentioned embodiment provided by the present application, in the process of determining the first mask image according to the third mask image and the entity noun, M fourth mask images are selected from the mask image library according to the entity noun, where M is a positive integer greater than 1; the area intersection-and-union ratio of each fourth mask image to the third mask image is determined according to the area of ​​the third mask image, the area of ​​each fourth mask image, the intersection area of ​​each fourth mask image and the third mask image, and the union area of ​​each fourth mask image and the third mask image; and the first mask image is determined from the M fourth mask images according to the area intersection-and-union ratio. In this way, the area intersection-and-union ratio of each fourth mask image to the third mask image is calculated taking into account the boundary conditions, thereby ensuring the accuracy of the area intersection-and-union ratio calculation, and further ensuring the accuracy of the determined first mask image.

[0347] Optionally, the processor 710 is specifically used to: adjust the size of the first mask image according to the size of the target face frame; display the adjusted first mask image; and adjust the first mask image according to the user's adjustment input to the first mask image to obtain the second mask image.

[0348] In the above embodiment provided by the present application, in the process of determining the second mask image according to the first mask image, the size of the first mask image is adjusted according to the size of the target face frame; the adjusted first mask image is displayed; and according to the user's adjustment input to the first mask image, the first mask image is adjusted to obtain the second mask image. In this way, on the one hand, the matching degree between the first mask image and the third mask image is improved, and the accuracy of the second mask image determined subsequently is guaranteed; on the other hand, the interactive generation of the second mask image is realized, and the second mask image is guaranteed to meet the user's requirements, and the accuracy of the subsequent intelligent generation and editing of the image based on the second mask image is guaranteed.

[0349] Optionally, the processor 710 is specifically configured to: determine a first image feature according to the prompt information; determine a second image feature according to the first image and the second mask image; and generate a second image according to the first image feature and the second image feature.

[0350] In the above-mentioned embodiment provided by the present application, in the process of generating the second image according to the first image, the second mask image and the prompt information, the first image feature is determined according to the prompt information; the second image feature is determined according to the first image and the second mask image; and the second image is generated according to the first image feature and the second image feature. In this way, the image is intelligently edited and generated by combining the user's expected features for the generated image and the features to be edited of the input image, thereby improving the quality of the intelligently generated image and ensuring the followability of the user's instructions in the process of intelligently generating the image.

[0351] Optionally, the prompt information also includes expected image information, and the processor 710 is specifically used to: extract the text feature vector of the generated instruction information and the image feature vector of the expected image information; perform linear mapping on the image feature vector; and determine the first image feature based on the text feature vector and the mapped image feature vector.

[0352] In the above-mentioned embodiment provided by the present application, the prompt information also includes the expected image information. In the process of determining the first image feature according to the prompt information, the text feature vector of the generation instruction information and the image feature vector of the expected image information are extracted; the image feature vector is linearly mapped; and the first image feature is determined according to the text feature vector and the mapped image feature vector. In this way, the user's expected features for the generated image are determined based on the user's prompt information, which facilitates the control of generating the image required by the user and improves the effect of intelligently generating the image.

[0353] Optionally, the processor 710 is specifically used to: determine a fourth image based on the first image and the second mask image; encode the fourth image to obtain a first eigenvector; adjust the size of the second mask image, and determine a second eigenvector based on the mask information in the adjusted second mask image; determine a third eigenvector based on Gaussian noise data; and splice the target dimensions of the first eigenvector, the second eigenvector, and the third eigenvector to obtain second image features.

[0354] In the above embodiment provided by the present application, in the process of determining the second image feature according to the first image and the second mask image, the fourth image is determined according to the first image and the second mask image; the fourth image is encoded to obtain the first feature vector; the size of the second mask image is adjusted, and the second feature vector is determined according to the mask information in the adjusted second mask image; the third feature vector is determined according to the Gaussian noise data; the target dimensions of the first feature vector, the second feature vector and the third feature vector are spliced ​​to obtain the second image feature. In this way, the accuracy of the determined second image feature is guaranteed, and the accuracy of the subsequent intelligent editing and generation of the image based on the second image feature is guaranteed.

[0355] Optionally, the processor 710 is specifically used to: determine a rectangular area image corresponding to the mask information in the second mask image; increase the resolution of the rectangular area image to obtain a processed rectangular area image; and determine a fourth image based on the processed rectangular area image and the first image.

[0356] In the above embodiment provided by the present application, in the process of determining the fourth image according to the first image and the second mask image, the rectangular area image corresponding to the mask information in the second mask image is determined; the resolution of the rectangular area image is increased to obtain the processed rectangular area image; and the fourth image is determined according to the processed rectangular area image and the first image. In this way, it is easy to implement local editing of the area of ​​interest to the user in the first image without generating the entire image, which reduces the amount of calculation and processing and improves the efficiency of image processing.

[0357] Optionally, the processor 710 is specifically used to: input the first image feature and the second image feature into a trained image editing model, the image editing model including multiple network modules; use the multiple network modules in sequence to process the first image feature and the second image feature to generate a second image; wherein the output information of the Nth network module is the input information of the N+1th network module, and N is a positive integer.

[0358] In the above-mentioned embodiment provided by the present application, in the process of generating the second image according to the first image feature and the second image feature, the first image feature and the second image feature are input into the trained image editing model, and the image editing model includes multiple network modules; the first image feature and the second image feature are processed in sequence by the multiple network modules to generate the second image; wherein the output information of the Nth network module is the input information of the N+1th network module, and N is a positive integer. In this way, the image editing model is used to intelligently edit and generate images, which improves the quality of the intelligently generated images and ensures the followability of user instructions in the process of intelligently generating images.

[0359] Optionally, each network module includes a residual network, a first attention module, a second attention module and a sampling module connected in sequence, and the processor 710 is specifically used to: for each network module, process the second image feature through the residual network to obtain a first query vector, a first key vector and a first value vector; process the first query vector, the first key vector and the first value vector through the first attention module to obtain a second value vector; process the first image feature and the second value vector through the second attention module to obtain third output information; sample the resolution of the feature map corresponding to the third output information through the sampling module, and adjust the number of channels of the feature map to obtain the output information of each network module, until the last network module completes processing the first image feature and the second image feature to obtain the second image.

[0360] In the above-mentioned embodiment provided by the present application, each network module includes a residual network, a first attention module, a second attention module and a sampling module connected in sequence. In the process of sequentially processing the first image feature and the second image feature using multiple network modules to generate the second image, for each network module, the second image feature is processed by the residual network to obtain the first query vector, the first key vector and the first value vector; the first query vector, the first key vector and the first value vector are processed by the first attention module to obtain the second value vector; the first image feature and the second value vector are processed by the second attention module to obtain the third output information; the resolution of the feature map corresponding to the third output information is sampled by the sampling module, and the number of channels of the feature map is adjusted to obtain the output information of each network module, until the last network module completes the processing of the first image feature and the second image feature to obtain the second image. In this way, the image editing model is used to intelligently edit and generate images, the quality of intelligently generated images is improved, and the followability of user instructions in the process of intelligently generating images is ensured.

[0361] Optionally, the processor 710 is specifically used to: perform linear processing on the first query vector, the first key vector and the first value vector, respectively, to obtain a second query vector, a second key vector and a third value vector; perform activation processing and normalization processing on the second query vector and the second key vector to obtain a third query vector and a third key vector; perform multiplication operation on the third query vector and the third key vector to obtain a target vector; perform normalization processing on the target vector in the row dimension to obtain a weight matrix; adjust the weight of the third value vector according to the weight matrix to obtain a fourth value vector; perform linear processing on the fourth value vector to obtain a fifth value vector; perform element-wise addition on the first value vector and the fifth value vector to obtain a second value vector; wherein, when the dimension of the second value vector meets the second condition, the second value vector is nonlinearly mapped to adjust the dimension of the second value vector.

[0362] In the above embodiment provided by the present application, in the process of processing the first query vector, the first key vector and the first value vector by the first attention module to obtain the second value vector, the first query vector, the first key vector and the first value vector are linearly processed respectively to obtain the second query vector, the second key vector and the third value vector; the second query vector and the second key vector are activated and normalized to obtain the third query vector and the third key vector; the third query vector and the third key vector are multiplied to obtain the target vector; the target vector is normalized in the row dimension to obtain the weight matrix; the weight of the third value vector is adjusted according to the weight matrix to obtain the fourth value vector; the fourth value vector is linearly processed to obtain the fifth value vector; the first value vector and the fifth value vector are element-wise added to obtain the second value vector; wherein, when the dimension of the second value vector meets the second condition, the second value vector is nonlinearly mapped to adjust the dimension of the second value vector. In this way, by re-weighting the value vector input therein through the attention module, the image editing model can focus on more useful information, thereby improving the accuracy of the image editing model.

[0363] Optionally, the processor 710 is specifically used to: perform Gaussian blur processing on the edge of the first area in the second image corresponding to the second mask image to obtain the processed first area; and perform weighted fusion on the processed first area and the second area in the first image corresponding to the second mask image to obtain a third image.

[0364] In the above embodiment provided by the present application, in the process of fusing the first image and the second image to obtain the third image, Gaussian blur processing is performed on the edge of the first area of ​​the second image corresponding to the second mask image to obtain the processed first area; and weighted fusion is performed on the processed first area and the second area of ​​the first image corresponding to the second mask image to obtain the third image. In this way, the edge of the mask area can be feathered while retaining the generated content, which can improve the color difference problem of the intelligently generated image.

[0365] Optionally, the processor 710 is also used to: obtain a sample data set, the sample data set including multiple sample images and description information corresponding to each sample image; encode each sample image and add noise information to obtain an expected processing result corresponding to each sample image; process each sample image using an image editing model to obtain a model output result corresponding to each sample image; determine a loss value of the image editing model based on the model output result and the expected processing result corresponding to each sample image; and iteratively update the weights of the model parameters of the image editing model based on the loss value using a back propagation algorithm until the loss value of the image editing model converges to obtain a trained image editing model.

[0366] The above-mentioned embodiment provided by the present application obtains a sample data set, which includes multiple sample images and description information corresponding to each sample image; encodes each sample image and adds noise information to obtain the expected processing result corresponding to each sample image; uses the image editing model to process each sample image to obtain the model output result corresponding to each sample image; determines the loss value of the image editing model according to the model output result and the expected processing result corresponding to each sample image; uses the back propagation algorithm to iteratively update the weights of the model parameters of the image editing model according to the loss value until the loss value of the image editing model converges to obtain the trained image editing model. In this way, the image editing model is trained based on images of people and objects, which ensures the accuracy of the training of the image editing model and improves the image processing performance of the image editing model.

[0367] It should be understood that in the embodiment of the present application, the input unit 704 may include a graphics processor (Graphics Processing Unit, GPU) 7041 and a microphone 7042, and the graphics processor 7041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 706 may include a display panel 7061, and the display panel 7061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 707 includes a touch panel 7071 and at least one of other input devices 7072. The touch panel 7071 is also called a touch screen. The touch panel 7071 may include two parts: a touch detection device and a touch controller. Other input devices 7072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0368] The memory 709 can be used to store software programs and various data. The memory 709 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 709 may include a volatile memory or a non-volatile memory, or the memory 709 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 709 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0369] The processor 710 may include one or more processing units; optionally, the processor 710 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 710.

[0370] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned image processing method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0371] The processor is the processor in the electronic device in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0372] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0373] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0374] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned image processing method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0375] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0376] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods of each embodiment of the present application.

[0377] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An image processing method, characterized in that: include: receiving a first image and prompt information, wherein the prompt information includes generating instruction information; Determining a first mask image according to the first image and the generation instruction information; determining a second mask image according to the first mask image; generating a second image according to the first image, the second mask image and the prompt information; The first image and the second image are fused to obtain a third image.

2. The image processing method according to claim 1, characterized in that: The step of determining a first mask image according to the first image and the generation instruction information includes: Performing face detection and face segmentation on the first image to obtain a third mask image, wherein the third mask image corresponds to the portrait area in the first image; Obtaining entity nouns in the generation instruction information; The first mask image is determined according to the third mask image and the entity noun.

3. The image processing method according to claim 2, characterized in that: The performing face detection and face segmentation on the first image to obtain a third mask image includes: Performing face detection on the first image to determine a face detection frame; Constructing a target face frame according to the coordinate information and size information of the face detection frame; When the size of the target face frame does not meet the first condition, adjusting the size of the target face frame through a first process until the size of the target face frame meets the first condition; When the size of the target face frame meets the first condition, the image within the target face frame is segmented to obtain the third mask image.

4. The image processing method according to claim 3, characterized in that: When the size of the target face frame does not meet the first condition, adjusting the size of the target face frame through a first process includes: When the area of ​​the target face frame is larger than the first area or the area ratio of the target face frame in the first image is larger than a first value, reducing the size information of the first image to reduce the area of ​​the target face frame until the area of ​​the target face frame meets the first condition, and adjusting the pixel information in the target face frame according to the scaling factor of the first image; When the area of ​​the target face frame is smaller than the second area or the area proportion of the target face frame in the first image is smaller than a second value, increasing the size information of the face detection frame to increase the area of ​​the target face frame until the area of ​​the target face frame meets the first condition; When the boundary of the target face frame coincides with the boundary of the first image, the coordinates of the center point of the face detection frame are adjusted to adjust the position information of the target face frame.

5. The image processing method according to claim 2, characterized in that: The determining the first mask image according to the third mask image and the entity noun includes: According to the entity noun, M fourth mask images are selected from the mask image library, where M is a positive integer greater than 1; Determining an intersection-and-union ratio of the areas of each fourth mask image and the third mask image according to the area of ​​the third mask image, the area of ​​each fourth mask image, the intersection area of ​​each fourth mask image and the third mask image, and the union area of ​​each fourth mask image and the third mask image; The first mask image is determined from M fourth mask images according to the area intersection-to-union ratio.

6. The image processing method according to any one of claims 1 to 5, characterized in that: The step of generating a second image according to the first image, the second mask image and the prompt information includes: Determine a first image feature according to the prompt information; determining a second image feature according to the first image and the second mask image; The second image is generated according to the first image feature and the second image feature.

7. The image processing method according to claim 6, characterized in that: The prompt information further includes expected image information, and determining the first image feature according to the prompt information includes: Extracting a text feature vector of the generation instruction information and an image feature vector of the expected image information; Performing linear mapping on the image feature vector; The first image feature is determined according to the text feature vector and the mapped image feature vector.

8. The image processing method according to claim 6, characterized in that: The determining of the second image feature according to the first image and the second mask image includes: determining a fourth image according to the first image and the second mask image; Encoding the fourth image to obtain a first feature vector; adjusting a size of the second mask image, and determining a second feature vector according to mask information in the adjusted second mask image; determining a third eigenvector based on the Gaussian noise data; The target dimensions of the first feature vector, the second feature vector, and the third feature vector are concatenated to obtain the second image feature.

9. The image processing method according to claim 8, characterized in that: The determining of the fourth image according to the first image and the second mask image comprises: Determine a rectangular area image corresponding to the mask information in the second mask image; Increasing the resolution of the rectangular area image to obtain a processed rectangular area image; The fourth image is determined according to the processed rectangular area image and the first image.

10. The image processing method according to claim 6, characterized in that: The step of generating the second image according to the first image feature and the second image feature includes: Inputting the first image feature and the second image feature into a trained image editing model, wherein the image editing model includes a plurality of network modules; sequentially processing the first image feature and the second image feature using a plurality of the network modules to generate the second image; The output information of the Nth network module is the input information of the N+1th network module, and N is a positive integer.

11. The image processing method according to claim 10, characterized in that: Each of the network modules includes a residual network, a first attention module, a second attention module, and a sampling module connected in sequence, and sequentially using the plurality of network modules to process the first image features and the second image features to generate the second image includes: For each of the network modules, processing the second image feature through the residual network to obtain a first query vector, a first key vector and a first value vector; Processing the first query vector, the first key vector, and the first value vector through the first attention module to obtain a second value vector; Processing the first image feature and the second value vector through the second attention module to obtain third output information; The resolution of the feature map corresponding to the third output information is sampled by the sampling module, and the number of channels of the feature map is adjusted to obtain the output information of each network module, until the last network module completes processing the first image features and the second image features to obtain the second image.

12. The image processing method according to claim 11, characterized in that: The processing of the first query vector, the first key vector, and the first value vector by the first attention module to obtain a second value vector includes: Performing linear processing on the first query vector, the first key vector, and the first value vector respectively to obtain a second query vector, a second key vector, and a third value vector; Performing activation processing and normalization processing on the second query vector and the second key vector to obtain a third query vector and a third key vector; Performing a multiplication operation on the third query vector and the third key vector to obtain a target vector; Normalizing the target vector in the row dimension to obtain a weight matrix; Adjusting the weight of the third value vector according to the weight matrix to obtain a fourth value vector; Performing linear processing on the fourth value vector to obtain a fifth value vector; Add the first value vector and the fifth value vector element by element to obtain the second value vector; Wherein, when the dimension of the second value vector meets the second condition, nonlinear mapping is performed on the second value vector to adjust the dimension of the second value vector.

13. The image processing method according to any one of claims 1 to 5, characterized in that: The fusing the first image and the second image to obtain a third image includes: Performing Gaussian blur processing on an edge of a first region in the second image corresponding to the second mask image to obtain a processed first region; The processed first region and a second region in the first image corresponding to the second mask image are weightedly fused to obtain the third image.

14. An image processing device, characterized in that: include: A receiving unit, configured to receive a first image and prompt information, wherein the prompt information includes generating instruction information; A processing unit, configured to determine a first mask image according to the first image and the generation instruction information; The processing unit is further used to determine a second mask image according to the first mask image; The processing unit is further configured to generate a second image according to the first image, the second mask image and the prompt information; The processing unit is further used to perform fusion processing on the first image and the second image to obtain a third image.

15. The image processing device according to claim 14, characterized in that: The processing unit is specifically used for: Performing face detection and face segmentation on the first image to obtain a third mask image, wherein the third mask image corresponds to the portrait area in the first image; Obtaining entity nouns in the generation instruction information; The first mask image is determined according to the third mask image and the entity noun.

16. The image processing device according to claim 15, characterized in that: The processing unit is specifically used for: Performing face detection on the first image to determine a face detection frame; Constructing a target face frame according to the coordinate information and size information of the face detection frame; When the size of the target face frame does not meet the first condition, adjusting the size of the target face frame through a first process until the size of the target face frame meets the first condition; When the size of the target face frame meets the first condition, the image within the target face frame is segmented to obtain the third mask image.

17. The image processing device according to claim 16, characterized in that: The processing unit is specifically used for: When the area of ​​the target face frame is larger than the first area or the area ratio of the target face frame in the first image is larger than a first value, reducing the size information of the first image to reduce the area of ​​the target face frame until the area of ​​the target face frame meets the first condition, and adjusting the pixel information in the target face frame according to the scaling factor of the first image; When the area of ​​the target face frame is smaller than the second area or the area proportion of the target face frame in the first image is smaller than a second value, increasing the size information of the face detection frame to increase the area of ​​the target face frame until the area of ​​the target face frame meets the first condition; When the boundary of the target face frame coincides with the boundary of the first image, the coordinates of the center point of the face detection frame are adjusted to adjust the position information of the target face frame.

18. The image processing device according to any one of claims 14 to 17, characterized in that: The processing unit is specifically used for: Determine a first image feature according to the prompt information; determining a second image feature according to the first image and the second mask image; The second image is generated according to the first image feature and the second image feature.

19. The image processing device according to claim 18, characterized in that: The processing unit is specifically used for: determining a fourth image according to the first image and the second mask image; Encoding the fourth image to obtain a first feature vector; adjusting a size of the second mask image, and determining a second feature vector according to mask information in the adjusted second mask image; determining a third eigenvector based on the Gaussian noise data; The target dimensions of the first feature vector, the second feature vector, and the third feature vector are concatenated to obtain the second image feature.

20. The image processing device according to claim 18, characterized in that: The processing unit is specifically used for: Inputting the first image feature and the second image feature into a trained image editing model, wherein the image editing model includes a plurality of network modules; sequentially processing the first image feature and the second image feature using a plurality of the network modules to generate the second image; The output information of the Nth network module is the input information of the N+1th network module, and N is a positive integer.

21. The image processing device according to claim 20, characterized in that: Each of the network modules includes a residual network, a first attention module, a second attention module and a sampling module connected in sequence, and the processing unit is specifically used for: For each of the network modules, processing the second image feature through the residual network to obtain a first query vector, a first key vector and a first value vector; Processing the first query vector, the first key vector, and the first value vector through the first attention module to obtain a second value vector; Processing the first image feature and the second value vector through the second attention module to obtain third output information; The resolution of the feature map corresponding to the third output information is sampled by the sampling module, and the number of channels of the feature map is adjusted to obtain the output information of each network module, until the last network module completes processing the first image features and the second image features to obtain the second image.

22. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the image processing method according to any one of claims 1 to 13 are implemented.

23. A readable storage medium, characterized in that: The readable storage medium stores a program or an instruction, and when the program or the instruction is executed by the processor, the steps of the image processing method according to any one of claims 1 to 13 are implemented.

Images