Image generation method and device, electronic equipment, storage medium and program product

Abstract

The invention discloses an image generation method and device, electronic equipment, a storage medium and a program product, and belongs to the technical field of artificial intelligence. The method comprises the steps of obtaining a first image and a second image; extracting a first portrait segmentation area in the first image and a second portrait segmentation area in the second image; determining a position relationship and a hierarchical relationship between the first portrait segmentation region and the second portrait segmentation region; and displaying the image content of the first portrait segmentation region in the second image according to the position relationship and the hierarchical relationship, and generating a target image.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and specifically relates to an image generation method, device, electronic device, storage medium and program product. Background Art

[0002] When users take group photos, their expressions may be stiff or unnatural, their eyes may be closed, or their gestures or postures may be uncoordinated, which may result in the image not meeting the user's expectations. The image needs to be edited, spliced, and processed to obtain a satisfactory group photo. Currently, the method of generating group photos mainly relies on users to manually edit and modify the image content to continuously adjust the image content, and finally obtain a group photo with better effect. The operation is complicated and inefficient. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide an image generation method, device, electronic device, storage medium and program product, which can solve the problem of cumbersome image generation operation and low efficiency.

[0004] In a first aspect, an embodiment of the present application provides an image generation method, the method comprising:

[0005] Acquire a first image and a second image;

[0006] Extracting a first portrait segmentation region in the first image and a second portrait segmentation region in the second image;

[0007] Determine the positional relationship and hierarchical relationship between the first portrait segmentation region and the second portrait segmentation region;

[0008] According to the position relationship and the hierarchical relationship, the image content of the first portrait segmentation area is displayed in the second image to generate a target image.

[0009] In a second aspect, an embodiment of the present application provides an image generating device, the device comprising:

[0010] An acquisition module, used for acquiring a first image and a second image;

[0011] An extraction module, used for extracting a first portrait segmentation region in the first image and a second portrait segmentation region in the second image;

[0012] A determination module, used to determine the positional relationship and hierarchical relationship between the first portrait segmentation region and the second portrait segmentation region;

[0013] The generating module is used to display the image content of the first portrait segmentation area in the second image according to the position relationship and the layer relationship to generate a target image.

[0014] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0015] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0016] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0017] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.

[0018] In the embodiment of the present application, the first image and the second image can be acquired, and the first image and the second image can be automatically extracted to obtain the first portrait segmentation area in the first image and the second portrait segmentation area in the second image. The positional relationship and hierarchical relationship between the first portrait segmentation area and the second portrait segmentation area can be determined, and the image content of the first portrait segmentation area can be displayed in the second image according to the positional relationship and hierarchical relationship to generate a target image. There is no need for the user to manually edit and adjust the image content. The target image can be automatically generated according to the positional relationship and hierarchical relationship between the portrait segmentation areas in the first image and the second image, which simplifies the operation process of image generation and improves the efficiency of image generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a flowchart of an image generation method provided in an embodiment of the present application;

[0020] Figure 2 It is a schematic diagram of a single-person photo uploading interface in the image generation method provided in an embodiment of the present application;

[0021] Figure 3 is a schematic diagram of a multi-person photo uploading interface in the image generation method provided in an embodiment of the present application;

[0022] Figure 4 is a schematic diagram of a portrait segmentation region in an image generation method provided in an embodiment of the present application;

[0023] Figure 5 is a schematic diagram of a parameter estimation model in the image generation method provided in an embodiment of the present application;

[0024] Figure 6 yes Figure 5 Schematic diagram of the attention learning module;

[0025] Figure 7 is a schematic diagram of a setting interface in the image generation method provided in an embodiment of the present application;

[0026] Figure 8 is a schematic diagram of the first area in the image generation method provided in an embodiment of the present application;

[0027] Fig. 9 is a working principle diagram of the image generation model in the image generation method provided in the embodiment of the present application;

[0028] Fig.10 yes Fig. 9 Schematic diagram of the main structure in the U-shaped network;

[0029] Fig.11 is a schematic diagram of a specific scenario embodiment of the image generation method provided in an embodiment of the present application;

[0030] Fig.12 is a structural schematic diagram of an image generating device provided in an embodiment of the present application;

[0031] Fig.13 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0032] Fig.14 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0034] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0035] The image generation method provided in the embodiment of the present application is described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0036] Figure 1 : is a flow chart of an image generation method provided in an embodiment of the present application. The image generation method may include:

[0037] Step 101: Acquire a first image and a second image.

[0038] In step 101, a first image and a second image may be obtained, wherein the first image may be a material image in image generation, and the second image may be a background image in image generation. That is, the portrait in the first image needs to be cut out and spliced ​​into the second image to generate a group photo.

[0039] In some specific scenarios, multiple users need to take a group photo, but only one user needs to take the photo, which means that the user cannot appear in the group photo. Therefore, a separate photo can be taken for the user, and the portrait in the photo can be spliced ​​into the group photo taken by the user to obtain a group photo including all users. In other specific scenarios, user A and user B may be separated from each other, but if a group photo needs to be taken, user A and user B can take their own photos separately, use user A's photo as the first image, use user B's photo as the second image, and splice the portrait in user A's photo into user B's photo to obtain a photo of two people.

[0040] To facilitate understanding of the technical solution provided in the embodiments of the present application, the first image will be referred to as a single-person photo, and the second image will be referred to as a multi-person photo, and the example of splicing the portrait content in the single-person photo into the multi-person photo to generate a target image, i.e., a group photo, will be used for explanation.

[0041] like Figure 2 As shown, obtaining the first image may be that the electronic device displays a single-person photo upload interface 201, and the user may select a first image from the album and input it into the single-person photo upload interface 201 for uploading. Alternatively, the user may take any first image in the shooting preview interface and input it into the single-person photo upload interface 201 for uploading.

[0042] like Figure 3 As shown, obtaining the second image may be that the electronic device displays a multi-person photo upload interface 301 , and the user may select a second image from the album and input it into the multi-person photo upload interface 301 for uploading.

[0043] Step 102: extracting a first portrait segmentation region in the first image and a second portrait segmentation region in the second image.

[0044] In step 102, human body position information can be extracted from the image by using an image segmentation algorithm, an image segmentation model, etc., to obtain a portrait segmentation region for each person. The image can be a first image or a second image, the portrait segmentation region in the first image can be a first portrait segmentation region, and the portrait segmentation map in the second image can be a second portrait segmentation region. It can be understood that the number of the first portrait segmentation region and the second portrait segmentation region can be one or more.

[0045] Take the image as the first image as an example, Figure 4 As shown, a first portrait segmentation area 402 can be extracted from the first image 401 .

[0046] Step 103: determine the positional relationship and hierarchical relationship between the first portrait segmentation region and the second portrait segmentation region.

[0047] In step 103, the positional relationship between the first portrait segmentation area and the second portrait segmentation area can be determined based on the user's input. The positional relationship can include position coordinates, and can also include information such as scaling size and rotation angle. For example, the user drags the first portrait segmentation area to a certain position in the second image to obtain the position coordinates, the user scales the first portrait segmentation area through a two-finger operation to obtain the scaling size, and the user rotates the first portrait segmentation area by long pressing the first portrait segmentation area to obtain the rotation angle. For another example, the user can directly input parameters such as position coordinates, scaling size, and rotation angle in the setting interface, and determine the positional relationship between the first portrait segmentation area and the second portrait segmentation area based on the parameters.

[0048] The electronic device may also automatically match the corresponding positional relationship based on the feature information of the first portrait segmentation area and the second portrait segmentation area. For example, a target portrait segmentation area closest to the first portrait segmentation area may be determined from multiple second portrait segmentation areas, the first portrait segmentation area may be placed in an adjacent area of ​​the target portrait segmentation area, and the scaling size and rotation angle of the first portrait segmentation area may be adjusted according to the target portrait segmentation area to obtain the positional relationship between the first portrait segmentation area and the second portrait segmentation area.

[0049] The hierarchical relationship between the first portrait segmentation area and the second portrait segmentation area can be determined based on the user input, wherein the hierarchical relationship can be used to characterize whether the first portrait segmentation area is displayed before or after the second portrait segmentation area when it is spliced ​​into the second image.

[0050] Step 104 , displaying the image content of the first portrait segmentation area in the second image according to the position relationship and the hierarchical relationship, to generate a target image.

[0051] In step 104, the image content of the first portrait segmentation area can be spliced ​​to the position in the second image that matches the positional relationship, and the display effects of the first portrait segmentation area and the second portrait segmentation area are determined according to the hierarchical relationship to generate a target image.

[0052] In the embodiment of the present application, the first image and the second image can be acquired, and the first image and the second image can be automatically extracted to obtain the first portrait segmentation area in the first image and the second portrait segmentation area in the second image. The positional relationship and hierarchical relationship between the first portrait segmentation area and the second portrait segmentation area can be determined, and the image content of the first portrait segmentation area can be displayed in the second image according to the positional relationship and hierarchical relationship to generate a target image. There is no need for the user to manually edit and adjust the image content. The target image can be automatically generated according to the positional relationship and hierarchical relationship between the portrait segmentation areas in the first image and the second image, which simplifies the operation process of image generation and improves the efficiency of image generation.

[0053] In some embodiments, extracting a portrait segmentation region in an image may include:

[0054] Detect the face detection frame in the image through the face detector;

[0055] Detecting a human body detection frame in an image using a human body detector;

[0056] Determine the center point of the face according to the face detection frame, and determine the center point of the body according to the body detection frame;

[0057] Extract the portrait segmentation area in the image according to the human body detection frame, the center point of the face and the center point of the body;

[0058] Wherein, when the image is a first image, the portrait segmentation region is a first portrait segmentation region, and when the image is a second image, the portrait segmentation region is a second portrait segmentation region.

[0059] In this embodiment, a face detector, such as a facial detection model RetinaFace, can be used to detect all face regions and face key points in an image to obtain a face detection frame. The i-th face detection frame in the image can be recorded as box_face_i = [x1, y1, x2, y2], where (x1, y1) can represent the upper left corner coordinates of the face detection frame in the image, and (x2, y2) can represent the lower right corner coordinates of the face detection frame in the image.

[0060] A human body detector, such as the target detection algorithm YOLOv8, can be used to detect the human body region corresponding to each face and obtain a human body detection frame. The i-th human body detection frame in the image can be recorded as box_person_i = [x'1, y'1, x'2, y'2]. Among them, (x'1, y'1) can represent the coordinates of the upper left corner of the human body detection frame in the image, and (x'2, y'2) can represent the coordinates of the lower right corner of the human body detection frame in the image.

[0061] The center point of the face and the center point of the body can be calculated. The calculation formula of the center point of the face can be: face_center_i = [(x1+x2) / 2, (y1+y2) / 2]. The calculation formula of the center point of the body can be: person_center_i = [(x'1+x'2) / 2, (y'1+y'2) / 2].

[0062] The center point of the face and the center point of the body can be used as the center point of the segmentation model. The segmentation model can be a pre-trained model for image segmentation. For example, the segmentation algorithm can be used to segment the image. The segmentation algorithm will segment the area belonging to the human body according to the position of the human body detection frame, the center point of the human body and the center point of the face, and obtain the portrait segmentation area of ​​the i-th person, which is recorded as person_mask_i.

[0063] The above segmentation operation is performed on all human bodies in the first image and the second image to obtain a set of all portrait segmentation regions, which is recorded as mask_all = {person_mask_i, i∈all portraits}, where i = 0 represents the portrait in the first image. In other words, mask_all[0] can be the first portrait segmentation region, and mask_all[i=1:] can be the second portrait segmentation region.

[0064] In this way, the face detection frame and the body detection frame can be detected based on the face detector and the body detector, and the center point of the face and the center point of the body can be calculated based on the face detection frame and the body detection frame, and a more accurate portrait segmentation area can be extracted according to the center point of the face and the center point of the body, laying the foundation for the subsequent generation of group photos, so that the generated group photos have better effects.

[0065] In some embodiments, determining the positional relationship between the first portrait segmentation region and the second portrait segmentation region may include:

[0066] The first image is subjected to a two-dimensional convolution operation to obtain the first feature and the second feature.

[0067] acquiring a third feature of the second image;

[0068] Inputting the first feature, the second feature, and the third feature into an attention learning module in a self-attention mechanism feature block of a parameter estimation model; wherein the parameter estimation model includes N self-attention mechanism feature blocks and a fully connected layer, each self-attention mechanism feature block includes an attention learning module and a feedforward neural network module, and N is an integer greater than 1;

[0069] Using the attention learning module to process the first feature, the second feature, and the third feature, to obtain the features of the dimension of the transformation matrix;

[0070] A feedforward neural network module is used to perform nonlinear spatial mapping on the dimensional features of the transformation matrix to obtain the fourth feature;

[0071] The fourth feature is input into the fully connected layer to obtain the position coordinates, scaling size and rotation angle of the first portrait segmentation area in the second image; wherein the position relationship includes the position coordinates, scaling size and rotation angle.

[0072] In this embodiment, if Figure 5 and Figure 6 As shown, the parameter estimation model may include N self-attention mechanism feature blocks Transformer and a fully connected layer Linear, wherein each self-attention mechanism feature block Transformer includes an attention learning M-Attention module and a feed-forward neural network (FFN) module.

[0073] The first image can be subjected to a two-dimensional convolution operation (Conv2D) to obtain a first feature and a second feature, which are input into the M-Attention module. The second image can be subjected to a normalization operation (Norm) to obtain a third feature, which is input into the M-Attention module.

[0074] like Figure 6 As shown in the figure, the M-Attention module has three inputs, namely Q (query), K (Key), and V (Value). Among them, the first feature and the second feature can be used as Q and K, and the third feature can be used as V, where Q, K, and V are all feature vectors. The M-Attention module can process the first feature, the second feature, and the third feature, and then re-weight the third feature, so that the parameter estimation model can focus on more useful information, thereby improving the accuracy of the parameter estimation model.

[0075] The M-Attention module can include a Linear layer, a layernorm&Activation layer, a BMM layer, a Normalize layer, and a Reweight layer.

[0076] Among them, the Linear layer is a fully connected layer, which receives the input tensor and converts it into an output tensor. Its mathematical formula is y=x·AT+b, where x is the input, A is the weight matrix of Linear, AT is the transposed matrix of A, b is the bias matrix of Linear, and y is the output of Linear.

[0077] The Layernorm&Activation layer is a layer normalization and activation layer. The Layernorm&Activation layer is used to normalize the Nd dimension of the input [batch, Nd, dim]. For example, in the Nd dimension, the mean of each pixel is subtracted and then divided by the variance, and then the activated function such as the Relu activation function is used to process the processed data, such as Relu(x)=max(0,x).

[0078] The BMM layer is used to implement BMM operations between matrices. BMM operations are three-dimensional matrix multiplication operations. For example, A b×n1×n2 *B b×n2×n3 =C b×n1×n3 , which means that a new three-dimensional matrix C is obtained by multiplying a three-dimensional matrix A by a three-dimensional matrix B, where the third dimension of A and the second dimension of B must be the same.

[0079] The Normalize layer is used to normalize a three-dimensional matrix by row. For example, by normalizing the position of the i-th row and j-th column of the two-dimensional matrix A, we can get:

[0080] The Reweight layer is used to adjust the weight of the input feature matrix V according to a three-dimensional weight matrix W, which is the output of the Normalize layer. For example, the Reweight layer calls the BMM operation to implement the matrix multiplication between the weight matrix W and the feature matrix V.

[0081] On this basis, in the working process of the M-Attention module, such as Figure 6 As shown, for input Q, K, V, the output is The entire calculation process of the M-Attention module can include the following formulas (1) to (6):

[0082]

[0083] Among them, Q, K, and V are all eigenvectors. and is the feature vector after LayerNorm&Activation processing, is the feature vector after Linear processing, KT is the transposed vector of the eigenvector K, is the feature vector K T The transposed vector, W is a three-dimensional matrix, W b,i,j represents the matrix element of the jth row, ith column, and bth layer in the three-dimensional matrix W, is the feature vector W after LayerNorm&Activation processing b,i,j .

[0084] The M-Attention module can extract the features of the transformation matrix dimension [batch,Nd,dim], denoted as max(0,xW1+b1).

[0085] The FFN module can be used to perform nonlinear spatial mapping on the features of the transformation matrix dimensions to obtain the features extracted by the self-attention mechanism feature block, i.e., the fourth feature. The calculation formula can be shown as formula (7):

[0086] FFN(x)=max(0,xW1+b1)W2+b2 (7)

[0087] like Figure 5 As shown in the figure, after N self-attention mechanism feature blocks are extracted, for example, N can be 3, the fourth feature extracted by the three Transformers is input into the fully connected layer Linear, and the output is converted into a four-channel floating-point number using Linear to obtain the position coordinates (x, y), scaling size scale and rotation angle rotate of the first portrait segmentation area in the second image.

[0088] It is understandable that it is difficult to place the first portrait segmentation area at a suitable position in the second image, and it is necessary to provide suitable parameters such as position coordinates, scaling size and rotation angle of the first portrait segmentation area to ensure that a more natural and better photo is generated.

[0089] The embodiment of the present application uses a parameter estimation model to output parameters such as the position coordinates, scaling size, and rotation angle of the first portrait segmentation area based on image features in the first image and the second image, thereby obtaining a more accurate positional relationship between the first portrait segmentation area and the second portrait segmentation area, thereby generating a better group photo effect.

[0090] In some embodiments, determining the positional relationship between the first portrait segmentation region and the second portrait segmentation region may include:

[0091] Receiving a first input from a user, where the first input is a drag input on the first portrait segmentation area, or a parameter editing input on a setting interface;

[0092] In response to the first input, a positional relationship between the first portrait segmentation region and the second portrait segmentation region is determined.

[0093] In this embodiment, the positional relationship between the first portrait segmentation region and the second portrait segmentation region may also be determined based on user interaction.

[0094] In some examples, the user can directly drag the first portrait segmentation area to a certain position in the second image. When the user drags the first portrait segmentation area to a certain position in the second image, the user can also adjust the size and angle of the first portrait segmentation area. Based on the position, size and angle, the positional relationship between the first portrait segmentation area and the second portrait segmentation area can be determined.

[0095] In other examples, such as Figure 7 As shown, the electronic device may display a setting interface 701, which may include an x-coordinate control 702, a y-coordinate control 703, a zoom size control 704, etc. Based on the user's input to these controls, the corresponding parameters may be edited, and the positional relationship between the first portrait segmentation area and the second portrait segmentation area may be determined based on the edited parameters.

[0096] In this way, based on the user's interaction and in response to the received first input, the positional relationship between the first portrait segmentation area and the second portrait segmentation area can be determined so that the positional relationship better meets the user's personalized needs, thereby improving the interactivity and flexibility of image generation.

[0097] In some embodiments, determining the hierarchical relationship between the first portrait segmentation region and the second portrait segmentation region may include:

[0098] receiving a second input from the user;

[0099] In response to the second input, a hierarchical relationship between the first portrait segmentation region and the second portrait segmentation region is determined.

[0100] In this embodiment, if Figure 7 As shown, the setting interface 701 may also include a control 705 related to the hierarchical relationship. For example, the option of "Put behind others" may be displayed in the setting interface 701. If the user selects this option, the hierarchical relationship may be determined as the level of the first portrait segmentation area is located behind the level of the second portrait segmentation area, and the overlapped portion of the first portrait segmentation area and the second portrait segmentation area will be blocked. If the user does not select this option, the hierarchical relationship may be determined as the level of the first portrait segmentation area is located before the level of the second portrait segmentation area, and the first portrait segmentation area is completely displayed in the second image, and the overlapped portion of the second portrait segmentation area and the first portrait segmentation area will be blocked.

[0101] The user can also determine the hierarchical relationship between the first portrait segmentation area and the second portrait segmentation area by clicking the first portrait segmentation area and the second portrait segmentation area in sequence. The specific setting can be made according to actual needs and is not specifically limited here.

[0102] In this way, based on the user's interaction and in response to the received second input, the hierarchical relationship between the first portrait segmentation area and the second portrait segmentation area can be determined so that the hierarchical relationship can better meet the user's personalized needs and improve the interactivity and flexibility of image generation.

[0103] In some embodiments, displaying the image content of the first portrait segmentation area in the second image according to the position relationship and the hierarchical relationship to generate the target image may include:

[0104] Displaying the image content of the first portrait segmentation area in the second image according to the position relationship and the hierarchical relationship to generate a third image;

[0105] cropping a first region from the third image, where the first region is a region where the first portrait segmentation region is located;

[0106] The first region is edge corrected, and the edge-corrected first region and the third image are subjected to light and shadow reconstruction processing to generate a target image.

[0107] In this embodiment, the image content of the first portrait segmentation area can be directly displayed in the second image according to the position relationship and the layer relationship to generate the third image. At this time, since the first portrait segmentation area is suspended in the second image, there will be white edges at its edges, resulting in an unnatural photo effect of the third image.

[0108] Based on this, Figure 8 As shown, the first area 802 can be cropped from the third image 801. The first area 802 can be the area where the first portrait segmentation area is located. In other words, the first area 802 includes the image content of the first portrait segmentation area and may also include part of the content of the second image.

[0109] According to the content of part of the second image in the first area, edge correction can be performed on the edge of the first portrait segmentation area in the first area, for example, by filling, feathering, etc., to eliminate the white edges on the edges, and then the first area after edge correction and the third image are subjected to light and shadow reconstruction processing, so that the light, brightness, etc. of the image content of the first area and the rest of the image content in the third image are more coordinated, thereby generating a target image with a more natural effect.

[0110] In this way, the image generation effect can be improved through edge correction, light and shadow reconstruction and other processing, so as to obtain a more natural target image that meets the user's expectations.

[0111] In some embodiments, displaying the image content of the first portrait segmentation area in the second image according to the position relationship and the hierarchical relationship to generate the target image may include:

[0112] Determine the portrait display area of ​​the first portrait segmentation area in the second image according to the position relationship and the layer relationship;

[0113] Inputting the portrait segmentation map in the first image and the background area in the second image into the image generation model, and using the encoder in the image generation model to obtain a first vector corresponding to the portrait segmentation map and a second vector corresponding to the background area; wherein the portrait segmentation map is determined based on the first image and the first portrait segmentation area, and the background area is other areas of the second image except the portrait display area;

[0114] The first vector, the second vector, the third vector corresponding to the portrait display area, and any noise vector in the Gaussian noise are concatenated to obtain a first feature vector and input it into a U-shaped network in the image generation model;

[0115] The central area of ​​the human body is cropped from the portrait segmentation image and input into the contrastive language-image pre-trained CLIP encoder in the image generation model to obtain the second feature vector and input into the U-shaped network;

[0116] Use the U-shaped network to obtain the latent space vector;

[0117] The decoder in the image generation model is used to decode the latent space vector to obtain the target image.

[0118] In this embodiment, the portrait display area of ​​the first portrait segmentation area in the second image can be determined based on the positional relationship and the hierarchical relationship. It can be understood that the portrait display area can be the first portrait segmentation area in the generated group photo, which can be displayed in at least a partial area of ​​the second image. For example, if the hierarchical relationship of the first portrait segmentation area is located before the second portrait segmentation area, the entire area of ​​the first portrait segmentation area can be displayed in the second image at a position corresponding to the positional relationship. If the hierarchical relationship of the first portrait segmentation area is located after the first portrait segmentation area, only a partial area of ​​the first portrait segmentation area may be displayed in the second image at a position corresponding to the positional relationship, and another partial area may be blocked by the second portrait segmentation area.

[0119] A portrait segmentation map may be determined based on the first image and the first portrait segmentation region, wherein the portrait segmentation map=the first image*the first portrait segmentation region.

[0120] The second image may also be inverted in the portrait display area to obtain a background area, wherein the background area=the second image*(1-the portrait display area).

[0121] The portrait segmentation map in the first image and the background area in the second image can be input into the image generation model. Fig. 9 As shown, the image generation model 900 may include an encoder 901 , a Unet network (U-shaped network) 902 , a CLIP (Contrastive Language–Image Pre-training) encoder 903 and a decoder 904 .

[0122] For example, the portrait segmentation map may need to be subjected to edge corrosion and color enhancement to improve the robustness of the image generation model, and the enhanced portrait segmentation map is used as the input of the encoder 901. After passing through the encoder 901, such as the vae encoder, an output of the dimension of B*128*128*4 can be obtained, i.e., the first vector. After passing the background area through the same vae encoder, the output of the background area can be obtained, and its size is also B*128*128*4, i.e., the second vector. It can be understood that the B in the dimension is the number of images processed by the image generation model at one time.

[0123] The portrait display area can also be directly resized to a size of B*128*128*1 to obtain a third vector, and a noise vector of B*128*128*4 is randomly sampled from Gaussian noise. The first vector, the second vector, the third vector, and the noise vector are concatenated according to the last dimension, so as to obtain a first eigenvector of B*128*128*13, which is used as the photo condition input of the Unet network 902.

[0124] In order to make the Unet network 902 applicable to the generation of group photos, the portrait segmentation map is cropped with the center point of the human body, and a rectangular area with the human body as the center point and a width and height that are the maximum values ​​of the length and width of the human body detection frame is cropped, that is, the human body center area is used as the input of the CLIP encoder 903, and the second eigenvector output by the CLIP encoder 903 is used as the control condition input of the Unet network 902.

[0125] The Unet network 902 is the main network in the image generation model 900 responsible for generating portrait photos. Its function is to generate the final conditional output, i.e., the latent space vector, based on the input of the photo conditions and the control conditions, and decode the latent space vector into an image through the decoder 904 and then display it to the user.

[0126] like Fig.10As shown, the main structure of the Unet network 902 uses an M-attention network block that does not require normalization of rectangles (i.e., softmax operation), and concatenates the feature map of the first image and the feature map of the second image and directly sends them to the network to generate the group photo content. Resnet is a convolutional residual network, which is composed of Skip-Connect connection and convolution block. The first M-attention network block uses the output of Resnet, that is, the output obtained after the group photo condition input passes through Resnet, as the input of Q and K, and the second M-attention network block uses the features of the portrait segmentation map, that is, the output obtained after the control condition input passes through the CLIP encoder 903, as the input of Q and K. The DownSample or Upsample module is used to upsample or downsample the resolution of the feature map using convolution operations, and at the same time increase or decrease the number of channels.

[0127] After stacking the six main structures repeatedly, a latent space vector is obtained. After the latent space vector passes through the decoder, the group image generated by the model is obtained, that is, the model image. In some examples, the model image can be determined as the target image.

[0128] In this embodiment, an image generation model can be used to achieve more natural image generation based on the group photo condition input and the control condition input, which further simplifies the image generation operation and improves the image generation efficiency.

[0129] In some examples, for the image generation model, an image and standard sentence data corresponding to the image may be obtained, and training may be performed based on the image and the standard sentence data. The training scheme may include self-supervised training and training strategies corresponding to the diffusion model.

[0130] The self-supervised training can be to perform body segmentation on each person in the multi-person image to obtain the segmentation set mask_set. During the training process, one of the masks is randomly sampled as a single-person photo and the rest as multi-person photos. Then, an image pair of single-person photo and multi-person photo <single-person photo, multi-person photo> can be constructed and used as the input of the image generation network, thereby realizing data self-construction.

[0131] The training strategy of the diffusion model is to obtain the latent space vector Latent after vae encoding for any image, then randomly sample a time step from the time range 1-T, and then sample a noise vector Noise from Gaussian noise, add Noise to Latent, and use the Unet network to predict Noise, and obtain loss = mean (predict-Noise), so as to calculate the loss. Repeat the iteration until the model converges, and the training is completed.

[0132] In some embodiments, decoding the latent space vector using a decoder in the image generation model to obtain a target image may include:

[0133] Use the decoder in the image generation model to decode the latent space vector to obtain the model image;

[0134] The model image is fused with the second image to generate a target image.

[0135] In this embodiment, the decoder in the image generation model can be used to decode the latent space vector to obtain a model image, that is, a group photo image output by the model.

[0136] It is understandable that when the image generation model generates a group photo, it may need to crop the image into an image of a predetermined size for subsequent processing. However, in actual application scenarios, the image sizes taken by different electronic devices may be different, resulting in the group photo generated by the image generation model being based on the second image, and some content may be cropped.

[0137] In order for the finally generated group photo to fully display the user's desired content, the model image and the second image may be fused, that is, the model image may be pasted back to the corresponding area in the second image to generate a target image.

[0138] In this way, the model image and the second image can be fused to obtain a group photo image with better display effect and better meeting the user's expectations.

[0139] In some embodiments, fusing the model image with the second image to generate a target image may include:

[0140] cropping a second region from the model image, where the second region is the region where the portrait display region is located;

[0141] According to the second area and the second image, feathering processing is performed on the edge of the second area to generate a target image.

[0142] In this embodiment, the second area where the portrait display area is located can be cropped from the model image, and the edge of the second area can be feathered according to the second area and the second image.

[0143] In other words, only the second area can be replied to in the second image. In order to avoid the color difference problem caused by replying, the edge of the second area can be feathered. For example, the edge of the second area can be Gaussian blurred to make the edge softer as a whole, and the second area after edge blurring and the area at the corresponding position in the second image can be replaced according to the preset weight to obtain the edge area after feathering, so that the edge after replying is more natural.

[0144] In this way, edge feathering can be used to generate a more natural group photo image.

[0145] In some embodiments, determining the first portrait segmentation area in the portrait display area in the second image according to the position relationship and the hierarchical relationship may include:

[0146] When the hierarchical relationship indicates that the portrait display region and the second portrait segmentation region have no intersection, determining the first portrait segmentation region as the portrait display region according to the position relationship;

[0147] When the hierarchical relationship indicates that the portrait display area intersects with the M second portrait segmentation areas, the portrait display area is determined by an intersection-over-union (IOU) operation based on the first portrait segmentation area, the M second portrait segmentation areas, and the positional relationship between the first portrait segmentation area and the M second portrait segmentation areas, where M is a positive integer.

[0148] In this embodiment, if Figure 7 As shown, if the hierarchical relationship indicates that the portrait display area has no intersection with the second portrait segmentation area, that is, the level of the first portrait segmentation area is located before the level of the second portrait segmentation area, then the first portrait segmentation area can be directly placed at the corresponding position in the second image according to the positional relationship to form a portrait display area.

[0149] If the hierarchical relationship indicates that the portrait display area intersects with the M second portrait segmentation areas, that is, the level of the first portrait segmentation area is located after the level of the M second portrait segmentation areas, it is necessary to perform an IOU operation on the first portrait segmentation area and the M second portrait segmentation areas.

[0150] That is, mask_all[0] and mask_all[i=1:] in mask_all perform intersection over union (IOU) calculation to obtain all mask_all[j]_ that intersect with mask_all[0]. The calculation formula can be:

[0151] mask_all[j]=IOU(mask_all[0],mask_all[j])>0, for j in mask_all[1:].

[0152] The calculation formula of IOU is: IOU = (A∩B) / (A∪B). A∩B represents the intersection area of ​​two portrait segmentation areas, and A∪B represents the union area of ​​two portrait segmentation areas. The larger the IOU value, the more similar the two portrait segmentation areas are. Here, A and B can be replaced by the human body detection frame corresponding to the portrait segmentation area to calculate the minimum enclosing rectangle of the portrait segmentation area.

[0153] After obtaining mask_all[j], a difference operation is performed between mask_all[0] and each mask_all[j], that is, the area in mask_all[0] but not in any mask_all[j] is taken as the portrait display area.

[0154] In this way, the portrait display area of ​​the first portrait segmentation area displayed in the second image can be determined through IOU calculation, and the result is more accurate, which improves the accuracy of the input of the subsequent image generation model, and thus enables the image generation model to output a more accurate group photo image.

[0155] In a specific scenario embodiment, Fig.11 As shown, the image generation method may include the following steps:

[0156] Step 1101, receiving a first image and a second image uploaded by a user;

[0157] Step 1102, extracting a first portrait segmentation region in the first image and a second portrait segmentation region in the second image;

[0158] Step 1103, receiving an input from a user dragging the first portrait segmentation region to a suitable position in the second image, and determining a positional relationship between the first portrait segmentation region and the second portrait segmentation region;

[0159] Step 1104, receiving a user-specified level input, and determining a level relationship between the first portrait segmentation region and the second portrait segmentation region;

[0160] Step 1105, generating a model image through an image generation model;

[0161] Step 1106, fusing the model image with the second image to generate a target image;

[0162] Step 1107, determine whether the user continues editing, if not, execute step 1108, if yes, return to step 1103;

[0163] Step 1108, save the target image.

[0164] For example, a user uploads a first image with a size of 1080*1440. The system detects the position of the human body in the first image. The detected human body position can be expressed as person_center_i[x1,y1,x2,y2]=[795.0,561.0,970.0,1274.0]. The human body in the human body area in the first image is segmented using a segmentation network to obtain a first portrait segmentation area. The user uploads a second image, and the system performs the same steps to obtain a second portrait segmentation area for each person in the second image. The user adjusts the position and level of the first portrait segmentation area in the second image, and displays the image content of the first portrait segmentation area in the second image to obtain a portrait display area and a background area. A portrait segmentation map is obtained based on the first image and the first portrait segmentation area, thereby obtaining the input of the image generation model. Assume that the size of the portrait segmentation image is 1024*1024, and the size of the corresponding background area is 1024*1024. After passing through the encoder, the outputs are 1*128*128*4, where 1 is the number of images processed, 128 is the width and height of the latent space, and 4 is the number of latent space channels. That is, the first vector is VAE_single, whose size is 1*128*128*4, the second vector is VAE_background, whose size is 1*128*128*4, and the portrait display area is resized to a size of 1*128*128*1, recorded as the third vector mask_resized, and these three vectors are added with the randomly initialized noise vector Noise, whose size is 1*128*128*4. The channels are spliced ​​together to form the photo condition input in the Unet network, that is, the first feature vector Unet_input = concat ([Noise, VAE_single, mask_resized, VAE_background]), with a size of 1*128*128*13. Similarly, the control condition input of the Unet network is the second feature vector of 1*1024*1024 extracted from the portrait segmentation map by the CLIP encoder, denoted as clip_feature.

[0165] Clip_feature and Unet_input are used as the two inputs of the Unet network. The output of the Unet network is 1*128*128*4, which is the latent space vector of the content of the generated group photo. After the latent space vector is decoded by the decoder, the group photo image generated by the model is obtained, whose size is 1*1024*1024*3. The image is pasted back to the corresponding position in the second image to obtain the final target image.

[0166] The image generation method provided in the embodiment of the present application can be executed by an image generation device. In the embodiment of the present application, the image generation device provided in the embodiment of the present application is described by taking the image generation method executed by the image generation device as an example.

[0167] like Fig.12 As shown, the embodiment of the present application provides an image generating device 1200, and the image generating device 1200 may include:

[0168] An acquisition module 1201 is used to acquire a first image and a second image;

[0169] An extraction module 1202 is used to extract a first portrait segmentation region in the first image and a second portrait segmentation region in the second image;

[0170] A determination module 1203 is used to determine a positional relationship and a hierarchical relationship between the first portrait segmentation region and the second portrait segmentation region;

[0171] The generating module 1204 is used to display the image content of the first portrait segmentation area in the second image according to the position relationship and the layer relationship, so as to generate a target image.

[0172] In the embodiment of the present application, the first image and the second image can be acquired, and the first image and the second image can be automatically extracted to obtain the first portrait segmentation area in the first image and the second portrait segmentation area in the second image. The positional relationship and hierarchical relationship between the first portrait segmentation area and the second portrait segmentation area can be determined, and the image content of the first portrait segmentation area can be displayed in the second image according to the positional relationship and hierarchical relationship to generate a target image. There is no need for the user to manually edit and adjust the image content. The target image can be automatically generated according to the positional relationship and hierarchical relationship between the portrait segmentation areas in the first image and the second image, which simplifies the operation process of image generation and improves the efficiency of image generation.

[0173] In some embodiments, the extraction module 1202 may also be used to:

[0174] Detect the face detection frame in the image through the face detector;

[0175] Detecting a human body detection frame in an image using a human body detector;

[0176] Determine the center point of the face according to the face detection frame, and determine the center point of the body according to the body detection frame;

[0177] Extract the portrait segmentation area in the image according to the human body detection frame, the center point of the face and the center point of the body;

[0178] Wherein, when the image is a first image, the portrait segmentation region is a first portrait segmentation region, and when the image is a second image, the portrait segmentation region is a second portrait segmentation region.

[0179] In this way, the face detection frame and the body detection frame can be detected based on the face detector and the body detector, and the center point of the face and the center point of the body can be calculated based on the face detection frame and the body detection frame, and a more accurate portrait segmentation area can be extracted according to the center point of the face and the center point of the body, laying the foundation for the subsequent generation of group photos, so that the generated group photos have better effects.

[0180] In some embodiments, the determination module 1203 may also be used to:

[0181] The first image is subjected to a two-dimensional convolution operation to obtain the first feature and the second feature.

[0182] acquiring a third feature of the second image;

[0183] Inputting the first feature, the second feature, and the third feature into an attention learning module in a self-attention mechanism feature block of a parameter estimation model; wherein the parameter estimation model includes N self-attention mechanism feature blocks and a fully connected layer, each self-attention mechanism feature block includes an attention learning module and a feedforward neural network module, and N is an integer greater than 1;

[0184] Using the attention learning module to process the first feature, the second feature, and the third feature, to obtain the features of the dimension of the transformation matrix;

[0185] A feedforward neural network module is used to perform nonlinear spatial mapping on the dimensional features of the transformation matrix to obtain the fourth feature;

[0186] The fourth feature is input into the fully connected layer to obtain the position coordinates, scaling size and rotation angle of the first portrait segmentation area in the second image; wherein the position relationship includes the position coordinates, scaling size and rotation angle.

[0187] The embodiment of the present application uses a parameter estimation model to output parameters such as the position coordinates, scaling size, and rotation angle of the first portrait segmentation area based on image features in the first image and the second image, thereby obtaining a more accurate positional relationship between the first portrait segmentation area and the second portrait segmentation area, thereby generating a better group photo effect.

[0188] In some embodiments, the determination module 1203 may also be used to:

[0189] Receiving a first input from a user, where the first input is a drag input on the first portrait segmentation area, or a parameter editing input on a setting interface;

[0190] In response to the first input, a positional relationship between the first portrait segmentation region and the second portrait segmentation region is determined.

[0191] In this way, based on the user's interaction and in response to the received first input, the positional relationship between the first portrait segmentation area and the second portrait segmentation area can be determined so that the positional relationship better meets the user's personalized needs, thereby improving the interactivity and flexibility of image generation.

[0192] In some embodiments, the determination module 1203 may also be used to:

[0193] receiving a second input from the user;

[0194] In response to the second input, a hierarchical relationship between the first portrait segmentation region and the second portrait segmentation region is determined.

[0195] In this way, based on the user's interaction and in response to the received second input, the hierarchical relationship between the first portrait segmentation area and the second portrait segmentation area can be determined so that the hierarchical relationship can better meet the user's personalized needs and improve the interactivity and flexibility of image generation.

[0196] In some embodiments, the generating module 1204 may also be used to:

[0197] Displaying the image content of the first portrait segmentation area in the second image according to the position relationship and the hierarchical relationship to generate a third image;

[0198] cropping a first region from the third image, where the first region is a region where the first portrait segmentation region is located;

[0199] The first region is edge corrected, and the edge-corrected first region and the third image are subjected to light and shadow reconstruction processing to generate a target image.

[0200] In this way, the image generation effect can be improved through edge correction, light and shadow reconstruction and other processing, so as to obtain a more natural target image that meets the user's expectations.

[0201] In some embodiments, the generating module 1204 may also be used to:

[0202] Determine the portrait display area of ​​the first portrait segmentation area in the second image according to the position relationship and the layer relationship;

[0203] Inputting the portrait segmentation map in the first image and the background area in the second image into the image generation model, and using the encoder in the image generation model to obtain a first vector corresponding to the portrait segmentation map and a second vector corresponding to the background area; wherein the portrait segmentation map is determined based on the first image and the first portrait segmentation area, and the background area is other areas of the second image except the portrait display area;

[0204] The first vector, the second vector, the third vector corresponding to the portrait display area, and any noise vector in the Gaussian noise are concatenated to obtain a first feature vector and input it into a U-shaped network in the image generation model;

[0205] The central area of ​​the human body is cropped from the portrait segmentation image and input into the contrastive language-image pre-trained CLIP encoder in the image generation model to obtain the second feature vector and input into the U-shaped network;

[0206] Use the U-shaped network to obtain the latent space vector;

[0207] The decoder in the image generation model is used to decode the latent space vector to obtain the target image.

[0208] In this embodiment, an image generation model can be used to achieve more natural image generation based on the group photo condition input and the control condition input, which further simplifies the image generation operation and improves the image generation efficiency.

[0209] In some embodiments, the generating module 1204 may also be used to:

[0210] Use the decoder in the image generation model to decode the latent space vector to obtain the model image;

[0211] The model image is fused with the second image to generate a target image.

[0212] In this way, the model image and the second image can be fused to obtain a group photo image with better display effect and better meeting the user's expectations.

[0213] In some embodiments, the generating module 1204 may also be used to:

[0214] cropping a second region from the model image, where the second region is the region where the portrait display region is located;

[0215] According to the second area and the second image, feathering processing is performed on the edge of the second area to generate a target image.

[0216] In this way, edge feathering can be used to generate a more natural group photo image.

[0217] In some embodiments, the generating module 1204 may also be used to:

[0218] When the hierarchical relationship indicates that the portrait display region and the second portrait segmentation region have no intersection, determining the first portrait segmentation region as the portrait display region according to the position relationship;

[0219] When the hierarchical relationship indicates that the portrait display area intersects with the M second portrait segmentation areas, the portrait display area is determined by an intersection-over-union (IOU) operation based on the first portrait segmentation area, the M second portrait segmentation areas, and the positional relationship between the first portrait segmentation area and the M second portrait segmentation areas, where M is a positive integer.

[0220] In this way, the portrait display area of ​​the first portrait segmentation area displayed in the second image can be determined through IOU calculation, and the result is more accurate, which improves the accuracy of the input of the subsequent image generation model, and thus enables the image generation model to output a more accurate group photo image.

[0221] The image generating device in the embodiment of the present application can be a device or a component in the device, such as an integrated circuit or a chip. The device can be a terminal or other devices other than a terminal. Exemplarily, the device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc. It can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0222] The image generation device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0223] The image generation device provided in the embodiment of the present application can achieve Figures 1 to 11 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0224] Alternatively, if Fig.13As shown, an embodiment of the present application further provides an electronic device 1300, including a processor 1301 and a memory 1302, wherein the memory 1302 stores programs or instructions that can be executed on the processor 1301, and when the program or instructions are executed by the processor 1301, the various steps of the above-mentioned image generation method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0225] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0226] Fig.14 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0227] The electronic device 1400 includes but is not limited to: a radio frequency unit 1401, a network module 1402, an audio output unit 1403, an input unit 1404, a sensor 1405, a display unit 1406, a user input unit 1407, an interface unit 1408, a memory 1409, and a processor 1410 and other components.

[0228] Those skilled in the art will appreciate that the electronic device 1400 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 1410 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig. 9 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0229] The processor 1410 may be used for:

[0230] Acquire a first image and a second image;

[0231] Extracting a first portrait segmentation region in the first image and a second portrait segmentation region in the second image;

[0232] Determine the positional relationship and hierarchical relationship between the first portrait segmentation region and the second portrait segmentation region;

[0233] According to the position relationship and the hierarchical relationship, the image content of the first portrait segmentation area is displayed in the second image to generate a target image.

[0234] In the embodiment of the present application, the first image and the second image can be acquired, and the first image and the second image can be automatically extracted to obtain the first portrait segmentation area in the first image and the second portrait segmentation area in the second image. The positional relationship and hierarchical relationship between the first portrait segmentation area and the second portrait segmentation area can be determined, and the image content of the first portrait segmentation area can be displayed in the second image according to the positional relationship and hierarchical relationship to generate a target image. There is no need for the user to manually edit and adjust the image content. The target image can be automatically generated according to the positional relationship and hierarchical relationship between the portrait segmentation areas in the first image and the second image, which simplifies the operation process of image generation and improves the efficiency of image generation.

[0235] In some embodiments, the processor 1410 may also be configured to:

[0236] Detect the face detection frame in the image through the face detector;

[0237] Detecting a human body detection frame in an image using a human body detector;

[0238] Determine the center point of the face according to the face detection frame, and determine the center point of the body according to the body detection frame;

[0239] Extract the portrait segmentation area in the image according to the human body detection frame, the center point of the face and the center point of the body;

[0240] Wherein, when the image is a first image, the portrait segmentation region is a first portrait segmentation region, and when the image is a second image, the portrait segmentation region is a second portrait segmentation region.

[0241] In this way, the face detection frame and the body detection frame can be detected based on the face detector and the body detector, and the center point of the face and the center point of the body can be calculated based on the face detection frame and the body detection frame, and a more accurate portrait segmentation area can be extracted according to the center point of the face and the center point of the body, laying the foundation for the subsequent generation of group photos, so that the generated group photos have better effects.

[0242] In some embodiments, the processor 1410 may also be configured to:

[0243] The first image is subjected to a two-dimensional convolution operation to obtain the first feature and the second feature.

[0244] acquiring a third feature of the second image;

[0245] Inputting the first feature, the second feature, and the third feature into an attention learning module in a self-attention mechanism feature block of a parameter estimation model; wherein the parameter estimation model includes N self-attention mechanism feature blocks and a fully connected layer, each self-attention mechanism feature block includes an attention learning module and a feedforward neural network module, and N is an integer greater than 1;

[0246] Using the attention learning module to process the first feature, the second feature, and the third feature, to obtain the features of the dimension of the transformation matrix;

[0247] A feedforward neural network module is used to perform nonlinear spatial mapping on the dimensional features of the transformation matrix to obtain the fourth feature;

[0248] The fourth feature is input into the fully connected layer to obtain the position coordinates, scaling size and rotation angle of the first portrait segmentation area in the second image; wherein the position relationship includes the position coordinates, scaling size and rotation angle.

[0249] The embodiment of the present application uses a parameter estimation model to output parameters such as the position coordinates, scaling size, and rotation angle of the first portrait segmentation area based on image features in the first image and the second image, thereby obtaining a more accurate positional relationship between the first portrait segmentation area and the second portrait segmentation area, thereby generating a better group photo effect.

[0250] In some embodiments, the processor 1410 may also be configured to:

[0251] Receiving a first input from a user, where the first input is a drag input on the first portrait segmentation area, or a parameter editing input on a setting interface;

[0252] In response to the first input, a positional relationship between the first portrait segmentation region and the second portrait segmentation region is determined.

[0253] In this way, based on the user's interaction and in response to the received first input, the positional relationship between the first portrait segmentation area and the second portrait segmentation area can be determined so that the positional relationship better meets the user's personalized needs, thereby improving the interactivity and flexibility of image generation.

[0254] In some embodiments, the processor 1410 may also be configured to:

[0255] receiving a second input from the user;

[0256] In response to the second input, a hierarchical relationship between the first portrait segmentation region and the second portrait segmentation region is determined.

[0257] In this way, based on the user's interaction and in response to the received second input, the hierarchical relationship between the first portrait segmentation area and the second portrait segmentation area can be determined so that the hierarchical relationship can better meet the user's personalized needs and improve the interactivity and flexibility of image generation.

[0258] In some embodiments, the processor 1410 may also be configured to:

[0259] Displaying the image content of the first portrait segmentation area in the second image according to the position relationship and the hierarchical relationship to generate a third image;

[0260] A first area is cropped from the third image, where the first area is where the first portrait segmentation area is located; edge correction is performed on the first area, and light and shadow reconstruction is performed on the edge-corrected first area and the third image to generate a target image.

[0261] In this way, the image generation effect can be improved through edge correction, light and shadow reconstruction and other processing, so as to obtain a more natural target image that meets the user's expectations.

[0262] In some embodiments, the processor 1410 may also be configured to:

[0263] Determine the portrait display area of ​​the first portrait segmentation area in the second image according to the position relationship and the layer relationship;

[0264] Inputting the portrait segmentation map in the first image and the background area in the second image into the image generation model, and using the encoder in the image generation model to obtain a first vector corresponding to the portrait segmentation map and a second vector corresponding to the background area; wherein the portrait segmentation map is determined based on the first image and the first portrait segmentation area, and the background area is other areas of the second image except the portrait display area;

[0265] The first vector, the second vector, the third vector corresponding to the portrait display area, and any noise vector in the Gaussian noise are concatenated to obtain a first feature vector and input it into a U-shaped network in the image generation model;

[0266] The central area of ​​the human body is cropped from the portrait segmentation image and input into the contrastive language-image pre-trained CLIP encoder in the image generation model to obtain the second feature vector and input into the U-shaped network;

[0267] Use the U-shaped network to obtain the latent space vector;

[0268] The decoder in the image generation model is used to decode the latent space vector to obtain the target image.

[0269] In this embodiment, an image generation model can be used to achieve more natural image generation based on the group photo condition input and the control condition input, which further simplifies the image generation operation and improves the image generation efficiency.

[0270] In some embodiments, the processor 1410 may also be configured to:

[0271] Use the decoder in the image generation model to decode the latent space vector to obtain the model image;

[0272] The model image is fused with the second image to generate a target image.

[0273] In this way, the model image and the second image can be fused to obtain a group photo image with better display effect and better meeting the user's expectations.

[0274] In some embodiments, the processor 1410 may also be configured to:

[0275] cropping a second region from the model image, where the second region is the region where the portrait display region is located;

[0276] According to the second area and the second image, feathering processing is performed on the edge of the second area to generate a target image.

[0277] In this way, edge feathering can be used to generate a more natural group photo image.

[0278] In some embodiments, the processor 1410 may also be configured to:

[0279] When the hierarchical relationship indicates that the portrait display region and the second portrait segmentation region have no intersection, determining the first portrait segmentation region as the portrait display region according to the position relationship;

[0280] When the hierarchical relationship indicates that the portrait display area intersects with the M second portrait segmentation areas, the portrait display area is determined by an intersection-over-union (IOU) operation based on the first portrait segmentation area, the M second portrait segmentation areas, and the positional relationship between the first portrait segmentation area and the M second portrait segmentation areas, where M is a positive integer.

[0281] In this way, the portrait display area of ​​the first portrait segmentation area displayed in the second image can be determined through IOU calculation, and the result is more accurate, which improves the accuracy of the input of the subsequent image generation model, and thus enables the image generation model to output a more accurate group photo image.

[0282] It should be understood that in the embodiment of the present application, the input unit 1404 may include a graphics processor (Graphics Processing Unit, GPU) 14041 and a microphone 14042, and the graphics processor 14041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 1406 may include a display panel 14061, and the display panel 14061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1407 includes a touch panel 14071 and at least one of other input devices 14072. The touch panel 14071 is also called a touch screen. The touch panel 14071 may include two parts: a touch detection device and a touch controller. Other input devices 14072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0283] The memory 1409 can be used to store software programs and various data. The memory 1409 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 1409 may include a volatile memory or a non-volatile memory, or the memory 1409 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 1409 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0284] The processor 1410 may include one or more processing units; optionally, the processor 1410 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 1410.

[0285] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned image generation method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0286] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0287] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image generation method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0288] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0289] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned image generation method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0290] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0291] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0292] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An image generation method, characterized in that: The method comprises: Acquire a first image and a second image; Extracting a first portrait segmentation region in the first image and a second portrait segmentation region in the second image; Determining a positional relationship and a hierarchical relationship between the first portrait segmentation region and the second portrait segmentation region; According to the positional relationship and the hierarchical relationship, the image content of the first portrait segmentation area is displayed in the second image to generate a target image.

2. The method according to claim 1, characterized in that The step of displaying the image content of the first portrait segmentation area in the second image according to the position relationship and the hierarchical relationship to generate a target image includes: According to the positional relationship and the hierarchical relationship, displaying the image content of the first portrait segmentation area in the second image to generate a third image; Cropping a first area from the third image, where the first area is the area where the first portrait segmentation area is located; The first region is edge corrected, and light and shadow reconstruction processing is performed on the edge-corrected first region and the third image to generate a target image.

3. The method according to claim 1, characterized in that The step of displaying the image content of the first portrait segmentation area in the second image according to the position relationship and the hierarchical relationship to generate a target image includes: Determining a portrait display region of the first portrait segmentation region in the second image according to the position relationship and the hierarchical relationship; Inputting the portrait segmentation map in the first image and the background area in the second image into an image generation model, and using an encoder in the image generation model to obtain a first vector corresponding to the portrait segmentation map and a second vector corresponding to the background area; wherein the portrait segmentation map is determined based on the first image and the first portrait segmentation area, and the background area is other areas of the second image except the portrait display area; splicing the first vector, the second vector, the third vector corresponding to the portrait display area, and any noise vector in the Gaussian noise to obtain a first feature vector and input it into the U-shaped network in the image generation model; The central area of ​​the human body is cropped from the portrait segmentation image and input into the contrastive language-image pre-trained CLIP encoder in the image generation model to obtain a second feature vector and input into the U-shaped network; Obtaining a latent space vector using the U-shaped network; The latent space vector is decoded using a decoder in the image generation model to obtain a target image.

4. The method according to claim 3, characterized in that The method of decoding the latent space vector by using the decoder in the image generation model to obtain a target image includes: Decoding the latent space vector using a decoder in the image generation model to obtain a model image; The model image is fused with the second image to generate a target image.

5. The method according to claim 4, characterized in that The step of fusing the model image with the second image to generate a target image includes: Cutting out a second area from the model image, where the second area is the area where the portrait display area is located; According to the second area and the second image, feathering processing is performed on the edge of the second area to generate a target image.

6. The method according to claim 3, characterized in that The determining, according to the position relationship and the hierarchical relationship, a portrait display region of the first portrait segmentation region in the second image includes: When the hierarchical relationship indicates that the portrait display area and the second portrait segmentation area have no intersection, determining the first portrait segmentation area as the portrait display area according to the positional relationship; When the hierarchical relationship indicates that the portrait display area has an intersection with M second portrait segmentation areas, the portrait display area is determined by an intersection-over-union (IOU) operation based on the first portrait segmentation area, the M second portrait segmentation areas, and the positional relationship between the first portrait segmentation area and the M second portrait segmentation areas, where M is a positive integer.

7. The method according to claim 1, characterized in that The determining the positional relationship between the first portrait segmentation area and the second portrait segmentation area includes: The first image is subjected to a two-dimensional convolution operation to obtain a first feature and a second feature, acquiring a third feature of the second image; Inputting the first feature, the second feature and the third feature into an attention learning module in a self-attention mechanism feature block of a parameter estimation model; wherein the parameter estimation model comprises N self-attention mechanism feature blocks and a fully connected layer, each self-attention mechanism feature block comprises the attention learning module and a feedforward neural network module, and N is an integer greater than 1; Using the attention learning module to process the first feature, the second feature, and the third feature to obtain features of the dimensions of the transformation matrix; Using the feedforward neural network module to perform nonlinear spatial mapping on the features of the dimension of the transformation matrix to obtain a fourth feature; The fourth feature is input into the fully connected layer to obtain the position coordinates, scaling size and rotation angle of the first portrait segmentation area in the second image; wherein the position relationship includes the position coordinates, the scaling size and the rotation angle.

8. The method according to claim 1, characterized in that Determining a positional relationship between the first portrait segmentation region and the second portrait segmentation region includes: receiving a first input from a user, where the first input is a drag input on the first portrait segmentation area, or a parameter editing input on a setting interface; In response to the first input, a positional relationship between the first portrait segmentation region and the second portrait segmentation region is determined.

9. The method according to claim 1, characterized in that: Extract the portrait segmentation area in the image, including: Detect the face detection frame in the image through the face detector; Detecting a human body detection frame in an image using a human body detector; Determine the center point of the face according to the face detection frame, and determine the center point of the human body according to the human body detection frame; Extracting a portrait segmentation area in the image according to the human body detection frame, the center point of the human face and the center point of the human body; Wherein, when the image is a first image, the portrait segmentation region is a first portrait segmentation region, and when the image is a second image, the portrait segmentation region is a second portrait segmentation region.

10. An image generating device, characterized in that: The device comprises: An acquisition module, used for acquiring a first image and a second image; An extraction module, used for extracting a first portrait segmentation region in the first image and a second portrait segmentation region in the second image; A determination module, used to determine the positional relationship and hierarchical relationship between the first portrait segmentation region and the second portrait segmentation region; A generating module is used to display the image content of the first portrait segmentation area in the second image according to the position relationship and the hierarchical relationship to generate a target image.

11. The device according to claim 10, characterized in that The generating module is further used for: According to the positional relationship and the hierarchical relationship, displaying the image content of the first portrait segmentation area in the second image to generate a third image; Cropping a first area from the third image, where the first area is the area where the first portrait segmentation area is located; The first region is edge corrected, and light and shadow reconstruction processing is performed on the edge-corrected first region and the third image to generate a target image.

12. The device according to claim 10, characterized in that The generating module is further used for: Determining a portrait display region of the first portrait segmentation region in the second image according to the position relationship and the hierarchical relationship; Inputting the portrait segmentation map in the first image and the background area in the second image into an image generation model, and using an encoder in the image generation model to obtain a first vector corresponding to the portrait segmentation map and a second vector corresponding to the background area; wherein the portrait segmentation map is determined based on the first image and the first portrait segmentation area, and the background area is other areas of the second image except the portrait display area; splicing the first vector, the second vector, the third vector corresponding to the portrait display area, and any noise vector in the Gaussian noise to obtain a first feature vector and input it into the U-shaped network in the image generation model; The central area of ​​the human body is cropped from the portrait segmentation image and input into the contrastive language-image pre-trained CLIP encoder in the image generation model to obtain a second feature vector and input into the U-shaped network; Obtaining a latent space vector using the U-shaped network; The latent space vector is decoded using a decoder in the image generation model to obtain a target image.

13. The device according to claim 12, characterized in that The generating module is further used for: When the hierarchical relationship indicates that the portrait display area and the second portrait segmentation area have no intersection, determining the first portrait segmentation area as the portrait display area according to the positional relationship; When the hierarchical relationship indicates that the portrait display area has an intersection with M second portrait segmentation areas, the portrait display area is determined by an intersection-over-union (IOU) operation based on the first portrait segmentation area, the M second portrait segmentation areas, and the positional relationship between the first portrait segmentation area and the M second portrait segmentation areas, where M is a positive integer.

14. The device according to claim 10, characterized in that The determining module is further used for: The first image is subjected to a two-dimensional convolution operation to obtain a first feature and a second feature, acquiring a third feature of the second image; Inputting the first feature, the second feature and the third feature into an attention learning module in a self-attention mechanism feature block of a parameter estimation model; wherein the parameter estimation model comprises N self-attention mechanism feature blocks and a fully connected layer, each self-attention mechanism feature block comprises the attention learning module and a feedforward neural network module, and N is an integer greater than 1; Using the attention learning module to process the first feature, the second feature and the third feature to obtain features of the dimensions of the transformation matrix; Using the feedforward neural network module, nonlinear spatial mapping is performed on the features of the dimension of the transformation matrix to obtain a fourth feature; The fourth feature is input into the fully connected layer to obtain the position coordinates, scaling size and rotation angle of the first portrait segmentation area in the second image; wherein the position relationship includes the position coordinates, the scaling size and the rotation angle.

15. The device according to claim 10, characterized in that The determining module is further used for: receiving a first input from a user, where the first input is a drag input on the first portrait segmentation area, or a parameter editing input on a setting interface; In response to the first input, a positional relationship between the first portrait segmentation region and the second portrait segmentation region is determined.

16. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the method according to any one of claims 1 to 9 are implemented.

17. A readable storage medium, characterized in that: The readable storage medium stores programs or instructions, and when the programs or instructions are executed by the processor, the steps of the method according to claims 1-9 are implemented.

18. A computer program product, the program product being stored in a storage medium, the program product being executed by at least one processor to implement the steps of the method according to claims 1 to 9.

Images