Image synthesis method, device, equipment and storage medium

Abstract

The present application provides an image synthesis method, apparatus, device, and storage medium. The method includes: obtaining an outline image of a target object to be synthesized and a background image; first, determining the size range of the target object in the background image based on the background image; and then constructing multiple candidate synthetic images based on the size range. The multiple candidate synthetic images include synthetic images of target objects of one or more sizes at different positions in the background image. Subsequently, a target synthetic image with a composition rationality coefficient greater than or equal to a threshold is determined from the multiple candidate synthetic images, and then the target synthetic image is post-processed to obtain a synthetic image that meets the image requirements. Based on the above processing process, the efficiency and effect of image synthesis can be improved, and the generated synthetic image is more natural and realistic.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an image synthesis method, apparatus, device and storage medium. Background Art

[0002] Image synthesis involves cutting out the foreground of one image and synthesizing it into another. For example, extracting a person from one image and synthesizing that person into another. In related art, after obtaining the foreground image, professional image makers typically use image processing software to adjust the foreground and background images at the pixel level. This method is labor-intensive and inefficient. Summary of the Invention

[0003] The embodiments of the present application provide an image synthesis method, apparatus, device, and storage medium to improve the efficiency and effect of image synthesis.

[0004] A first aspect of the embodiments of the present application provides an image synthesis method, including:

[0005] Obtaining a target object outline image and a background image to be synthesized;

[0006] Determining a size range of the target object in the background image according to the background image;

[0007] constructing a plurality of candidate composite images, wherein the target objects in the plurality of candidate composite images meet the size range and are located at different positions of the background image;

[0008] Determining a target composite image whose composition rationality coefficient is greater than or equal to a threshold from the plurality of candidate composite images;

[0009] Perform image post-processing on the target composite image to obtain a composite image that meets image requirements.

[0010] In an optional embodiment of the first aspect of the present application, the plurality of candidate composite images include target object outline images of different sizes; and constructing the plurality of candidate composite images includes:

[0011] The target object contour image corresponding to the target size is synthesized at different positions of the background image by window sliding to obtain the multiple candidate synthesized images; the target size includes multiple different sizes within the size range, and each candidate synthesized image includes a mask image of the target object.

[0012] In an optional embodiment of the first aspect of the present application, determining a target composite image having a reasonable coefficient greater than or equal to a threshold from the multiple candidate composite images includes:

[0013] Inputting the first candidate composite image into a preset composition evaluation model to obtain a composition rationality coefficient corresponding to the first candidate composite image, wherein the composition rationality coefficient is used to indicate the rationality of the size and position of the target object in the composite image;

[0014] If the composition rationality coefficient corresponding to the first candidate composite image is greater than or equal to a threshold, taking the first candidate composite image as the target composite image;

[0015] The first candidate composite image is any one of the multiple candidate composite images, and the composition evaluation model is obtained by training a convolutional neural network.

[0016] In an optional embodiment of the first aspect of the present application, the training process of the composition evaluation model includes:

[0017] Obtaining a first training sample set, the first training sample set including a plurality of first sample images and a first label value for each first sample image, the first sample image including a composite image containing a target object, the first label value being used to indicate a reasonableness of a size and a position of the target object in the first sample image;

[0018] inputting the plurality of first sample images into the composition evaluation model in sequence to obtain a first prediction value corresponding to each first sample image;

[0019] Determine the loss function of the composition evaluation model based on the first prediction value and the first label value corresponding to each first sample image. When the loss function of the composition evaluation model converges, stop training the composition evaluation model and obtain the model parameters of the composition evaluation model.

[0020] In an optional embodiment of the first aspect of the present application, determining a target composite image having a composition rationality coefficient greater than or equal to a threshold from the multiple candidate composite images includes:

[0021] If there are multiple composite images with a composition rationality coefficient greater than or equal to the threshold, the composite image with the largest composition rationality coefficient is used as the target composite image.

[0022] In an optional embodiment of the first aspect of the present application, performing image post-processing on the target composite image to obtain a composite image that meets image requirements includes:

[0023] Inputting the target composite image into an encoder of a preset image fusion model to extract high-dimensional image features of the target composite image;

[0024] The high-dimensional image features are input into the decoder of the image fusion model. In the decoder, an attention mechanism is used to perform image restoration on the high-dimensional image features to obtain a composite image that meets image requirements; meeting the image requirements includes meeting the image feature requirements of color, lighting, and brightness and darkness of the target object in the background image.

[0025] In an optional embodiment of the first aspect of the present application, the image fusion model further includes a discriminator, and the training process of the image fusion model includes:

[0026] Obtaining a second training sample set, where the second training sample set includes a plurality of second sample images and a second label value for each second sample image, the second sample images including synthetic images and / or real images containing a target object, and the second label value is used to indicate whether the target object in the second sample image is real;

[0027] Using the plurality of second sample images as inputs of the image fusion model, and sequentially passing through an encoder, a decoder, and a discriminator of the image fusion model to obtain a second prediction value corresponding to each second sample image;

[0028] According to the second prediction value and the second label value corresponding to each second sample image, the loss function of the image fusion model is determined. When the loss function of the image fusion model converges, the training of the image fusion model is stopped, and the relevant parameters of the encoder and decoder in the image fusion model are obtained.

[0029] In an optional embodiment of the first aspect of the present application, determining a size range of the target object in the background image according to the background image includes:

[0030] Acquire point cloud data of a reference object in the background image, and determine scale information of the background image according to the point cloud data of the reference object;

[0031] The size range of the target object in the background image is determined according to the scale information.

[0032] In an optional embodiment of the first aspect of the present application, determining a size range of the target object in the background image according to the background image includes:

[0033] Determining scale information of the background image according to size information of the reference object in the background image;

[0034] The size range of the target object in the background image is determined according to the scale information.

[0035] A second aspect of the embodiments of the present application provides an image synthesis method, including:

[0036] In response to a first operation performed on the first interface, obtaining a character image and a home decoration design image uploaded by a user;

[0037] In response to a second operation performed on the first interface, a person outline image is obtained from the person image; a size range of the person in the home decoration design drawing is determined based on the home decoration design drawing; a target composite image having a composition rationality coefficient greater than or equal to a threshold is determined from the constructed multiple candidate composite images, and image post-processing is performed on the target composite image to obtain a home decoration design drawing containing the person that meets image requirements; the people in the multiple candidate composite images are within the size range and are located at different positions in the home decoration design drawing;

[0038] Displaying a home design that includes the character.

[0039] A third aspect of the embodiments of the present application provides an image synthesis device, including:

[0040] An acquisition module is used to obtain a contour image of a target object and a background image to be synthesized;

[0041] A processing module, configured to determine a size range of a target object in the background image according to the background image;

[0042] constructing a plurality of candidate composite images, wherein the target objects in the plurality of candidate composite images meet the size range and are located at different positions of the background image;

[0043] Determining a target composite image whose composition rationality coefficient is greater than or equal to a threshold from the plurality of candidate composite images;

[0044] Perform image post-processing on the target composite image to obtain a composite image that meets image requirements.

[0045] A fourth aspect of the embodiments of the present application provides an image synthesis device, including:

[0046] an acquisition module, configured to respond to a first operation on the first interface and acquire a character image and a home decoration design image uploaded by a user;

[0047] a processing module configured to, in response to a second operation performed on the first interface, obtain a person outline from the person image; determine a size range of the person within the home decoration design drawing based on the home decoration design drawing; determine a target composite image having a composition rationality coefficient greater than or equal to a threshold value from the constructed multiple candidate composite images, and perform image post-processing on the target composite image to obtain a home decoration design drawing containing the person that meets image requirements; the people in the multiple candidate composite images conform to the size range and are located at different positions in the home decoration design drawing;

[0048] The display module is used to display a home decoration design drawing including the character.

[0049] The fifth aspect of an embodiment of the present application provides an electronic device, comprising: a memory, a processor, and a computer program; the computer program is stored in the memory and is configured to be executed by the processor to implement the method as described in any one of the first aspects of the present application, or the method as described in the second aspect of the present application.

[0050] A sixth aspect of an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the method as described in any one of the first aspects of the present application, or the method as described in the second aspect of the present application.

[0051] A seventh aspect of an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the method described in any one of the first aspects of the present application, or the method described in the second aspect of the present application.

[0052] The embodiments of the present application provide an image synthesis method, apparatus, device, and storage medium. The method includes: obtaining an outline image of a target object to be synthesized and a background image, first determining the size range of the target object in the background image based on the background image, and then constructing multiple candidate synthetic images based on the size range. The multiple candidate synthetic images include synthetic images of target objects of one or more sizes at different positions in the background image. Subsequently, a target synthetic image with a reasonable composition coefficient greater than or equal to a threshold is determined from the multiple candidate synthetic images, and then the target synthetic image is post-processed to obtain a synthetic image that meets the image requirements. Based on the above processing process, the efficiency and effect of image synthesis can be improved, and the generated synthetic image is more natural and realistic. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0054] Figure 1 A schematic diagram of an application scenario of the image synthesis method provided in an embodiment of the present application;

[0055] Figure 2 Schematic diagram of the process of the image synthesis method provided in the embodiment of the present application Figure 1 ;

[0056] Figure 3 A schematic diagram of constructing a candidate composite graph provided in an embodiment of the present application;

[0057] Figure 4 Schematic diagram of the process of the image synthesis method provided in the embodiment of the present application Figure 2 ;

[0058] Figure 5 A schematic diagram of a training method for a composition evaluation model provided in an embodiment of the present application;

[0059] Figure 6 A schematic diagram of a training method for an image fusion model provided in an embodiment of the present application;

[0060] Figure 7 Schematic diagram of the process of the image synthesis method provided in the embodiment of the present application Figure 3 ;

[0061] Figures 8a to 8b A schematic diagram of interface changes provided in an embodiment of the present application;

[0062] Figure 9 A diagram showing the processing process and effects of the image synthesis method provided in an embodiment of the present application;

[0063] Figure 10 Schematic diagram of the structure of the image synthesis device provided in the embodiment of the present application Figure 1 ;

[0064] Figure 11 Schematic diagram of the structure of the image synthesis device provided in the embodiment of the present application Figure 2 ;

[0065] Figure 12 This is a hardware structure diagram of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0066] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0067] The terms "first," "second," and the like in the description, claims, and accompanying drawings of the embodiments of the present application are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, such that the embodiments of the present application described herein can be practiced in an order other than that illustrated or described herein.

[0068] It should be understood that the terms "include" and "have" and any variations thereof as used herein are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus that includes a series of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, product or apparatus.

[0069] In the description of the embodiments of the present application, the term "corresponding" may indicate a direct or indirect correspondence between the two, or an association relationship between the two, or a relationship between indication and being indicated, configuration and being configured, etc.

[0070] The technical solution provided in the embodiments of the present application can be applied to any image synthesis scenario, such as synthesizing character materials into landscape images, synthesizing furniture materials into indoor scene images taken by users, and synthesizing character materials into interior design images.

[0071] For example, interior design image synthesis scenarios are often presented as spatial renderings. Adding human figures to these renderings can help users better understand the design scene, enhance the realism of the interior design, and better showcase the interior design and rendering effects. For example, a family of three can be shown in a living room, a child in a nursery, or a user's photo can be added to a rendering of an interior design scene.

[0072] Currently, after obtaining the stock images of people to be added, professional illustrators typically use image processing software to perform pixel-level adjustments on the stock images and interior scene images. This method is labor-intensive and inefficient. With the advancement of image processing technology, rule-based image synthesis algorithms can optimize the synthesized image. These algorithms adjust the pixel parameters of the spliced image parts according to preset pixel parameter adjustment rules. However, for interior home design drawings with complex scene content, these algorithms cannot achieve the desired results, resulting in poor visual effects, a noticeable sense of synthesis, and a lack of realism.

[0073] In response to the above problems, an embodiment of the present application proposes an image synthesis method, the main inventive idea of which is as follows: first, identify the foreground target to be synthesized, determine the size range of the foreground target in the background image to be synthesized, and use a traversal strategy and composition evaluation to generate multiple candidate composite images that meet the composition requirements. Then, by adjusting the image features of each candidate composite image, a composite image with the best synthesis effect is determined from the multiple adjusted candidate composite images, and the composite image is used as the final composite image and displayed to the user. In the above scheme, the intelligent generation of the target composite image is achieved by presetting the traversal strategy and composition evaluation. The position and size of the foreground target in the target composite image are optimal. Then, by post-processing the image of the target composite image, the foreground target and the background image are merged, and the final composite image is more natural.

[0074] The following first briefly introduces the application scenarios of the technical solutions provided in the embodiments of the present application.

[0075] Figure 1 Schematic diagram of the application scenario of the image synthesis method provided in the embodiment of the present application. Figure 1 As shown, the application scenario of this embodiment includes a client 101 and a server 102, which are in communication with each other. A user uploads at least two images to be synthesized through the client 101, and the server 102 synthesizes the at least two images and sends the synthesized image that meets the fusion effect requirements to the client 101.

[0076] For example, taking two pictures to be synthesized as an example, the server 102 receives a first picture and a second picture. The first picture contains the target object to be synthesized, which can be a person or any object. The second picture can be a picture containing any information, that is, the second picture is regarded as a background picture. The server 102 first obtains the image block corresponding to the target object from the first picture, that is, obtains the outline of the target object, and then synthesizes the outline of the target object into the second picture with a reasonable size and position. In order to better integrate the target object into the second picture, the synthesized image can also be post-processed to further improve the synthesis effect and obtain a more natural synthesized image.

[0077] In one image synthesis scenario, the first image is a person / animal stock image, and the second image is a landscape image. The user uploads the person / animal stock image and the landscape image to be synthesized through the client. The server first obtains the person / animal outline from the person / animal stock image and then synthesizes the person / animal outline into the landscape image at a reasonable size and position.

[0078] In one image synthesis scenario, the first image is a still life image, which includes but is not limited to various types of furniture, ornaments, and supplies, and the second image is an indoor scene image, which includes but is not limited to home and office scenes. The user uploads the still life image and indoor scene image to be synthesized through the client. The server first obtains the still life outline image from the still life image, and then synthesizes the still life outline image into the indoor scene image at a reasonable size and position.

[0079] Optionally, the first image can be a real photo or a model / design drawing. The second image can be a real photo or a model / design drawing. For example, if the first image is a real photo of a person and the second image is an interior design drawing, the person in the real photo is composited into the interior design drawing. For another example, if the first image is a furniture design drawing and the second image is a real photo of a living room, the furniture in the furniture design drawing is composited into the real photo of the living room.

[0080] The above-mentioned image synthesis scenario is only an example, and the embodiments of this application do not limit the image synthesis scenario. The technical solution provided by the embodiments of this application is also applicable to the scenario of synthesizing different target objects in multiple pictures into the same picture, which only requires target recognition and synthesis of multiple pictures in sequence.

[0081] In the embodiments of the present application, the execution device of the image synthesis method is not limited. Optionally, the image synthesis method can realize the overall image synthesis function with the help of a cloud computing system. For example, the image synthesis method can be applied to a cloud server so as to run various neural network models that generate synthetic images by taking advantage of the resources on the cloud. In addition to being applied to the cloud, the image synthesis method can also be applied to server-side devices such as conventional servers or server arrays. Optionally, the image synthesis method can also be applied to the client. For example, the client performs image synthesis by remotely calling the execution algorithm of the server, or the client performs image synthesis by pre-setting the execution algorithm locally.

[0082] Based on the above application scenarios, the technical solutions provided by the embodiments of the present application are described in detail below through specific embodiments. It should be noted that the technical solutions provided by the embodiments of the present application may include part or all of the following contents, and the following specific embodiments may be combined with each other. For the same or similar concepts or processes, some embodiments may not be repeated.

[0083] Figure 2 Schematic diagram of the process of the image synthesis method provided in the embodiment of the present application Figure 1 The image synthesis method of this embodiment can be applied to any image synthesis scenario, and the execution subject of the image synthesis method can be Figure 1 The client 101 or the server 102 shown. Figure 2 As shown, the image synthesis method of this embodiment includes the following steps:

[0084] Step 201: Obtain a target object outline image and a background image to be synthesized.

[0085] In this embodiment, the target object outline includes a person's outline and / or an arbitrary object's outline. The target object outline can be obtained from a real-life image of the target object or from a model design drawing of the target object, and this embodiment imposes no limitations on this. The background image can be a real-life image or a design drawing.

[0086] For example, to synthesize a person into a landscape image, the target object is the person and the background image is the landscape. Another example is to synthesize furniture into an interior design image, the target object is the furniture and the background image is the interior design. Another example is to synthesize a vehicle into a traffic map, the target object is the vehicle and the background image is the traffic map.

[0087] Step 202: Determine the size range of the target object in the background image according to the background image.

[0088] In this embodiment, the size range of the target object in the background image refers to the pixel size range of the target object in the background image. For example, the vertical pixel value range of the target person in the background image is determined. For another example, the vertical and / or horizontal pixel value range of the target vehicle in the background image is determined.

[0089] In one optional embodiment, by acquiring point cloud data of a reference object in the background image, the scale of the background image is first determined based on the point cloud data of the reference object, and then the size range of the target object within the background image is determined based on the scale information. In this embodiment, the background image is typically a real-life image, i.e., an image captured using a depth camera. The scale information of the image can be determined by analyzing the point cloud data of the reference object in the image. The reference object can be any object, such as a window or the intersection of a wall and the floor / ceiling in a real-life living room image, or a small house or tree in a landscape image.

[0090] In one optional embodiment, by obtaining the size information of a reference object in the background image, the scale information of the background image is first determined based on the size information of the reference object, and then the size range of the target object within the background image is determined based on the scale information. In this embodiment, the background image is typically a design drawing, and the scale information of the image can be determined by obtaining the size information of the reference object in the design drawing. For example, if a wardrobe is pre-installed in a home decoration design drawing, the scale information of the home decoration design drawing can be determined based on the size information of the wardrobe.

[0091] Step 203: construct multiple candidate composite images, where the target objects in the multiple candidate composite images meet the size range and are located at different positions of the background image.

[0092] In this embodiment, the plurality of candidate composite images include composite images of target object outline images of one or more sizes at different positions of the background image. It should be noted that the one or more sizes refer to the sizes of the target object determined in step 202 within the size range of the background image.

[0093] It should be understood that the size range of the target object in the background image will not be too large. If the target object is too large or too small, it will lead to unrealistic image synthesis. Therefore, when constructing a candidate synthetic image, it is necessary to determine a reasonable size range of the target object in the real scene image based on the physical size of the target object in the real scene image and the scale information of the real scene image.

[0094] In an optional embodiment, the target object outline corresponding to the target size is synthesized at different positions of the background image by sliding a window to obtain multiple candidate synthesized images. Optionally, the target size includes multiple different sizes of the target object within the size range of the background image.

[0095] For ease of understanding, the following Figure 3 The construction process of multiple candidate composite images is graphically displayed.

[0096] Figure 3 This is a schematic diagram of constructing a candidate composite graph provided in an embodiment of the present application. Figure 3 As shown, the first picture contains the target object. First, the image block of the target object, i.e., the target object contour map, is extracted through an image recognition algorithm. Then, the size range of the target object in the second picture to be synthesized is determined based on the scale information of the second picture. For example, the size of the target object in the second picture is determined to include size 1, size 2, and size 3. Then, for target object contour maps of different sizes, candidate synthesized images of target object contour maps of different sizes at different positions in the second picture are obtained by sliding a window. Figure 3 Taking the target object outline of size 1 in the second image as an example, starting from the upper right corner of the second image, from left to right and from top to bottom, obtain candidate composite images of the target object outline of size 1 at different positions in the second image, and ultimately obtain n candidate composite images containing the target object of size 1. Similarly, obtain m candidate composite images containing the target object of size 2, and p candidate composite images containing the target object of size 3. Where n, m, and p are all positive integers.

[0097] It should be noted that the window sliding method is not limited to the above example, and candidate composite images can also be obtained according to other preset sliding methods, which is not specifically limited in this embodiment.

[0098] It should also be noted that Figure 3This is only an example of the principle of constructing multiple candidate composite images. In actual processing, the size change of the target object in the candidate composite image will not be as obvious as shown in the figure.

[0099] Optionally, in some embodiments, each candidate composite image includes a mask image of the target object, where the mask image of the target object is used to indicate the position information of the target object in the background image. It should be understood that the mask image of the target object of each candidate composite image can be obtained during the window sliding process.

[0100] Step 204: Determine a target composite image whose composition rationality coefficient is greater than or equal to a threshold value from the multiple candidate composite images.

[0101] In this embodiment, the composition rationality coefficient is used to indicate the rationality of the size and position of the target object in the composite image. Specifically, the target composite image can be determined through the following optional implementations.

[0102] In an optional implementation, a composition rationality coefficient of each candidate composite image among a plurality of candidate composite images is obtained, and a target composite image having a composition rationality coefficient greater than or equal to a threshold is determined therefrom.

[0103] In an optional implementation, the composition rationality coefficient of each candidate composite image among multiple candidate composite images is obtained. If there are multiple composite images with a composition rationality coefficient greater than or equal to a threshold, the composite image with the largest composition rationality coefficient is used as the target composite image.

[0104] In an optional embodiment, multiple candidate composite images are combined into a set of images, and a candidate composite image is randomly selected from the set to determine the composition rationality coefficient of the candidate composite image. In one possible scenario, if the composition rationality coefficient of the candidate composite image is less than a threshold, a new candidate composite image is randomly selected from the remaining candidate composite images in the set, and the composition rationality coefficient of the new candidate composite image is determined, until a target composite image with a composition rationality coefficient greater than or equal to the threshold is determined from the set. In another possible scenario, if the composition rationality coefficient of the candidate composite image is greater than or equal to the threshold, the candidate composite image is used as the target composite image.

[0105] Step 205: Perform image post-processing on the target composite image to obtain a composite image that meets image requirements.

[0106] In an optional implementation, performing image post-processing on the target composite image only includes processing image features of the target object in the target composite image, such as adjusting the image color, lighting, and brightness of the target object.

[0107] In an optional embodiment, performing image post-processing on the target composite image includes processing image features of the target object in the target composite image and processing image features of an edge portion where the target object and the background image are spliced together in the target composite image.

[0108] In an optional embodiment, image post-processing of the target composite image includes processing image features of the target object in the target composite image, processing image features of the edge portion where the target object and the background image are spliced together in the target composite image, and processing image features of the background image in the target composite image.

[0109] The image synthesis method illustrated in the embodiment of the present application obtains an outline image of the target object to be synthesized and a background image, first determining the size range of the target object in the background image based on the background image, and then constructing multiple candidate synthetic images based on the size range. The multiple candidate synthetic images include synthetic images of target objects of one or more sizes at different positions in the background image. Subsequently, a target synthetic image with a composition rationality coefficient greater than or equal to a threshold is determined from the multiple candidate synthetic images, and the target synthetic image is then post-processed to obtain a synthetic image that meets the image requirements. Based on the above processing process, the efficiency and effect of image synthesis can be improved, and the generated synthetic image is more natural and realistic.

[0110] Based on the above embodiments, several specific embodiments are used below to describe in detail how to obtain the composition rationality coefficient of a candidate composite image and how to perform image post-processing.

[0111] Figure 4 Schematic diagram of the process of the image synthesis method provided in the embodiment of the present application Figure 2 On the basis of the above embodiments, Figure 4 As shown, the image synthesis method of this embodiment includes the following steps:

[0112] Step 401: Obtain a target object outline image and a background image to be synthesized.

[0113] Step 402: Determine the size range of the target object in the background image according to the background image.

[0114] Step 403: construct multiple candidate composite images, where the target objects in the multiple candidate composite images meet the size range and are located at different positions of the background image.

[0115] In this embodiment, steps 401 to 403 are similar to steps 201 to 203 in the above embodiment, and reference may be made to the above, and no further details will be given here.

[0116] Step 404: Input the first candidate composite image into a preset composition evaluation model to obtain a composition rationality coefficient corresponding to the first candidate composite image.

[0117] In this embodiment, the first candidate composite image is any one of multiple candidate composite images, and the composition rationality coefficient is used to indicate the rationality of the size and position of the target object in the composite image. The composition evaluation model is obtained by training a convolutional neural network and is used to output the composition rationality coefficient of the composite image. The composition rationality coefficient is also called the composition harmony. Optionally, the composition evaluation model can be trained using the network structure of Resnet-18. Resnet-18 is a network structure composed of stacked residual blocks, including 1 convolutional layer, 8 residual blocks and 1 fully connected layer, and each residual block has 2 convolutional layers.

[0118] The following is combined with Figure 5 The training process of the composition evaluation model is explained.

[0119] Figure 5 Schematic diagram of the training method of the composition evaluation model provided in the embodiment of the present application. Figure 5 As shown, the training method for the composition evaluation model in this embodiment includes the following steps:

[0120] Step 501: Obtain a first training sample set, where the first training sample set includes multiple first sample images and a first label value for each first sample image. The first sample image includes a composite image containing a target object, and the first label value is used to indicate a composition rationality coefficient of the first sample image.

[0121] In this embodiment, the first label value can be manually annotated. Optionally, the composition rationality coefficient ranges from 0 to 1, and a larger value indicates a more rational composition of the sample image, that is, a larger value indicates a more rational size and position of the target object in the sample image.

[0122] Step 502: Input a plurality of first sample images into the composition evaluation model in sequence to obtain a first prediction value corresponding to each first sample image.

[0123] Step 503: Determine the loss function of the composition evaluation model based on the first prediction value and the first label value corresponding to each first sample image. When the loss function of the composition evaluation model converges, stop training the composition evaluation model and obtain the model parameters of the composition evaluation model.

[0124] Through the above training process, the composition evaluation model can score the composition rationality of any input composite image and judge whether the current composite image meets the preset composition rationality requirements, such as whether the composition rationality coefficient is greater than the threshold, so as to quickly screen out the target composite image for image post-processing.

[0125] Step 405: If the composition rationality coefficient corresponding to the first candidate composite image is greater than or equal to the threshold, the first candidate composite image is used as the target composite image.

[0126] In this embodiment, by setting a threshold for the composition rationality coefficient, it is determined whether the position and size of the target object in the first candidate composite image meet the requirements. If the composition rationality coefficient of the first candidate composite image is greater than or equal to the threshold (for example, the threshold is 0.8), it indicates that the position of the target object in the first candidate composite image is reasonable and the size is moderate.

[0127] Optionally, in some embodiments, if the composition rationality coefficient corresponding to the first candidate composite image is less than a threshold value, indicating that the position and size of the target object in the first candidate composite image do not meet the requirements, that is, the position and / or size are unreasonable, it is necessary to reselect a candidate composite image from the remaining candidate composite images of the multiple candidate composite images, and re-execute step 404 until a candidate composite image with a composition rationality coefficient greater than or equal to the threshold value is found and used as the target composite image.

[0128] Step 406: Perform image post-processing on the target composite image to obtain a composite image that meets image requirements.

[0129] In an optional embodiment, the target composite image is post-processed using a preset image fusion model to obtain a composite image that meets the image requirements. In this embodiment, the image fusion model can be trained using an encoder-decoder model architecture, also known as a U-Net model architecture. The first half of the U-Net network architecture (i.e., the encoder) is primarily used for feature extraction, while the second half (i.e., the decoder) is primarily used for upsampling.

[0130] The image features of the target composite image are optimized through the image fusion model to obtain a more natural composite image.

[0131] In one optional embodiment, post-processing of the target composite image is performed using a preset image fusion model. This includes: inputting the target composite image into an encoder of the preset image fusion model to extract high-dimensional image features of the target composite image; inputting the high-dimensional image features into a decoder of the image fusion model, where an attention mechanism is employed to restore the high-dimensional image features to obtain a composite image that meets image requirements. In this embodiment, meeting image requirements includes meeting image feature requirements for color, lighting, and brightness of the target object within the background image.

[0132] Optionally, in some embodiments, in addition to the encoder and decoder, the image fusion model also includes a discriminator, which is used to assist in the training of the encoder and decoder. By judging whether the input sample image is a synthetic image or a real image, the discriminator guides the image fusion model to generate a synthetic image with a data distribution close to the data distribution of the real image.

[0133] Optionally, the discriminator can adopt a lightweight convolutional neural network architecture containing 7 convolutional layers.

[0134] It should be noted that the discriminator is only used in the training phase of the image fusion model and does not need to be run during actual deployment and inference. Figure 6 The training process of the image fusion model is described in detail.

[0135] Figure 6 Schematic diagram of the training method of the image fusion model provided in the embodiment of the present application. Figure 6 As shown, the training method for the image fusion model in this embodiment includes the following steps:

[0136] Step 601: Obtain a second training sample set, where the second training sample set includes multiple second sample images and a second label value for each second sample image. The second sample images include synthetic images and / or real images containing a target object. The second label value is used to indicate whether the target object in the second sample image is real.

[0137] Optionally, the second label value includes 0 and 1. For example, 0 indicates that the target object in the second sample image is real, and 1 indicates that the target object in the second sample image is not real. Optionally, the second label value can be manually labeled.

[0138] Step 602: Use the multiple second sample images as inputs of the image fusion model respectively, and pass them through the encoder, decoder and discriminator of the image fusion model in sequence to obtain a second prediction value corresponding to each second sample image.

[0139] Step 603: Determine the loss function of the image fusion model based on the second prediction value and the second label value corresponding to each second sample image. When the loss function of the image fusion model converges, stop training the image fusion model and obtain relevant parameters of the encoder and decoder in the image fusion model.

[0140] The above training process updates the relevant parameters of the encoder and decoder in the image fusion model, so that the image fusion model can better optimize the image features of the target composite image to obtain a more natural composite image.

[0141] The image synthesis method shown in the embodiment of the present application obtains the outline image of the target object to be synthesized and the background image, first determines the size range of the target object in the background image based on the background image, and then constructs multiple candidate synthetic images based on the size range. The multiple candidate synthetic images include synthetic images of target objects of one or more sizes at different positions in the background image. Subsequently, the multiple candidate synthetic images are input into the composition evaluation model in sequence to obtain the composition rationality coefficient corresponding to each candidate synthetic image, and the target synthetic image with a composition rationality coefficient greater than or equal to a threshold is determined from the multiple candidate synthetic images. Finally, the target synthetic image is subjected to image post-processing by the image fusion model to obtain a synthetic image that meets the image requirements. Among them, the composition evaluation model can be obtained by training a convolutional neural network to evaluate the composition rationality of each candidate synthetic image. The image fusion model can be obtained by training an encoder-decoder model architecture to optimize the synthesis effect of the synthetic image. Based on the above processing process, the overall effect of the synthetic image can be improved, making the synthetic image more natural and realistic.

[0142] The following is an example of the application of the image synthesis method provided in the embodiment of the present application in conjunction with a specific image synthesis scenario. Figure 7 Schematic diagram of the process of the image synthesis method provided in the embodiment of the present application Figure 3 The image synthesis method of this embodiment is applied to the character image synthesis scene of home decoration design drawings. The execution subject of the image synthesis method can be Figure 1 The client 101 or the server 102 shown. Figure 7 As shown, the image synthesis method of this embodiment includes the following steps:

[0143] Step 701: In response to a first operation performed on a first interface, obtain a character image and a home decoration design drawing uploaded by a user.

[0144] In this embodiment, the first interface is also called the picture import interface. The first interface includes a picture upload control. The picture upload control can be one or more. For details, see Figure 8a and Figure 8b The first operation may be a touch operation performed by the user on the first interface using a finger or a mouse, such as a click operation.

[0145] Figures 8a to 8b A schematic diagram of interface changes provided in an embodiment of the present application.

[0146] like Figure 8aAs shown, the picture import interface 800 includes a picture upload control 802 and a picture synthesis control 803. When the user clicks the picture upload control 802, he can select at least two pictures to be synthesized at the same time and upload them with one click. Furthermore, the user clicks the picture synthesis control 803, and the picture synthesis control 803 is used to trigger the intelligent processing of at least two pictures to be synthesized. After the processing is completed, the interface jumps to the synthesis preview interface 801. Optionally, in addition to displaying the final synthesized image, such as a home decoration design drawing containing characters, the synthesis preview interface 801 also includes a save control, a share control, and a return control. Among them, the save control is used to save the synthesized image to a user-specified location, such as an album or a file system; the share control is used to share the synthesized image with other users; the return control is used to return to the picture import interface 800, and the user can re-add pictures and synthesize the pictures again until the user is satisfied.

[0147] like Figure 8b As shown, the image import interface 800 includes two image upload controls: a first control 804 and a second control 805. The first control 804 is used to upload an image containing a target person, and the second control 805 is used to upload a background image. The user clicks on the first control 804 and the second control 805, respectively, to upload the two images to be synthesized. Furthermore, the user clicks on the image synthesis control 803, triggering intelligent processing of the two images to be synthesized. After processing is complete, the interface jumps to the synthesis preview interface 801.

[0148] It should be pointed out that the illustrated interface takes the mobile phone interface as an example. The computer interface can also be designed according to actual needs. The control layout of the computer interface is similar to that of the mobile phone interface.

[0149] Optionally, if there are multiple target persons to be synthesized, and the multiple target persons are located in different pictures, the user clicks the first control 804 to simultaneously select multiple pictures containing different target persons. The user then clicks the second control 805 to upload a background picture. The user then clicks the picture synthesis control 803 to synthesize the multiple target persons in the multiple pictures selected by the user into the same background picture.

[0150] Optionally, if there are multiple target persons to be synthesized, and the multiple target persons are in the same picture, the user clicks the first control 804 to upload a picture containing the multiple target persons, and clicks the second control 805 to upload a background picture. The user then clicks the picture synthesis control 803 to synthesize the multiple target persons into the same background picture.

[0151] Step 702: In response to a second operation performed on the first interface, obtain a character outline image from the character image.

[0152] In this embodiment, the first interface also includes an image synthesis control, which is used to trigger the synthesis of two or more images uploaded by the user. For example, it can trigger the synthesis of a person image and a home design image uploaded by the user, and synthesize the person in the person image into the home design image. The second operation can be a touch operation on the first interface by the user using a finger or a mouse, such as a click operation.

[0153] In one optional embodiment, a person outline image is obtained from a person image using a person recognition model. The recognition process of the person recognition model includes: first, using a deep learning-based person cutout technique to obtain masks of the foreground, background, and uncertain regions, i.e., obtaining a rough outline of the person; then, inputting these masks, along with the original red, green, and blue (RGB) image, a total of six channels, into a codec architecture network, such as a U-Net network, to obtain a detailed alpha transparency channel of the foreground person, and finally outputting a person outline image.

[0154] Step 703: Determine the size range of the character in the home decoration design drawing according to the home decoration design drawing.

[0155] In this embodiment, the size range of the person in the home decoration design drawing refers to the pixel size range of the person in the home decoration design drawing, for example, the pixel value range of the person in the vertical direction of the home decoration design drawing.

[0156] In an optional implementation, by obtaining the size information of the preset furniture models in the home decoration design drawing, the scale information of the home decoration design drawing is first determined based on the size information of the preset furniture models, and then the size range of the characters in the home decoration design drawing is determined based on the scale information.

[0157] Optionally, in some embodiments, the user can also manually adjust the size and position of the characters in the home design drawing to assist the device in composition analysis and image post-processing.

[0158] Step 704: construct multiple candidate composite images, where the characters in the multiple candidate composite images meet the size range and are located at different positions in the home decoration design drawing.

[0159] In this embodiment, the plurality of candidate composite images include composite images of the person's outline images of one or more sizes at different positions. It should be noted that the one or more sizes refer to the sizes of the person determined in step 703 within the size range of the home decoration design drawing.

[0160] In an optional embodiment, the character outline corresponding to the target size is synthesized at different positions of the home decoration design drawing by sliding a window to obtain multiple candidate synthesized images. Optionally, the target size includes multiple different sizes of the character within the size range of the home decoration design drawing.

[0161] Optionally, in some embodiments, each candidate composite image includes a mask image of a person, and the mask image of the person is used to indicate the position information of the person in the home design image. It should be understood that the mask image of the person in each candidate composite image can be obtained during the window sliding process.

[0162] Step 705: Determine a target composite image whose composition rationality coefficient is greater than or equal to a threshold value from multiple candidate composite images.

[0163] In this embodiment, the composition rationality coefficient is used to indicate the rationality of the size and position of the characters in the composite image.

[0164] In one optional implementation, multiple candidate composite images are sequentially input into a preset composition evaluation model to obtain a composition plausibility coefficient for each candidate composite image. A target composite image with a composition plausibility coefficient greater than or equal to a preset value is determined from the multiple candidate composite images. Optionally, if multiple composite images have a composition plausibility coefficient greater than or equal to a threshold, the composite image with the largest composition plausibility coefficient is selected as the target composite image. For details on the composition evaluation model, please refer to the above text and will not be elaborated on here.

[0165] Step 706: perform image post-processing on the target composite image to obtain a home decoration design drawing containing people that meets the image requirements.

[0166] In an optional embodiment, the target composite image is post-processed using a preset image fusion model to obtain a home design drawing containing people that meets the image requirements. For the relevant content of the image fusion model, please refer to the above and will not be repeated here.

[0167] Step 707: Display the home decoration design drawing including the person.

[0168] In response to the second operation performed on the first interface, the character image and the home decoration design image are synthesized to obtain a home decoration design image containing the character that meets the image requirements, and then the home decoration design image containing the character is displayed on the second interface. The second interface is also called a synthesis preview interface.

[0169] It should be pointed out that the picture import interface and synthesis preview interface provided in this embodiment are user operation interfaces provided by the home decoration design platform (server side). Users can access the home decoration design platform through a browser, mini-program or application APP, and import pictures to be synthesized and preview the synthesized pictures on the user operation interface provided by the home decoration design platform.

[0170] The image synthesis method illustrated in the embodiment of the present application is based on the person image and home decoration design drawing to be synthesized, which are imported by the user in the image import interface. First, the size range of the person in the home decoration design drawing is determined based on the home decoration design drawing. Then, multiple candidate synthesis images are constructed based on the size range. The multiple candidate synthesis images include synthesis images of people of one or more sizes at different positions in the home decoration design drawing. Then, a target synthesis image with a composition rationality coefficient greater than or equal to a threshold is determined from the multiple candidate synthesis images. The target synthesis image is then subjected to image post-processing to obtain a home decoration design drawing containing a person that meets the image requirements. Finally, the home decoration design drawing containing the person is displayed in the synthesis preview interface. Existing image synthesis algorithms cannot achieve ideal synthesis effects for person synthesis in home decoration design drawings with complex scene content. There are problems such as severe synthesis effect, image distortion, and lack of realism in the synthesis effect image. In contrast, the processing process based on this case can stably and efficiently generate high-quality images, realize intelligent synthesis of people into reasonable positions in the home decoration design drawing, and generate more lifelike home decoration design drawings, enhance the realism of home decoration design, and enable users to better understand the home decoration design scene.

[0171] Next, we will combine Figure 9 Graphically display the image synthesis scheme of home decoration design scenes. Figure 9 This is a diagram showing the processing and effect of the image synthesis method provided in the embodiment of this application. Figure 9As shown in the figure, a user-uploaded image of a person and a living room design are obtained. The living room design features a sofa and a floor lamp. First, the person's main subject is extracted from the person's image and used as the foreground of the composite image. After determining the size of the person's main subject within the living room design, a traversal strategy is used to construct multiple candidate composite images. The composition rationality of these candidate composite images is then evaluated using a composition evaluation model. Specifically, within the given size range of the person's main subject image, traversal begins at the maximum size of the person's main subject and is composited onto the interior design image using a sliding window method, resulting in candidate composite images and a person's foreground position mask. The candidate composite images and the person's foreground position mask are then input into the composition evaluation model (not shown) to obtain the composition rationality coefficient of the candidate composite image. In one case, if the composition rationality coefficient of a candidate composite image is less than a threshold, traversal proceeds to the next candidate composite image. In another case, if the composition rationality coefficient of a candidate composite image is greater than or equal to the threshold, evaluation of further candidate composite images ceases. Because the foreground and background are acquired under different environmental conditions, the composite image suffers from brightness and color mismatches, lacking a sense of realism. Therefore, the composite image and the person foreground position mask must be fed into an image fusion model. This model, consisting of an encoder and a decoder, adjusts the color, lighting, and brightness of the person (foreground) in the composite image to better integrate it with the interior design image (background). The image fusion model optimizes the composite image and ultimately outputs a composite image of the person and scene that meets the required image quality.

[0172] The above describes the image synthesis method provided by the embodiment of the present application. The following describes the image synthesis device provided by the embodiment of the present application.

[0173] In the embodiment of the present application, the image synthesis device can be divided into functional modules according to the above-mentioned method embodiment. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above-mentioned integrated modules can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical functional division. There may be other division methods in actual implementation. The following is an example of dividing each functional module according to each function.

[0174] Figure 10 Schematic diagram of the structure of the image synthesis device provided in the embodiment of the present application Figure 1 .like Figure 10 As shown, the image synthesis device 1000 of this embodiment includes: an acquisition module 1001 and a processing module 1002 .

[0175] An acquisition module 1001 is used to acquire a target object outline image and a background image to be synthesized;

[0176] Processing module 1002, configured to determine a size range of a target object in the background image according to the background image;

[0177] constructing a plurality of candidate composite images, wherein the target objects in the plurality of candidate composite images meet the size range and are located at different positions of the background image;

[0178] Determining a target composite image whose composition rationality coefficient is greater than or equal to a threshold from the plurality of candidate composite images;

[0179] Perform image post-processing on the target composite image to obtain a composite image that meets image requirements.

[0180] In an optional embodiment of this embodiment, the multiple candidate synthetic images include target object contour images of different sizes; the processing module 1002 is used to synthesize the target object contour images corresponding to the target size into different positions of the background image by window sliding to obtain the multiple candidate synthetic images; the target size includes multiple different sizes within the size range, and each candidate synthetic image includes a mask image of the target object.

[0181] In an optional embodiment of this embodiment, the processing module 1002 is configured to:

[0182] Inputting the first candidate composite image into a preset composition evaluation model to obtain a composition rationality coefficient corresponding to the first candidate composite image, wherein the composition rationality coefficient is used to indicate the rationality of the size and position of the target object in the composite image;

[0183] If the composition rationality coefficient corresponding to the first candidate composite image is greater than or equal to a threshold, taking the first candidate composite image as the target composite image;

[0184] The first candidate composite image is any one of the multiple candidate composite images, and the composition evaluation model is obtained by training a convolutional neural network.

[0185] In an optional embodiment of this embodiment, the acquisition module 1001 is configured to acquire a first training sample set, where the first training sample set includes a plurality of first sample images and a first label value for each first sample image, where the first sample image includes a composite image containing a target object, and the first label value is used to indicate a reasonableness of a size and position of the target object in the first sample image;

[0186] The processing module 1002 is configured to sequentially input the plurality of first sample images into the composition evaluation model to obtain a first prediction value corresponding to each first sample image;

[0187] Determine the loss function of the composition evaluation model based on the first prediction value and the first label value corresponding to each first sample image. When the loss function of the composition evaluation model converges, stop training the composition evaluation model and obtain the model parameters of the composition evaluation model.

[0188] In an optional embodiment of this embodiment, the processing module 1002 is configured to:

[0189] If there are multiple composite images with a composition rationality coefficient greater than or equal to the threshold, the composite image with the largest composition rationality coefficient is used as the target composite image.

[0190] In an optional embodiment of this embodiment, the processing module 1002 is configured to:

[0191] Inputting the target composite image into an encoder of a preset image fusion model to extract high-dimensional image features of the target composite image;

[0192] The high-dimensional image features are input into the decoder of the image fusion model. In the decoder, an attention mechanism is used to perform image restoration on the high-dimensional image features to obtain a composite image that meets image requirements; meeting the image requirements includes meeting the image feature requirements of color, lighting, and brightness and darkness of the target object in the background image.

[0193] In an optional embodiment of this embodiment, the image fusion model further includes a discriminator, and the acquisition module 1001 is further used to obtain a second training sample set, where the second training sample set includes multiple second sample images and a second label value for each second sample image, where the second sample images include synthetic images and / or real images containing the target object, and the second label value is used to indicate whether the target object in the sample image is real;

[0194] The processing module 1002 is further configured to use the plurality of second sample images as inputs of the image fusion model, and sequentially pass the plurality of second sample images through the encoder, decoder, and discriminator of the image fusion model to obtain a prediction value corresponding to each second sample image;

[0195] According to the second prediction value and the second label value corresponding to each second sample image, the loss function of the image fusion model is determined. When the loss function of the image fusion model converges, the training of the image fusion model is stopped, and the relevant parameters of the encoder and decoder in the image fusion model are obtained.

[0196] In an optional embodiment of this embodiment, the processing module 1002 is configured to:

[0197] Acquire point cloud data of a reference object in the background image, and determine scale information of the background image according to the point cloud data of the reference object;

[0198] The size range of the target object in the background image is determined according to the scale information.

[0199] In an optional embodiment of this embodiment, the processing module 1002 is configured to:

[0200] Determining scale information of the background image according to size information of the reference object in the background image;

[0201] The size range of the target object in the background image is determined according to the scale information.

[0202] The image synthesis device provided in this embodiment can perform Figure 2 or Figure 4 The technical solution of the method embodiment shown has similar implementation principles and technical effects, which will not be repeated here.

[0203] Figure 11 Schematic diagram of the structure of the image synthesis device provided in the embodiment of the present application Figure 2 .like Figure 11 As shown, the image synthesis device 1100 of this embodiment includes: an acquisition module 1101 , a processing module 1102 and a display module 1103 .

[0204] An acquisition module 1101 is configured to respond to a first operation on the first interface and acquire a character image and a home decoration design drawing uploaded by a user;

[0205] Processing module 1102 is configured to, in response to a second operation performed on the first interface, obtain a person outline from the person image; determine a size range of the person within the home decoration design drawing based on the home decoration design drawing; determine a target composite image having a composition rationality coefficient greater than or equal to a threshold from the constructed multiple candidate composite images, and perform image post-processing on the target composite image to obtain a home decoration design drawing that meets image requirements and includes the person; the people in the multiple candidate composite images are within the size range and are located at different positions in the home decoration design drawing;

[0206] The display module 1103 is configured to display a home decoration design drawing including the character.

[0207] The image synthesis device provided in this embodiment can perform Figure 7 The technical solution of the method embodiment shown has similar implementation principles and technical effects, which will not be repeated here.

[0208] Figure 12 This is a hardware structure diagram of the electronic device provided in the embodiment of this application. Figure 12 As shown, the electronic device 1200 provided in this embodiment includes:

[0209] Memory 1201, processor 1202 and computer program; wherein, the computer program is stored in memory 1201 and is configured to be executed by processor 1202 to implement the technical solution of any of the aforementioned method embodiments, and its implementation principles and technical effects are similar and will not be repeated here.

[0210] Optionally, the memory 1201 may be independent or integrated with the processor 1202. When the memory 1201 is a device independent of the processor 1202, the electronic device 1200 further includes a bus 1203 for connecting the memory 1201 and the processor 1202.

[0211] An embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. The computer program is executed by the processor 1202 to implement the technical solution of any of the aforementioned method embodiments.

[0212] An embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the technical solution of any of the aforementioned method embodiments.

[0213] An embodiment of the present application provides a chip, including: a processing module and a communication interface, wherein the processing module can execute the technical solution of any of the aforementioned method embodiments.

[0214] Optionally, the chip also includes a storage module (such as a memory), the storage module is used to store instructions, the processing module is used to execute the instructions stored in the storage module, and the execution of the instructions stored in the storage module enables the processing module to execute the technical solution of any of the aforementioned method embodiments.

[0215] It should be understood that the processor described above may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), or application-specific integrated circuits (ASICs). A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the present invention may be directly executed by a hardware processor or by a combination of hardware and software modules within the processor.

[0216] The memory may include a high-speed RAM memory, and may also include non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk.

[0217] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. Buses can be classified into address buses, data buses, and control buses. For ease of illustration, the buses in the drawings of this application are not limited to just one bus or just one type of bus.

[0218] The storage medium may be implemented by any type of volatile or non-volatile memory device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium may be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0219] An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and storage medium can be located in an application-specific integrated circuit (ASIC). Of course, the processor and storage medium can also exist as discrete components in an electronic device.

[0220] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An image synthesis method, characterized in that: include: Obtaining a target object outline image and a background image to be synthesized; Determining a size range of the target object in the background image according to the background image; constructing a plurality of candidate composite images, wherein the target objects in the plurality of candidate composite images meet the size range and are located at different positions of the background image; Determining a target composite image having a composition rationality coefficient greater than or equal to a threshold value from the plurality of candidate composite images; wherein the composition rationality coefficient is obtained by inputting the first candidate composite image into a preset composition evaluation model; Performing image post-processing on the target composite image to obtain a composite image that meets image requirements; The performing image post-processing on the target composite image to obtain a composite image that meets image requirements includes: Inputting the target composite image into an encoder of a preset image fusion model to extract high-dimensional image features of the target composite image; The high-dimensional image features are input into the decoder of the image fusion model. In the decoder, an attention mechanism is used to perform image restoration on the high-dimensional image features to obtain a synthetic image that meets the image requirements.

2. The method according to claim 1, characterized in that The plurality of candidate composite images include target object outline images of different sizes; The constructing of multiple candidate composite images includes: synthesizing the target object contour images corresponding to the target size into different positions of the background image by means of window sliding, to obtain the plurality of candidate synthesized images; The target size includes a plurality of different sizes within the size range, and each candidate synthetic image includes a mask image of the target object.

3. The method according to claim 1, characterized in that Determining a target composite image having a reasonable coefficient greater than or equal to a threshold from the plurality of candidate composite images includes: Inputting the first candidate composite image into a preset composition evaluation model to obtain a composition rationality coefficient corresponding to the first candidate composite image, wherein the composition rationality coefficient is used to indicate the rationality of the size and position of the target object in the composite image; If the composition rationality coefficient corresponding to the first candidate composite image is greater than or equal to a threshold, taking the first candidate composite image as the target composite image; The first candidate composite image is any one of the multiple candidate composite images, and the composition evaluation model is obtained by training a convolutional neural network.

4. The method according to claim 3, characterized in that The training process of the composition evaluation model includes: Obtaining a first training sample set, the first training sample set including a plurality of first sample images and a first label value for each first sample image, the first sample image including a composite image containing a target object, the first label value being used to indicate a reasonableness of a size and a position of the target object in the first sample image; inputting the plurality of first sample images into the composition evaluation model in sequence to obtain a first prediction value corresponding to each first sample image; Determine the loss function of the composition evaluation model based on the first prediction value and the first label value corresponding to each first sample image. When the loss function of the composition evaluation model converges, stop training the composition evaluation model and obtain the model parameters of the composition evaluation model.

5. The method according to any one of claims 1 to 4, characterized in that The step of determining a target composite image having a reasonable composition coefficient greater than or equal to a threshold value from the plurality of candidate composite images includes: If there are multiple composite images with a composition rationality coefficient greater than or equal to the threshold, the composite image with the largest composition rationality coefficient is used as the target composite image.

6. The method according to claim 1, characterized in that The image fusion model further includes a discriminator, and the training process of the image fusion model includes: Obtaining a second training sample set, where the second training sample set includes a plurality of second sample images and a second label value for each second sample image, the second sample images including synthetic images and / or real images containing a target object, and the second label value is used to indicate whether the target object in the second sample image is real; Using the plurality of second sample images as inputs of the image fusion model, and sequentially passing through an encoder, a decoder, and a discriminator of the image fusion model to obtain a second prediction value corresponding to each second sample image; Determine the loss function of the image fusion model based on the second prediction value and the second label value corresponding to each second sample image; when the loss function of the image fusion model converges, stop training the image fusion model and obtain relevant parameters of the encoder and decoder in the image fusion model.

7. The method according to any one of claims 1 to 4, characterized in that Determining the size range of the target object in the background image according to the background image includes: Acquire point cloud data of a reference object in the background image, and determine scale information of the background image according to the point cloud data of the reference object; The size range of the target object in the background image is determined according to the scale information.

8. The method according to any one of claims 1 to 4, characterized in that Determining the size range of the target object in the background image according to the background image includes: Determining scale information of the background image according to size information of the reference object in the background image; The size range of the target object in the background image is determined according to the scale information.

9. An image synthesis method, characterized in that: include: In response to a first operation performed on the first interface, obtaining a character image and a home decoration design image uploaded by a user; In response to a second operation performed on the first interface, obtaining a character outline from the character image; and determining a size range of the character in the home decoration design drawing according to the home decoration design drawing; Determining a target composite image having a composition rationality coefficient greater than or equal to a threshold value from the constructed multiple candidate composite images, performing image post-processing on the target composite image to obtain a home decoration design image that meets the image requirements and includes the person; the people in the multiple candidate composite images are within the size range and are located at different positions in the home decoration design image; wherein the composition rationality coefficient is obtained by inputting the first candidate composite image into a preset composition evaluation model; displaying a home design drawing including the character; The performing image post-processing on the target composite image to obtain a home decoration design drawing containing the person that meets image requirements includes: Inputting the target composite image into an encoder of a preset image fusion model to extract high-dimensional image features of the target composite image; The high-dimensional image features are input into the decoder of the image fusion model. In the decoder, an attention mechanism is used to perform image restoration on the high-dimensional image features to obtain a home decoration design drawing containing the person that meets the image requirements.

10. An image synthesis device, characterized in that: include: An acquisition module is used to obtain a contour image of a target object and a background image to be synthesized; A processing module, configured to determine a size range of a target object in the background image according to the background image; constructing a plurality of candidate composite images, wherein the target objects in the plurality of candidate composite images meet the size range and are located at different positions of the background image; Determining a target composite image having a composition rationality coefficient greater than or equal to a threshold value from the plurality of candidate composite images; wherein the composition rationality coefficient is obtained by inputting the first candidate composite image into a preset composition evaluation model; The target composite image is input into the encoder of a preset image fusion model to extract high-dimensional image features of the target composite image; the high-dimensional image features are input into the decoder of the image fusion model, and in the decoder, an attention mechanism is used to perform image restoration on the high-dimensional image features to obtain a composite image that meets the image requirements.

11. An image synthesis device, characterized in that: include: an acquisition module, configured to respond to a first operation on the first interface and acquire a character image and a home decoration design image uploaded by a user; A processing module is configured to, in response to a second operation performed on the first interface, obtain a character outline from the character image; and determine a size range of the character in the home decoration design drawing according to the home decoration design drawing; Determining a target composite image whose composition rationality coefficient is greater than or equal to a threshold from the constructed multiple candidate composite images, inputting the target composite image into an encoder of a preset image fusion model, and extracting high-dimensional image features of the target composite image; Inputting the high-dimensional image features into a decoder of the image fusion model, wherein the decoder employs an attention mechanism to perform image restoration on the high-dimensional image features to obtain a home decoration design drawing containing the person that meets the image requirements; the person(s) in the multiple candidate composite images fit within the size range and are located at different positions in the home decoration design drawing; wherein the composition rationality coefficient is obtained by inputting the first candidate composite image into a preset composition evaluation model; The display module is used to display a home decoration design drawing including the character.

12. An electronic device, characterized in that: include: A memory, a processor, and a computer program; the computer program is stored in the memory and configured to be executed by the processor to implement the method according to any one of claims 1 to 9.

13. A computer-readable storage medium, characterized in that A computer program is stored thereon, and the computer program is executed by a processor to implement the method according to any one of claims 1 to 9.

14. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 9.

Images