A common-sense-enhanced image synthesis method and apparatus

Abstract

This invention proposes a common-sense-enhanced image synthesis method, comprising: a data preparation step, in which a scene description model is selected to process known multimodal data to obtain common-sense knowledge of the placement of objects in the environment; a model training step, in which known images are labeled with positional descriptions to generate a training set, and the synthesis model is trained using the training set; the known images include known background images and known foreground images; and an image synthesis step, in which the synthesis model, based on the common-sense knowledge, synthesizes the target background image and the target foreground image into a target image. This invention also proposes a common-sense-enhanced image synthesis apparatus and an electronic device for common-sense-enhanced image synthesis.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, and in particular relates to a common sense-enhanced image synthesis method and apparatus. Background Technology

[0002] In the field of computer vision, the goal of image synthesis is to merge multiple images or image components into a unified and harmonious visual scene. A key subtask is object placement, which involves accurately determining the optimal position and proportion of foreground objects within a given scene to accomplish the image synthesis task. Currently, some representative methods include:

[0003] 1. A generative adversarial network (GAN)-based approach, as described in the paper "Shashank Tripathi, Siddhartha Chandra, Amit Agrawal, Ambrish Tyagi, James M Rehg, and Visesh Chari. Learning to generate synthetic data via compositing. CVPR 2019". Its main technique involves two networks: a generator network and a discriminator network. The generator is responsible for determining the foreground placement and proportions to generate a seemingly reasonable synthetic image, while the discriminator distinguishes between reasonable and unreasonable images. In this way, the generator learns how to improve object placement to make the generated image as plausible as possible. Problems and drawbacks: This method lacks an embedded common-sense fusion mechanism, so object placement may lack logic and realism, potentially resulting in objects appearing to float in mid-air or placed in impossible positions. Furthermore, GANs themselves are difficult to converge, and the training process is unstable, often leading to long training times.

[0004] 2. A reinforcement learning-based approach, as described in the paper "Shengping Zhang, Quanling Meng, Qinglin Liu, Liqiang Nie, Bineng Zhong, Xiaopeng Fan, and Rongrong Ji. Inter-active object placement with reinforcement learning. ICML 2023." Its main technique involves the model continuously trying and failing during its interaction with the environment, scoring its current state, and using a reward mechanism to identify the optimal placement location and method. This method can learn and adapt to various scenarios and backgrounds, thereby improving the accuracy and rationality of object placement. Problems and drawbacks: First, training reinforcement learning models typically requires a large amount of data and computational resources, resulting in long training times and high costs. Second, because reinforcement learning relies on trial and error, the model needs to undergo numerous failed attempts to learn an effective placement strategy, which may lead to low learning efficiency and difficulty in convergence. Finally, since the goal of reinforcement learning is to maximize cumulative rewards, the model may overemphasize short-term rewards while neglecting long-term stability and sustainability, potentially leading to insufficient generalization ability in practical applications.

[0005] 3. A heatmap-based method, the relevant paper being "Sijie Zhu, Zhe Lin, Scott Cohen, Jason Kuen, Zhifei Zhang, and Chen Chen. Topnet: Transformer-based object placement network for image compositing. CVPR 2023". Its main technique involves using a single-path transformer network to extract features from the foreground and background images, and then using heatmaps to predict the appropriate positions of objects in the synthesized image. This method effectively utilizes the self-attention mechanism in the transformer architecture, enabling the model to understand the contextual relationships in the image and generate the most reasonable possible results. Problems and drawbacks: The lack of consideration for physical laws and real-world semantic norms may lead to objects being placed in inappropriate positions, such as books suspended in mid-air. Furthermore, because this method relies entirely on visual features, it cannot fully understand and handle complex scenes and interactions between objects, resulting in insufficient plausibility and harmony in the generated images. Summary of the Invention

[0006] To address the aforementioned problems, this invention proposes a common-sense-enhanced image synthesis method, comprising: a data preparation step, in which a scene description model is selected to process known multimodal data to obtain common-sense knowledge about the placement of objects in the environment; a model training step, in which known images are labeled with location descriptions to generate a training set, and the synthesis model is trained using the training set; the known images include known background images and known foreground images; and an image synthesis step, in which the synthesis model, based on the common-sense knowledge, synthesizes the target background image and the target foreground image into a target image.

[0007] Furthermore, the model training steps specifically include: a training set generation step, which involves labeling the positional description of the known foreground image within the known background image and generating guiding prompts for the positional description; generating the training set from the labeled known background image and the known foreground image; and a training step, which uses the training set to train the synthetic model, generating multiple prediction heads and scorers for the synthetic model, with the training objective L = λ1L. cap +λ2L gt Where λ1 and λ2 are weighting coefficients, and L cap To generate the objective function for this location description annotation, w i Let i be the true probability distribution of the i-th word labeled at this location. Let L be the predicted probability distribution of the i-th word; gt For the objective function, L gt =L gt.pos +L gt.neg L gt.pos Let L be the objective function that restricts the prediction results to approach positive samples. gt.neg To limit the prediction results from deviating from the objective function of negative samples, N gt Let δ be the number of negative samples, (r, x, y) be the descriptive triple, r be the scaling parameter of the known foreground image in the known background image, x and y be the positional parameters of the known foreground image in the known background image, and δ be the safety boundary between the predicted result and the negative samples.

[0008]

[0009] Furthermore, the data preparation step also includes an annotation step, which involves annotating the known multimodal data with location information. The location annotation includes concise annotations and detailed annotations. The concise annotations include basic information about where the object should be placed, and the detailed annotations include logical information about the object's placement behavior.

[0010] Furthermore, the image synthesis step specifically includes: a description step, which uses the synthesis model to generate image description information for placing the target foreground image in the target background image; a scoring step, which uses the image description information to generate at least one set of description triples for placing the target foreground image in the target background image, and uses the scorer of the synthesis model to select a target triple from the description triples; and a texturing step, which uses the target triple to obtain the target image through texturing.

[0011] This invention also proposes a common-sense-enhanced image synthesis device, comprising: a data preparation module for selecting a scene description model to process known multimodal data to obtain common-sense knowledge of the placement of objects in the environment; a model training module for annotating the known images with location descriptions to generate a training set, and using the training set to train the synthesis model; the known images include a known background image and a known foreground image; and an image synthesis module for using the synthesis model to synthesize the target background image and the target foreground image into a target image based on the common-sense knowledge.

[0012] Furthermore, the model training module specifically includes: a training set generation module, used to annotate the positional description of the known foreground image in the known background image and generate guiding prompts for the positional description; generating the training set from the annotated known background image and the known foreground image; and a training module, used to train the synthetic model with the training set, generating multiple prediction heads and scorers for the synthetic model, with the training objective L = λ1L. cap +λ2L gt Where λ1 and λ2 are weighting coefficients, and L cap To generate the objective function for this location description annotation, w i Let i be the true probability distribution of the i-th word labeled at this location. Let L be the predicted probability distribution of the i-th word; gt For the objective function, L gt =L gt.pos +L gt.neg L gt.pos Let L be the objective function that restricts the prediction results to approach positive samples. gt.neg To limit the prediction results from deviating from the objective function of negative samples, N gt Let δ be the number of negative samples, (r, x, y) be the descriptive triple, r be the scaling parameter of the known foreground image in the known background image, x and y be the positional parameters of the known foreground image in the known background image, and δ be the safety boundary between the predicted result and the negative samples.

[0013]

[0014] Furthermore, the data preparation module includes an annotation module for annotating the known multimodal data at location. The location annotation includes concise annotations and detailed annotations. The concise annotation includes basic information about where the object should be placed, and the detailed annotation includes logical information about the object's placement behavior.

[0015] Furthermore, the image synthesis module includes: a description module, used to generate image description information of the target foreground image placed in the target background image using the synthesis model; a scoring module, used to generate at least one set of description triples of the target foreground image placed in the target background image based on the image description information, and to select a target triple from the description triples using the scorer of the synthesis model; and a texture module, used to obtain the target image by texture mapping based on the target triple.

[0016] The present invention also proposes a computer-readable storage medium storing computer-executable instructions, which, when executed, implement the common-sense-enhanced image synthesis method as described above.

[0017] The present invention also proposes an electronic device including the common sense-enhanced image synthesis apparatus as described above.

[0018] The common sense-enhanced image synthesis method of the present invention does not employ adversarial networks, has a simple training objective, few hyperparameters, and a very fast model training and convergence process, saving computational resources while improving the model's usability in more data environments. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the technical route of the image synthesis method for enhancing common sense according to the present invention.

[0020] Figure 2 This is a flowchart of the common sense-enhanced image synthesis method of the present invention.

[0021] Figure 3 This is a diagram illustrating the effect of the image synthesis method for enhancing common sense according to the present invention.

[0022] Figure 4 This is a schematic diagram of the image synthesis device for enhancing common sense according to the present invention.

[0023] Figure 5 This is a schematic diagram of the data preparation module.

[0024] Figure 6 This is a schematic diagram of the model training module.

[0025] Figure 7 This is a schematic diagram of the image synthesis module.

[0026] Figure 8 This is a schematic diagram of an electronic device according to the present invention.

[0027] Figure 9 This is a schematic diagram of the hardware structure of an electronic device according to the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0029] To address the limitations of existing image synthesis technologies, this invention proposes a common-sense-enhanced object placement and image synthesis method. This method aims to overcome the following problems in existing technologies: irrational object placement in a scene, lack of awareness of object size and orientation in different environments, and difficulty in model training.

[0030] The core principle of this method is "think before you place." It utilizes a multimodal large model to learn common-sense knowledge through pre-training, and then guides the model with manually labeled location descriptions. This allows the large model to better learn the specific constraints of object placement during text description, thereby significantly reducing the likelihood of unreasonable synthesized images during parameter prediction. Through the design of a scorer and multiple regression prediction heads, this method can simultaneously satisfy the tasks of finding the optimal placement location and finding multiple possible locations, thus increasing the method's versatility.

[0031] This invention employs a common-sense knowledge-enhanced object placement and image synthesis method. First, it utilizes the common-sense knowledge learned through pre-training of a multimodal large-scale model. During specific predictions, the model accepts the image and spontaneously uses language to describe the correct placement location, thus better leveraging the capabilities gained through pre-training tasks. Then, it guides the large-scale model to rationally utilize common-sense knowledge by providing manually labeled location descriptions. Using the image itself and the previously obtained location descriptions, it makes specific predictions of location parameters, thereby obtaining the thought process to guide the object placement process and achieve a more rational image synthesis process. This allows the model to generate predicted location descriptions that conform to real-world common sense, physical laws, and content harmony. Based on these descriptions, reasonable image placement parameters are further obtained, resulting in a reasonable image. This invention can simultaneously satisfy the need to generate the optimal synthesis location and provide multiple reasonable synthesis locations. The method incorporates a final scorer and multiple regression prediction heads. For tasks requiring multiple reasonable placement locations, each prediction head, after training, can provide highly reasonable prediction results. For tasks requiring only the optimal prediction location, the final scorer scores the results of each prediction head to obtain the best candidate from among many results.

[0032] Figure 1 This is a schematic diagram of the technical route of the image synthesis method for enhancing common sense according to the present invention. For example... Figure 1 As shown, the core technical approach of this invention is "think before you place," aiming to improve semantic consistency and diversity in object placement tasks by combining a large-scale multimodal model and a common-sense enhancement module. This invention uses a large multimodal model as its foundation, providing a foreground image and a background image to be placed. First, the large model provides a verbal description of the reasonable placement position of the foreground image within the background image. Then, based on the corresponding description, it provides specific placement coordinates, realizing image placement and generation, and thereby improving the image's rationality and semantic consistency.

[0033] Figure 2 This is a flowchart of the common-sense-enhanced image synthesis method of the present invention. Figure 2 As shown, the common sense-enhanced image synthesis method of the present invention specifically includes:

[0034] Step S1: Data Preparation

[0035] The multimodal large model selected in this invention acquires common-sense knowledge about objects and their relationships by processing a large amount of multimodal data during the pre-training phase. This makes it particularly suitable for tasks requiring the rational placement of objects in specific environments. Furthermore, to improve the accuracy and diversity of object placement tasks, additional annotations are added to the original image dataset, including two types of image descriptions: concise and detailed. Concise annotations provide basic information about where the object should be placed, such as "on a table" or "in a field." Detailed annotations provide richer scene descriptions, such as "on a table instead of on the floor," or "on a table away from the plate," emphasizing the thought process behind the placement.

[0036] Step S2, Model Training

[0037] During the training phase, the model takes a background image, a processed foreground image, a manually labeled foreground image location description, and a guiding prompt as input. This input data is processed by a "common sense reinforcement" module. This module uses common sense reasoning to output an image description for potential placement locations, learning to fit the logic of the given standard description. This step is crucial to the entire model because it integrates image data and verbal prompts, providing the foundation for subsequent object placement. Next, the model parses a specific state description triple (r, x, y) from the regression labels t based on the image and the obtained reasonable location description. Here, r describes the image size scaling information, and x and y describe the quantiles of the foreground image's position on the x and y axes in the background image, respectively. This process involves accurate inference of the object's position and size in the image, a vital part of the model, and represents the model's specific quantitative expression of the object's correct placement location.

[0038] To improve the diversity of the generated images, this invention uses multiple prediction heads to predict triples. Increasing the number of prediction heads within a certain range can simultaneously improve the rationality and diversity of the predicted positions. For optimal performance, a single-layer fully connected network is used to fit positive and negative samples to obtain a scorer for the generated images. This scorer provides a rationality prediction of the generated images, and the model selects the most reasonable position for the object from the candidate positions based on the scorer's results. The scorer's role is to evaluate the rationality of each candidate position, ensuring that the final selected position is visually and logically the most suitable. By choosing whether to apply the scorer for filtering, the model of this invention can simultaneously meet the requirements of generating the optimal position and generating images from multiple possible positions at once.

[0039] From the perspective of the training objective, this invention combines two types of loss. First, cross-entropy loss is used to encourage the model to generate appropriate descriptions, ensuring that the model provides contextual information relevant to object placement.

[0040]

[0041] Among them, w i Let be the true probability distribution of the i-th word. Let be the predicted probability distribution of the i-th word;

[0042] For positive samples, the goal is to align the predicted location with the triple (r, x, y) representing the ideal location. In the proposed method, each output head generates a candidate location, represented as a triple (r, x, y), to represent the candidate location. The Euclidean distance of each candidate location is calculated with the nearest ground truth location, and the mean squared error (MSE) between these paired locations is calculated. The average MSE of all these pairs reflects the accuracy of the alignment while emphasizing the diversity of results. Secondly, the method designs a loss function to handle negative samples. Since the scaling size cannot be considered incorrect simply because a location r is incorrect, this function focuses only on the (x, y) coordinates, aiming to ensure that the predicted location is sufficiently far from the negative sample. If the MSE between the prediction and the negative sample is less than a set boundary threshold, a penalty is incurred; if the MSE is equal to or exceeds the boundary, the distance is considered safe, and the loss is zero. When the penalties generated for positive and negative samples conflict, the method prioritizes producing results that are closer to the positive sample, harmonizing the weights of both to generate the final location training objective.

[0043]

[0044] Among them, L gt.pos Let L be the objective function that restricts the prediction results to approach positive samples. gt.negTo limit the prediction results from deviating from the objective function of negative samples, N gt δ represents the number of negative samples, (r,x,y) is the descriptive triple, r is the scaling parameter of the known foreground image in the known background image, x and y are the position parameters of the known foreground image in the known background image, and δ is the safety boundary between the prediction result and the negative samples.

[0045] Combining the above objectives linearly results in the final training objective that simultaneously satisfies "thinking correctly and placing correctly," as shown below.

[0046]

[0047] Among them, L gt =L gt.pos +L gt.neg λ1 and λ2 are weighting coefficients.

[0048] Step S3: Image Synthesis

[0049] After training, the model can be used for image synthesis, which consists of two stages. In the first stage, two images are input (a background image and the foreground image to be placed). The trained model uses language to describe the specific location to be placed. In the second stage, the images and the image descriptions obtained in the first stage are input back into the model. The model will provide predicted triples (r, x, y). Based on these three parameters, the image can be scaled, the location selected, and the final synthesized image obtained through texturing. To obtain the optimal solution, a pre-trained scorer is added to score all candidate locations, and the highest-scoring location is selected for repeated texturing. Specific results are shown below. Figure 3 As shown.

[0050] The following are system embodiments corresponding to the above method embodiments. This embodiment can be implemented in conjunction with the above embodiments. The relevant technical details mentioned in the above embodiments are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiments.

[0051] Figure 4 This is a schematic diagram of the image synthesis apparatus for enhancing common sense according to the present invention. (See diagram below.) Figure 4 As shown, in a second embodiment of the present invention, an image synthesis apparatus is proposed, comprising:

[0052] Data preparation module 10 is used to select a scene description model to process known multimodal data in order to obtain common sense knowledge about the placement of objects in the environment. Figure 5 This is a schematic diagram of the data preparation module, as shown below. Figure 5 As shown, the data preparation module 10 includes:

[0053] Annotation module 11 is used to annotate the location of known multimodal data. The location annotation includes concise annotations and detailed annotations. The concise annotation includes basic information about where the object should be placed, such as "on a table" or "in a field". The detailed annotation includes logical information about the object's placement behavior, such as "on a table instead of on the floor" or "on a table away from the plate", emphasizing the thought process behind the placement behavior.

[0054] The model training module 20 is used to perform location description annotation on the known image, generate a training set, and use the training set to train the synthetic model; the known image includes a known background image and a known foreground image; Figure 6 This is a schematic diagram of the model training module, as shown below. Figure 6 As shown, the model training module 20 includes:

[0055] The training set generation module 21 is used to annotate the position description of the known foreground image in the known background image and generate a guiding prompt for the position description; the annotated known background image and the known foreground image are used to generate the training set;

[0056] Training module 22 is used to train the synthetic model with the training set to generate multiple prediction heads and scorers for the synthetic model.

[0057] Image synthesis module 30 is used to synthesize the target background image and the target foreground image into a target image based on the common knowledge using the synthesis model; Figure 7 This is a schematic diagram of the image synthesis module, as shown below. Figure 7 As shown, the image synthesis module 30 includes:

[0058] Description module 31 is used to generate image description information of the target foreground image placed in the target background image using the synthesis model;

[0059] The scoring module 32 is used to generate at least one set of description triples of the target foreground image placed in the target background image based on the image description information, and to select a target triple from the description triples using the scorer of the synthetic model.

[0060] The texture module 33 is used to obtain the target image by texture based on the target triplet.

[0061] In a third embodiment of the present invention, a computer-readable storage medium is provided. The common-sense-enhanced image synthesis apparatus of the present invention, if its functions are implemented as software functional units and sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a computer-readable storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. Therefore, in the third embodiment of the present invention, a computer-readable storage medium is provided for storing a computer program that performs a common-sense-enhanced image synthesis method. It should be understood that the computer-readable storage medium in the embodiments of the present invention can be volatile memory and / or non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0062] Figure 8 This is a schematic diagram of an electronic device according to the present invention. Figure 8As shown, in the fourth embodiment of the present invention, an electronic device is proposed, including the common-sense enhanced image synthesis apparatus as described above. Those skilled in the art will understand that all or part of the steps in the above methods can be implemented by a program instructing related hardware (e.g., processor, FPGA, ASIC, etc.). All or part of the steps in the above embodiments can also be implemented using one or more integrated circuits. Accordingly, each module in the above embodiments can be implemented in hardware, for example, by implementing its corresponding function through an integrated circuit, or it can be implemented as a software functional module, for example, by a processor executing a program / instruction stored in memory to implement its corresponding function. The embodiments of the present invention are not limited to any particular combination of hardware and software.

[0063] It should be noted that the structure of the electronic device shown in the accompanying drawings of this invention does not constitute a limitation thereof. The actual knowledge structure recognition device may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0064] The electronic device of the present invention can be any device with data processing capabilities, such as a computer or other similar device. The device embodiment can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution. Figure 9 This is a schematic diagram of the hardware structure of an electronic device according to the present invention. Figure 9 As shown, from a hardware perspective, this is a hardware structure diagram of any device with data processing capabilities, including the image synthesis device for enhancing common knowledge of this invention. (Except for...) Figure 9 In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0065] When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0066] The common-sense-enhanced image synthesis method proposed in this invention has the following advantages:

[0067] 1. Common sense-enhanced object placement and synthesis method: By using a multimodal large model to learn the language description of the correct position labeled by humans, and describing the specific position by "thinking" in the prediction stage, the position parameters can be predicted on the basis of this, so as to obtain a more reasonable placement position and ultimately generate a more harmonious image.

[0068] 2. Object position prediction function that meets multiple needs: The method is designed with both a scorer and a multi-regression prediction head. When multiple possible results are required, each prediction head can provide position parameters with high reasonableness. When the best prediction result is required, the scorer can identify the best candidate option among multiple candidate position parameters, so that the method has the ability to meet multiple needs at the same time.

[0069] 3. Training efficiency: Compared with other methods, this method uses a simple loss function and a non-adversarial model, with few hyperparameters, avoiding complex training steps. The model training and convergence are both fast, saving computational overhead and time costs.

[0070] The above embodiments are only used to illustrate the present invention and are not intended to limit the present invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, all equivalent technical solutions also fall within the scope of the present invention, and the patent protection scope of the present invention should be defined by the claims.

Claims

1. A common sense enhanced image synthesis method, characterized by, The method comprises: a data preparation step of selecting a scene description model to process known multi-modal data to obtain common sense knowledge of a placement position of an object in an environment; a model training step of labeling a position description of a known foreground image in a known background image and generating a guiding prompt for the position description; The known background image and the known foreground image after being labeled are used to generate a training set; the synthetic model is trained by using the training set, and a plurality of prediction heads and a scorer of the synthetic model are generated, and a training target ; λ1, λ2 are weight coefficients, L cap is a target function for generating the position description, i is a true probability distribution of the i-th word of the position description, gt is a predicted probability distribution of the i-th word; L gt gt.pos gt.neg gt.pos gt.neg gt is a target function for limiting the prediction result to approach the positive sample, Lis a target function for limiting the prediction result to be far away from the negative sample, Nis a number of negative samples, (r, x, y) is a description triplet, r is a scaling parameter of the known foreground image in the known background image, x, y are position parameters of the known foreground image in the known background image, and δ is a safety boundary between the prediction result and the negative sample.​​​​​ ， ； an image synthesis step of generating, by using the synthesis model, image description information of a target foreground image placed in a target background image; generating at least one set of description triplets of the target foreground image placed in the target background image according to the image description information, and selecting a target triplet from the description triplets by using a scorer of the synthesis model; and obtaining a target image by pasting according to the target triplet.

2. The common sense augmented image compositing method of claim 1, wherein, The data preparation step further comprises: an annotation step of performing position annotation on the known multi-modal data, wherein the position annotation comprises concise annotation and detailed annotation, the concise annotation comprises basic information of an object to be placed, and the detailed annotation comprises logical information of a placement behavior of the object.

3. A common sense enhanced image compositing apparatus characterized by comprising: The method comprises: a data preparation module configured to select a scene description model to process known multi-modal data to obtain common sense knowledge of a placement position of an object in an environment; a model training module configured to label a position description of a known foreground image in a known background image and generate a guiding prompt for the position description; The known background image and the known foreground image are used to generate a training set. The generative model is trained using the training set to generate a plurality of prediction heads and scorers of the generative model, and a training target ; λ1, λ2 are weight coefficients, L cap is a target function for generating the position description, , w i is a real probability distribution of an i-th word of the position description, is a predicted probability distribution of the i-th word; L gt is a target function of L gt = L gt.pos + L gt.neg , L gt.pos is a target function in which the prediction result is limited to approach the positive sample, L gt.neg is a target function in which the prediction result is limited to be away from the negative sample, N gt is a number of negative samples, (r, x, y) is a description triplet, r is a scaling parameter of the known foreground image in the known background image, x and y are position parameters of the known foreground image in the known background image, and δ is a safety boundary between the prediction result and the negative sample. ， ； an image synthesis module configured to generate, by using the synthesis model, image description information of a target foreground image placed in a target background image; generate at least one set of description triplets of the target foreground image placed in the target background image according to the image description information, and select a target triplet from the description triplets by using a scorer of the synthesis model; and obtain a target image by pasting according to the target triplet.

4. The common sense augmented image compositing apparatus of claim 3, wherein, The data preparation module comprises: an annotation module configured to perform position annotation on the known multi-modal data, wherein the position annotation comprises concise annotation and detailed annotation, the concise annotation comprises basic information of an object to be placed, and the detailed annotation comprises logical information of a placement behavior of the object.

5. A computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, cause the processor to perform operations comprising: When the computer executable instructions are executed, the method for synthesizing an image with common sense enhancement is implemented.

6. An electronic device comprising the apparatus for synthesizing an image with common sense enhancement according to claim 3 or 4.

Images