Model training method, image synthesis method, device and electronic equipment

Abstract

The embodiment of the present application provides a model training method, an image synthesis method, a device and an electronic device, which relates to the field of computer technology. The model training method includes: obtaining multiple groups of sample data; inputting the image data in the multiple groups of sample data into the image synthesis model to be trained, and obtaining the image synthesis result and predicted position information corresponding to each group of sample data; determining the target loss of the image synthesis model based on the sample position information in each group of sample data, and the image synthesis result and predicted position information corresponding to each group of sample data; and adjusting the model parameters of the image synthesis model when it is determined that the image synthesis model has not converged based on the target loss. Through this solution, an image synthesis model that can automatically synthesize images without manual intervention can be trained, thereby providing a basis for automatic image synthesis to improve synthesis efficiency.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a model training method, an image synthesis method, a device and an electronic device. Background Art

[0002] Image synthesis can synthesize objects in an image (original foreground image) into another image (original background image), and can be used in a variety of fields such as data augmentation, scene synthesis, and artistic creation. In addition, deep learning models are usually highly dependent on training data. The more training data there is, the higher the performance of the trained deep learning model. Therefore, in order to improve the performance of deep learning models, image synthesis is usually performed on existing image data to achieve data augmentation and obtain more training data.

[0003] In the related art, a pre-trained image synthesis model is usually used to synthesize a foreground image into a background image to obtain a synthesized image. The foreground image is an image obtained by separating the foreground object in the original foreground image and containing the foreground object, and the background image is an image formed after the background object in the original background image is removed.

[0004] However, in the related art, during the application process of the pre-trained image synthesis model, the position of the foreground image to be synthesized into the background image needs to be manually given each time, so that the image synthesis model performs image synthesis according to the given position.

[0005] It can be seen that how to provide a model training method to train an image synthesis model that can automatically synthesize images without the need for manual work is a technical problem that needs to be solved urgently. Summary of the invention

[0006] The purpose of the embodiments of the present application is to provide a model training method, an image synthesis method, a device and an electronic device to train an image synthesis model that can automatically synthesize images without manual intervention. The specific technical solution is as follows:

[0007] In a first aspect, an embodiment of the present application provides a model training method, the method comprising:

[0008] Acquire multiple groups of sample data; wherein each group of sample data includes image data and sample position information, the image data includes a foreground image, a background image and a composite image, and the sample position information is information of a composite position based on which the foreground image and the background image in the corresponding sample data are composited to obtain the composite image;

[0009] Input the image data in the plurality of groups of sample data into the image synthesis model to be trained, and obtain the image synthesis result and predicted position information corresponding to each group of sample data; wherein the image synthesis model includes a position prediction network and an image synthesis network; the position prediction network is used to generate, for each group of sample data, a feature fusion result of a foreground image and a background image in the group of sample data, obtain a fusion feature corresponding to the group of sample data, and determine the predicted position information corresponding to the group of sample data based on the obtained fusion feature; the image synthesis network is used to predict the image synthesis result for each group of sample data based on the foreground image, background image and synthesized image in the group of sample data, and using the fusion feature corresponding to the group of sample data as auxiliary information; the predicted position information corresponding to the group of sample data is the information of the predicted synthesis position based on which the foreground image and the background image in the group of sample data are synthesized;

[0010] Determining a target loss of the image synthesis model based on sample position information in each set of sample data, and an image synthesis result and predicted position information corresponding to each set of sample data;

[0011] When it is determined based on the target loss that the image synthesis model has not converged, model parameters of the image synthesis model are adjusted.

[0012] In a second aspect, an embodiment of the present application provides an image synthesis method, the method comprising:

[0013] Obtain a target background image and a target foreground image;

[0014] Acquire a synthetic image template that matches the size of the target background image; wherein the synthetic image template is a noise image;

[0015] Inputting the target background image, the target foreground image and the synthetic image template into an image synthesis model, so that the image synthesis model performs image synthesis processing based on the target background image, the target foreground image and the synthetic image template;

[0016] The image synthesis result output by the image synthesis model is obtained to obtain a synthesized image after the target background image and the target foreground image are synthesized; wherein the image synthesis model is a model trained according to the model training method.

[0017] In a third aspect, an embodiment of the present application provides a model training device, the device comprising:

[0018] A first acquisition module is used to acquire multiple groups of sample data; each group of sample data includes image data and sample position information, the image data includes a foreground image, a background image and a composite image, and the sample position information is information of a composite position based on which the foreground image and the background image in the sample data are composited to obtain the composite image;

[0019] A first input module is used to input the image data in the plurality of groups of sample data into the image synthesis model to be trained, and obtain the image synthesis result and predicted position information corresponding to each group of sample data; wherein the image synthesis model includes a position prediction network and an image synthesis network; the position prediction network is used to generate, for each group of sample data, a feature fusion result of a foreground image and a background image in the group of sample data, obtain a fusion feature corresponding to the group of sample data, and determine the predicted position information corresponding to the group of sample data based on the obtained fusion feature; the image synthesis network is used to predict the image synthesis result for each group of sample data based on the foreground image, background image and synthesized image in the group of sample data, and using the fusion feature corresponding to the group of sample data as auxiliary information; the predicted position information corresponding to the group of sample data is the information of the predicted synthesis position based on which the foreground image and the background image in the group of sample data are synthesized;

[0020] A determination module, configured to determine a target loss of the image synthesis model based on the synthesized image and sample position information in each set of sample data, and the image synthesis result and predicted position information corresponding to each set of sample data;

[0021] An adjustment module is used to adjust the model parameters of the image synthesis model when it is determined based on the target loss that the image synthesis model has not converged.

[0022] In a fourth aspect, an embodiment of the present application provides an image synthesis device, the device comprising:

[0023] The second acquisition module is used to acquire a target background image and a target foreground image;

[0024] A third acquisition module is used to acquire a synthetic image template that matches the size of the target background image; wherein the synthetic image template is a noise image;

[0025] A second input module, used for inputting the target background image, the target foreground image and the synthetic image template into the image synthesis model, so that the image synthesis model performs image synthesis processing based on the target background image, the target foreground image and the synthetic image template;

[0026] The fourth acquisition module is used to obtain the image synthesis result output by the image synthesis model to obtain a composite image after the target background image and the target foreground image are synthesized; wherein the image synthesis model is a model trained according to the model training method.

[0027] In a fifth aspect, an embodiment of the present application provides an electronic device, including:

[0028] Memory, used to store computer programs;

[0029] The processor is used to implement any of the model training methods or image synthesis methods when executing the program stored in the memory.

[0030] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements any of the model training methods or image synthesis methods described above.

[0031] An embodiment of the present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the above-mentioned model training methods or image synthesis methods.

[0032] Beneficial effects of the embodiments of the present application:

[0033] In the model training method provided by the embodiment of the present application, each group of sample data includes image data and sample position information, the image data includes a foreground image, a background image and a synthesized image, and the sample position information is the synthesis position information based on which the foreground image and the background image in the sample data to which it belongs are synthesized to obtain the synthesized image; the present application can first obtain multiple groups of sample data, and input the image data in the multiple groups of sample data into the image synthesis model to be trained, and the position prediction network in the image synthesis model generates the feature fusion result of the foreground image and the background image in the group of sample data for each group of sample data, obtains the fusion feature corresponding to the group of sample data, and determines the predicted position information corresponding to the group of sample data based on the obtained fusion feature. Since the fusion feature corresponding to each group of sample data can imply the synthesis position information based on which the foreground image and the background image in the group of sample data are synthesized, the image synthesis network predicts the image synthesis result based on the foreground image, the background image and the synthesized image in the group of sample data, using the fusion feature corresponding to the group of sample data as auxiliary information. In the present application, the position prediction network in the image synthesis model can generate fusion features corresponding to each set of sample data, and the fusion features corresponding to each set of sample data can implicitly contain the synthesis position information based on which the foreground image and the background image in the set of sample data are synthesized. The position prediction network can determine the predicted position information corresponding to the set of sample data based on the fusion features corresponding to each set of sample data; the image synthesis network can use the fusion features corresponding to each set of sample data as auxiliary information to predict the image synthesis result, that is, the fusion features corresponding to each set of sample data, which implicitly contain the synthesis position information based on which the foreground image and the background image in the set of sample data are synthesized, can act on the image synthesis process; and it is not necessary to manually give the position of the foreground image synthesized into the background image every time. Furthermore, the target loss of the image synthesis model can be determined based on the sample position information in each set of sample data, as well as the image synthesis result and predicted position information corresponding to each set of sample data. When it is determined based on the target loss that the image synthesis model has not converged, the model parameters of the image synthesis model are adjusted to obtain a trained image synthesis model. It can be seen that through this solution, an image synthesis model that can automatically synthesize images without the help of manual work can be trained, thereby providing a basis for automatic image synthesis to improve synthesis efficiency.

[0034] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0036] Figure 1a A schematic diagram of the process of texture synthesis provided in an embodiment of the present application;

[0037] Figure 1b A schematic diagram of a flow chart of a generation model provided in an embodiment of the present application;

[0038] Figure 2 A flowchart of a model training method provided in an embodiment of the present application;

[0039] Figure 3 A schematic diagram of a flow chart of an image synthesis method provided in an embodiment of the present application;

[0040] Figure 4 A schematic diagram of a process for constructing a data set provided in an embodiment of the present application;

[0041] Figure 5 A schematic diagram of a process of data amplification during data set construction provided in an embodiment of the present application;

[0042] Figure 6 A schematic diagram of the training process of the image synthesis model provided in an embodiment of the present application;

[0043] Figure 7 A schematic diagram of a process of performing image synthesis using an image synthesis model provided in an embodiment of the present application;

[0044] Figure 8 A schematic diagram of a process for foreground separation provided in an embodiment of the present application;

[0045] Fig. 9 A schematic diagram of the background repair process provided in the embodiment of the present application;

[0046] Fig.10 A schematic diagram of the process of synthesizing a new image provided in an embodiment of the present application;

[0047] Fig.11 A schematic diagram of a specific structure of an image synthesis model provided in an embodiment of the present application;

[0048] Fig.12 A schematic diagram of the structure of a model training device provided in an embodiment of the present application;

[0049] Fig.13A schematic diagram of the structure of an image synthesis device provided in an embodiment of the present application;

[0050] Fig.14 A block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0052] For ease of understanding, the professional terms involved in this application are first introduced below.

[0053] Image Composition: A technique in the field of computer graphics and computer vision that involves combining multiple images or parts of images together, for example: compositing an object in an image (the original foreground image) into another image (the original background image) to create a new image. Image synthesis can be used in a variety of fields such as data augmentation, scene synthesis, and artistic creation.

[0054] Diffusion Model: A generative model that generates new data by simulating the diffusion process of data. The diffusion model includes two main processes: forward diffusion process and reverse denoising process. The diffusion model can generate high-quality and diverse data, so it has a wide range of applications in image synthesis, data completion, denoising and other fields.

[0055] Deep learning performs well in classification, segmentation, detection, and recognition tasks. However, to avoid overfitting of deep learning models, deep learning models are usually highly dependent on training data. When the training data is small and unbalanced, the performance of deep learning models will be greatly reduced. Generally speaking, the final performance of deep learning models is positively correlated with the amount of data. At the same time, when the performance of deep learning models is high, the amount of data required to improve the same performance is increasing. In order to improve the performance of deep learning models, data augmentation (image synthesis) is usually performed on existing data to obtain more training data.

[0056] There are two main approaches to image synthesis:

[0057] 1) Map synthesis: Through the object detection task, the bbox of the original foreground image (i.e., the position box of the foreground object in the original foreground image) can be cropped to obtain the foreground image; then the position of the map to the background image is predicted by the position prediction model, and the foreground image is mapped to the background image using this position to obtain the map image. Map synthesis is simple and easy, but the obtained map image has very poor authenticity and is difficult to use for subsequent training (difficult to use for training deep learning models). Therefore, some synthesis techniques are usually added to the map image, such as edge harmony, shadow supplementation, background fusion, etc., to obtain the final synthesized image, but these synthesis techniques are usually cumbersome, and different synthesis techniques usually require training different models.

[0058] 2) Generative model: Use VAE (Variational Auto-Encoder), GAN (Generative Adversarial Networks) or diffusion model to input foreground and background images, and then directly generate synthetic images. However, this method requires a large amount of training data and is not applicable when there is a lack of data. However, if you have a large amount of training data and want to customize and optimize for certain scenarios, then using a generative model for image synthesis is a good choice.

[0059] The following introduces the map synthesis and generation models by comparison.

[0060] The technical comparison of map synthesis and generation model is shown in Table 1:

[0061] Table 1

[0062]

[0063] The advantage of map synthesis technology is that most models (such as object detection models, etc.) are highly versatile, so the data volume requirements are moderate; however, there are some defects. For example, the obtained synthetic images have poor authenticity and the process of map synthesis is cumbersome. The applicable scenarios of map synthesis are scenarios with a small amount of data to train the model (such as the object detection model required in the process of map synthesis) and scenarios with low performance requirements, that is, low requirements on the quality of the synthesized images.

[0064] The advantage of generative model technology is its high authenticity and simple process; however, the disadvantage is that training a generative model requires manually giving the synthesis positions of a large amount of training data in order to obtain a converged synthetic model. Therefore, the applicable scenario of the generative model is a scenario where you have a large amount of training data and want to customize and optimize it for certain scenarios.

[0065] Specifically, the process of texture synthesis and model generation is as follows Figure 1a and Figure 1b As shown, it can be seen that in the mapping synthesis technology, for the foreground image and the background image (obtained by segmenting the original foreground image and the original background image respectively through the object detection task), the foreground image is first placed at the appropriate position of the background image through object placement, and the steps of image mixing, image coordination and shadow generation are performed to obtain the final synthetic image. Among them, image mixing and image coordination are the above-mentioned synthesis technologies such as edge harmony and background fusion, and shadow generation is the above-mentioned synthesis technology of shadow supplementation; it can be seen that the authenticity of the mapping image obtained by the mapping synthesis technology is poor. If some synthesis technologies are used to generate synthetic images, the steps are relatively cumbersome.

[0066] In the generative model technology, for the foreground image and the background image, the foreground image and the background image can be directly input into the generative model, and the generative model synthesizes the foreground image and the background image based on the artificially given position C, and can directly obtain the synthesized image. Among them, the generative model requires a large amount of labeled data with positions for training, and the positions in the labeled data are artificially given synthetic positions, which is costly.

[0067] According to the above comparison, mapping synthesis and generative modeling each have their own advantages, but they also have their own problems. Based on this, the embodiments of the present application provide a model training method, an image synthesis method, a device, and an electronic device, which combine mapping synthesis and generative modeling technologies, effectively utilize their respective advantages, avoid each other's shortcomings, and train an image synthesis model that automatically synthesizes images without the need for manual labor.

[0068] The following is an introduction to a model training method provided by this application.

[0069] Among them, a model training method provided by the present application can be applied to an electronic device, which can be a client, or a server, and the client can be a mobile phone or a computer, etc. The present application does not limit the specific form of the electronic device. In addition, a model training method provided in an embodiment of the present application can be applied to the training scenario of an image synthesis model that automatically synthesizes images without the help of artificial means. The trained image synthesis model can be applied to a variety of image synthesis scenarios such as data augmentation, scene synthesis, and artistic creation. The present application does not limit the specific scenario.

[0070] In addition, the image synthesis model trained in the present application may include a diffusion model. The diffusion model may be a module in the image synthesis model in the present application. In order to realize automatic image synthesis without manual assistance, the image synthesis model trained in the present application also includes a position prediction network.

[0071] Among them, an embodiment of the present application provides a model training method, the method comprising:

[0072] Acquire multiple groups of sample data; wherein each group of sample data includes image data and sample position information, the image data includes a foreground image, a background image and a composite image, and the sample position information is information of a composite position based on which the foreground image and the background image in the corresponding sample data are composited to obtain the composite image;

[0073] Input the image data in the plurality of groups of sample data into the image synthesis model to be trained, and obtain the image synthesis result and predicted position information corresponding to each group of sample data; wherein the image synthesis model includes a position prediction network and an image synthesis network; the position prediction network is used to generate, for each group of sample data, a feature fusion result of a foreground image and a background image in the group of sample data, obtain a fusion feature corresponding to the group of sample data, and determine the predicted position information corresponding to the group of sample data based on the obtained fusion feature; the image synthesis network is used to predict the image synthesis result for each group of sample data based on the foreground image, background image and synthesized image in the group of sample data, and using the fusion feature corresponding to the group of sample data as auxiliary information; the predicted position information corresponding to the group of sample data is the information of the predicted synthesis position based on which the foreground image and the background image in the group of sample data are synthesized;

[0074] Determining a target loss of the image synthesis model based on sample position information in each set of sample data, and an image synthesis result and predicted position information corresponding to each set of sample data;

[0075] When it is determined based on the target loss that the image synthesis model has not converged, model parameters of the image synthesis model are adjusted.

[0076] In the model training method provided by the embodiment of the present application, each group of sample data includes image data and sample position information, the image data includes a foreground image, a background image and a synthesized image, and the sample position information is the synthesis position information based on which the foreground image and the background image in the sample data to which it belongs are synthesized to obtain the synthesized image; the present application can first obtain multiple groups of sample data, and input the image data in the multiple groups of sample data into the image synthesis model to be trained, and the position prediction network in the image synthesis model generates the feature fusion result of the foreground image and the background image in the group of sample data for each group of sample data, obtains the fusion feature corresponding to the group of sample data, and determines the predicted position information corresponding to the group of sample data based on the obtained fusion feature. Since the fusion feature corresponding to each group of sample data can imply the synthesis position information based on which the foreground image and the background image in the group of sample data are synthesized, the image synthesis network predicts the image synthesis result based on the foreground image, the background image and the synthesized image in the group of sample data, using the fusion feature corresponding to the group of sample data as auxiliary information. In the present application, the position prediction network in the image synthesis model can generate fusion features corresponding to each set of sample data, and the fusion features corresponding to each set of sample data can implicitly contain the synthesis position information based on which the foreground image and the background image in the set of sample data are synthesized. The position prediction network can determine the predicted position information corresponding to the set of sample data based on the fusion features corresponding to each set of sample data; the image synthesis network can use the fusion features corresponding to each set of sample data as auxiliary information to predict the image synthesis result, that is, the fusion features corresponding to each set of sample data, which implicitly contain the synthesis position information based on which the foreground image and the background image in the set of sample data are synthesized, can act on the image synthesis process; and it is not necessary to manually give the position of the foreground image synthesized into the background image every time. Furthermore, the target loss of the image synthesis model can be determined based on the sample position information in each set of sample data, as well as the image synthesis result and predicted position information corresponding to each set of sample data. When it is determined based on the target loss that the image synthesis model has not converged, the model parameters of the image synthesis model are adjusted to obtain a trained image synthesis model. It can be seen that through this solution, an image synthesis model that can automatically synthesize images without the help of manual work can be trained, thereby providing a basis for automatic image synthesis to improve synthesis efficiency.

[0077] The following is an exemplary introduction to a model training method provided in this application in conjunction with the accompanying drawings.

[0078] like Figure 2 As shown, a model training method provided in an embodiment of the present application may include the following steps:

[0079] S201: Acquire multiple groups of sample data;

[0080] Each set of sample data includes image data and sample position information, the image data includes a foreground image, a background image and a composite image, and the sample position information is information of a composite position based on which the foreground image and the background image in the sample data are composited to obtain the composite image;

[0081] In the present application, multiple sets of sample data are pre-constructed for the image synthesis model to be trained. Before training the image synthesis model to be trained, multiple sets of sample data can be obtained. Each set of sample data includes image data and sample position information. The image data includes a foreground image, a background image, and a synthesized image. The image data of each set of sample data can be used to train the image synthesis model to be trained. The sample position information of each set of sample data is the synthesis position information based on the foreground image and the background image in the sample data to which it belongs when performing image synthesis to obtain a synthesized image.

[0082] In addition, the foreground image can be understood as an image containing a foreground object obtained after foreground separation of the corresponding original foreground image, and the background image can be understood as an image after the foreground object is removed from the corresponding original background image; of course, the area in the foreground image other than the foreground object can be a mask area, and the mask area in the background image where the foreground object is removed can be integrated into the background of the background image through image restoration, so as to perform image synthesis through the foreground image and the background image.

[0083] It should be noted that the sample position information in each set of sample data does not directly affect the training process of the image synthesis model to be trained, but serves as supervision and can be used for the calculation of the target loss of the subsequent image synthesis model; and the sample position information in each set of sample data can be automatically determined according to a predetermined position determination strategy, or can be given manually, and this application does not limit this.

[0084] The construction method of multiple groups of sample data and the predetermined position determination strategy will be described in detail in subsequent embodiments and will not be elaborated here.

[0085] S202: inputting the image data in the plurality of groups of sample data into the image synthesis model to be trained, and obtaining the image synthesis result and predicted position information corresponding to each group of sample data;

[0086] The image synthesis model includes a position prediction network and an image synthesis network; the position prediction network is used to generate, for each set of sample data, a feature fusion result of a foreground image and a background image in the set of sample data, obtain a fusion feature corresponding to the set of sample data, and determine the predicted position information corresponding to the set of sample data based on the obtained fusion feature; the image synthesis network is used to predict the image synthesis result for each set of sample data based on the foreground image, background image and synthesized image in the set of sample data, and using the fusion feature corresponding to the set of sample data as auxiliary information; the predicted position information corresponding to the set of sample data is the information of the predicted synthesis position based on which the foreground image and the background image in the set of sample data are synthesized;

[0087] After obtaining multiple groups of sample data, the image data in the multiple groups of sample data can be input into the image synthesis model to be trained, and the image synthesis model can output the image synthesis result and predicted position information corresponding to each group of sample data.

[0088] Specifically, the image synthesis model may include a position prediction network and an image synthesis network. The position prediction network may learn, for each group of sample data, the position and size relationship of the foreground image in the group of sample data in the background image, generate a feature fusion result of the foreground image and the background image in the group of sample data, and obtain a fusion feature corresponding to the group of sample data. The fusion feature corresponding to the group of sample data implicitly contains information on the predicted synthesis position based on which the foreground image and the background image in the group of sample data are synthesized. Of course, the position prediction network may also determine the position prediction information corresponding to the group of sample data based on the obtained fusion feature. The position prediction information is the predicted synthesis position based on which the foreground image and the background image in the group of sample data are synthesized.

[0089] It should be noted that the fusion features of each group of sample data do not directly represent the information of the synthesis position, but implicitly contain the information of the synthesis position, which can act on the image synthesis network as a feature indicating the synthesis position to assist the image synthesis of the image synthesis network; the predicted position information corresponding to each group of sample data is the predicted information of the synthesis position based on which the foreground image and background image in the group of sample data are synthesized. The predicted position information corresponding to each group of sample data and the sample position information in each group of sample data are used to calculate the target loss of the image synthesis model. At this time, the predicted position information is the synthesis position predicted by the image synthesis model, and the sample position information can be understood as the calibrated true value of the synthesis position.

[0090] For each set of sample data, the image synthesis network predicts the image synthesis result based on the foreground image, background image and synthesized image in the set of sample data, and uses the fusion features corresponding to the set of sample data as auxiliary information, and the synthesis position implied by the fusion features, that is, the image synthesis result corresponding to the set of sample data is obtained. The image synthesis result corresponding to each set of sample data can be used to calculate the target loss of the image synthesis model. At this time, the image synthesis result is the result of image synthesis of the foreground image and background image predicted by the image synthesis model.

[0091] The predicted position information output by the position prediction network does not directly affect the image synthesis in the image synthesis network. The fusion features obtained in the position prediction network can be used as auxiliary information that implies the synthesis position to instruct the image synthesis network to perform image synthesis. The predicted position information output by the position prediction network can be used for loss calculation of the image synthesis model, and this application does not limit this.

[0092] Exemplarily, in one implementation of the present application, the image synthesis network includes a first encoder, a splicing module, and an image synthesis module;

[0093] The first encoder is used to compress the foreground image, the background image and the synthesized image after adding noise in each group of sample data, and input the compressed image into the stitching module;

[0094] The stitching module is used to stitch the compressed images in the image channel dimension and input the stitched image obtained by stitching into the image synthesis module;

[0095] The image synthesis module is used to receive the stitched image, calculate the cross-attention value of the fusion feature corresponding to the group of sample data, obtain the calculation result corresponding to the group of sample data, and use the calculation result as a guidance content for the synthesis position, perform image synthesis processing based on the received stitched image, and obtain the image synthesis result corresponding to the group of sample data.

[0096] The image synthesis module can receive a stitched image, calculate the cross-attention value of the fusion feature corresponding to the group of sample data, obtain the calculation result of the information corresponding to the group of sample data, which represents the predicted synthesis position based on the foreground image and background image in the group of sample data, and use the calculation result as a guide for the synthesis position, perform image synthesis processing based on the received stitched image, and obtain the image synthesis result corresponding to the group of sample data.

[0097] Among them, the first encoder can add noise to the synthesized image. Exemplarily, the synthesized image can be denoised according to the current time t. The specific implementation method can be similar to the prior art and will not be described in detail here.

[0098] In addition, when the image synthesis model performs image synthesis processing, it is equivalent to considering the information of the synthesis position based on which the foreground image and the background image in the set of sample data are predicted by the calculation results. Therefore, it is not necessary to manually give the synthesis position. Based on the synthesis position represented by the calculation results, the spliced ​​image can be subjected to image synthesis processing of the forward diffusion process and the reverse denoising process to obtain the image synthesis result obtained by synthesizing the foreground image and the background image according to the synthesis position represented by the calculation results. It can be understood that the obtained image synthesis result is a highly realistic synthetic image, and no further processing of synthesis techniques such as edge harmony, shadow supplementation, and background fusion is required.

[0099] Optionally, the first encoder is a compression module VAE in a predetermined diffusion model, and the image synthesis module is a main structure Unet in the diffusion model.

[0100] The first encoder can be a compression module VAE in a predetermined diffusion model, which can add noise to the synthetic image in each set of sample data according to time, and compress the foreground image, background image and noisy synthetic image in each set of sample data, and input the compressed image to the stitching module.

[0101] The image synthesis module may be the main structure Unet in the diffusion model. The main structure Unet is a symmetrical structure that can realize image synthesis processing of the forward diffusion process and the reverse denoising process on the spliced ​​image. The reverse denoising process is the process of generating a new synthetic image.

[0102] Exemplarily, in one implementation, the position prediction network includes a second encoder, a position prediction module, and a fully connected layer;

[0103] The second encoder is used for extracting features of the foreground image and the background image in each group of sample data, respectively, and inputting the extracted foreground image features and background image features into the position prediction module;

[0104] The position prediction module is used to generate a feature fusion result of the foreground image and the background image in the group of sample data based on the received foreground image features and background image features, obtain a fusion feature corresponding to the group of sample data, and input the obtained fusion feature corresponding to the group of sample data into the fully connected layer;

[0105] The fully connected layer is used to perform position prediction processing on the fusion features corresponding to the group of sample data to obtain predicted position information corresponding to the group of sample data.

[0106] The position prediction module first learns the position and size relationship of the foreground image in the background image based on the received foreground image features and background image features, generates a fusion feature of the foreground image and the background image in the group of sample data that implicitly contains the synthesis position, and inputs the fusion feature corresponding to the group of sample data into the fully connected layer; the fully connected layer performs position prediction processing according to the fusion feature corresponding to the group of sample data to obtain specific predicted position information corresponding to the group of sample data.

[0107] It should be noted that, in another implementation, the fully connected layer may belong to the position prediction module, that is, the fully connected layer may be integrated into the position prediction module, and the position prediction module may generate a fusion feature of the foreground image and the background image in the group of sample data that implicitly contains a synthetic position, and perform position prediction processing according to the fusion feature corresponding to the group of sample data through the fully connected layer in the position prediction module to obtain predicted position information; at this time, the position prediction module may output the fusion feature corresponding to the group of sample data and the predicted position information.

[0108] In addition, it should be noted that the predicted position information can be multi-dimensional coordinate information about position and size, for example: (x, y, h, w), where x and y represent the two-dimensional coordinate position of the synthesis position in the background image, which can be the coordinates of the upper left corner or the lower right corner of the synthesis position in the background image, but are not limited to this; h and w represent the height and width information of the foreground image at the synthesis position. Moreover, the extracted foreground image features and background image features of the group of sample data, as well as the fusion features corresponding to the group of sample data, have the same number of dimensions, and the dimensions correspond to each other, for example: the foreground image features and the background image features are both 1024-dimensional, and the obtained fusion features are fusion features of 1024 dimensions obtained by fusing the foreground image features and the background image features in the corresponding 1024 dimensions.

[0109] S203: Determine the target loss of the image synthesis model based on the sample position information in each set of sample data, and the image synthesis result and predicted position information corresponding to each set of sample data;

[0110] After obtaining the image synthesis results and position prediction information corresponding to each group of sample data through the image synthesis model, the target loss of the image synthesis model can be determined based on the sample position information in each group of sample data, as well as the image synthesis results and predicted position information corresponding to each group of sample data.

[0111] It should be noted that the target loss of the image synthesis model can be calculated based on the network loss of the position prediction network and the network loss of the image synthesis network. The network loss of the image synthesis network can be calculated based on the image synthesis result corresponding to each group of sample data. The network loss of the position prediction network can be calculated based on the sample position information in each group of sample data and the predicted position information corresponding to each group of sample data, thereby calculating the target loss of the image synthesis model.

[0112] Specifically, determining the target loss of the image synthesis model based on the sample position information in each set of sample data, and the image synthesis result and predicted position information corresponding to each set of sample data includes:

[0113] Determine a first type of loss based on the image synthesis result corresponding to each set of sample data; wherein the first type of loss is used to characterize the network loss of the image synthesis network;

[0114] Determine a second type of loss based on the sample location information in each set of sample data and the predicted location information corresponding to each set of sample data; wherein the second type of loss is used to characterize the network loss of the location prediction network;

[0115] The first type of loss and the second type of loss are subjected to predetermined operation processing to obtain a target loss of the image synthesis model.

[0116] When determining the target loss of the image synthesis model, the first type of loss of the image synthesis network can be determined based on the image synthesis result corresponding to each group of sample data; and the second type of loss of the position prediction network can be determined based on the sample position information in each group of sample data and the predicted position information corresponding to each group of sample data; and the target loss of the image synthesis model is obtained by performing predetermined operations on the first type of loss and the second type of loss.

[0117] Exemplarily, the predetermined operation processing may be direct addition processing, or respectively assigning corresponding importance coefficients to the first-category losses and the second-category losses, and then performing addition processing on the first-category losses and the second-category losses assigned with the importance coefficients, which are both reasonable.

[0118] In addition, when determining the first category of losses and the second category of losses, other features may need to be utilized, and this application does not limit this.

[0119] Optionally, determining the first type of loss based on the image synthesis result corresponding to each set of sample data includes:

[0120] Calculate the mean square error between the feature vector of the image synthesis result corresponding to each set of sample data and the random vector representing the standard Gaussian noise to obtain the first type of loss;

[0121] The standard Gaussian noise is Gaussian noise that conforms to the standard normal distribution.

[0122] When determining the first type of loss, the mean square error between the feature vector of the image synthesis result corresponding to each group of sample data and the random vector representing the standard Gaussian noise can be calculated to obtain the first type of loss; of course, since the synthesized image is denoised according to the current moment, and the fusion features corresponding to each group of sample data act on the generation process of the image synthesis result, the fusion features corresponding to the current moment and each group of sample data can also be considered when determining the first type of loss, that is, the fusion features corresponding to the current moment and each group of sample data can also act on the calculation of the first type of loss.

[0123] Specifically, the first category of losses can be determined according to the following formula:

[0124] ;

[0125] in, The first type of loss, Refers to calculating the expectation of the function or variable in the brackets. In the subscript parameter: z refers to the hidden variable of the original image to be synthesized (that is, the synthesized image in each set of sample data), is the fusion feature corresponding to each set of sample data, is a random vector representing standard Gaussian noise, The random vector representing the standard Gaussian noise is a random vector of Gaussian noise that conforms to the standard normal distribution. is the current moment. refers to the mean square error, where It can be understood as an image synthesis module (i.e., the main structure Unet in the diffusion model). For each set of sample data corresponding to the The noise predicted under (i.e. the image synthesis result predicted by each set of sample data), is the image after adding noise to the synthesized image at the current time t. In addition, it should be emphasized that the overall meaning of the above formula is "calculating the difference between the predicted noise of the diffusion model (i.e., the feature vector of the predicted image synthesis result) and the random vector of the standard Gaussian noise", which is used to reduce the difference between the predicted noise and the standard Gaussian noise.

[0126] Optionally, determining the second type of loss based on the sample position information in each group of sample data and the predicted position information corresponding to each group of sample data includes:

[0127] Based on the sample position information in each group of sample data and the predicted position information corresponding to each group of sample data, the mean square error loss is calculated according to a predetermined mean square error loss calculation method to obtain the second type of loss.

[0128] When determining the second type of loss, the mean square error loss may be calculated based on the sample position information in each group of sample data and the predicted position information corresponding to each group of sample data in accordance with a predetermined mean square error calculation method to obtain the second type of loss.

[0129] Specifically, based on the first eigenvector of the sample position information in each group of sample data and the second eigenvector of the predicted position information corresponding to each group of sample data, the mean square error loss can be calculated according to a predetermined mean square error calculation formula to obtain the second type of loss.

[0130] Exemplarily, in one implementation, the predetermined mean square error calculation formula is as follows:

[0131] ;

[0132] in, The second type of loss is is the calculation function of mean square error, is the first eigenvector representing the sample position information in each set of sample data, is the second eigenvector that represents the predicted position information corresponding to each set of sample data. Of course, It can be understood that after calculating the mean square error corresponding to each group of sample data, the mean square error of each group of sample data as a whole is obtained by summing and averaging.

[0133] After obtaining the first type of loss and the second type of loss, a predetermined operation process can be performed on the first type of loss and the second type of loss to obtain a target loss of the image synthesis model. For example, the target loss The calculation formula is as follows:

[0134] ;

[0135] It should be noted that the above calculation formulas for calculating the first type of loss, the second type of loss and the target loss are only examples and should not constitute a limitation on the present application.

[0136] S204: When it is determined based on the target loss that the image synthesis model has not converged, adjust model parameters of the image synthesis model.

[0137] After determining the target loss of the image synthesis model, it is possible to judge whether the image synthesis model has converged based on the target loss. If it has converged, such as the target loss is less than the preset threshold, the trained image synthesis model is directly obtained; if it has not converged, such as the target loss is not less than the preset threshold, it is necessary to continue training the image synthesis model. The network parameters of the position prediction network and the image synthesis network in the image synthesis model can be adjusted first, and multiple sets of sample data can be re-acquired for training.

[0138] Among them, for each sub-network in the position prediction network and the image synthesis network, the network parameters of the sub-network can be adjusted so that the image synthesis result output based on the adjusted network parameters is closer to the synthesized image in the group of sample data, and the predicted position information output based on the adjusted network parameters is closer to the sample position information in the group of sample data. The specific adjustment method of the network parameters can be similar to the prior art and will not be repeated here.

[0139] In the technical solution of the present application, the operations of acquiring, storing, using, processing, transmitting, providing and disclosing multiple sets of sample data, various features, various location information, etc., are all carried out with the user's authorization.

[0140] It should be noted that the multiple groups of sample data in this embodiment are from public data sets.

[0141] In the model training method provided by the embodiment of the present application, each group of sample data includes image data and sample position information, the image data includes a foreground image, a background image and a synthesized image, and the sample position information is the synthesis position information based on which the foreground image and the background image in the sample data to which it belongs are synthesized to obtain the synthesized image; the present application can first obtain multiple groups of sample data, and input the image data in the multiple groups of sample data into the image synthesis model to be trained, and the position prediction network in the image synthesis model generates the feature fusion result of the foreground image and the background image in the group of sample data for each group of sample data, obtains the fusion feature corresponding to the group of sample data, and determines the predicted position information corresponding to the group of sample data based on the obtained fusion feature. Since the fusion feature corresponding to each group of sample data can imply the synthesis position information based on which the foreground image and the background image in the group of sample data are synthesized, the image synthesis network predicts the image synthesis result based on the foreground image, the background image and the synthesized image in the group of sample data, using the fusion feature corresponding to the group of sample data as auxiliary information. In the present application, the position prediction network in the image synthesis model can generate fusion features corresponding to each set of sample data, and the fusion features corresponding to each set of sample data can implicitly contain the synthesis position information based on which the foreground image and the background image in the set of sample data are synthesized. The position prediction network can determine the predicted position information corresponding to the set of sample data based on the fusion features corresponding to each set of sample data; the image synthesis network can use the fusion features corresponding to each set of sample data as auxiliary information to predict the image synthesis result, that is, the fusion features corresponding to each set of sample data, which implicitly contain the synthesis position information based on which the foreground image and the background image in the set of sample data are synthesized, can act on the image synthesis process; and it is not necessary to manually give the position of the foreground image synthesized into the background image every time. Furthermore, the target loss of the image synthesis model can be determined based on the sample position information in each set of sample data, as well as the image synthesis result and predicted position information corresponding to each set of sample data. When it is determined based on the target loss that the image synthesis model has not converged, the model parameters of the image synthesis model are adjusted to obtain a trained image synthesis model. It can be seen that through this solution, an image synthesis model that can automatically synthesize images without the help of manual work can be trained, thereby providing a basis for automatic image synthesis to improve synthesis efficiency.

[0142] Exemplarily, in another embodiment of the present application, the plurality of groups of sample data are data in a pre-constructed training data set;

[0143] The training data set is constructed in the following manner:

[0144] The images in the foreground database and the images in the background database are combined according to a predetermined combination method to obtain multiple groups of initial data; wherein each group of initial data includes an original foreground image in the foreground database and an original background image in the background database;

[0145] For each group of initial data, determine the foreground image corresponding to the original foreground image in the group of initial data, and the background image corresponding to the original background image in the group of initial data; and map the determined foreground image to the determined background image according to the synthesis position corresponding to the group of initial data, and repair the mapped image to obtain the synthesized image corresponding to the group of initial data; wherein the synthesis position corresponding to the group of initial data is a position determined according to a predetermined position determination strategy and used to indicate that the determined foreground image is mapped to the determined background image; the foreground image corresponding to any original foreground image is: an image containing a foreground object obtained after foreground separation of the original foreground image, and the background image corresponding to any original background image is: an image after the foreground object is removed from the original background image;

[0146] For each group of initial data, a foreground image corresponding to an original foreground image in the group of initial data, a background image corresponding to an original background image in the group of initial data, and sample position information and a synthesized image corresponding to the group of initial data are constructed into a group of sample data; wherein the sample position information corresponding to the group of initial data is position information of a synthesized position corresponding to the group of initial data;

[0147] Construct a training data set containing each group of sample data.

[0148] In the present application, multiple groups of sample data are data in a pre-constructed training data set. In the present application, a foreground database and a background database are established. The foreground database is used to improve the diversity of foreground images, and the background database is used to improve the diversity of background images. First, the images in the foreground database and the images in the background database can be combined in a predetermined combination method, such as: permutation and combination, etc., to obtain multiple groups of initial data, each group of output data includes an original foreground image in the foreground database and a background image in the background database; in this way, the number of subsequent data pairs of foreground images and background images is greatly increased, and the data scale and diversity of multiple groups of sample data are increased. Of course, the foreground database and the background database can be interchangeable to further improve the data scale and diversity of multiple groups of sample data, that is, the foreground database is used as a new background database, and the background database is used as a new foreground database, which is reasonable.

[0149] It should be noted that each set of initial data is a unique data pair of an original foreground image and an original background image. An original foreground image can be combined with different original background images to form multiple sets of initial data. An original background image can be combined with different original foreground images to form multiple sets of initial data. Under the condition that each set of initial data is unique, the present application does not limit the combination method of the images in the foreground database and the images in the background database.

[0150] After obtaining multiple groups of initial data, for each group of initial data, determine the foreground image corresponding to the original foreground image in the group of initial data, and the background image corresponding to the original background image in the group of initial data, and map the determined foreground image to the determined background image according to the synthesis position corresponding to the group of initial data. Since the image after mapping may have a mask area, etc., which leads to low authenticity of the image after mapping, the present application can repair the image after mapping to obtain the synthetic image corresponding to the group of initial data. It should be noted that the synthesis position corresponding to each group of initial data is determined according to a predetermined position determination strategy and is used to indicate the position where the determined foreground image is mapped to the determined background image. Of course, the synthesis position corresponding to each group of initial data can also be an artificially given position, which is not limited in the present application. The foreground image corresponding to any original foreground image is an image containing a foreground object obtained after the foreground object is separated from the original foreground image; the background image corresponding to any original background image is an image from which the foreground object is removed from the original image.

[0151] Afterwards, for each group of initial data, the foreground image corresponding to the original foreground object in the group of initial data, the background image corresponding to the original background image in the group of initial data, the synthetic image corresponding to the group of initial data and the sample position information representing the synthetic position can be constructed into a group of sample data; thus, a training data set is constructed based on each group of sample data.

[0152] Specifically, the method for determining the foreground image corresponding to the original foreground image includes:

[0153] Performing object detection on the original foreground image to obtain a position frame of the foreground object in the original foreground image;

[0154] From the original foreground image, according to the obtained position frame of the foreground object, intercepting a sub-image containing the foreground object;

[0155] Segmenting an object region in the sub-image;

[0156] Setting the area between the segmentation boundary of the object area and the image edge in the sub-image as a mask area to obtain a foreground image corresponding to the original foreground image; wherein the segmentation boundary is a segmentation boundary belonging to the foreground object;

[0157] The method for determining the background image corresponding to the original background image includes:

[0158] Performing object detection on the original background image to obtain a position frame of the background object in the original background image;

[0159] Setting the area of ​​the position box of the background object in the original background image as a mask area to obtain an intermediate background image;

[0160] Background restoration is performed on the masked area of ​​the intermediate background image to obtain a background image corresponding to the original background image.

[0161] When determining the foreground image corresponding to the original foreground image, the original foreground image may be firstly subjected to object detection to obtain the position frame of the foreground object in the original foreground image. According to the position frame of the foreground object, a sub-image containing the foreground object may be intercepted from the original foreground image. For example, if the foreground object is an electric box, it is equivalent to cutting out the complete electric box image from the original foreground image. At this time, the obtained sub-image may contain the background area belonging to the original foreground image except the foreground object (that is, the area between the object area of ​​the foreground object and the edge of the image is the background area belonging to the original foreground image). The object area in the sub-image may be segmented, that is, the segmentation boundary of the object area of ​​the foreground object is identified, and the segmentation boundary is the outline of the foreground object; the area between the segmentation boundary of the object area and the edge of the image in the sub-image may be set as a mask area to obtain the foreground image corresponding to the original foreground image. At this time, the obtained foreground image does not contain the background area belonging to the original foreground image. Although the foreground image contains the mask area, a synthetic image with higher authenticity can be obtained by performing image restoration after subsequent mapping.

[0162] Exemplarily, a method of setting an area in a sub-image that is located between a segmentation boundary of an object area and an edge of an image as a mask area may include: setting the object area in the segmented sub-image as a mask area, for example: setting the area of ​​an electrical box as a mask area; at this time, the area between the segmentation boundary of the object area in the segmented sub-image and the edge of the image is a background area belonging to the original foreground image, and the segmented sub-image may be inverted, and the object area that is a mask area is reset to a foreground object, and the area between the segmentation boundary of the object area and the edge of the image that is a background area is set as a mask area.

[0163] When determining the background image corresponding to the original background image, object detection can also be performed on the original background image to obtain the position box of the background object in the original background image, and the area in the original background image that belongs to the position box of the background object is set as the mask area to obtain the intermediate background image; in order to improve the authenticity of the background image and facilitate subsequent mapping, the mask area of ​​the intermediate background image can be repaired so that the mask area is integrated into the background area in the intermediate background image. For example, if the background area in the intermediate background image is a grassland area, the mask area can be repaired to the grassland area to obtain the background image corresponding to the original background image.

[0164] It should be noted that the detection and segmentation model can be used to identify the foreground object and the background object in the original foreground image and the original background image respectively, and the detection and segmentation model can be used to capture a sub-image containing the foreground object from the original foreground image according to the obtained position box of the foreground object, and the detection and segmentation model can be used to set the area in the original background image that belongs to the position box of the background object as a mask area, for example: the area of ​​the position box of the background object is removed from the original background image; the existing diffusion model, for example: the repair model, can be used to perform background repair on the mask area of ​​the intermediate background image, and the present application does not limit its specific implementation method.

[0165] It should be noted that the foreground object and the background object are corresponding objects belonging to the same class of objects. For example, the foreground object and the background object are both objects about an electric box, but the foreground object and the background object are not exactly the same. Therefore, a new synthetic image can be obtained by mapping and synthesizing the foreground image and the background image. Of course, the number of foreground objects and background objects can be the same or different. For example, there is one electric box in the original foreground image, and there are two electric boxes in the original background image. The foreground images corresponding to the original foreground image can be mapped to the background images corresponding to the original background image.

[0166] Exemplarily, in one implementation, the predetermined position determination strategy includes: a first strategy for indicating a randomly selected position, a second strategy for indicating a position determination based on a position box of a background object in the original background image, or a third strategy for indicating a position prediction based on a predetermined position prediction model.

[0167] The predetermined position determination strategy may include a first strategy for indicating a randomly selected position, a second strategy for indicating a position determination based on a position box of a background object in an original background image, or a third strategy for indicating a position prediction based on a predetermined position prediction model.

[0168] It should be noted that when the predetermined position determination strategy includes the first strategy, a position can be randomly selected from the specified area of ​​the background image corresponding to the original background image in the group of initial data as the synthesis position corresponding to the group of initial data. Any position in the specified area can make the synthetic image after image restoration not distorted if the mapping is performed according to the position. For example, if the foreground object is an electric box, since the electric box should be placed on a pole, the specified area of ​​the background image is the pole area; if the foreground object is a vehicle, since the vehicle should be located on the road, the specified area of ​​the background image is the road area. When the predetermined position determination strategy includes the second strategy, the position of the position box of the background object in the original background image can be directly determined as the synthesis position corresponding to the group of initial data. When the predetermined position determination strategy includes the third strategy, the position of the background image corresponding to the original background image in the group of initial data can be predicted based on a predetermined position prediction model to obtain the synthesis position corresponding to the group of initial data.

[0169] It can be seen that in the present application, a training data set can be constructed in advance. In the present application, a foreground database and a background database are established. The foreground database is used to improve the diversity of foreground images, and the background database is used to improve the diversity of background images. First, the images in the foreground database and the images in the background database can be combined according to a predetermined combination method, and each group of output data includes an original foreground image in the foreground database and a background image in the background database; in this way, the number of subsequent data pairs of foreground images and background images is greatly improved, and the data scale and diversity of multiple groups of sample data are improved; and the foreground image, background image, synthetic image and sample position information corresponding to each group of initial data are determined, so as to construct a group of sample data, thereby constructing a training data set with rich data volume and high diversity, which contains each group of sample data.

[0170] Based on the above model training method, the present application embodiment also provides an image synthesis method, such as Figure 3 As shown, the method may include the following steps:

[0171] S301: Acquire a target background image and a target foreground image;

[0172] S302: Acquire a synthetic image template that matches the size of the target background image; wherein the synthetic image template is a noise image;

[0173] S303: inputting the target background image, the target foreground image and the synthetic image template into an image synthesis model, so that the image synthesis model performs image synthesis processing based on the target background image, the target foreground image and the synthetic image template;

[0174] S304: Obtain the image synthesis result output by the image synthesis model to obtain a synthesized image after the target background image and the target foreground image are synthesized; wherein the image synthesis model is a model trained according to the model training method.

[0175] Among them, the target background image and the target foreground image, and the corresponding synthetic image template can be understood as image data in a set of sample data. The target background image, the target foreground image and the synthetic image template can be input into the image synthesis model. The image synthesis model performs image synthesis processing based on the position prediction network and the image synthesis network. The obtained image synthesis result is the predicted image synthesis result mentioned above. The specific image synthesis process can be similar to the process of predicting the image synthesis result in the above-mentioned model training method, which will not be repeated here.

[0176] An image synthesis method provided by an embodiment of the present application can obtain a target background image and a target foreground image, and obtain a synthetic image template belonging to a noise image and matching the size of the target background image, and input the target background image, the target foreground image and the synthetic image model into the image synthesis model trained by the model training method, and the image synthesis model performs image synthesis processing based on the auxiliary information obtained by the target background image, the target foreground image and the synthetic image model, and the position prediction model in the image synthesis model, and the image synthesis model outputs the image synthesis result, and the synthetic image after the target background image and the target foreground image are synthesized can be obtained. It can be seen that since the auxiliary information obtained by the position prediction module can implicitly contain the synthetic position information based on which the target foreground image and the target background image are synthesized, it is not necessary to manually give the position for image synthesis, and the image synthesis model can directly perform image synthesis processing on the target background image, the target foreground image and the synthetic image template with the auxiliary information, and output the image synthesis result of the synthetic image. It can be seen that through this solution, images can be automatically synthesized without the help of manual labor.

[0177] The following is a detailed introduction to a model training method and an image synthesis method provided by the present application in conjunction with a specific embodiment.

[0178] This application mainly consists of 3 steps:

[0179] 1. Dataset construction & expansion: construct a data set (training data set) from existing data for subsequent model training. The constructed data set includes multiple sets of sample data, each set of sample data includes: <foreground image (foreground image), background image (background image), the position of the foreground in the background (coordinates, i.e., sample position information), and synthetic image>;

[0180] 2. Image synthesis model training: Use the dataset constructed in step 1 to train the image synthesis model;

[0181] 3. Image synthesis: Use the image synthesis model trained in step 2 to perform image synthesis;

[0182] like Figure 4 As shown, dataset construction can include the following steps:

[0183] Step 1. Foreground separation: image a (original foreground image) to image c (foreground image), that is, use the detection and segmentation model Ground-SAM (an open source model that can be replaced by other similar detection and segmentation models) to detect the electric box bbox (location box) in image a, and according to the segmentation boundary of the electric box (the segmentation boundary of the object area), use the area between the bbox of the electric box and the segmentation boundary of the electric box as the mask area Mask to obtain image c; among them, image a and image b can be understood as a set of initial data.

[0184] Step 2. Background restoration: image b (original background image) to image d to image f (background image), use the detection and segmentation model Ground-SAM to detect the electric box bbox in image b, and use the bbox of the electric box as the mask area Mask to obtain image d; use a diffusion model, such as the restoration model SD-inpaint (an open source model that can be replaced by other similar models) to perform background restoration to obtain image f;

[0185] Step 3. New image synthesis: Image e to synthetic image, enlarge image c and map it to image f to get image e; use the repair model SD-inpaint to repair the mask area Mask (black edge) to get a synthetic image. The repair model SD-inpaint is a diffusion model; while generating the synthetic image, it outputs the mapping position information (x, y, h, w) (i.e. the predicted position information mentioned above).

[0186] When this application is used for data amplification, it can also be used according to Figure 5The process shown in the figure is used for data augmentation. The present application pre-establishes a foreground library (foreground database) and a background library (background database), wherein the foreground library is used to increase the diversity of foreground objects, and the background library is used to increase the diversity of backgrounds. The foreground library and the background library can be arranged and combined, which greatly increases the number of "foreground-background" data pairs, and ultimately increases the data scale and diversity. Among them, both the foreground library and the background library include domain data sets, associated data sets, and network data sets. Foreground objects in the original foreground images in the foreground library are vehicles, and the background in the original background images in the background library is roads. For example, the domain data set in the foreground library is the original foreground image including vehicles in the city, the associated data set is the original foreground image including vehicles in the countryside, and the network data set is the original foreground image obtained by means of web crawlers and the like, including more diverse vehicles; the domain data set in the background library includes the original background image of urban roads, the associated data set includes the original background image of rural roads, and the network data set is the original background image obtained by means of web crawlers and the like, including more diverse roads. Among them, the original foreground image in the foreground library (or the foreground image corresponding to the original foreground image) can be expanded by rotation, scaling, and the like. Of course, the foreground library and the background library are interchangeable, that is, the foreground library can be used as the background library, and the background library can be used as the foreground library, further improving the diversity of the data.

[0187] After permutation and combination, multiple groups of initial data are obtained. For each group of initial data, the foreground image corresponding to the original foreground image in the group of initial data and the background image corresponding to the original background image in the group of initial data are determined; and according to the synthesis position corresponding to the group of initial data, the determined foreground image is mapped to the determined background image, and the mapped image is repaired, that is, the foreground image and the background image are synthesized to obtain the synthesized image corresponding to the group of initial data, thereby expanding the data set to construct a training data set.

[0188] The training process of the image synthesis model is as follows Figure 6 As shown, the image synthesis model is based on Stable Diffusion (an open source diffusion model). In this application, a position prediction module (i.e., the position prediction network mentioned above) is added to the image synthesis model to predict the synthesis position of the foreground image c in the background image f. The fusion feature implying the synthesis position will be input into the SD model (the original diffusion model Unet) through the Cross Attention module to guide the synthesis of the new image. The image synthesis model includes:

[0189] 1. Position prediction module: The position prediction module consists of the CLIP model (Contrastive Language-Image Pre-Training, multimodal pre-training neural network model) and FC (Full Connection, fully connected layer), that is, the foreground image features and background image features of the foreground image c and the background image f are extracted through CLIP, and after splicing (i.e. fusion features), they are input into FC, and the mapping position information (x, y, h, w) is output, that is, the upper left coordinate, length and width of the foreground image in the background image.

[0190] 2. Original diffusion model Unet (i.e. this module can be an existing diffusion model): This module can guide the noise addition process during training, and input the synthetic image (that is, when the data set is constructed, the composite image corresponding to the foreground image c and the background image f in the set of sample data) and the time information t, then the image after noise addition at that time is obtained , and input the foreground image c and background image f for diffusion training at the same time.

[0191] Based on the map position information (x, y, h, w) corresponding to each set of sample data output by the image synthesis model and the image synthesis result (i.e., the new image synthesized by the image synthesis model), as well as the sample position information in each set of sample data, and based on Gaussian noise, the position prediction loss (i.e., the second type of loss) and the diffusion generation loss (i.e., the first type of loss) are calculated to obtain the target loss of the image synthesis model.

[0192] After the image synthesis model is trained, the steps of using the image synthesis model to synthesize images are as follows: Figure 7 As shown in FIG. 1 , by inputting the foreground image and the background image into the image synthesis model, the final synthesis result can be directly obtained. During the image synthesis process, the image synthesis model will automatically predict the synthesis position of the foreground image in the background image, thereby eliminating the need for manual specification.

[0193] The following will introduce the details of each step of dataset construction (foreground separation, background restoration and new image synthesis).

[0194] The process of foreground separation is as follows Figure 8As shown, for a complete original foreground image, Ground-SAM is used to detect the bounding-box (i.e., bbox, the target box of the electric box), and then the complete electric box image (i.e., the sub-image containing the foreground object) is cropped. Ground-SAM is then used to segment the cropped electric box image to obtain the mask Mask of the object area of ​​the electric box; then, the complete image of the electric box and the mask Mask corresponding to the object area of ​​the electric box are used to set the area between the segmentation boundary of the object area of ​​the electric box and the bbox (image boundary) as the mask area through a negation operation to obtain the foreground image with the mask, that is, the foreground image corresponding to the original foreground image (i.e., the method of determining the foreground image corresponding to the original foreground image mentioned above). It should be noted that there is no restriction on the model for detecting and segmenting objects, and different models can be used. For the convenience of this application, the open source detection and segmentation large model Ground-SAM is used.

[0195] The process of background repair is as follows Fig. 9 As shown, for a complete original background image, Ground-SAM is used to detect the bounding-box of the electric box, and the bounding-box is masked, that is, the target box of the detected electric box is set as the mask area Mask, and then the open source image restoration model Stable Diffusion is used to repair the mask area (that is, the above-mentioned method of determining the background image corresponding to the original background image). It should be noted that there is no restriction on the restoration model used, and different models can be used, and this application does not limit this. However, when using the restoration model, it is necessary to input a guide text to guide the restoration process of the mask area. Therefore, the guide text needs to be adjusted accordingly according to the different restoration models used. The core of the guide text is "there is no object in the restoration area and it is in harmony with the background". For example, the guide text can be: photograph of an empty scene without anything, harmony with the background, highest quality settings.

[0196] The process of new image synthesis is as follows Fig.10 As shown, the obtained foreground image can be appropriately enlarged, such as rotated, scaled, etc. (the specific method and intensity depend on actual needs, and the mask area Mask at the edge of the foreground image needs to be adjusted synchronously), and then mapped to the background image, and the final composite image (i.e., the composite image in each set of sample data) is obtained through background repair.

[0197] The synthesis position of the foreground image map to the background image can be determined by the following strategy:

[0198] 1) Randomly select a position (the first strategy): You can randomly select a position in the background image. It should be noted that the selected position must ensure that the foreground image can be stored and does not exceed the range of the image; and after the foreground image is mapped to the background image through this position, the resulting composite image is not distorted.

[0199] 2) Determine the position based on the position box of the background object in the background image (second strategy): The background image is obtained by background repair, and the position of the background repair can be used as the synthesis position of the current foreground image. Of course, the synthesis position is not uniquely determined, and the synthesis position can be any position within the predetermined range of the background repair position, thereby improving the diversity of mapping.

[0200] 3) Position prediction using a position prediction model (third strategy): The position prediction model can be used to predict the appropriate mapping position of the foreground image in the background image. The position prediction model can use an open source model or can be developed and trained by itself according to needs. This application does not limit this.

[0201] The specific structure of the image synthesis model is as follows Fig.11 As shown, the various modules of the image synthesis model are described below:

[0202] VAE (i.e. the first encoder): compresses the input original image (synthesized image), background image (background image) and foreground image (foreground image). This application uses the VAE that comes with Stable Diffusion to compress the length and width of the image eight times, thereby improving the training speed of the image synthesis model.

[0203] Image Encoder (i.e. the second encoder): extracts image features. This application uses the CLIP model, and the extracted feature dimension is 1x1024.

[0204] Unet (i.e. image synthesis module): the main body of the diffusion model, which is a symmetrical structure with the same input and output spatial size and the same input and output dimensions. Usually, the image input to Unet is first reduced in dimension and then increased in dimension until it reaches the size of its input, thus generating a synthetic image.

[0205] Fusion Module (i.e., feature fusion module included in the position prediction network): mainly used to fuse the foreground image features and background image features of the foreground image and the background image, so as to learn the position and size relationship of the foreground image in the background image. The structure consists of 6 to 8 layers of Transformer Encoder (i.e., N encoders in the figure), with input and output dimensions of 1024.

[0206] FC: Fully connected layer (which can be understood as belonging to the location prediction network), used to predict location coordinates (x, y, h, w), with an input dimension of 1024 and an output dimension of 4 (predicting four values ​​of x, y, h, w).

[0207] The training process of the image synthesis model is as follows:

[0208] 1. Input the original image, background image and foreground image, use the first encoder VAE to compress, and add noise to the original image according to the time step t to obtain the noisy original image . Use the first encoder VAE to compress the background image and foreground image to obtain and Then we splice in the image channel dimension to get The image synthesis module Unet is based on the calculation result of the cross attention value. Predict the image synthesis result and get a new synthetic image .

[0209] 2. Use the second encoder Image Encoder to extract background image features and foreground image features , and then add them and input them into the feature fusion module to get the fusion feature . First, it is input into the image synthesis module Unet for cross attention calculation, and the calculation result is used to guide the generation of new images (new synthetic images). The position prediction result (x, y, h, w) will be output through the fully connected layer FC. The position prediction result is used to calculate the MSE loss of the position prediction module (i.e. the position prediction network mentioned above). :

[0210] ;

[0211] On the other hand, the diffusion model loss needs to be calculated (i.e. the loss of the image synthesis network):

[0212] ;

[0213] Finally and Add them together and use gradient descent to train the image synthesis model.

[0214] In the embodiment of the present application, the image synthesis model introduces a position prediction module, which can automatically predict the position of the foreground image in the background image; during the data set construction process, the foreground image and the background image can be arranged and combined to maximize the data set; the repair (synthesis) process is based on the diffusion model, which has a powerful image generation capability; the process is clear, simple and easy to use. In addition, it should be noted that Figure 1a , Figure 1b , Figure 4 as well as Figures 6 to 11 , it is only used to illustrate the relationship between the various graphs, and does not focus on the content of the image itself.

[0215] This application independently constructs a training data set by arranging and combining foreground and background on existing data, and combines foreground augmentation (rotation, scaling, offset, etc.), which greatly improves the diversity of generated data; it synthesizes new images at one time based on the diffusion model, which is more efficient than traditional methods. The position prediction module is introduced, so the image synthesis model can predict the position of the foreground in the background by itself, without the need to manually give the synthesis position. The traditional image synthesis scheme includes multiple sub-processes (such as: edge harmony, shadow supplementation, light generation, background fusion, etc.), the process is cumbersome and the authenticity is low. This application uses a diffusion model to repair (synthesize), without the need for each sub-process and the effect is real. This scheme can complete augmentation on existing data, produce a large amount of new image data with high authenticity, and at the same time ensure the diversity of generated data. This application does not require manual data annotation (that is, the synthesis position is manually given during the training process, and the synthesis position can be automatically determined by a predetermined position determination strategy) training, and can be completed using open source tools, which is low-cost and effective.

[0216] Based on the above method embodiment, the present application also provides a model training device, such as Fig.12 As shown, the device comprises:

[0217] The first acquisition module 1210 is used to acquire multiple groups of sample data; each group of sample data includes image data and sample position information, the image data includes a foreground image, a background image and a composite image, and the sample position information is information of a composite position based on which the foreground image and the background image in the sample data are composited to obtain the composite image;

[0218] The first input module 1220 is used to input the image data in the plurality of groups of sample data into the image synthesis model to be trained, and obtain the image synthesis result and predicted position information corresponding to each group of sample data; wherein the image synthesis model includes a position prediction network and an image synthesis network; the position prediction network is used to generate, for each group of sample data, a feature fusion result of a foreground image and a background image in the group of sample data, obtain a fusion feature corresponding to the group of sample data, and determine the predicted position information corresponding to the group of sample data based on the obtained fusion feature; the image synthesis network is used to predict the image synthesis result for each group of sample data based on the foreground image, background image and synthesized image in the group of sample data, and using the fusion feature corresponding to the group of sample data as auxiliary information; the predicted position information corresponding to the group of sample data is the information of the predicted synthesis position based on which the foreground image and the background image in the group of sample data are synthesized;

[0219] A determination module 1230, configured to determine a target loss of the image synthesis model based on sample position information in each set of sample data, and an image synthesis result and predicted position information corresponding to each set of sample data;

[0220] The adjustment module 1240 is used to adjust the model parameters of the image synthesis model when it is determined based on the target loss that the image synthesis model has not converged.

[0221] In the model training device provided by the embodiment of the present application, each group of sample data includes image data and sample position information, the image data includes a foreground image, a background image and a synthesized image, and the sample position information is the synthesis position information based on which the foreground image and the background image in the sample data to which it belongs are synthesized to obtain the synthesized image; the present application can first obtain multiple groups of sample data, and input the image data in the multiple groups of sample data into the image synthesis model to be trained, and the position prediction network in the image synthesis model generates the feature fusion result of the foreground image and the background image in the group of sample data for each group of sample data, obtains the fusion feature corresponding to the group of sample data, and determines the predicted position information corresponding to the group of sample data based on the obtained fusion feature. Since the fusion feature corresponding to each group of sample data can imply the synthesis position information based on which the foreground image and the background image in the group of sample data are synthesized, the image synthesis network predicts the image synthesis result based on the foreground image, the background image and the synthesized image in the group of sample data, using the fusion feature corresponding to the group of sample data as auxiliary information. In the present application, the position prediction network in the image synthesis model can generate fusion features corresponding to each set of sample data, and the fusion features corresponding to each set of sample data can implicitly contain the synthesis position information based on which the foreground image and the background image in the set of sample data are synthesized. The position prediction network can determine the predicted position information corresponding to the set of sample data based on the fusion features corresponding to each set of sample data; the image synthesis network can use the fusion features corresponding to each set of sample data as auxiliary information to predict the image synthesis result, that is, the fusion features corresponding to each set of sample data, which implicitly contain the synthesis position information based on which the foreground image and the background image in the set of sample data are synthesized, can act on the image synthesis process; and it is not necessary to manually give the position of the foreground image synthesized into the background image every time. Furthermore, the target loss of the image synthesis model can be determined based on the sample position information in each set of sample data, as well as the image synthesis result and predicted position information corresponding to each set of sample data. When it is determined based on the target loss that the image synthesis model has not converged, the model parameters of the image synthesis model are adjusted to obtain a trained image synthesis model. It can be seen that through this solution, an image synthesis model that can automatically synthesize images without the help of manual work can be trained, thereby providing a basis for automatic image synthesis to improve synthesis efficiency.

[0222] Based on the above method embodiment, the present application also provides an image synthesis device, such as Fig.13 As shown, the device comprises:

[0223] The second acquisition module 1310 is used to acquire a target background image and a target foreground image;

[0224] A third acquisition module 1320 is used to acquire a synthetic image template that matches the size of the target background image; wherein the synthetic image template is a noise image;

[0225] A second input module 1330, configured to input the target background image, the target foreground image and the synthetic image template into an image synthesis model, so that the image synthesis model performs image synthesis processing based on the target background image, the target foreground image and the synthetic image template;

[0226] The fourth acquisition module 1340 is used to obtain the image synthesis result output by the image synthesis model, and obtain a composite image after the target background image and the target foreground image are synthesized; wherein the image synthesis model is a model trained according to the model training method.

[0227] An image synthesis device provided by an embodiment of the present application can obtain a target background image and a target foreground image, and obtain a synthetic image template belonging to a noise image and matching the size of the target background image, and input the target background image, the target foreground image and the synthetic image model into the image synthesis model trained by the model training method, and the image synthesis model performs image synthesis processing based on the target background image, the target foreground image and the synthetic image model, and the auxiliary information obtained by the position prediction model in the image synthesis model, and the image synthesis model outputs the image synthesis result, and the synthetic image after the target background image and the target foreground image are synthesized can be obtained. It can be seen that since the auxiliary information obtained by the position prediction module can implicitly contain the synthetic position information based on which the target foreground image and the target background image are synthesized, it is not necessary to manually give the position for image synthesis, and the image synthesis model can directly perform image synthesis processing on the target background image, the target foreground image and the synthetic image template with the auxiliary information, and output the image synthesis result of the synthetic image. It can be seen that through this solution, images can be automatically synthesized without the help of manual labor.

[0228] The present application also provides an electronic device, such as Fig.14 As shown, including:

[0229] Memory 1401, used for storing computer programs;

[0230] The processor 1402 is used to implement any of the above-mentioned model training methods or image synthesis methods when executing the program stored in the memory 1401.

[0231] In addition, the electronic device may further include a communication bus and / or a communication interface, and the processor 1402, the communication interface, and the memory 1401 communicate with each other via the communication bus.

[0232] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0233] The communication interface is used for communication between the above electronic device and other devices.

[0234] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0235] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0236] In another embodiment provided in the present application, a computer-readable storage medium is also provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned model training methods or image synthesis methods are implemented.

[0237] In another embodiment provided in the present application, a computer program product comprising instructions is also provided, which, when executed on a computer, enables the computer to execute any model training method or image synthesis method in the above-mentioned embodiments.

[0238] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state hard disk (SSD), etc.

[0239] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0240] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0241] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A model training method, characterized in that: The method comprises: Acquire multiple groups of sample data; wherein each group of sample data includes image data and sample position information, the image data includes a foreground image, a background image and a composite image, and the sample position information is information of a composite position based on which the foreground image and the background image in the corresponding sample data are composited to obtain the composite image; Input the image data in the plurality of groups of sample data into the image synthesis model to be trained, and obtain the image synthesis result and predicted position information corresponding to each group of sample data; wherein the image synthesis model includes a position prediction network and an image synthesis network; the position prediction network is used to generate, for each group of sample data, a feature fusion result of a foreground image and a background image in the group of sample data, obtain a fusion feature corresponding to the group of sample data, and determine the predicted position information corresponding to the group of sample data based on the obtained fusion feature; the image synthesis network is used to predict the image synthesis result for each group of sample data based on the foreground image, background image and synthesized image in the group of sample data, and using the fusion feature corresponding to the group of sample data as auxiliary information; the predicted position information corresponding to the group of sample data is the information of the predicted synthesis position based on which the foreground image and the background image in the group of sample data are synthesized; Calculate the mean square error between the feature vector of the image synthesis result corresponding to each group of sample data and the random vector representing the standard Gaussian noise to obtain the first type of loss; based on the sample position information in each group of sample data and the predicted position information corresponding to each group of sample data, calculate the mean square error loss according to the predetermined mean square error loss calculation method to obtain the second type of loss; perform predetermined operation processing on the first type of loss and the second type of loss to obtain the target loss of the image synthesis model; wherein the first type of loss is used to characterize the network loss of the image synthesis network; the second type of loss is used to characterize the network loss of the position prediction network; the standard Gaussian noise is Gaussian noise that conforms to the standard normal distribution; When it is determined based on the target loss that the image synthesis model has not converged, model parameters of the image synthesis model are adjusted.

2. The method according to claim 1, characterized in that The image synthesis network includes a first encoder, a splicing module and an image synthesis module; The first encoder is used to compress the foreground image, the background image and the synthesized image after adding noise in each group of sample data, and input the compressed image into the stitching module; The stitching module is used to stitch the compressed images in the image channel dimension and input the stitched image obtained by stitching into the image synthesis module; The image synthesis module is used to receive the stitched image, calculate the cross-attention value of the fusion feature corresponding to the group of sample data, obtain the calculation result corresponding to the group of sample data, and use the calculation result as a guidance content for the synthesis position, perform image synthesis processing based on the received stitched image, and obtain the image synthesis result corresponding to the group of sample data.

3. The method according to claim 2, characterized in that The first encoder is a compression module VAE in a predetermined diffusion model, and the image synthesis module is a main structure Unet in the diffusion model.

4. The method according to any one of claims 1 to 3, characterized in that: The position prediction network includes a second encoder, a position prediction module and a fully connected layer; The second encoder is used for extracting features of the foreground image and the background image in each group of sample data, respectively, and inputting the extracted foreground image features and background image features into the position prediction module; The position prediction module is used to generate a feature fusion result of the foreground image and the background image in the group of sample data based on the received foreground image features and background image features, obtain a fusion feature corresponding to the group of sample data, and input the obtained fusion feature corresponding to the group of sample data into the fully connected layer; The fully connected layer is used to perform position prediction processing on the fusion features corresponding to the group of sample data to obtain predicted position information corresponding to the group of sample data.

5. The method according to claim 1, characterized in that The multiple groups of sample data are data in a pre-constructed training data set; The training data set is constructed in the following manner: The images in the foreground database and the images in the background database are combined according to a predetermined combination method to obtain multiple groups of initial data; wherein each group of initial data includes an original foreground image in the foreground database and an original background image in the background database; For each group of initial data, determine the foreground image corresponding to the original foreground image in the group of initial data, and the background image corresponding to the original background image in the group of initial data; and map the determined foreground image to the determined background image according to the synthesis position corresponding to the group of initial data, and repair the mapped image to obtain the synthesized image corresponding to the group of initial data; wherein the synthesis position corresponding to the group of initial data is a position determined according to a predetermined position determination strategy and used to indicate that the determined foreground image is mapped to the determined background image; the foreground image corresponding to any original foreground image is: an image containing a foreground object obtained after foreground separation of the original foreground image, and the background image corresponding to any original background image is: an image after the foreground object is removed from the original background image; For each group of initial data, a foreground image corresponding to an original foreground image in the group of initial data, a background image corresponding to an original background image in the group of initial data, and sample position information and a synthesized image corresponding to the group of initial data are constructed into a group of sample data; wherein the sample position information corresponding to the group of initial data is position information of a synthesized position corresponding to the group of initial data; Construct a training data set containing each group of sample data.

6. The method according to claim 5, characterized in that The method for determining the foreground image corresponding to the original foreground image includes: Performing object detection on the original foreground image to obtain a position frame of the foreground object in the original foreground image; From the original foreground image, according to the obtained position frame of the foreground object, intercepting a sub-image containing the foreground object; Segmenting an object region in the sub-image; Setting the area between the segmentation boundary of the object area and the image edge in the sub-image as a mask area to obtain a foreground image corresponding to the original foreground image; wherein the segmentation boundary is a segmentation boundary belonging to the foreground object; The method for determining the background image corresponding to the original background image includes: Performing object detection on the original background image to obtain a position frame of the background object in the original background image; Setting the area of ​​the position box of the background object in the original background image as a mask area to obtain an intermediate background image; Background restoration is performed on the masked area of ​​the intermediate background image to obtain a background image corresponding to the original background image.

7. The method according to claim 5, characterized in that The predetermined position determination strategy includes: a first strategy for indicating a randomly selected position, a second strategy for indicating position determination based on a position box of a background object in the original background image, or a third strategy for indicating position prediction based on a predetermined position prediction model.

8. An image synthesis method, characterized in that: The method comprises: Obtain a target background image and a target foreground image; Acquire a synthetic image template that matches the size of the target background image; wherein the synthetic image template is a noise image; Inputting the target background image, the target foreground image and the synthetic image template into an image synthesis model, so that the image synthesis model performs image synthesis processing based on the target background image, the target foreground image and the synthetic image template; The image synthesis result output by the image synthesis model is obtained to obtain a composite image after the target background image and the target foreground image are synthesized; wherein the image synthesis model is a model trained according to the model training method of any one of claims 1 to 7.

9. A model training device, characterized in that: The device comprises: A first acquisition module is used to acquire multiple groups of sample data; each group of sample data includes image data and sample position information, the image data includes a foreground image, a background image and a composite image, and the sample position information is information of a composite position based on which the foreground image and the background image in the sample data are composited to obtain the composite image; A first input module is used to input the image data in the plurality of groups of sample data into the image synthesis model to be trained, and obtain the image synthesis result and predicted position information corresponding to each group of sample data; wherein the image synthesis model includes a position prediction network and an image synthesis network; the position prediction network is used to generate, for each group of sample data, a feature fusion result of a foreground image and a background image in the group of sample data, obtain a fusion feature corresponding to the group of sample data, and determine the predicted position information corresponding to the group of sample data based on the obtained fusion feature; the image synthesis network is used to predict the image synthesis result for each group of sample data based on the foreground image, background image and synthesized image in the group of sample data, and using the fusion feature corresponding to the group of sample data as auxiliary information; the predicted position information corresponding to the group of sample data is the information of the predicted synthesis position based on which the foreground image and the background image in the group of sample data are synthesized; A determination module is used to calculate the mean square error between the feature vector of the image synthesis result corresponding to each group of sample data and the random vector representing the standard Gaussian noise to obtain a first type of loss; based on the sample position information in each group of sample data and the predicted position information corresponding to each group of sample data, the mean square error loss is calculated according to a predetermined mean square error loss calculation method to obtain a second type of loss; the first type of loss and the second type of loss are subjected to predetermined operation processing to obtain the target loss of the image synthesis model; wherein the first type of loss is used to characterize the network loss of the image synthesis network; the second type of loss is used to characterize the network loss of the position prediction network; the standard Gaussian noise is Gaussian noise that conforms to the standard normal distribution; An adjustment module is used to adjust the model parameters of the image synthesis model when it is determined based on the target loss that the image synthesis model has not converged.

10. An image synthesis device, characterized in that: The device comprises: The second acquisition module is used to acquire a target background image and a target foreground image; A third acquisition module is used to acquire a synthetic image template that matches the size of the target background image; wherein the synthetic image template is a noise image; A second input module, used for inputting the target background image, the target foreground image and the synthetic image template into the image synthesis model, so that the image synthesis model performs image synthesis processing based on the target background image, the target foreground image and the synthetic image template; The fourth acquisition module is used to obtain the image synthesis result output by the image synthesis model, and obtain a composite image after the target background image and the target foreground image are synthesized; wherein the image synthesis model is a model trained according to the model training method of any one of claims 1-7.

11. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, for implementing the method described in any one of claims 1 to 7, or the method described in claim 8, when executing a program stored in a memory.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 or the method according to claim 8 is implemented.

Images