Image generation method and device, electronic equipment and computer program product

Abstract

The embodiment of the invention is suitable for the technical field of computers, and provides an image generation method and device, electronic equipment and a computer program product, and the method comprises the steps: adding a to-be-corrected model of a target object in an original image to a layer of a scene image corresponding to a target scene, and generating a to-be-corrected image of the target object in the target scene; inputting the to-be-corrected image and a standard model corresponding to the target object into the first light effect identification model to obtain material information of the target object and first light and shadow information of the target object in the target scene; and generating a target image of the target object in the target scene based on the material information, the first light and shadow information and the to-be-corrected image. According to the method provided by the embodiment of the invention, the electronic equipment can extract the material information of the target object and the first light and shadow information of the target scene through the first light effect identification model to generate the target image, so that the method provided by the embodiment of the invention can improve the fusion effect between the foreground product and the background in the target image; and the display effect of the target image is improved.

Description

Technical Field

[0001] The embodiments of the present application belong to the field of computer technology, and in particular, relate to an image generation method, device, electronic device, and computer program product. Background Art

[0002] As one of the ways to display products, product display pictures play an important role in the communication between enterprises and customers. Due to the high cost of real shooting and the need for a lot of manpower costs, the product display pictures provided by most companies are currently mainly generated by traditional modeling and rendering methods. For example, automobile companies can generate automobile display pictures by traditional modeling and rendering methods. When staff generate product display pictures by traditional modeling and rendering methods, they need to accurately build product models based on information such as the appearance, structure, and material of the product, and then add lighting effects to the product model through rendering algorithms and the lighting information of the background environment in the display picture to make the product model look more realistic. However, when the existing rendering algorithm adds lighting effects to the product model, it mainly depends on the staff to set the lighting parameters by themselves. Therefore, the lighting effects added to the product model by the existing rendering algorithm are quite different from the lighting information of the background environment in the display picture. There is a large sense of separation between the foreground product and the background in the display picture, which affects the overall display effect of the display picture. Summary of the invention

[0003] In view of this, the embodiments of the present application provide an image generation method, device, electronic device and computer program product to improve the sense of integrity between the target object and the background in the target image, thereby improving the overall display effect of the target image.

[0004] A first aspect of an embodiment of the present application provides an image generation method, comprising:

[0005] Adding the model to be corrected of the target object in the original image to the layer of the scene image corresponding to the target scene, to generate the image to be corrected of the target object in the target scene;

[0006] Input the image to be corrected and the standard model corresponding to the target object into the first light effect recognition model to obtain the material information of the target object and the first light and shadow information of the target object in the target scene; the standard model is collected when the target object is not affected by any reflective background objects;

[0007] A target image of the target object in the target scene is generated based on the material information, the first light and shadow information, and the image to be corrected.

[0008] Compared with the prior art, the embodiments of the present application have the following advantages:

[0009] In an embodiment of the present application, when generating a target image, the electronic device may first add the model to be corrected corresponding to the target object in the original image to the layer of the scene image corresponding to the target scene to generate the image to be corrected of the target object in the target scene; then, the electronic device may input the image to be corrected and the standard model corresponding to the target object into the first light effect recognition model to obtain the material information of the target object and the first light and shadow information of the target object in the target scene through the first light effect recognition model; after obtaining the material information and the first light and shadow information, the electronic device may generate the target image of the target object in the target scene based on the image to be corrected, the material information and the first light and shadow information. Different from the prior art method of directly generating an image by modeling and rendering, the first light and shadow information in this embodiment is obtained by the first light effect recognition model according to the image to be corrected of the target object in the target scene. Therefore, compared with the light and shadow information determined only according to the target scene in the prior art, the light and shadow information determined in this embodiment not only includes the information of the light distribution in the target scene, but also includes the light and shadow information corresponding to the specific position of the target object in the target scene. It can be seen that the method provided in this embodiment can make the first light and shadow information obtained by the electronic device consistent with the position of the target object in the target scene and the specific light distribution in the target scene, so that the target image generated by the electronic device based on the first light and shadow information has a strong integrity.

[0010] In addition, since the electronic device can also obtain the material information of the target object through the standard model of the target object, and the standard model of the target object is collected when the target object is not affected by any reflective background objects, the method provided in this embodiment can also make the material information obtained by the electronic device not be affected by the light effect of the target scene, thereby improving the accuracy of the material information obtained by the first light effect recognition model. Finally, the electronic device generates a target image based on the first light and shadow information and material information respectively obtained. Therefore, the target image generated by the method provided in this embodiment can not only ensure the overall sense between the target object and the target scene, but also retain the material details of the target object, thereby generating a target image with better display effect.

[0011] In a possible implementation manner of the first aspect, the method further includes:

[0012] If the shooting scene corresponding to the original image is different from the target scene, the original image is input into a light effect correction algorithm to generate the standard model corresponding to the target object.

[0013] Compared with the prior art, the embodiments of the present application have the following advantages:

[0014] Through the method provided in this embodiment, the electronic device can automatically generate a standard model of the target object without any influence of reflective background objects, thereby preventing the target object in the target image from being affected by the light and shadow effects of other scenes, thereby improving the overall sense of the target image.

[0015] In a possible implementation manner of the first aspect, the light effect correction algorithm includes a second light effect recognition model;

[0016] The step of inputting the original image into a light effect correction algorithm to generate the standard model corresponding to the target object includes:

[0017] Inputting the original image into the second light effect recognition model to obtain second light and shadow information of the target object in the shooting scene;

[0018] The standard model is generated according to the second light and shadow information, the original image and a background mask corresponding to the target object in the original image.

[0019] Compared with the prior art, the embodiments of the present application have the following advantages:

[0020] According to the method provided in this embodiment, the electronic device can extract the second light and shadow information of the target object in the shooting scene through the second light effect recognition model, and perform image subtraction operation based on the second light and shadow information to generate a standard model. Therefore, the method provided in this embodiment can ensure that there is no light and shadow effect caused by the shooting scene on the standard model of the target object, thereby ensuring that there is no light and shadow effect caused by other scenes on the target image, and improving the overall sense of the target image generated by the electronic device based on the standard model.

[0021] In a possible implementation manner of the first aspect, generating the standard model according to the second light and shadow information, the original image, and a background mask corresponding to the target object in the original image includes:

[0022] Performing an image subtraction operation on the original image based on the second light and shadow information to determine a corrected image corresponding to the target object;

[0023] The calibration image is subjected to image multiplication operation with a product mask generated based on the target object contour to generate the standard model.

[0024] In a possible implementation manner of the first aspect, the method further includes:

[0025] If the original image is obtained based on the three-dimensional model of the target object, performing image rendering on the three-dimensional model based on preset natural lighting parameters to obtain a natural light rendering model;

[0026] The standard model corresponding to the target object is generated according to the shooting angle of the target object in the original image and the natural light rendering model; the natural lighting parameters are used to represent the lighting without the influence of any reflective background objects.

[0027] In a possible implementation manner of the first aspect, generating a target image of the target object in the target scene based on the material information, the first light and shadow information, and the image to be corrected includes:

[0028] Performing an image addition operation on the standard model of the target object based on the material information and the first light and shadow information to obtain a target object model of the target object in the target scene;

[0029] The target image is generated based on the target object model and the image to be corrected.

[0030] In a possible implementation manner of the first aspect, generating the target image based on the target object model and the image to be corrected includes:

[0031] The target object model, the image to be corrected and the background mask of the target object in the original image are input into a preset Poisson fusion algorithm to generate the target image.

[0032] A second aspect of the embodiments of the present application provides an image generating device, including:

[0033] A first image generation module, used for adding a model of a target object to be corrected in the original image to a layer of a scene image corresponding to a target scene, to generate an image of the target object to be corrected in the target scene;

[0034] A light and shadow information acquisition module, used for inputting the image to be corrected and the standard model corresponding to the target object into a first light effect recognition model to obtain material information of the target object and first light and shadow information of the target object in the target scene; the standard model is acquired when the target object is not affected by any reflective background objects;

[0035] The second image generation module is used to generate a target image of the target object in the target scene based on the material information, the first light and shadow information and the image to be corrected.

[0036] A third aspect of an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the image generation method as described in the first aspect above is implemented.

[0037] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the image generation method as described in the first aspect above is implemented.

[0038] A fifth aspect of the embodiments of the present application provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute the image generation method described in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or prior art descriptions. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0040] Figure 1 is a schematic diagram of an image generation method provided in an embodiment of the present application;

[0041] Figure 2 It is a schematic diagram of a target image generation process provided by an embodiment of the present application;

[0042] Figure 3 is a schematic diagram of another image generation method provided in an embodiment of the present application;

[0043] Figure 4 It is a schematic diagram of a specific generation process of a target image provided in an embodiment of the present application;

[0044] Figure 5 It is a schematic diagram of an implementation flow of a light effect correction algorithm provided in an embodiment of the present application;

[0045] Figure 6 is a schematic diagram of an image generation method and device provided in an embodiment of the present application;

[0046] Figure 7 It is a schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] In the following description, specific details such as specific system structures, technologies, etc. are proposed for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from hindering the description of the present application.

[0048] Artificial Intelligence Generated Content (AIGC) technology refers to the technology that automatically generates various types of content through artificial intelligence algorithms, such as deep learning algorithms. AI-generated content technology can be used to automatically generate images, text, music and other content. With the development of AI-generated content technology, its ability to generate high-quality images is getting stronger and stronger. For example, the diffusion model is a deep learning method based on a probabilistic generative model. The diffusion model can gradually convert data into noise by simulating the physical diffusion process, and then learn the reverse process to gradually recover the original data from the noise, thereby gradually generating clear images from the noise. In the direction of high-quality image generation, the diffusion model shows excellent ability to generate realistic and colorful images.

[0049] Since AI content generation technology can quickly generate high-quality images, AI content generation technology is often used in the creative industry and product promotion industry. In order to reduce the cost of publicity, more and more companies are beginning to automatically generate product display pictures through AI content generation technology. However, in the prior art, compared with the real picture, when the product display picture automatically generated by AI content generation technology cannot accurately determine the imaging effect of the product in the corresponding background through the material information of the product and the light and shadow information of the product in the background, it increases the sense of separation between the product and the background, and affects the quality of the generated image. In view of this, the embodiment of the present application provides a method that can identify the material information of the target object and the first light and shadow information of the target object in the target scene, and correct the image to be corrected in the target scene by the material information and the first light and shadow information, thereby generating a target image. Therefore, after the staff uses the AI ​​content generation technology to generate the image to be corrected, the method provided by the embodiment of the present application can be used to correct the image to be corrected, thereby generating a product display picture that can accurately depict the material information and the light and shadow information of the product in the background.

[0050] The technical solution of the present application is described below through specific embodiments.

[0051] Reference Figure 1 , shows a schematic diagram of an image generation method provided by an embodiment of the present application. The method can be applied to electronic devices such as computers, tablet computers, mobile devices, large servers, etc. The above-mentioned image generation method can specifically include the following steps:

[0052] S101, adding a model to be corrected of a target object in an original image to a layer of a scene image corresponding to a target scene, to generate an image to be corrected of the target object in the target scene.

[0053] In this embodiment, when a staff member needs to generate a product display image of a certain product, for example, a staff member of an automobile company needs to generate a vehicle display image corresponding to a certain model of vehicle, the staff member can initiate an image generation instruction to the electronic device. The electronic device can respond to the user's initiation of the image generation instruction, obtain the original image corresponding to the target object, and add the model to be corrected corresponding to the target object in the original image to the layer of the scene image corresponding to the target scene, so as to generate the image to be corrected of the target object in the target scene. Among them, the original image can be an image of the target object photographed in a certain shooting scene or an image obtained by the staff rendering the three-dimensional model of the target object.

[0054] In a possible implementation, the staff can input the original image corresponding to the target object, the product mask of the target object in the original image, and the scene information corresponding to the target scene into the electronic device, and initiate an image generation instruction to the electronic device. Among them, the scene information input by the staff can be a description text corresponding to the target scene or a scene image corresponding to the target scene. In response to the image generation instruction initiated by the staff, the electronic device can first determine whether the scene information is a scene image corresponding to the target scene. If the electronic device determines that the scene information is a scene image corresponding to the target scene, the electronic device can segment the model to be corrected corresponding to the target object from the original image through the product mask of the target object in the original image. Then, the electronic device can add the model to be corrected to the layer of the scene image corresponding to the target scene to generate an image to be corrected of the target object in the target scene.

[0055] If the electronic device determines that the scene information is a description text corresponding to the target scene, the electronic device can input the description text, the original image and the product mask into the image extension model pre-set by the R&D personnel to generate the image to be corrected of the target object in the target scene through the image extension model. Specifically, the image extension model can first segment the model to be corrected corresponding to the target object from the original image through the product mask of the target object in the original image. Then, the image extension model can expand outward along the edge of the product in the product mask according to the description text to generate a scene image corresponding to the target scene. Then, the image extension model can add the model to be corrected to the layer of the scene image corresponding to the target scene to generate the image to be corrected. Exemplarily, the image extension model can be any model that can expand outward along the edge of the target object to generate a new image, such as an inpainting model.

[0056] S102: Input the image to be corrected and the standard model corresponding to the target object into a first light effect recognition model to obtain material information of the target object and first light and shadow information of the target object in a target scene.

[0057] In this embodiment, since the lighting effect of the target object in the image to be corrected does not take into account the lighting environment in the target scene, that is, there may be a certain sense of separation between the target object and the background in the image to be corrected. Therefore, after generating the image to be corrected, the electronic device can input the product mask of the target object in the original image, the image to be corrected, and the standard model corresponding to the target object into the first light effect recognition model, so as to obtain the material information of the target object and the first light and shadow information of the target object in the target scene through the first light effect recognition model. Among them, the first light and shadow information of the target object in the target scene can be used to represent the light and shadow effects such as highlights, reflections, and reflections formed on the surface of the target object when the target object is located in the lighting and environment of the target scene. Specifically, when generating the first light and shadow information, the first light effect recognition model can determine the area to be identified from the image to be corrected where the light and shadow information needs to be extracted according to the product mask of the target object in the original image, and then generate the first light and shadow information of the target object in the target scene according to the position information of the area to be identified in the image to be corrected and the environmental information of the target scene in the image to be corrected.

[0058] The standard model is obtained when the target object is not affected by any reflective background objects, that is, the surface of the target object in the standard model does not have any light and shadow effects such as highlights, reflections, and inversions. Exemplarily, the standard model can be obtained when the target object is illuminated by white diffuse light with uniform illumination intensity and there are no obstructions around the target object.

[0059] In a possible implementation, the first light effect recognition model can be formed by training the first model to be trained by the first training data set. Specifically, the first model to be trained can be formed by connecting the first variational autoencoder, the forward diffusion network and the reverse diffusion network in sequence. Exemplarily, the first model to be trained can be designed by a staff member according to the model structure of the diffusion model. Among them, the first variational autoencoder is a generative deep learning model, which structurally includes an encoder and a decoder. The first variational autoencoder can learn the potential distribution information of the input data by minimizing the reconstruction loss. The forward diffusion network can be connected by multiple convolutional layers. The forward diffusion network can gradually add Gaussian noise to the image to be corrected and the standard model according to the potential distribution information input by the variational autoencoder until a pure noise image is finally obtained. The reverse diffusion network can be used to gradually denoise the pure noise generated by the forward diffusion network until an image that can be used to represent the first light and shadow information of the target object in the target scene and the material information of the target object is obtained.

[0060] When training the first model to be trained, the first training data set may include a first training image, a standard model of the target object, a product mask of the target object, and an expected image that can represent the material information and light and shadow information of the target object. Among them, the first training image can be an image of the target object photographed in a certain shooting scene, or it can be an image obtained by rendering the three-dimensional model of the target object by a staff member. After the first training data set is input into the first model to be trained, the potential distribution information of the training image and the training model can be obtained by the first variational autoencoder in the first model to be trained. Then, the potential distribution information is input into the forward diffusion network to perform a noise operation on the training image and the standard model through the forward diffusion network to generate a pure noise image. Then, the forward diffusion network can input the pure noise image into the reverse diffusion network to obtain the initial image.

[0061] Then, the electronic device can input the initial image, training image, standard model, product mask of the target object and expected image into the first loss function pre-set by the R&D personnel to determine the first loss value between the initial image and the expected image. The electronic device can back-propagate the calculated first loss value to the first model to be trained, and update the parameters in the first model to be trained according to the first loss value, and train the updated first model to be trained through the first training data set. When the first loss value reaches the first stop condition set by the user, the electronic device can use the current first network to be trained as the first light effect recognition model. The first loss function can be specifically as follows.

[0062]

[0063] Among them, L R Can be used to represent the first loss value. ε Can be used to represent the cross entropy of the calculated variable ε. ε~Ν(0,1) can be used to represent the normal distribution function. t can be used to represent the first weight coefficient. ε θ Used to represent the b ,I p ,I a ,M) constitutes the Gaussian distribution function. t1 It can be used to represent the first initial distribution information generated by the first variational self-encoder. Used to represent the divergence function of two Gaussian distributions. M can be used to represent the product mask of the target object. I a Can be used to represent the desired image. p A standard model that can be used to represent the target object. b It can be used to represent the first training image. t1 can be used to represent the initial image.

[0064] Specifically, the first variational autoencoder may be composed of three subnetworks. The first subnetwork in the first variational autoencoder may be used to perform image operations on the product mask, the training model, and the expected image according to the first function, generate a first image operation result, and obtain first distribution information corresponding to the first image operation result. The first function may be as follows.

[0065] z1=(1-M)×(I a -I p )

[0066] Wherein, z1 can be used to represent the first image operation result. M can be used to represent the product mask of the target object. a Can be used to represent the desired image. p A standard model that can be used to represent an object.

[0067] The second subnetwork in the first variational autoencoder can be used to perform image operations on the product mask, the training model, and the training image according to the second function, generate a second image operation result, and obtain second distribution information corresponding to the second image operation result. The second function can be as follows.

[0068] z2=MI b +I p

[0069] Wherein, z2 can be used to represent the second image operation result. b Can be used to represent the first training image.

[0070] The third subnetwork in the first variational autoencoder can be used to perform image operations on the product mask according to the three functions, generate a third image operation result, and obtain third distribution information corresponding to the third image operation result.

[0071] Among them, the third function can be as follows.

[0072] z3=(1-M)

[0073] Among them, z3 can be used to represent the third image operation result.

[0074] Then, the first variational autoencoder can input the first distribution information, the second distribution information and the third distribution information into the forward diffusion network. Accordingly, the first convolution layer in the forward diffusion network can include 9 input channels. Four of the 9 input channels of the forward diffusion network are used to receive the first distribution information, the other four are used to receive the second distribution information, and the last one is used to receive the third distribution information, so as to gradually add Gaussian noise to the image to be corrected and the standard model according to the first distribution information, the second distribution information and the third distribution information. Among them, the specific structure of the forward diffusion network and the reverse diffusion network can be similar to the structure in the diffusion model, which will not be repeated here.

[0075] S103: Generate a target image of the target object in a target scene based on the material information, the first light and shadow information, and the image to be corrected.

[0076] In this embodiment, after acquiring the material information of the target object and the first light and shadow information of the target object in the target scene, the electronic device can perform image processing on the target object in the image to be corrected according to the material information and the first light and shadow information to generate a target image of the target object in the target scene.

[0077] In this embodiment, since the first light and shadow information is obtained by the first light effect recognition model according to the image to be corrected of the target object in the target scene, the method provided by this embodiment enables the electronic device to not only obtain the light and shadow information according to the light distribution of the target scene, but also further determine the light and shadow information of the surface of the target object in combination with the specific position of the target object in the target scene. Furthermore, since the light effect presented on the surface of the object is not only affected by the light distribution in the scene, but also by the occlusion relationship between the target object and other objects in the scene, the specific material of the surface of the target object, and the relative position relationship between the target object and the light source in the scene. It can be seen that, compared with the prior art of directly determining the light and shadow information according to the target scene, the method provided by this embodiment can make the first light and shadow information determined by the electronic device consistent with the specific position of the target object in the target scene, the reflective characteristics of the surface material of the target object, and the actual light distribution in the target scene, so that the target image generated by the electronic device based on the first light and shadow information has a strong integrity.

[0078] In addition, since the electronic device further combines the material information of the target object for processing when generating the target image, the method provided in this embodiment can not only improve the overall sense between the target object and the target scene, but also improve the material details of the target object displayed by the target image. Therefore, the method provided in this embodiment can generate an image with better display effect, so that the user can accurately know the specific material of the target object based on the target image.

[0079] In one possible implementation, see Figure 2 , shows a schematic diagram of a target image generation process provided by an embodiment of the present application. Figure 2 As shown, after acquiring the image to be corrected, the electronic device can input the image to be corrected, the standard model of the target object, and the product mask used to represent the contour of the target object into the first light effect recognition model to generate material information of the target object and first light and shadow information of the target object in the target scene. After the electronic device generates the material information and the first light and shadow information through the first light effect recognition model, it can perform image addition operation on the standard model of the target object according to the material information and the first light and shadow information to obtain the target object model of the target object in the target scene.

[0080] Among them, the material information and the first light and shadow information generated by the first light effect recognition model can be represented as image information of the shape of the target object. When the material information and the first light and shadow information represent image information, that is, the material information is a material image of the shape of the target object and the first light and shadow information is a first light and shadow image of the shape of the target object, the electronic device can add the standard model, the material image and the first light and shadow image pixel by pixel to obtain the target object model of the target object in the target scene. Exemplarily, when the electronic device performs an image addition operation on a certain pixel, the electronic device can add the pixel value at a certain position in the standard model, the pixel value at the same position in the material image, and the pixel value at the same position in the first light and shadow image, and then the electronic device can normalize the result of the addition at the position to obtain the image addition operation result of the pixel at the position.

[0081] After acquiring the target object model of the target object in the target scene, the electronic device can generate the target image of the target object in the target scene according to the target object model and the image to be corrected. Specifically, after generating the target object model, the electronic device can input the target object model, the image to be corrected, and the background mask of the target object in the original image into the Poisson fusion algorithm pre-set by the R&D personnel to generate the target image through the Poisson fusion algorithm. Among them, when the electronic device performs Poisson fusion on the target object model and the image to be corrected, it can first identify the area to be fused with the target object model from the image to be corrected according to the background mask, that is, the area to be fused.

[0082] Then, the electronic device can construct the Poisson equation according to the gradient information of the area to be fused in the image to be corrected and the gradient information of the target object model. Among them, the Poisson equation constructed by the electronic device can be used to describe the pixel value constraint relationship between the area to be fused in the image to be corrected and the target object model. Then, the electronic device can solve the constructed Poisson equation through an iterative method (such as Gauss-Seidel iteration method) pre-set by the R&D personnel to fuse the target object model into the area to be fused to generate a target image.

[0083] In this embodiment, the standard model generated by the electronic device is a model obtained by collecting the target object without the influence of any reflective background objects. Therefore, the method provided by this embodiment can make the target model generated on the basis of the standard model, and the light and shadow effects presented by it are not affected by the illumination in scenes other than the target scene. In addition, since the first light effect recognition model can not only extract the first light and shadow information of the target object in the target scene, but also extract the material information of the target object, when the electronic device generates the target model, it can not only show the light and shadow effects on the surface of the target object in the target scene on the target model according to the first light and shadow information, but also show the actual material of the target object on the target model according to the material information. Since the target model can display the light and shadow effects and the actual material at the same time, the method provided by this embodiment can improve the display effect of the target image.

[0084] Furthermore, since the first light effect recognition model is formed by the electronic device training the first model to be trained based on the expected image in the training data set that can represent the material information of the target object, and in the process of the electronic device training the first model to be trained, the first distribution information corresponding to the expected image can be calculated by the first function, that is, the first distribution information corresponding to the material information of the target object is extracted by the first function. Then, the electronic device can train the first model to be trained including the first distribution information of the material information. Therefore, the first light effect recognition model trained by the method provided in this embodiment can be used not only to extract light and shadow information, but also to extract the material information of the target object. Compared with the light effect recognition model in the prior art that can only be used to extract light and shadow information, the method provided in this embodiment can improve the availability of the light effect recognition model.

[0085] On the basis of the first embodiment of the present application, in order to further improve the accuracy of the standard model generated by the electronic device, the second embodiment of the present application provides a specific method for generating a standard model according to the source of the original image. Figure 3 FIG. 1 shows a specific implementation flow chart of an image generation method before S102 provided in the second embodiment of the present application. Figure 3 , compared to Figure 1 In the embodiment, an image generation method provided in this embodiment includes before S102: S301 to S302, which are described in detail as follows:

[0086] S301: If the original image is not generated based on the three-dimensional model rendering of the target object and the shooting scene corresponding to the original image is different from the target scene, the original image is input into the light effect correction algorithm to generate a standard model corresponding to the target object.

[0087] In this embodiment, before acquiring the material information and the first light and shadow information of the target object, the electronic device can also automatically generate a standard model corresponding to the target object based on the original image input by the staff. Specifically, after acquiring the original image, the electronic device can determine whether the original image is generated by rendering the three-dimensional model of the target object. If the electronic device determines that the original image is not generated based on the rendering of the three-dimensional model of the target object and the shooting scene corresponding to the original image is different from the target scene, that is, the original image may be obtained by shooting in a shooting scene different from the target scene, then the electronic device can input the original image into the light effect correction algorithm pre-set by the R&D personnel, so as to delete the light and shadow effects on the target object in the original image through the light effect correction algorithm, and generate a standard model corresponding to the target object.

[0088] Through the method provided by this embodiment, when the original image is an image obtained by shooting, in this case, since the electronic device cannot directly adjust the light and shadow effects of the surface of the target object in the original image by image rendering, the standard model corresponding to the target object is constructed by the light effect correction algorithm, which can improve the usability of this solution, so that the electronic device can obtain the standard model of the target object under any circumstances. In addition, the light reflection of the shooting scene can easily interfere with the surface of the target object. The light effect correction algorithm does not simply and roughly remove the reflective background, but corrects the original image according to the second light and shadow information of the target object in the shooting scene to generate a standard model. Therefore, the standard model generated by the method provided by this embodiment can both eliminate the influence of the light of the shooting scene and ensure the realism of the target object, and has high practicality.

[0089] See also Figure 4 , shows a schematic diagram of a specific generation process of a target image provided by an embodiment of the present application. Figure 4As shown, when the staff needs to generate a display image of a certain model of car in another scene based on the original image obtained by shooting a certain model of car in a certain shooting scene, the staff can input the original image of the car, the product mask corresponding to the car, and the description text used to describe the target scene into the electronic device, and initiate an image generation instruction to the electronic device. The electronic device can input the original image, the product mask and the description text into the image extension model in response to the image generation instruction, so as to generate the image to be corrected of the car of the model in the target scene through the image extension model. The electronic device can also input the original image and the product mask into the light effect correction algorithm to generate the standard model corresponding to the car of the model through the light effect correction algorithm. Then, the electronic device can input the image to be corrected and the standard model into the second light effect recognition model to obtain the material information of the car of the model and the first light and shadow information of the car of the model in the target scene through the second light effect recognition model. The electronic device can generate the target image of the car of the model in the target scene according to the material information and the first light and shadow information.

[0090] In a possible implementation, the light effect correction algorithm may include a second light effect recognition model. The second light effect recognition model may be used to recognize the light and shadow effects formed on the surface of the target object due to the influence of the lighting of the shooting scene and surrounding obstructions when the target object is located in the shooting scene. Figure 5 , shows a schematic diagram of the implementation flow of a light effect correction algorithm provided by an embodiment of the present application. Figure 5 As shown, after acquiring the original image, the electronic device can input the original image and the background mask corresponding to the target object in the original image into the second light effect recognition model, so as to obtain the second light and shadow information of the target object in the shooting scene through the second light effect recognition model. Among them, the background mask can be a mask that identifies the outline of the background in the product image, thereby separating the background from the scene in the original image. Exemplarily, the background mask can be a binary image, in which the pixel value of the area where the target object is located can be 0, and the pixel value of other areas except the target object can be 1. The second light and shadow information can be used to represent the light and shadow effect on the surface of the target object under the influence of the lighting of the shooting scene. Then, the electronic device can generate a standard model corresponding to the target object based on the second light and shadow information, the original image, and the background mask of the target object in the original image.

[0091] Specifically, Figure 5As shown, after acquiring the second light and shadow information, the electronic device can perform an image subtraction operation on the original image based on the second light and shadow information to determine the corrected image corresponding to the target object. Among them, the specific method of image subtraction operation is similar to the specific method of image addition operation. The reader can refer to the relevant process of image addition operation performed by the electronic device in the embodiment of the present application, and replace addition with subtraction and addition with subtraction to understand the specific method of image subtraction operation performed by the electronic device. Then, the electronic device can perform an image multiplication operation on the corrected image and the product mask of the target object to generate a standard model corresponding to the target object. Among them, the product mask can be used to represent the outline of the target object in the original image.

[0092] In a possible implementation, the second light effect recognition model can be formed by training the second model to be trained by the second training data set. Specifically, the second model to be trained can be formed by sequentially connecting a second variational autoencoder, a forward diffusion network, and a reverse diffusion network. Among them, the second variational autoencoder, the forward diffusion network, and the reverse diffusion network in the second light effect recognition model are similar to the first variational autoencoder, the forward diffusion network, and the reverse diffusion network in the first light effect recognition model. Please refer to the relevant description in the first light effect recognition model, and no further description will be given here.

[0093] When training the second model to be trained, the second training data set may include a second training image, a product mask of the target object, and a standard model of the target object. Among them, the first training image may be an image of the target object taken in a certain shooting scene, or an image obtained by the staff rendering the three-dimensional model of the target object. The electronic device may input the second training image and the product mask of the target object into the second model to be trained to generate an initial model of the target object. Then, the electronic device may input the initial model, the second training image, the standard model, and the product mask of the target object into the second loss function pre-set by the R&D personnel to determine the second loss value between the initial model and the standard model. The electronic device may back-propagate the calculated second loss value to the second model to be trained, and update the parameters in the second model to be trained according to the second loss value. The electronic device may train the updated second model to be trained through the second training data set until the second loss value reaches the second stop condition set by the user. When the second loss value reaches the second stop condition set by the user, the electronic device may use the current second network to be trained as the second light effect recognition model. Among them, the second loss function may be specifically as follows.

[0094]

[0095] Among them, L p Can be used to represent the second loss value. εCan be used to represent the cross entropy of the calculated variable ε. ε~Ν(0,1) can be used to represent the normal distribution function. t can be used to represent the first weight coefficient. ε θ It can be used to represent the p ,M) constitutes the Gaussian distribution function. t2 Used to represent the second initial distribution information generated by the second variational autoencoder. Used to represent the divergence function. M can be used to represent the product mask of the target object. I p It can be used to represent the standard model of the target object. I can be used to represent the second training image. t2 can be used to represent the initial model.

[0096] Specifically, the second variational autoencoder can also be composed of three subnetworks. The fourth subnetwork in the second variational autoencoder can be used to perform image operations on the product mask, the training model, and the expected image according to the fourth function, generate a fourth image operation result, and obtain fourth distribution information corresponding to the fourth image operation result. The fourth function can be as follows.

[0097] z4=(1-M)×(II p )

[0098] Wherein, z4 can be used to represent the fourth image operation result. M can be used to represent the product mask of the target object. I can be used to represent the second training image. p A standard model that can be used to represent an object.

[0099] The fifth subnetwork in the second variational autoencoder can be used to obtain fifth distribution information corresponding to the second training image. The sixth subnetwork in the second variational autoencoder can be used to perform image operation on the product mask according to the fifth function, generate a fifth image operation result, and obtain sixth distribution information corresponding to the fifth image operation result. The fifth function can be as follows.

[0100] z5=(1-M)

[0101] Among them, z5 can be used to represent the fifth image operation result.

[0102] Then, the second variational autoencoder can input the fourth distribution information, the fifth distribution information and the sixth distribution information into the forward diffusion network. Accordingly, the first convolution layer in the forward diffusion network can include 9 input channels. Four of the 9 input channels of the forward diffusion network are used to receive the fourth distribution information, the other four are used to receive the fifth distribution information, and the last one is used to receive the sixth distribution information, so as to gradually add Gaussian noise to the image to be corrected and the standard model according to the fourth distribution information, the fifth distribution information and the sixth distribution information. Among them, the specific structure of the forward diffusion network and the reverse diffusion network can be similar to the structure in the diffusion model, which will not be repeated here.

[0103] Through the method provided in this embodiment, since the electronic device can extract the second light and shadow information of the target object in the shooting scene through the second light effect recognition model, and perform image subtraction operation based on the second light and shadow information to generate a standard model. Therefore, the electronic device can accurately delete the light and shadow effects on the surface of the target object caused by the light distribution of the shooting scene in the original image, so that the light and shadow effects caused by the shooting scene do not exist on the standard model of the target object. Therefore, the method provided in this embodiment can generate a standard model that is not affected by any reflective background objects, so that when the electronic device generates an image of the target object in the target scene, it will not be affected by light effects unrelated to the target scene, thereby achieving the effect of improving the overall sense of the target image.

[0104] S302. If the original image is obtained based on a three-dimensional model of the target object, the three-dimensional model is rendered based on preset natural light parameters to obtain a natural light rendering model, and a standard model corresponding to the target object is generated according to the shooting angle of the target object in the original image and the natural light rendering model.

[0105] In this embodiment, if the electronic device determines that the original image is generated based on the rendering of the three-dimensional model of the target object, the electronic device can obtain the three-dimensional model corresponding to the target object to generate a standard model corresponding to the target object through the three-dimensional model.

[0106] Specifically, after the electronic device obtains the three-dimensional model of the target object, it can render the three-dimensional model according to the natural lighting parameters preset by the staff to obtain the natural light rendering model corresponding to the target object. Among them, the preset natural lighting parameters can be used to represent the lighting without the influence of any reflective background objects, that is, the preset natural lighting parameters can be used to represent the lighting under white diffuse light with uniform illumination intensity. After the electronic device obtains the natural light rendering model of the target object, it can determine the shooting angle of the target object in the original image according to the product mask of the target object in the original image. Then, the electronic device can generate a standard model corresponding to the target object according to the shooting angle of the target object and the natural light rendering model.

[0107] Through the method provided in this embodiment, when the source of the image is a three-dimensional model, since in this case the electronic device can directly render the image of the three-dimensional model corresponding to the target object, in addition, rendering the image of the three-dimensional model according to natural lighting parameters can enable the model to more accurately display the contour, texture, material and other detailed information of the target object, and avoid the problem of loss of detailed information due to insufficient lighting. Therefore, the method provided in this embodiment can accurately outline the appearance of the target object in an ideal non-reflective background.

[0108] It should be noted that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0109] Reference Figure 6 , shows a schematic diagram of an image generating device provided in an embodiment of the present application, which may specifically include a first image generating module 601, a light and shadow information acquiring module 602, and a second image generating module 603, wherein:

[0110] The first image generation module 601 is used to add the model to be corrected of the target object in the original image to the layer of the scene image corresponding to the target scene, so as to generate the image to be corrected of the target object in the target scene;

[0111] The light and shadow information acquisition module 602 is used to input the image to be corrected and the standard model corresponding to the target object into the first light effect recognition model to obtain the material information of the target object and the first light and shadow information of the target object in the target scene; the standard model is obtained by collecting the target object without any influence of reflective background objects;

[0112] The second image generating module 603 is configured to generate a target image of the target object in the target scene based on the material information, the first light and shadow information and the image to be corrected.

[0113] The image generating device may also include a standard model generating module, which is used to input the original image into a light effect correction algorithm to generate the standard model corresponding to the target object if the shooting scene corresponding to the original image is different from the target scene.

[0114] The standard model generation module can also be used to input the original image into the second light effect recognition model to obtain the second light and shadow information of the target object in the shooting scene; and generate the standard model according to the second light and shadow information, the original image and the background mask corresponding to the target object in the original image.

[0115] The standard model generation module can also be used to perform image subtraction operation on the original image based on the second light and shadow information to determine the corrected image corresponding to the target object; perform image multiplication operation on the corrected image and the product mask generated based on the target object contour to generate the standard model.

[0116] The standard model generation module can also be used to render the three-dimensional model based on preset natural lighting parameters to obtain a natural light rendering model if the original image is obtained based on the three-dimensional model of the target object; generate the standard model corresponding to the target object according to the shooting angle of the target object in the original image and the natural light rendering model; the natural lighting parameters are used to represent the lighting without the influence of any reflective background objects.

[0117] The second image generation module can also be used to perform image addition operation on the standard model of the target object based on the material information and the first light and shadow information to obtain the target object model of the target object in the target scene; and generate the target image based on the target object model and the image to be corrected.

[0118] The second image generation module can also be used to input the target object model, the image to be corrected and the background mask of the target object in the original image into a preset Poisson fusion algorithm to generate the target image.

[0119] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiment part.

[0120] Reference Figure 7 , shows a schematic diagram of an electronic device provided by an embodiment of the present application. Figure 7 As shown, the electronic device 700 in the embodiment of the present application includes: a processor 710, a memory 720, and a computer program 721 stored in the memory 720 and executable on the processor 710. When the processor 710 executes the computer program 721, the steps in each embodiment of the above-mentioned image generation method are implemented, for example Figure 1 Alternatively, when the processor 710 executes the computer program 721, the functions of each module / unit in the above-mentioned device embodiments are realized, for example Figure 3 Functions of modules 301 to 305 are shown.

[0121] Exemplarily, the computer program 721 may be divided into one or more modules / units, which are stored in the memory 720 and executed by the processor 710 to complete the present application. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which may be used to describe the execution process of the computer program 721 in the electronic device 700. For example, the computer program 721 may be divided into a first image generation module, a light and shadow information acquisition module, and a second image generation module, and the specific functions of each module are as follows:

[0122] A first image generation module, used for adding a model of a target object to be corrected in the original image to a layer of a scene image corresponding to a target scene, to generate an image of the target object to be corrected in the target scene;

[0123] A light and shadow information acquisition module, used for inputting the image to be corrected and the standard model corresponding to the target object into a first light effect recognition model to obtain material information of the target object and first light and shadow information of the target object in the target scene; the standard model is acquired when the target object is not affected by any reflective background objects;

[0124] The second image generation module is used to generate a target image of the target object in the target scene based on the material information, the first light and shadow information and the image to be corrected.

[0125] The electronic device 700 may be a computing device such as a desktop computer or a cloud server. The electronic device 700 may include, but is not limited to, a processor 710 and a memory 720. Those skilled in the art will appreciate that Figure 7 It is only an example of the electronic device 700 and does not constitute a limitation of the electronic device 700. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device 700 may also include input and output devices, network access devices, buses, etc.

[0126] The processor 710 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0127] The memory 720 may be an internal storage unit of the electronic device 700, such as a hard disk or memory of the electronic device 700. The memory 720 may also be an external storage device of the electronic device 700, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 700. Further, the memory 720 may also include both an internal storage unit of the electronic device 700 and an external storage device. The memory 720 is used to store the computer program 721 and other programs and data required by the electronic device 700. The memory 720 may also be used to temporarily store data that has been output or is to be output.

[0128] An embodiment of the present application also discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the image generation method described in the aforementioned embodiments when executing the computer program.

[0129] The embodiments of the present application further disclose a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the image generating method as described in the above embodiments is implemented.

[0130] The embodiments of the present application further disclose a computer program product. When the computer program product is run on a computer, the computer is enabled to execute the image generating method described in the above embodiments.

[0131] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application is described in detail with reference to the above-mentioned embodiments, a person skilled in the art should understand that the technical solutions described in the above-mentioned embodiments can still be modified, or some of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. An image generation method, characterized in that: include: Adding the model to be corrected of the target object in the original image to the layer of the scene image corresponding to the target scene, to generate the image to be corrected of the target object in the target scene; Inputting the image to be corrected and the standard model corresponding to the target object into a first light effect recognition model to obtain material information of the target object and first light and shadow information of the target object in the target scene; The standard model is acquired by collecting the target object without any influence of reflective background objects; A target image of the target object in the target scene is generated based on the material information, the first light and shadow information, and the image to be corrected.

2. The method according to claim 1, characterized in that: Also includes: If the shooting scene corresponding to the original image is different from the target scene, the original image is input into a light effect correction algorithm to generate the standard model corresponding to the target object.

3. The method according to claim 2, characterized in that The light effect correction algorithm includes a second light effect recognition model; The step of inputting the original image into a light effect correction algorithm to generate the standard model corresponding to the target object includes: Inputting the original image into the second light effect recognition model to obtain second light and shadow information of the target object in the shooting scene; The standard model is generated according to the second light and shadow information, the original image and a background mask corresponding to the target object in the original image.

4. The method according to claim 3, characterized in that The step of generating the standard model according to the second light and shadow information, the original image, and a background mask corresponding to the target object in the original image includes: Performing an image subtraction operation on the original image based on the second light and shadow information to determine a corrected image corresponding to the target object; The calibration image is subjected to image multiplication operation with a product mask generated based on the target object contour to generate the standard model.

5. The method according to claim 1, characterized in that Also includes: If the original image is obtained based on the three-dimensional model of the target object, performing image rendering on the three-dimensional model based on preset natural lighting parameters to obtain a natural light rendering model; The standard model corresponding to the target object is generated according to the shooting angle of the target object in the original image and the natural light rendering model; the natural lighting parameters are used to represent the lighting without the influence of any reflective background objects.

6. The method according to claim 1, characterized in that The step of generating a target image of the target object in the target scene based on the material information, the first light and shadow information, and the image to be corrected includes: Performing an image addition operation on the standard model of the target object based on the material information and the first light and shadow information to obtain a target object model of the target object in the target scene; The target image is generated based on the target object model and the image to be corrected.

7. The method according to claim 6, characterized in that The step of generating the target image based on the target object model and the image to be corrected includes: The target object model, the image to be corrected and the background mask of the target object in the original image are input into a preset Poisson fusion algorithm to generate the target image.

8. An image generating device, characterized in that: include: A first image generation module, used for adding a model of a target object to be corrected in the original image to a layer of a scene image corresponding to a target scene, to generate an image of the target object to be corrected in the target scene; A light and shadow information acquisition module, used for inputting the image to be corrected and the standard model corresponding to the target object into a first light effect recognition model to obtain material information of the target object and first light and shadow information of the target object in the target scene; The standard model is acquired by collecting the target object without any influence of reflective background objects; The second image generation module is used to generate a target image of the target object in the target scene based on the material information, the first light and shadow information and the image to be corrected.

9. An electronic device, characterized in that: The electronic device comprises a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the electronic device implements the image generation method as described in any one of claims 1 to 7.

10. A computer program product, characterized in that The invention comprises a computer program, which enables the image generation method according to any one of claims 1 to 7 to be executed when the computer program is executed.

Images