Picture processing method, light perception model training method, emotion illumination mapping model training method, video display method and electronic equipment

Abstract

The invention discloses a picture processing method, a light perception model training method, an emotion illumination mapping model training method, a video display method and an electronic device, and aims to solve the problem of low authenticity of a target object in a picture. The method comprises the steps of obtaining a target object picture; the target object picture comprises a target object; determining a target light HDR texture picture according to at least one of the background picture and the emotional feature information of the target object; the emotion feature information is used for reflecting the emotion of the target object; the background picture is used for reflecting the environment of the target object; and performing fusion processing on the target object picture and the target light HDR texture picture to obtain a target picture.

Description

Technical Field

[0001] The present application relates to the field of image processing and video technology, and in particular to a picture processing method, a light perception model training method, an emotion lighting mapping model training method, a video display method and an electronic device. Background Art

[0002] In video scenarios such as live broadcast, training, monitoring, and games, when synthesizing videos, the target object image is synthesized with the background image, and video frames are synchronously generated in combination with animations, the target object's voice, etc., and a video stream is generated based on the video frames.

[0003] In the process of implementing the embodiments of the present application, the inventors found that the prior art has the following problems:

[0004] In the synthesized video, the light of different video frames rendered using the initial image of the target object is often consistent, resulting in poor authenticity of the target object. For example, taking the target object as a digital human, a digital task image close to a human image constructed using digital technology is used in live broadcasts, training and other scenes. In the related art, in the synthesized live broadcast or training video, the light of different video frames rendered using the initial image of the digital human is consistent, resulting in poor authenticity of the digital human, which is easy to cause the user's uncanny valley effect.

[0005] Therefore, how to process images to improve the authenticity of target objects in the images is one of the technical problems that need to be urgently solved in the prior art. Summary of the invention

[0006] The embodiments of the present application provide an image processing method, a light perception model training method, an emotion lighting mapping model training method, a video display method and an electronic device, so that the light of the target object image is more realistic and natural, thereby improving the authenticity of the target object in the image.

[0007] In a first aspect, an embodiment of the present application provides a method for processing an image, which can be applied to a server and may include the following steps:

[0008] The server obtains a target object picture; the target object picture includes a target object; a target light HDR texture picture is determined according to at least one of a background picture and emotional characteristic information of the target object; the emotional characteristic information is used to reflect the emotion of the target object; the background picture is used to reflect the environment in which the target object is located; the target object picture is fused with the target light HDR texture picture to obtain a target picture.

[0009] In the image processing method provided in the embodiment of the present application, after the server obtains the target object image, it can determine the target light HDR texture image according to at least one of the background image of the target object and the emotional characteristic information of the target object, and merge the target light HDR texture image with the target object image. Since the light changes can be identified in real time according to the constantly changing background image and / or the emotional characteristics of the target object, the corresponding target light HDR texture image is generated and superimposed and rendered on the target object image, so that the light of the target object image is more realistic and natural, and the authenticity of the target object is improved.

[0010] In a possible implementation, the server can determine the target light HDR texture picture through the following steps: the server inputs the background picture into a light perception model to obtain a first light HDR texture picture, and uses the first light HDR texture picture as the target light HDR texture picture; the light perception model is used to identify the light feature vector of the background picture, and generate the first light HDR texture picture based on the light feature vector.

[0011] In this implementation, the server can input the background image of the target object into the light perception model, identify the light feature vector of the background image, generate a first light HDR texture image based on the light feature vector, and use the first light HDR texture image as the target light HDR texture image to merge with the target object image. In this way, the light in the fused target image can be kept consistent with the light in the background image, making the light in the fused target image more realistic and natural.

[0012] In a possible implementation, the server can determine the target light HDR texture picture through the following steps: the server obtains an emotion feature vector corresponding to the emotion feature information; inputs the emotion feature vector into an emotion lighting mapping model to obtain a second light HDR texture picture, and uses the second light HDR texture picture as the target light HDR texture picture; the emotion lighting mapping model is used to identify the light feature vector corresponding to the emotion feature vector, and generate the second light HDR texture picture based on the light feature vector.

[0013] In this implementation, the server can input the emotion feature vector corresponding to the acquired emotion feature information of the target object into the emotion lighting mapping model, identify the light feature vector corresponding to the emotion feature vector, generate a second light HDR texture picture based on the light feature vector, and use the second light HDR texture picture as the target light HDR texture picture to merge with the target object picture, thereby associating the emotion and lighting of the target object, and making the light in the fused target picture more consistent with the emotion of the target object and more realistic and natural.

[0014] In a possible implementation, the server can determine the target light HDR texture picture through the following steps: the server inputs the background picture into a light perception model to obtain a first light HDR texture picture; the light perception model is used to identify the light feature vector in the background picture, and generate the first light HDR texture picture based on the light feature vector in the background picture; obtain the emotion feature vector corresponding to the emotion feature information; input the emotion feature vector into an emotion lighting mapping model to obtain a second light HDR texture picture; the emotion lighting mapping model is used to identify the light feature vector corresponding to the emotion feature vector, and generate the second light HDR texture picture based on the light feature vector corresponding to the emotion feature vector; based on the first light HDR texture picture and the second light HDR texture picture, determine the target light HDR texture picture.

[0015] In this implementation, the server can jointly determine the target light HDR texture picture according to the background picture of the target object and the emotional characteristic information of the target object. The server can input the background picture of the target object into the light perception model, identify the light feature vector of the background picture, generate a first light HDR texture picture based on the light feature vector of the background picture, input the acquired emotion feature vector corresponding to the emotional characteristic information of the target object into the emotional illumination mapping model, identify the light feature vector corresponding to the emotion feature vector, and generate a second light HDR texture picture based on the light feature vector corresponding to the emotion feature information, thereby associating the emotion and illumination of the target object, and thus, after the target light HDR texture picture obtained based on the first light HDR texture picture and the second light HDR texture picture are fused with the target object picture, the light in the obtained target picture can be consistent with the light in the background picture, so that the light of the fused target picture is more real and natural, and fits the emotion of the target object.

[0016] In a possible implementation manner, the server may obtain text or voice information of the target object; and convert the text or voice information of the target object into the emotion feature information based on a large language model.

[0017] In this implementation, the server can convert the text or voice information of the target object into emotional feature information reflecting the emotions of the target object based on the large language model. The large language model can understand and generate emotional feature information by learning the language pattern and semantics of the text data corresponding to the text data or voice data.

[0018] In a possible implementation, the target object image includes a normal map and a base map, and the server can fuse the target object image with the target light HDR texture image through the following steps to obtain the target image: the server obtains the normal map and the base map in the target object image; identifies the specular reflection light distribution map and the diffuse reflection light distribution map in the normal map; removes the specular reflection light and the diffuse reflection light in the normal map according to the specular reflection light distribution map and the diffuse reflection light distribution map to obtain a first normal map; superimposes the first normal map with the target light HDR texture image to obtain a target normal map; superimposes the target normal map with the base map to obtain the target image.

[0019] In this implementation, the server can remove the specular reflection light and diffuse reflection light in the target object image, and use the target light HDR texture image to replace the specular reflection light and diffuse reflection light in the target object image, so that the light of the fused target image is more realistic and natural.

[0020] In one embodiment, the method further includes: if the server determines that the background image of the target object is updated, and / or the emotional characteristic information is updated, determining an updated target light HDR texture image based on at least one of the updated background image of the target object and the updated emotional characteristic information; fusing the target object image with the updated target light HDR texture image to obtain an updated target image.

[0021] In this implementation, when the background image of the target object is updated, the server can redetermine the target light HDR texture image according to the updated background image, and re-fuse the target object image with the updated target light HDR texture image to obtain an updated target image. The server can also monitor the emotional characteristic information of the target object in real time. When the emotional characteristic information changes, the target light HDR texture image is redetermined according to the updated emotional characteristic information, and re-fuse the target object image with the updated target light HDR texture image to obtain an updated target image. Thus, it can be ensured that the target object image is updated at any time according to the changes in the background image and the changes in the emotional characteristic information of the target object, thereby improving the authenticity of the target object image.

[0022] In a second aspect, an embodiment of the present application provides a picture processing device, which can be applied to a server, and the device may include:

[0023] An acquisition module is used to acquire a target object picture; the target object picture contains the target object;

[0024] A light recognition module, used to determine a target light HDR texture image according to at least one of a background image and emotional characteristic information of the target object; the emotional characteristic information is used to reflect the emotion of the target object; the background image is used to reflect the environment where the target object is located;

[0025] The light fusion module is used to fuse the target object image with the target light HDR texture image to obtain a target image.

[0026] In a possible implementation, the light recognition module is specifically used to input the background image into a light perception model to obtain a first light HDR texture image, and use the first light HDR texture image as the target light HDR texture image; the light perception model is used to identify the light feature vector of the background image, and generate the first light HDR texture image based on the light feature vector.

[0027] In a possible implementation, the light recognition module is specifically used to obtain an emotion feature vector corresponding to the emotion feature information; input the emotion feature vector into an emotion lighting mapping model to obtain a second light HDR texture picture, and use the second light HDR texture picture as the target light HDR texture picture; the emotion lighting mapping model is used to identify the light feature vector corresponding to the emotion feature vector, and generate the second light HDR texture picture based on the light feature vector.

[0028] In a possible implementation, the light recognition module is specifically used to input the background image into a light perception model to obtain a first light HDR texture image; the light perception model is used to identify the light feature vector in the background image, and generate the first light HDR texture image based on the light feature vector in the background image; obtain the emotion feature vector corresponding to the emotion feature information; input the emotion feature vector into an emotion lighting mapping model to obtain a second light HDR texture image; the emotion lighting mapping model is used to identify the light feature vector corresponding to the emotion feature vector, and generate the second light HDR texture image based on the light feature vector corresponding to the emotion feature vector; based on the first light HDR texture image and the second light HDR texture image, determine the target light HDR texture image.

[0029] In a possible implementation, the acquisition module is further used to acquire text or voice information of the target object; and convert the text or voice information of the target object into the emotion feature information based on a large language model.

[0030] In a possible implementation, the target object image includes a normal map and a base map, and the light fusion module is specifically used to obtain the normal map and the base map in the target object image; identify the specular reflection light distribution map and the diffuse reflection light distribution map in the normal map; remove the specular reflection light and the diffuse reflection light in the normal map according to the specular reflection light distribution map and the diffuse reflection light distribution map to obtain a first normal map; superimpose the first normal map with the target light HDR texture image to obtain a target normal map; superimpose the target normal map with the base map to obtain the target image.

[0031] In one embodiment, the light recognition module is further used to determine an updated target light HDR texture picture according to at least one of the updated background picture of the target object and the updated emotional characteristic information if it is determined that the background picture of the target object is updated and / or the emotional characteristic information is updated;

[0032] The light fusion module is also used to fuse the target object image with the updated target light HDR texture image to obtain an updated target image.

[0033] In a third aspect, an embodiment of the present application provides a light perception model training method, which can be applied to a server. The method may include the following steps:

[0034] Get a set of training sample images and a light HDR texture image corresponding to each training sample image;

[0035] Labeling the light HDR texture image corresponding to each training sample image;

[0036] Based on each training sample image and the corresponding labeled light HDR texture image, the diffusion model is trained to obtain a light perception model, wherein the light perception model is used to identify the light feature vector of the background image of the target object, and generate a light HDR texture image based on the light feature vector, and the background image is used to reflect the environment in which the target object is located.

[0037] In a fourth aspect, an embodiment of the present application provides a light perception model training device, which can be applied to a server, and the device may include:

[0038] An acquisition module is used to acquire a set of training sample images and a light HDR texture image corresponding to each training sample image;

[0039] A labeling module, used to label the light HDR texture image corresponding to each training sample image;

[0040] A training module is used to train a diffusion model based on each training sample image and a corresponding labeled light HDR texture image to obtain a light perception model, wherein the light perception model is used to identify a light feature vector of a background image of a target object, and generate a light HDR texture image based on the light feature vector, wherein the background image is used to reflect the environment in which the target object is located.

[0041] In a fifth aspect, an embodiment of the present application provides an emotional lighting mapping model training method, which can be applied to a server. The method may include the following steps:

[0042] Obtain a set of training sample emotion feature vectors and a light HDR texture image corresponding to each training sample emotion feature vector;

[0043] Labeling the light HDR texture image corresponding to each training sample emotion feature vector;

[0044] Based on the emotion feature vector of each training sample and the corresponding labeled light HDR texture picture, the diffusion model is trained to obtain an emotion lighting mapping model, wherein the emotion lighting mapping model is used to identify the light feature vector corresponding to the emotion feature vector corresponding to the emotion feature information of the target object, and generate a light HDR texture picture based on the light feature vector, and the emotion feature information is used to reflect the emotion of the target object.

[0045] In a sixth aspect, an embodiment of the present application provides an emotional lighting mapping model training device, which can be applied to a server, and the device may include:

[0046] An acquisition module, used to acquire a set of training sample emotion feature vectors and a light HDR texture image corresponding to each training sample emotion feature vector;

[0047] A labeling module, used to label the light HDR texture image corresponding to each training sample emotion feature vector;

[0048] A training module is used to train a diffusion model based on the emotion feature vector of each training sample and the corresponding labeled light HDR texture picture to obtain an emotion lighting mapping model, wherein the emotion lighting mapping model is used to identify the light feature vector corresponding to the emotion feature vector corresponding to the emotion feature information of the target object, and generate a light HDR texture picture based on the light feature vector, and the emotion feature information is used to reflect the emotion of the target object.

[0049] In a seventh aspect, an embodiment of the present application provides a video display method, which can be applied in a terminal, and the method may include:

[0050] Receiving a video containing a target object sent by a server, wherein the video is generated by the server according to each target image obtained by using the image processing method provided by an embodiment of the present application;

[0051] Each video frame of the video is displayed.

[0052] In an eighth aspect, an embodiment of the present application provides a video display device, which can be applied to a terminal, and the device may include:

[0053] A receiving module, used to receive a video containing a target object sent by a server, wherein the video is generated by the server according to each target image obtained by the image processing method provided by an embodiment of the present application;

[0054] The display module is used to display each video frame of the video.

[0055] In the ninth aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the image processing method, light perception model training method, or emotion lighting mapping model training method described in the present application is implemented.

[0056] In the tenth aspect, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the image processing method, light perception model training method, emotion lighting mapping model training method or video display method described in the present application are implemented.

[0057] The technical effects that can be achieved in any of the second to tenth aspects mentioned above can refer to the description of the technical effects that can be achieved by any possible design in the first aspect mentioned above, and the repetitions will not be discussed here.

[0058] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0060] Figure 1 A schematic diagram of an application scenario of the image processing method provided in an embodiment of the present application;

[0061] Figure 2 A schematic diagram of the implementation process of the image processing method provided in the embodiment of the present application;

[0062] FIG3( a ) is an example of a background image provided in an embodiment of the present application;

[0063] FIG3( b ) is an example of a digital human image provided in an embodiment of the present application;

[0064] FIG3( c ) is an example diagram of a target image obtained by fusing a digital human image with a target light HDR texture image determined according to a background image, provided in an embodiment of the present application;

[0065] Figure 4 A schematic diagram of the training process of the light perception model provided in an embodiment of the present application;

[0066] Figure 5 is a schematic diagram of the structure of an exemplary stable diffusion model;

[0067] FIG6( a ) is a training sample image provided in an embodiment of the present application;

[0068] FIG6( b ) is a light HDR texture image corresponding to the training sample image shown in FIG6( a ) provided in an embodiment of the present application;

[0069] Figure 7 A schematic diagram of an implementation process for obtaining emotional characteristic information of a target object provided in an embodiment of the present application;

[0070] Figure 8 An example of a prompt text provided in an embodiment of the present application;

[0071] Fig. 9 A schematic diagram of an implementation process of determining a light HDR texture image according to emotional feature information of a target object provided in an embodiment of the present application;

[0072] Fig.10 A schematic diagram of the training process of the emotional lighting mapping model provided in an embodiment of the present application;

[0073] FIG. 11( a ) is an example of a light HDR texture picture corresponding to the emotional feature of “happy” provided in an embodiment of the present application;

[0074] FIG. 11( b ) is an example of a light HDR texture picture corresponding to the emotional feature of “gloomy” provided in an embodiment of the present application;

[0075] FIG. 11( c ) is an example diagram of a light HDR texture image corresponding to a situation characterized by “desire” provided in an embodiment of the present application;

[0076] Fig.12A schematic diagram of an implementation process for determining a light HDR texture image according to a background image and emotional feature information of a target object provided in an embodiment of the present application;

[0077] Fig.13 A schematic diagram of an implementation process of fusing a target object image with a light HDR texture image provided in an embodiment of the present application;

[0078] Fig.14 A schematic diagram of an implementation flow of a digital human image processing method provided in an embodiment of the present application;

[0079] Fig.15 A schematic diagram of the implementation process of another digital human image processing method provided in an embodiment of the present application;

[0080] Fig.16 A schematic diagram of an implementation flow of another digital human image processing method provided in an embodiment of the present application;

[0081] Fig.17 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0082] Fig.18 A schematic diagram of the structure of a light perception model training device provided in an embodiment of the present application;

[0083] Fig.19 A schematic diagram of the structure of the emotional lighting mapping model training device provided in an embodiment of the present application;

[0084] Fig. 20 A schematic diagram of an implementation flow of a video display method provided in an embodiment of the present application;

[0085] Fig.21 A schematic diagram of the structure of a video display device provided in an embodiment of the present application;

[0086] Fig. 22 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0087] The embodiments of the present application provide an image processing method, a light perception model training method, an emotion lighting mapping model training method, a video display method and an electronic device, which make the light of the target object image more realistic and natural, and improve the authenticity of the target object in the image.

[0088] In the embodiments of the present application, "at least one" refers to one or more, and "plurality" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can represent: a, b, c, a and b, a and c, b and c, or a, b and c, where a, b, c can be single or multiple.

[0089] In the embodiments of the present application, descriptions such as "when...", "in the case of...", "if" and "if" all mean that the electronic device will make corresponding processing under certain objective circumstances. It does not limit the time, nor does it require the electronic device to make a judgment when implementing it, nor does it mean that there are other limitations.

[0090] In addition, it should be understood that in the description of this application, words such as "first" and "second" are only used to distinguish the objects described, and cannot be understood as indicating or implying the relative importance of the objects described, nor can they be understood as indicating or implying the order. For example, "first light HDR texture picture" and "second light HDR texture picture" are used to distinguish different light HDR texture pictures, rather than describing the specific order or importance of light HDR texture pictures.

[0091] In summary of the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. The use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way.

[0092] The preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application. In addition, the embodiments and features in the embodiments of the present application may be combined with each other if there is no conflict.

[0093] In this article, it is necessary to understand that the technical terms involved in this application are:

[0094] Digital human: A digital human image close to human image created by digital technology. It is the product of the integration of information science and life science. It uses information science methods to simulate the human body at different levels of form and function. 2D (two-dimensional) digital human is a virtual character designed and drawn on a two-dimensional plane through computer graphics and animation technology, and can perform simple or complex actions.

[0095] WebSocket protocol: a protocol for full-duplex communication on a single TCP (Transmission Control Protocol) connection, which makes data exchange between the client and the server easier and allows the server to actively push data to the client. In the WebSocket API (Application Programming Interface), the client and the server only need to complete a handshake to directly create a persistent connection between the two and perform two-way data transmission.

[0096] Uncanny Valley Effect: When man-made objects (especially robots or animated characters) look and behave very similar to real humans, but not exactly the same, it can give people an uncomfortable or even terrifying feeling.

[0097] HDR (High Dynamic Range Imaging): A technology used in image processing to achieve a wider brightness range than ordinary digital image technology. It captures, processes and displays a wider range of brightness levels than standard digital imaging technology, thereby producing a real-world brightness range that is closer to what the human eye sees.

[0098] Diffusion model: A deep learning framework that is used to generate highly realistic images, audio, and other types of data. It is based on the diffusion process in physics and simulates the process by which data gradually transforms from a pure noise state to a target data state.

[0099] Large language model (LLM): A deep learning model that acquires extensive general knowledge based on massive text data training. It can deeply understand the meaning of text and can be used to handle complex language understanding tasks.

[0100] Normal Map: is a technology that simulates the lighting effect of bumps and depressions, and is an implementation of bump mapping. Normal mapping can add details to the model without adding polygons. A normal is drawn at each point on the bumpy surface of the original object, and the direction of the normal is marked by the RGB color channel, that is, the normal vector of the original object is expressed as a map using RGB colors, outlining the directionality of the lighting, and expressing bumps and body shadows through the direction of the light. Normal mapping is an important method to display more details of a 3D model, which calculates the details of the 3D model surface caused by lighting.

[0101] Cook-Torrance reflection model: The rough surface of an object is considered to be composed of many micro-planes. Each micro-plane can be regarded as an ideal mirror reflector. The roughness of the object surface is represented by the change in the slope of the micro-plane. The rougher the surface, the greater the change in slope, and vice versa. The Cook-Torrance reflection model considers light in two aspects: diffuse reflection intensity and specular reflection intensity: c-t =I diff +I spec , I diff is the diffuse reflection intensity, I spec The Cook-Torrance reflection model can be used to identify the distribution of diffuse and specular light in an image.

[0102] Video streaming: The process of transmitting video content to user terminals via the Internet in the form of a continuous data stream. Users can watch video content in real time or on demand without downloading the entire video file.

[0103] Video frame: A single image in a video stream, which is the basic unit of video. Video is composed of a series of video frames (or picture frames) played in sequence, which are played continuously at a certain rate (such as 24 or 30 frames per second) to produce a dynamic visual effect.

[0104] First reference Figure 1 , which is a schematic diagram of an application scenario of the image processing method provided in an embodiment of the present application, such as Figure 1The picture processing system shown may include a server 101 and a terminal 102, and the server 101 and the terminal 102 are connected through a communication network. For example, the communication network may include, but is not limited to, a mobile communication network, a local area network, a wide area network, an Internet of Things, etc. After the server 101 obtains the target object picture (such as a digital human picture), it may determine the light HDR texture picture according to at least one of the background picture reflecting the environment in which the target object is located and the emotional characteristic information reflecting the target object, and fuse the target object picture with the light HDR texture picture to replace the diffuse reflection light and the specular reflection light in the target object picture with the light HDR texture to obtain the target picture. Since the change of light can be identified in real time according to the constantly changing background picture and / or the emotional characteristics of the target object, the corresponding light HDR texture picture is generated and superimposed and rendered on the target object picture, so that the light of the target object picture is more real and natural, and the authenticity of the target object is improved. Furthermore, the server 101 may synthesize the sound animation of the target object with the obtained multi-frame target picture to generate a video stream, and output the video stream, and transmit the video stream to the terminal 102 according to the WebSocket protocol for loading by the terminal 102.

[0105] In the embodiment of the present application, the server can be an independent physical server or a cloud server that provides basic cloud computing services such as cloud servers, cloud databases, and cloud storage. A server cluster can also be used, which is not limited in the embodiment of the present application.

[0106] based on Figure 1 The application scenario shown below will refer to the attached Figures 2 to 16 The exemplary embodiments of the present application are described in more detail. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principle of the present application, and the implementation of the present application is not limited in any way. On the contrary, the implementation of the present application can be applied to any applicable scenario.

[0107] like Figure 2 As shown, it is a schematic diagram of the implementation process of the image processing method provided in the embodiment of the present application. The image processing method can be applied to Figure 1 The server 101 shown includes the following steps:

[0108] S21. The server obtains a target object image, where the target object image includes a target object.

[0109] In specific implementation, the server may store background images, target object images, text and / or voice information that the target object needs to explain, etc. in a local database. When the target object image needs to be processed, the server may obtain the target object image from the local database. The target object image contains the target object. For example, the digital human image is the target object. The background image is used to reflect the environment in which the target object is located. The background image contains light characteristics, but does not contain the target object.

[0110] Exemplarily, as shown in FIG3(a), it is an example of a background image, FIG3(b) is an example of a digital human image, and FIG3(c) is an example of a target image after the digital human image of FIG3(b) is fused with the target light HDR texture image determined according to the background image of FIG3(a) using the image processing method provided in an embodiment of the present application.

[0111] In one implementation, the user may input the target object image into the terminal and send it to the server through the terminal. The server may also obtain the target object image in other ways, which is not limited in the present embodiment.

[0112] For example, in the scenario of digital human live broadcast or training, a video of digital human explanation needs to be generated. The server can store the background image, digital human image, text and / or voice information that the digital human needs to explain in the local database. When the target object image needs to be processed, the digital human image can be obtained from the local database. The user can also send the digital human image to the server through the terminal.

[0113] S22. The server determines a target light HDR texture picture according to at least one of the background picture and the emotion characteristic information of the target object.

[0114] The target light HDR texture picture includes a first light HDR texture picture and / or a second light HDR texture picture.

[0115] In one implementation, the server may determine the first light HDR texture picture according to the background picture of the target object, and use the first light HDR texture picture as the target light HDR texture picture.

[0116] In specific implementation, after the server obtains the target object image, it can obtain the background image of the target object from the local database, and the user can also input the background image of the target object into the terminal and send it to the server through the terminal. The server inputs the background image into the light perception model to obtain the first light HDR texture image, and uses the first light HDR texture image as the target light HDR texture image. Among them, the light perception model is used to identify the light feature vector of the background image, and generate the first light HDR texture image based on the light feature vector.

[0117] During implementation, the server may pre-assign all background images of the target object to multiple time periods in the order of playback time, for example, 0 to 1 minute corresponds to background image 1, 1 to 3 minutes corresponds to background image 2, and so on. Initially, the server obtains the background image of the target object in the first time period, inputs the obtained background image of the target object into the light perception model, obtains the first light HDR texture image corresponding to the background image of the target object in the first time period, uses the first light HDR texture image as the target light HDR texture image, and then fuses the target object image with the target light HDR texture image. When entering the next time period for background image update, the background image of the target object in the next time period is obtained again, and the first light HDR texture image is continued to be determined based on the next background image. The first light HDR texture image is used as the target light HDR texture image, and the target object image is re-fused with the target light HDR texture image, and the previous fused image is updated until the last background image of the target object.

[0118] Specifically, the light perception model is obtained by training the server based on the training sample image set and the diffusion model. The training process of the light perception model is as follows: Figure 4 As shown, the following steps may be included:

[0119] S31, obtaining a set of training sample images and a light HDR texture image corresponding to each training sample image.

[0120] In specific implementation, the server collects multiple groups of background images and background video frames commonly used in scenes as training sample images, obtains a training sample image set including multiple training sample images, and obtains the light HDR texture image corresponding to each training sample image. The light HDR texture image corresponding to the training sample image can be drawn by professionals or drawing tools.

[0121] S32, labeling the light HDR texture image corresponding to each training sample image.

[0122] In a specific implementation, the server may label the light HDR texture image corresponding to each training sample image, so that the diffusion model can identify the light HDR texture image corresponding to each training sample image.

[0123] S33. Based on each training sample image and the corresponding labeled light HDR texture image, the diffusion model is trained to obtain a light perception model.

[0124] In the specific implementation, the server inputs each training sample image and the corresponding labeled light HDR texture image into the autoencoder, uses the autoencoder to encode each input training sample image, converts it into the corresponding training sample image feature vector, and encodes the light HDR texture image corresponding to each training sample image, converts it into the corresponding light HDR texture image feature vector, and then, each training sample image feature vector and each training sample image corresponding light HDR texture image feature vector are input into the diffusion model, combined with the UNet model in the diffusion model through denoising and denoising, learn the correspondence between the training sample image feature vector and the corresponding light HDR texture image feature vector, identify the light feature vector in the training sample image feature vector, complete the training of the diffusion model in the UNet conditional diffusion process, and obtain the trained diffusion model, which is: light perception model. Among them, the autoencoder can use variational autoencoders (VAEs), and the diffusion model can use a stable diffusion (StableDiffusion) model, such as InstructPix2Pix model, DreamShaper model, Lyriel model, Protogen model, etc., which is not limited in the embodiment of the present application. When determining the training sample image feature vector corresponding to the training sample image, and the light HDR texture image feature vector, any other encoder may also be used, such as but not limited to the following encoders: ViT (Visual Transformer) model, VAE (Variational AutoEncoder) model, VQ-VAE ((Vector Quantised-Variational AutoEncoder) model (a model that combines variational autoencoders and vector quantization technology), etc., which is not limited to the embodiments of the present application.

[0125] like Figure 5As shown, it is a schematic diagram of the structure of an exemplary stable diffusion model. The autoencoder adopts a variational autoencoder, and the diffusion model adopts a stable diffusion model. The stable diffusion model contains two parts: a reconstruction branch and a conditional branch. The reconstruction branch includes a pixel space and a latent space, and the conditional branch includes a conditional input module. The purpose of the reconstruction branch is to gradually remove noise from the noise data and reconstruct a clear image. The key to the reconstruction branch is to learn how to restore the original image from images with different noise levels. This process can be summarized as a reverse diffusion process, which starts from the initial pure noise, gradually reduces the noise through the steps of adding noise and removing noise, and finally obtains a clear image. Among them, the denoising process (Forward Process): During the training process, a series of images with different noise levels are obtained by gradually adding noise to the original image, and the denoising process (Reverse Process): During the generation process, starting from a random noise image, the noise is gradually removed through reverse diffusion to reconstruct a clear image. The conditional branch is used to introduce additional conditions or information when generating an image, so that the generated image meets specific requirements or descriptions. Conditional branches can include conditions in various forms, such as text descriptions, category labels, partial images, etc. These conditions affect the generation process in the following ways: (1) Conditional input: Conditional information (such as text descriptions, images, etc.) is provided to the model as additional input, and the model will guide the image generation process based on these conditions. (2) Conditional embedding: Conditional information is embedded into each layer of the generation model so that the generation process is affected by the conditions at each step. (3) Adjustment of the loss function: During the training process, the loss function can include condition-related terms so that the model can better meet the requirements of the conditions when generating images. Through conditional branches, the stable diffusion model can generate images that meet specific conditions, such as generating corresponding images based on a given text description, or completing the rest of the image based on the content of a part of the image. This structure enables the stable diffusion model to generate high-quality images and generate images that meet the requirements based on specific conditions.

[0126] In the implementation of this application, when training the light perception model, the light HDR texture image feature vector corresponding to the training sample image is used as the conditional input conditional branch, and the UNet model is combined with the noise addition and denoising process to learn the correspondence between the training sample image feature vector and the corresponding light HDR texture image feature vector, and identify the light feature vector in the training sample image feature vector.

[0127] As shown in Figure 6(a), it is a training sample image, and Figure 6(b) is the light HDR texture image corresponding to Figure 6(a).

[0128] Light HDR texture images contain richer colors and textures: Light HDR texture HDR images can capture more subtle changes in light and color, provide richer colors and details than ordinary images, and enhance the sense of reality and visual effects. Light HDR texture images can show detailed information of highlight and shadow areas in the same image, and can adapt to both bright sunlight and dark indoor environments, providing a wider dynamic range. In addition to ordinary RGB information, each pixel of the light HDR texture image also has the actual brightness information of the point, while ordinary graphic files only have a grayscale range of 0 to 255 for each pixel.

[0129] In the implementation of this application, the server inputs the background image of the target object into the light perception model, and obtains the light HDR texture image corresponding to the background image. When the target object image and the target light HDR texture image are subsequently fused (the fusion process is detailed in step S23), the light in the fused target image can be kept consistent with the light in the background image, making the light in the fused target image more realistic and natural.

[0130] In one implementation, the server may determine a second light HDR texture picture according to the emotion characteristic information of the target object, and use the second light HDR texture picture as the target light HDR texture picture, wherein the emotion characteristic information is used to reflect the emotion of the target object.

[0131] In specific implementation, the server can follow the following Figure 7 The process shown in the figure obtains the emotional characteristic information of the target object, including the steps of:

[0132] S41. Obtain text or voice information of the target object.

[0133] In specific implementation, the server may periodically obtain text or voice information of the target object, wherein the text of the target object is the text content corresponding to the voice segment to be played by the target object, and the voice information of the target object is the voice segment to be played by the target object.

[0134] During implementation, the server may pre-divide all the voices to be played by the target object into multiple voice segments in chronological order, and divide the corresponding time period according to each voice segment. Initially, the server obtains the text or voice information of the target object in the first time period, that is, the first voice segment to be played by the target object or the text content corresponding to the first voice segment, identifies the emotional feature information according to the obtained text or voice information of the target object, determines the second light HDR texture image according to the emotional feature information, and uses the second light HDR texture image as the target light HDR texture image, and then fuses the target object image with the target light HDR texture image. After the first voice segment is played, the text or voice information of the target object in the next time period is obtained, that is, the text content corresponding to the next voice segment or the next voice segment, and continues to identify the emotional feature information according to the text content corresponding to the next voice segment or the next voice segment. If the emotional feature information changes compared with the emotional feature information of the previous cycle, the target object image is re-fused with the target light HDR texture image, and the previous fused image is updated. If the emotional feature information does not change compared with the emotional feature information of the previous cycle, the target object image is not processed until the last voice segment of the target object or the text content corresponding to the last voice segment.

[0135] S42. Convert the text or voice information of the target object into emotional feature information based on the large language model.

[0136] During specific implementation, the server may input the text corresponding to the text or voice information of the target object into a pre-trained large language model to obtain the corresponding emotion feature information. The emotion feature information is the emotion description text corresponding to the text corresponding to the text or voice information of the target object, which is the text used to reflect the emotion of the target object. The large language model learns the language pattern and semantics of the text corresponding to the text data or voice data, so as to understand and generate the emotion feature information. The server may use large-scale text data to train the large language model in advance. The large language model learns the language pattern and semantics of the text data, so as to understand and generate natural language text. The large language model can use, but is not limited to, any of the following series of models: Qwen series model, GLM series model, DeepSeek series model, etc., which is not limited to the embodiments of the present application.

[0137] If the server obtains the voice information of the target object, it is necessary to first convert the voice information into corresponding text, and then convert the text corresponding to the voice information of the target object into emotional feature information (ie, emotional description text) based on the large language model.

[0138] When the server converts the text corresponding to the text or voice information of the target object into the emotion description text, it can simultaneously input several prompt texts into the large language model to assist the large language model in learning to convert the text corresponding to the text or voice information of the target object into the emotion description text, wherein the prompt text is a text with a prompt function. Figure 8 As shown in the figure, it is an example of a prompt text. It can be seen that the prompt large language model input is: the type is text or voice, the input data is "I'm so happy today, the weather is great and I got a promotion!" ("I am very happy today, the weather is great, and I got a promotion!"), the output format is "json" format, the output fields include: emotion, intensity, content (i.e., the content of the emotion description text), the language is English, and the intensity scale includes: low, medium, and high. The output of the large language model is specifically: the emotion is "happy", the intensity is "high", and the content is "Positive sentiment expressed due to good weather and a job promotion." ("Positive emotions expressed due to good weather and job promotion"). This prompt text only takes the input English text as an example. The language of the prompt text can be the same language as the text or voice information of the target object, but is not limited to this.

[0139] After the server obtains the emotional feature information of the target object through the large language model, it can determine the second light HDR texture picture according to the emotional feature vector corresponding to the emotional feature information of the target object and the emotional lighting mapping model, and use the second light HDR texture picture as the target light HDR texture picture.

[0140] When implementing it specifically, you can follow the following steps: Fig. 9 The process shown determines the second light HDR texture picture according to the emotional characteristic information of the target object, including the following steps:

[0141] S51. Obtain an emotion feature vector corresponding to the emotion feature information.

[0142] In specific implementation, after the server obtains the emotional feature information of the target object through a large language model, a text encoder can be used to convert the emotional feature information of the target object into an emotional feature vector, wherein the text encoder can be but is not limited to the following encoders: text vector model Text2vec model, BERT (Bidirectional Encoder Representations from Transformer, bidirectional encoder representation based on transformer) model, CLIP (Contrastive Language-Image Pre-training) model (which is a multimodal pre-training model), SBERT (Sentence-BERT) model (which is a pre-training model based on BERT), etc., and the embodiments of the present application are not limited to this.

[0143] S52: Input the emotion feature vector into the emotion lighting mapping model to obtain a second light HDR texture picture.

[0144] In a specific implementation, after obtaining the emotion feature vector corresponding to the emotion feature information of the target object, the server inputs the emotion feature vector corresponding to the emotion feature information of the target object into the emotion light mapping model to obtain the second light HDR texture picture, and uses the second light HDR texture picture as the target light HDR texture picture. The emotion light mapping model is used to identify the light feature vector corresponding to the emotion feature vector, and generate the second light HDR texture picture based on the light feature vector corresponding to the emotion feature vector.

[0145] In specific implementation, the emotional lighting mapping model is obtained by training the server based on the emotional feature vector set of training samples and the diffusion model. The training process of the emotional lighting mapping model is as follows: Fig.10 As shown, the following steps may be included:

[0146] S61, obtaining a set of training sample emotion feature vectors and a light HDR texture image corresponding to each training sample emotion feature vector.

[0147] In specific implementation, the server can collect a large number of voices and / or texts (that is, text content corresponding to the voices) in various types of movies and light HDR texture pictures that reflect the lighting effects of the corresponding scenes, wherein the lighting effects may include: cold light, warm light, backlight, contour light, etc. The light HDR texture pictures corresponding to the voices and / or texts can be drawn by professionals or drawing tools, and the light HDR texture pictures may include light HDR texture pictures with different lighting effects such as cold light HDR texture pictures, warm light HDR texture pictures, backlight HDR texture pictures, and contour light HDR texture pictures. Furthermore, the server converts the text and / or text corresponding to each voice into corresponding emotional feature information based on the large language model. The specific implementation process can be referred to in step S42, which will not be repeated here. After obtaining the emotional feature information corresponding to each voice and / or text, the emotional feature information corresponding to each voice and / or text is converted into an emotional feature vector using a text encoder. In this way, the corresponding relationship between the emotional feature vector corresponding to each voice and / or text and the light HDR texture picture can be obtained, and the emotional feature vector corresponding to each voice and / or text is used as a training sample emotional feature vector, which together constitutes a training sample emotional feature vector set.

[0148] Exemplarily, Figure 11(a) is an example of a light HDR texture picture corresponding to a situation where the emotional feature is "happy", Figure 11(b) is an example of a light HDR texture picture corresponding to a situation where the emotional feature is "gloomy", and Figure 11(c) is an example of a light HDR texture picture corresponding to a situation where the situation feature is "hope".

[0149] S62, labeling the light HDR texture image corresponding to each training sample emotion feature vector.

[0150] In specific implementation, the server may label the light HDR texture image corresponding to each training sample emotion feature vector, so that the diffusion model can identify the light HDR texture image corresponding to each training sample emotion feature vector.

[0151] S63. Based on the emotion feature vector of each training sample and the corresponding labeled light HDR texture picture, the diffusion model is trained to obtain an emotion lighting mapping model.

[0152] In the specific implementation, the server inputs each training sample emotion feature vector and the corresponding labeled light HDR texture picture into the autoencoder, uses the autoencoder to encode each input light HDR texture picture, and converts it into the corresponding light HDR texture picture feature vector, and then inputs the training sample emotion feature vector and the corresponding light HDR texture picture feature vector into the diffusion model, and combines the UNet model in the diffusion model through denoising and denoising to learn the correspondence between the training sample emotion feature vector and the corresponding light HDR texture picture feature vector, identify the light feature vector corresponding to the training sample emotion feature vector, and complete the training of the diffusion model in the UNet conditional diffusion process, and obtain the trained diffusion model, which is: the emotion illumination mapping model. Among them, the autoencoder can use variational autoencoders (VAEs), and the diffusion model can use a stable diffusion model, such as the InstructPix2Pix model, the DreamShaper model, the Lyriel model, the Protogen model, etc., which is not limited in the embodiment of the present application. When determining the feature vector of the light HDR texture image, any other encoder may also be used, such as but not limited to the following encoders: ViT model, VAE model, VQ-VAE model, etc., which is not limited to the embodiments of the present application.

[0153] In the implementation of this application, the server inputs the emotion feature vector corresponding to the emotion feature information of the target object into the emotion illumination mapping model, and obtains the light HDR texture image corresponding to the emotion feature information of the target object. Thus, the emotion and illumination (i.e., lighting effect) of the target object can be associated, and then,

[0154] When the target object image is subsequently fused with the light HDR texture image, the light in the fused target image can be more in line with the mood of the target object, and more realistic and natural.

[0155] In one implementation, the server may determine the light HDR texture image based on the background image and emotional characteristic information of the target object.

[0156] When implementing it specifically, you can follow the following steps: Fig.12 The process shown in the figure determines the target light HDR texture image according to the background image and emotional feature information of the target object, including the following steps:

[0157] S71. Input the background image into a light perception model to obtain a first light HDR texture image.

[0158] In a specific implementation, the server may input the background image of the target object into the light perception model to obtain a light HDR texture image corresponding to the background image of the target object, which may be recorded as the first light HDR texture image. The light perception model is used to identify the light feature vector in the background image, and generate the first light HDR texture image based on the light feature vector in the background image.

[0159] The implementation of this step can refer to the implementation of the first embodiment described above, and will not be described in detail here.

[0160] S72, obtaining an emotion feature vector corresponding to the emotion feature information; inputting the emotion feature vector into an emotion lighting mapping model to obtain a second light HDR texture picture.

[0161] In a specific implementation, the server obtains an emotion feature vector corresponding to the emotion feature information of the target object, inputs the emotion feature vector into the emotion light mapping model, and obtains a light HDR texture picture corresponding to the emotion feature information of the target object, which can be recorded as the second light HDR texture picture. The emotion light mapping model is used to identify the light feature vector corresponding to the emotion feature vector, and generate the second light HDR texture picture based on the light feature vector corresponding to the emotion feature vector.

[0162] The implementation of this step can refer to the implementation of the second embodiment described above, and will not be described in detail here.

[0163] S73. Determine a target light HDR texture picture based on the first light HDR texture picture and the second light HDR texture picture.

[0164] In specific implementation, the server may use the first light HDR texture picture and the second light HDR texture picture as light HDR texture pictures respectively, so that two light HDR texture pictures can be obtained for subsequent fusion processing with the target object picture.

[0165] The server may also superimpose the first light HDR texture picture with the second light HDR texture picture to form a light HDR texture picture, and use the superimposed light HDR texture picture as the target light HDR texture picture for subsequent fusion processing with the target object picture.

[0166] Therefore, after the server subsequently fuses the target light HDR texture picture obtained based on the first light HDR texture picture and the second light HDR texture picture with the target object picture, the light in the obtained target picture can be consistent with the light in the background picture, making the light of the fused target picture more realistic and natural, and in line with the emotion of the target object.

[0167] S23. The server fuses the target object image with the target light HDR texture image to obtain a target image.

[0168] When implementing it specifically, you can follow the following steps: Fig.13 The process shown fuses the target object image with the light HDR texture image, including the following steps:

[0169] S81. Obtain a normal map and a base map in a target object image.

[0170] In a specific implementation, the target object image includes a normal map and a base map, the base map is generated by removing the normal map from the target object image, and the base map is also: the target object image without the normal map. The server can obtain the normal map and the base map in the target object image based on the normal map recognition model.

[0171] Specifically, the server can input the target object image into the normal map recognition model, and output the normal map and the base map in the target object image. The normal map recognition model can be obtained by training a preset deep learning model based on a training sample set, and the preset deep learning model can be, but is not limited to, the following models: Normal Net model, GAN (Generative Adversarial Networks, Generative Adversarial Networks) model, DCNNs (Deep Convolutional Neural Networks, Deep Convolutional Neural Networks) model, etc., which is not limited in the embodiments of the present application. When training the normal map recognition model, the server can collect multiple sample object images containing different objects (such as people, objects, etc.) to obtain a training sample set, label the normal map and background map in each sample object image respectively, and use an encoder to encode the sample object image, normal map and background map respectively to obtain the corresponding sample object image vector, normal map vector and background map vector, input each sample object image vector, normal map vector and background map vector into a preset deep learning model for training, adjust the parameters of the preset deep learning model until the model converges, and obtain the trained normal map recognition model.

[0172] S82. Identify a specular reflection light distribution map and a diffuse reflection light distribution map in the normal map.

[0173] In specific implementation, the server can identify the specular reflection light distribution map and the diffuse reflection light distribution map in the normal map according to the reflection model. Among them, the reflection model can be but not limited to the following models: Cook-Torrance reflection model, Phong reflection model, Blinn-Phong reflection model, Ward reflection model, etc., which is not limited in the embodiment of the present application.

[0174] S83. According to the specular reflection light distribution map and the diffuse reflection light distribution map, remove the specular reflection light and the diffuse reflection light in the normal map to obtain a first normal map.

[0175] During specific implementation, the server can remove the specular reflection light and the diffuse reflection light from the normal map according to the obtained specular reflection light distribution map and the diffuse reflection light distribution map, and obtain a normal map from which the specular reflection light and the diffuse reflection light are removed, which can be recorded as the first normal map, and the first normal map is displayed in a grayish white state.

[0176] S84, superimposing the first normal map and the target light HDR texture image to obtain a target normal map.

[0177] In one embodiment, after obtaining the target object image with the normal map removed and the first normal map, the server may superimpose the first normal map with the first light HDR texture image determined according to the background image of the target object to re-light the first normal map to obtain the target normal map.

[0178] In one embodiment, after obtaining the target object image with the normal map removed and the first normal map, the server may superimpose the first normal map with a second light HDR texture image determined according to emotional characteristic information of the target object to re-light the first normal map to obtain a target normal map.

[0179] In one embodiment, after obtaining the target object image with the normal map removed and the first normal map, the server may also superimpose the first normal map with a target light HDR texture image determined based on the background image and emotional characteristic information of the target object to re-light the first normal map to obtain a target normal map.

[0180] In specific implementation, the server may superimpose the first normal map with a target light HDR texture picture obtained by superimposing the first light HDR texture picture and the second light HDR texture picture, so as to re-illuminate the first normal map and obtain a target normal map.

[0181] The server may also superimpose the first normal map with a first light HDR texture image determined according to a background image of the target object and a second light HDR texture image determined according to emotional characteristic information of the target object, so as to re-illuminate the first normal map and obtain a target normal map.

[0182] S85, superimposing the target normal map and the base map to obtain a target image.

[0183] In specific implementation, after obtaining the target normal map after re-illumination processing, the server superimposes the target normal map with the base map of the target object image to obtain the target image, which is the target object image whose specular reflection light and diffuse reflection light are replaced with the light in the light HDR texture image.

[0184] In one implementation, in order to improve the display effect of the fused target image, after obtaining the target normal map, the target normal map may be further enhanced in detail, and then the target normal map after the detail enhancement is superimposed on the base image of the target object image to obtain the target image. During implementation, any image enhancement processing method may be used to perform detail enhancement processing on the target normal map, which will not be described in detail here.

[0185] In the embodiment of the present application, since the server removes the specular reflection light and diffuse reflection light in the target object image, the specular reflection light and diffuse reflection light in the target object image are replaced by the light HDR texture image determined according to the background image of the target object and / or the emotional characteristic information, thereby making the light of the processed target object image more realistic and natural.

[0186] Then, the server will synchronize the target object image and the target object sound after re-synthesizing the light to generate an animation, output the video frame, and send the video frame to the terminal through the WebSocket protocol, which will be loaded and displayed by the terminal.

[0187] In one embodiment, when the server determines that the background image of the target object is updated and / or the emotional characteristic information of the target object is updated, the server determines an updated target light HDR texture image based on at least one of the updated background image of the target object and the updated emotional characteristic information; and fuses the target object image with the updated target light HDR texture image to obtain an updated target image.

[0188] In specific implementation, during the process of synthesizing video frames, when the background image of the target object is updated, the server can re-determine the target light HDR texture image according to the updated background image, and re-fuse the target object image with the updated target light HDR texture image to obtain the updated target image. The server can also monitor the emotional characteristic information of the target object in real time. When the emotional characteristic information changes, the target light HDR texture image is re-determined according to the updated emotional characteristic information, and re-fuse the target object image with the updated target light HDR texture image to obtain the updated target image. Thus, it can be ensured that the target object image is updated at any time according to the changes in the background image and the changes in the emotional characteristic information of the target object, thereby improving the authenticity of the target object image.

[0189] The following takes the target object image as a digital human image as an example to illustrate the processing of the digital human image.

[0190] like Fig.14 FIG. 1 is a schematic diagram of an implementation flow of a digital human image processing method provided in an embodiment of the present application, which may include the following steps:

[0191] S91. The server obtains the digital human image and the digital human background image.

[0192] S92: The server inputs the background image of the digital human into the light perception model to obtain a first light HDR texture image.

[0193] S93. The server inputs the digital human image into a normal map recognition model to obtain a normal map and a base map of the digital human image.

[0194] S94. The server identifies a specular reflection light distribution map and a diffuse reflection light distribution map in the normal map according to the reflection model.

[0195] S95. The server removes the specular reflection light and the diffuse reflection light in the normal map according to the specular reflection distribution map and the diffuse reflection distribution map to obtain a first normal map.

[0196] S96. The server superimposes the first normal map and the first light HDR texture image to obtain a target normal map.

[0197] S97. The server superimposes the target normal map with the base map to obtain a target image.

[0198] Furthermore, the server will synchronize the obtained re-illuminated digital human image with the digital human's voice to generate an animation, and use a 3D engine to map the digital human's voice to video frames such as the digital human's movements, postures, mouth shapes, and expressions in real time, and send each video frame to the terminal through the WebSocket protocol, which will be loaded and displayed by the terminal.

[0199] like Fig.15 FIG. 1 is a schematic diagram of an implementation flow of another digital human image processing method provided in an embodiment of the present application, which may include the following steps:

[0200] S101. The server obtains a digital human image, and text or voice information of the digital human.

[0201] S102: The server inputs the text corresponding to the text or voice information of the digital person into the large language model to obtain the emotional feature information of the digital person.

[0202] S103: The server uses a text encoder to convert the digital human emotion feature information into an emotion feature vector.

[0203] S104: The server inputs the emotion feature vector into an emotion lighting mapping model to obtain a second light HDR texture picture.

[0204] S105: The server inputs the digital human image into a normal map recognition model to obtain a normal map and a base map of the digital human image.

[0205] S106. The server identifies a specular reflection light distribution map and a diffuse reflection light distribution map in the normal map according to the reflection model.

[0206] S107: The server removes the specular reflection light and the diffuse reflection light in the normal map according to the specular reflection distribution map and the diffuse reflection distribution map to obtain a first normal map.

[0207] S108. The server superimposes the first normal map and the second light HDR texture image to obtain a target normal map.

[0208] S109: The server superimposes the target normal map with the base map to obtain a target image.

[0209] Furthermore, the server will synchronize the obtained re-illuminated digital human image with the digital human's voice to generate an animation, and use a 3D engine to map the digital human's voice to video frames such as the digital human's movements, postures, mouth shapes, and expressions in real time, and send each video frame to the terminal through the WebSocket protocol, which will be loaded and displayed by the terminal.

[0210] like Fig.16 As shown, it is a schematic diagram of the implementation process of another digital human image processing method provided in the embodiment of the present application, which may include the following steps:

[0211] S111. The server obtains the digital human image, the digital human background image, and the digital human text or voice information.

[0212] S112: The server inputs the light perception model of the background image of the digital human to obtain a first light HDR texture image.

[0213] S113: The server inputs the text corresponding to the text or voice information of the digital person into the large language model to obtain the emotional feature information of the digital person.

[0214] S114. The server uses a text encoder to convert the digital human emotion feature information into an emotion feature vector.

[0215] S115. The server inputs the emotion feature vector into the emotion lighting mapping model to obtain a second light HDR texture picture.

[0216] S116: The server inputs the digital human image into a normal map recognition model to obtain a normal map and a base map of the digital human image.

[0217] S117. The server identifies a specular reflection light distribution map and a diffuse reflection light distribution map in the normal map according to the reflection model.

[0218] S118. The server removes the specular reflection light and the diffuse reflection light in the normal map according to the specular reflection distribution map and the diffuse reflection distribution map to obtain a first normal map.

[0219] S119. The server superimposes the first normal map with the first light HDR texture image and the second light HDR texture image to obtain a target normal map.

[0220] S1110. The server superimposes the target normal map with the base map to obtain a target image.

[0221] Furthermore, the server will synchronize the obtained re-illuminated digital human image with the digital human's sound animation to generate an animation, and can use a 3D engine to map the digital human's voice to video frames such as the digital human's movements, postures, mouth shapes, and expressions in real time, and send each video frame to the terminal through the WebSocket protocol, which will be loaded and displayed by the terminal.

[0222] Based on the same inventive concept, an embodiment of the present application also provides an image processing device. Since the principle of solving the problem by the above-mentioned image processing device is similar to that of the above-mentioned image processing method, the implementation of the above-mentioned device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0223] like Fig.17 As shown, it is a structural schematic diagram of the image processing device provided in an embodiment of the present application, which may include:

[0224] An acquisition module 121 is used to acquire a target object picture; the target object picture includes a target object;

[0225] The light recognition module 122 is used to determine the target light HDR texture picture according to at least one of the background picture and the emotional characteristic information of the target object; the emotional characteristic information is used to reflect the emotion of the target object; the background picture is used to reflect the environment where the target object is located;

[0226] The light fusion module 123 is used to fuse the target object image with the target light HDR texture image to obtain a target image.

[0227] In a possible implementation, the light recognition module 122 is specifically used to input the background image into a light perception model to obtain a first light HDR texture image, and use the first light HDR texture image as the target light HDR texture image; the light perception model is used to identify the light feature vector of the background image, and generate the first light HDR texture image based on the light feature vector.

[0228] In a possible implementation, the light recognition module 122 is specifically used to obtain an emotion feature vector corresponding to the emotion feature information; input the emotion feature vector into an emotion lighting mapping model to obtain a second light HDR texture picture, and use the second light HDR texture picture as the target light HDR texture picture; the emotion lighting mapping model is used to identify the light feature vector corresponding to the emotion feature vector, and generate the second light HDR texture picture based on the light feature vector.

[0229] In a possible implementation, the light recognition module 122 is specifically used to input the background image into a light perception model to obtain a first light HDR texture image; the light perception model is used to identify the light feature vector in the background image, and generate the first light HDR texture image based on the light feature vector in the background image; obtain the emotion feature vector corresponding to the emotion feature information; input the emotion feature vector into an emotion lighting mapping model to obtain a second light HDR texture image; the emotion lighting mapping model is used to identify the light feature vector corresponding to the emotion feature vector, and generate the second light HDR texture image based on the light feature vector corresponding to the emotion feature vector; based on the first light HDR texture image and the second light HDR texture image, determine the target light HDR texture image.

[0230] In a possible implementation, the acquisition module 121 is further configured to acquire text or voice information of the target object; and convert the text or voice information of the target object into the emotion feature information based on a large language model.

[0231] In a possible implementation, the target object image includes a normal map and a base map, and the light fusion module 123 is specifically used to obtain the normal map and the base map in the target object image; identify the specular reflection light distribution map and the diffuse reflection light distribution map in the normal map; remove the specular reflection light and the diffuse reflection light in the normal map according to the specular reflection light distribution map and the diffuse reflection light distribution map to obtain a first normal map; superimpose the first normal map with the target light HDR texture image to obtain a target normal map; superimpose the target normal map with the base map to obtain the target image.

[0232] In one embodiment, the light recognition module 122 is further used to determine an updated target light HDR texture picture according to at least one of the updated background picture of the target object and the updated emotional characteristic information if it is determined that the background picture of the target object is updated and / or the emotional characteristic information is updated;

[0233] The light fusion module 123 is further used to fuse the target object image with the updated target light HDR texture image to obtain an updated target image.

[0234] Based on the same inventive concept, the present application also provides a light perception model training method, which can be applied to Figure 1 In the server 101 shown, the light perception model training method may include the following steps:

[0235] S131, obtaining a set of training sample images and a light HDR texture image corresponding to each training sample image.

[0236] S132, labeling the light HDR texture image corresponding to each training sample image.

[0237] S133: Based on each training sample image and the corresponding labeled light HDR texture image, the diffusion model is trained to obtain a light perception model.

[0238] The light perception model is used to identify the light feature vector of the background image of the target object, and generate a light HDR texture image based on the light feature vector, and the background image is used to reflect the environment in which the target object is located.

[0239] The implementation of steps S131 to S133 may refer to the implementation of steps S31 to S33, and will not be described in detail here.

[0240] Based on the same inventive concept, an embodiment of the present application also provides a light perception model training device. Since the principle of solving the problem by the above light perception model training device is similar to that of the above light perception model training method, the implementation of the above device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0241] like Fig.18 As shown, it is a schematic diagram of the structure of the light perception model training device provided in an embodiment of the present application. The light perception model training device can be applied to Figure 1 The server 101 shown may include:

[0242] An acquisition module 141 is used to acquire a set of training sample images and a light HDR texture image corresponding to each training sample image;

[0243] A labeling module 142 is used to label the light HDR texture image corresponding to each training sample image;

[0244] The training module 143 is used to train the diffusion model based on each training sample image and the corresponding labeled light HDR texture image to obtain a light perception model, wherein the light perception model is used to identify the light feature vector of the background image of the target object, and generate a light HDR texture image based on the light feature vector, and the background image is used to reflect the environment in which the target object is located.

[0245] Based on the same inventive concept, the present application also provides an emotional lighting mapping model training method, which can be applied to Figure 1 In the server 101 shown, the emotional lighting mapping model training method may include the following steps:

[0246] S151, obtaining a set of training sample emotion feature vectors and a light HDR texture image corresponding to each training sample emotion feature vector.

[0247] S152, labeling the light HDR texture image corresponding to each training sample emotion feature vector.

[0248] S153, based on each training sample emotion feature vector and the corresponding labeled light HDR texture picture, training the diffusion model to obtain an emotion lighting mapping model.

[0249] Among them, the emotional lighting mapping model is used to identify the light feature vector corresponding to the emotional feature vector corresponding to the emotional feature information of the target object, and generate a light HDR texture picture based on the light feature vector, and the emotional feature information is used to reflect the emotion of the target object.

[0250] The implementation of steps S151 to S153 may refer to the implementation of steps S61 to S63, and will not be described in detail here.

[0251] Based on the same inventive concept, an embodiment of the present application also provides an emotional lighting mapping model training device. Since the principle of solving the problem by the above-mentioned emotional lighting mapping model training device is similar to the above-mentioned emotional lighting mapping model training method, the implementation of the above-mentioned device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0252] like Fig.19 As shown, it is a schematic diagram of the structure of the emotional lighting mapping model training device provided in the embodiment of the present application. The emotional lighting mapping model training device can be applied to Figure 1 The server 101 shown may include:

[0253] An acquisition module 161 is used to acquire a set of training sample emotion feature vectors and a light HDR texture image corresponding to each training sample emotion feature vector;

[0254] A labeling module 162, used for labeling the light HDR texture picture corresponding to each training sample emotion feature vector;

[0255] The training module 163 is used to train the diffusion model based on the emotion feature vector of each training sample and the corresponding labeled light HDR texture picture to obtain an emotion lighting mapping model, wherein the emotion lighting mapping model is used to identify the light feature vector corresponding to the emotion feature vector corresponding to the emotion feature information of the target object, and generate a light HDR texture picture based on the light feature vector, and the emotion feature information is used to reflect the emotion of the target object.

[0256] like Fig. 20 As shown, an embodiment of the present application provides a video display method, which can be applied in a terminal. The method may include the following steps:

[0257] S171. The terminal receives a video containing a target object sent by a server.

[0258] Among them, the video is generated by the server based on each target image obtained by the image processing method provided in the embodiment of the present application, and the repeated parts are not repeated here.

[0259] S172: Display each video frame of the video.

[0260] like Fig.21 As shown, an embodiment of the present application provides a video display device, which can be applied to a terminal, and the device may include:

[0261] A receiving module 181 is used to receive a video containing a target object sent by a server, wherein the video is generated by the server according to each target image obtained by the image processing method provided by the embodiment of the present application;

[0262] The display module 182 is used to display each video frame of the video.

[0263] Based on the same technical concept, the present application embodiment also provides an electronic device 190, referring to Fig. 22As shown, the electronic device 190 is used to implement the image processing method described in the above method embodiment. The electronic device 190 of this embodiment may include: a memory 191, a processor 192, and a computer program stored in the memory and executable on the processor, such as an image processing program. When the processor executes the computer program, the steps in the above-mentioned various image processing method embodiments are implemented.

[0264] The specific connection medium between the memory 191 and the processor 192 is not limited in the embodiment of the present application. Fig. 22 In the embodiment, the memory 191 and the processor 192 are connected via a bus 193. The bus 193 is Fig. 22 The connection between other components is shown by bold lines, and is not intended to be limiting. The bus 193 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig. 22 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0265] The memory 191 may be a volatile memory, such as a random-access memory (RAM); the memory 191 may also be a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD), or the memory 191 is any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 191 may be a combination of the above memories.

[0266] Processor 192 is used to implement the image processing method, light perception model training method or emotion lighting mapping model training method provided in the embodiments of the present application.

[0267] An embodiment of the present application also provides a computer-readable storage medium that stores computer-executable instructions required to execute the above-mentioned processor, which includes a program required to execute the above-mentioned processor.

[0268] In some possible implementations, various aspects of the image processing method provided in the present application may also be implemented in the form of a program product, which includes a program code. When the program product is run on an electronic device, the program code is used to enable the electronic device to execute the steps of the image processing method according to various exemplary implementations of the present application described above in this specification.

[0269] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, devices, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0270] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (apparatus), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0271] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0272] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0273] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0274] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for processing an image, characterized in that: include: Get the target object image; The target object picture contains a target object; Determine a target light HDR texture picture according to at least one of a background picture and emotional characteristic information of the target object; the emotional characteristic information is used to reflect the emotion of the target object; and the background picture is used to reflect the environment in which the target object is located; The target object image is fused with the target light HDR texture image to obtain a target image.

2. The method according to claim 1, characterized in that Determine the target light HDR texture image, including: The background image is input into a light perception model to obtain a first light HDR texture image, and the first light HDR texture image is used as the target light HDR texture image; the light perception model is used to identify a light feature vector of the background image, and generate the first light HDR texture image based on the light feature vector.

3. The method according to claim 1, characterized in that Determine the target light HDR texture image, including: Obtaining an emotion feature vector corresponding to the emotion feature information; The emotion feature vector is input into the emotion lighting mapping model to obtain a second light HDR texture picture, and the second light HDR texture picture is used as the target light HDR texture picture; the emotion lighting mapping model is used to identify the light feature vector corresponding to the emotion feature vector, and generate the second light HDR texture picture based on the light feature vector.

4. The method according to claim 1, characterized in that Determine the target light HDR texture image, including: Inputting the background image into a light perception model to obtain a first light HDR texture image; the light perception model is used to identify a light feature vector in the background image, and generate the first light HDR texture image based on the light feature vector in the background image; Acquire an emotion feature vector corresponding to the emotion feature information; input the emotion feature vector into an emotion lighting mapping model to obtain a second light HDR texture picture; the emotion lighting mapping model is used to identify a light feature vector corresponding to the emotion feature vector, and generate the second light HDR texture picture based on the light feature vector corresponding to the emotion feature vector; The target light HDR texture picture is determined based on the first light HDR texture picture and the second light HDR texture picture.

5. The method according to claim 1, characterized in that The method further comprises: Acquiring text or voice information of the target object; The text or voice information of the target object is converted into the emotion feature information based on a large language model.

6. The method according to any one of claims 1 to 5, characterized in that: The target object image includes a normal map and a base map, and the target object image is fused with the target light HDR texture image to obtain a target image, specifically including: Obtaining the normal map and the base map in the target object picture; Identify a specular reflection light distribution map and a diffuse reflection light distribution map in the normal map; According to the specular reflection light distribution map and the diffuse reflection light distribution map, the specular reflection light and the diffuse reflection light in the normal map are removed to obtain a first normal map; Superimposing the first normal map with the target light HDR texture image to obtain a target normal map; The target normal map is superimposed on the base map to obtain the target image.

7. The method according to claim 1, characterized in that Also includes: If it is determined that the background image of the target object is updated, and / or the emotional characteristic information is updated, determining an updated target light HDR texture image according to at least one of the updated background image of the target object and the updated emotional characteristic information; The target object image is fused with the updated target light HDR texture image to obtain an updated target image.

8. A light perception model training method, characterized in that: include: Get a set of training sample images and a light HDR texture image corresponding to each training sample image; Labeling the light HDR texture image corresponding to each training sample image; Based on each training sample image and the corresponding labeled light HDR texture image, the diffusion model is trained to obtain a light perception model, wherein the light perception model is used to identify the light feature vector of the background image of the target object, and generate a light HDR texture image based on the light feature vector, and the background image is used to reflect the environment in which the target object is located.

9. A method for training an emotional lighting mapping model, characterized in that: include: Obtain a set of training sample emotion feature vectors and a light HDR texture image corresponding to each training sample emotion feature vector; Labeling the light HDR texture image corresponding to each training sample emotion feature vector; Based on the emotion feature vector of each training sample and the corresponding labeled light HDR texture picture, the diffusion model is trained to obtain an emotion lighting mapping model, wherein the emotion lighting mapping model is used to identify the light feature vector corresponding to the emotion feature vector corresponding to the emotion feature information of the target object, and generate a light HDR texture picture based on the light feature vector, and the emotion feature information is used to reflect the emotion of the target object.

10. A video display method, characterized in that: include: Receiving a video containing a target object sent by a server, wherein the video is generated by the server according to each target image obtained by using the image processing method according to any one of claims 1 to 7; Each video frame of the video is displayed.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, it implements the image processing method according to any one of claims 1 to 7, the light perception model training method according to claim 8, the emotion lighting mapping model training method according to claim 9, or the video display method according to claim 10.

Images