3D head model training method, 3D head model presenting method and electronic equipment

Abstract

The embodiment of the invention relates to the technical field of artificial intelligence, and discloses a 3D head model training method, a 3D head model presenting method and electronic equipment. The method comprises the steps of obtaining target video data, determining head control information corresponding to a target head image, obtaining Gaussian ball parameters output by a Gaussian ball learning model when the target head image serves as a training input sample, and inputting the Gaussian ball parameters and the head control information into a head rendering model to obtain a head rendering image, and determining image difference information of the head rendering image and the target head image, and training a 3D head model according to the image difference information. According to the embodiment of the invention, the output Gaussian ball parameters are automatically learned by using the Gaussian ball learning model, and the deviation of the head control information on the image generation is compensated, so that the performance of the 3D head model can be improved, and the 3D head model can output the head rendering image with a relatively high matching degree with the original head image.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a training method, presentation method and electronic device for a 3D head model. Background Technology

[0002] Immersive video conferencing is a new form of remote video conferencing that displays participants' heads in 3D on the meeting interface, increasing the sense of presence and immersion. The relevant technology requires setting up multiple cameras to capture participants' heads, obtaining numerous head images. Facial landmarks in these images are then manually annotated. Finally, the annotated facial landmarks are used to train a 3D head model. However, manual annotation is not accurate enough, easily leading to biases in the selection of facial landmarks. This results in poor matching between the rendered 3D head image and the actual head image, leading to suboptimal performance of the trained 3D head model. Summary of the Invention

[0003] One objective of this application is to provide a training method, a presentation method, and an electronic device for a 3D head model, in order to solve the technical problem of poor performance of 3D head models in related technologies.

[0004] In a first aspect, embodiments of this application provide a method for training a 3D head model, comprising:

[0005] Acquire target video data, which includes multiple original head images of the target head in different poses, and the 3D head model includes a Gaussian sphere learning model and a head rendering model.

[0006] Determine the head control information corresponding to the target head image, wherein the target head image is the original head image in the training state;

[0007] Obtain the Gaussian sphere parameters output by the Gaussian sphere learning model when the target head image is used as a training input sample. The Gaussian sphere parameters are used to constrain the Gaussian sphere of the head rendering model.

[0008] The Gaussian sphere parameters and the head control information are input into the head rendering model to obtain the head rendering image;

[0009] Determine the image difference information between the rendered head image and the target head image;

[0010] The 3D head model is trained based on the image difference information.

[0011] Optionally, the Gaussian sphere parameters include Gaussian sphere rendering parameters and position offset parameters; the Gaussian sphere learning model includes a neural network model; the head rendering model includes a human head FLAME model and a 3D Gaussian rendering model; and obtaining the Gaussian sphere parameters output by the Gaussian sphere learning model when the target head image is used as a training input sample includes:

[0012] When the target head image is used as a training input sample, the vertex features of the vertices in the human head FLAME model used to associate with the pose of the target head are determined.

[0013] The vertex features of the vertex are input into the neural network model to obtain Gaussian sphere rendering parameters and position offset parameters. The Gaussian sphere rendering parameters are used to represent the rendering properties of the Gaussian sphere in the 3D Gaussian rendering model, and the position offset parameters are used to indicate the position offset of the Gaussian sphere in the 3D Gaussian rendering model.

[0014] Optionally, the Gaussian sphere rendering parameters include spherical harmonic functions, opacity, rotation matrix, and scale vector, and the position offset parameters include the offset vector of the center point of the Gaussian sphere and the weight values ​​of the vertices associated with the pose of the target head.

[0015] Optionally, inputting the Gaussian sphere parameters and the head control information into the head rendering model to obtain the head rendering image includes:

[0016] The target position of the target vertex after the target head changes is determined based on the position offset parameter and the head control information, wherein the target vertex is a vertex in the target head image;

[0017] Input the target position and Gaussian sphere rendering parameters of each target vertex into the human head FLAME model to obtain the head rendering parameters corresponding to each target vertex;

[0018] The head rendering parameters corresponding to each target vertex are input into the 3D Gaussian rendering model to obtain the head rendering image.

[0019] Optionally, determining the target position of the target vertex after the target head changes, based on the position offset parameter and the head control information, includes:

[0020] The vertex position of the target vertex is updated according to the position offset parameter to obtain the static position of the target vertex before the target head changes.

[0021] The static position is moved according to the head control information to obtain the target position of the target vertex after the target head changes.

[0022] Optionally, the position offset parameter includes the offset vector of the center point of the Gaussian sphere, and updating the vertex position of the target vertex according to the position offset parameter to obtain the static position of the target vertex before the change in the target head includes:

[0023] The position of the target vertex is added to the offset vector of the center point of the Gaussian sphere to obtain the static position of the target vertex before the target head changes.

[0024] Optionally, the head control information includes camera pose and head key points of the target head image. The camera pose is used to indicate the pose of the target head captured by the camera. The position offset parameter includes the weight values ​​of vertices associated with the pose of the target head. Moving the static position according to the head control information to obtain the target position of the target vertex after the target head changes includes:

[0025] According to the preset linear hybrid skinning algorithm, the target vertex is updated using the static position, the camera pose, the head key points and the vertex weights to obtain the target position of the target vertex after the target head changes.

[0026] Optionally, determining the vertex features in the head FLAME model used to associate the vertices with the pose of the target head includes:

[0027] Obtain the vertex positions in the human head FLAME model used to associate the vertices with the pose of the target head;

[0028] Based on the vertex position, a feature sampling operation is performed in a preset sampling space to obtain the vertex features.

[0029] In a second aspect, embodiments of this application provide a method for rendering a 3D head, applied to an electronic device, comprising:

[0030] When the electronic device establishes a conference connection with the target device, it receives real-time head control information of the target user sent by the target device. The target user is the user participating in the conference through the target device.

[0031] The real-time head control information is input into the 3D head model corresponding to the target user, so that the 3D head model outputs a real-time head rendering image, wherein the 3D head model is the 3D head model obtained after the above training.

[0032] Generate a 3D head image based on the real-time head rendering image;

[0033] Present a 3D head image of the target user.

[0034] In a third aspect, embodiments of this application provide an electronic device, including a memory and a processor, wherein the memory is connected to the processor, and the processor is configured to execute one or more computer programs stored in the memory, wherein when the processor executes the one or more computer programs, the electronic device implements the 3D head model training method and the 3D head rendering method described above.

[0035] In a fourth aspect, embodiments of this application provide a computer-readable storage medium storing a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the above-described training method for a 3D head model.

[0036] The embodiments of this application can achieve the following technical effects: In the training method of the 3D head model provided in this application embodiment, firstly, this application embodiment acquires target video data, which consists of multiple original head images of the target head in different poses. Secondly, this application embodiment determines the head control information corresponding to the target head image, where the target head image is an original head image in a training state. In this step, this application embodiment can sequentially select each original head image in the training state as the target head image and add it to the training operation. Thirdly, this application embodiment acquires the Gaussian sphere parameters output by the Gaussian sphere learning model. The Gaussian sphere parameters are used to constrain the Gaussian sphere of the 3D Gaussian rendering model. In this step, this application embodiment can automatically select the Gaussian sphere parameters to compensate for the deviation caused by the head control information in image generation. Fourthly, this application embodiment inputs the Gaussian sphere parameters and head control information into the head rendering model to obtain a head rendering image. In this step, this application embodiment generates the trained head rendering image to adjust the 3D head model in conjunction with the target head image. Finally, this embodiment determines the image difference information between the rendered head image and the target head image, and trains a 3D head model based on this image difference information. In this step, the image difference information is used as supervisory information to feed back into the training of the 3D head model, so as to obtain the optimal 3D head model. Overall, this embodiment uses a Gaussian sphere learning model to automatically learn the output Gaussian sphere parameters, compensating for the deviation caused by the head control information in image generation. This improves the performance of the 3D head model and helps the 3D head model output a rendered head image with a high degree of matching with the original head image. Attached Figure Description

[0037] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 A schematic diagram of the system architecture of a conference system provided in this application embodiment;

[0039] Figure 2 A schematic diagram of the structure between modules involved in the electronic device or target device provided in the embodiments of this application for completing 3D head images;

[0040] Figure 3 This is a schematic diagram illustrating the interaction between an electronic device and a target device provided in an embodiment of this application.

[0041] Figure 4 An interactive diagram illustrating the exchange of 3D head models between an electronic device and a target device, provided in an embodiment of this application.

[0042] Figure 5a A schematic diagram illustrating the participation of an electronic device, a target device, and a server in a meeting, as provided in an embodiment of this application.

[0043] Figure 5b A schematic diagram of the meeting interface provided in this application embodiment;

[0044] Figure 6 A schematic diagram of the system architecture of a conference system provided in another embodiment of this application;

[0045] Figure 7 A flowchart illustrating a training method for a 3D head model provided in an embodiment of this application;

[0046] Figure 8 A schematic diagram of facial key points of a human head FLAME model provided in an embodiment of this application;

[0047] Figure 9 This is a schematic diagram illustrating the process of training a 3D head model as provided in an embodiment of this application.

[0048] Figure 10 A schematic diagram illustrating the learning of Gaussian sphere parameters when the neural network model provided in this embodiment is a multilayer MLP neural network;

[0049] Figure 11 This is a schematic diagram illustrating the learning of Gaussian sphere parameters when the neural network model provided in this embodiment is a multi-layer MLP neural network;

[0050] Figure 12A flowchart illustrating a method for rendering a 3D head according to an embodiment of this application;

[0051] Figure 13 A schematic diagram of the structure of a training device for a 3D head model provided in an embodiment of this application;

[0052] Figure 14 A schematic diagram of the structure of a 3D head rendering device provided in an embodiment of this application;

[0053] Figure 15 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0055] It should be noted that, unless there is a conflict, the various features in the embodiments of this application can be combined with each other, all of which are within the protection scope of this application. Furthermore, although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than the module division in the device or the order in the flowchart. Moreover, the terms "first," "second," and "third" used in this application do not limit the data or execution order, but only distinguish identical or similar items with essentially the same function and effect.

[0056] The head FLAME model is a parametric facial geometry model, typically used in 3D face reconstruction scenarios. Related technologies can use the head FLAME model in conjunction with multiple user head images to be trained, all of which have been manually annotated with numerous facial key points.

[0057] In the real physical world, the head does not possess biologically significant facial landmarks. The facial landmarks mentioned above are merely a concept proposed for artificially constructed human head FLAME models. Therefore, for a head image, the facial landmarks do not possess completely accurate biological locations. Even manual annotation of facial landmarks used as ground truth cannot guarantee complete accuracy; different designers annotating the same facial landmark location can easily yield different annotation results. Even the same designer annotating at different times can easily obtain different annotation results.

[0058] Even the gold standard Ground Truth annotation results cannot guarantee the accuracy and consistency of facial key points. Furthermore, 3D head models rely on the accuracy of facial key points. Therefore, 3D head models trained using facial key points obtained from Ground Truth are unreliable. The head rendering image output by the trained 3D head model will have image deviations relative to the original user image. These image deviations will cause severe distortions in the head rendering image output by the 3D head model, which will greatly reduce the user experience.

[0059] In another related technology, to improve the accuracy of manual facial landmark annotation, multiple cameras are set up in front of the user. These cameras can simultaneously capture multiple images of the user's head from different angles, such as using 16 or even hundreds of cameras. Facial landmarks are then annotated on these multiple head images obtained from the multiple cameras, and the annotated facial landmarks are relatively stable and reliable. However, this method has a problem: when a 3D head model needs to be built for each user, the technology requires setting up dozens or even hundreds of cameras in front of each user to simultaneously capture their head for training. However, most scenarios lack the resources to set up multiple cameras for each participant, especially in remote conferencing. Therefore, this technology has significant limitations, a limited application scope, and low efficiency in generating 3D head models.

[0060] This embodiment of the application only requires target video data collected by a single camera to help complete the training of a 3D head model, eliminating the need for multiple cameras and thus improving the generation efficiency and application scope of the 3D head model. Furthermore, this embodiment uses the image difference information between the rendered head image and the target head image as supervisory information to learn Gaussian sphere rendering parameters and position offset parameters. The position offset parameters can compensate for deviations caused by inaccurate facial key points. Therefore, this embodiment reduces the accuracy requirements for extracting facial key points, thereby reducing the number of cameras required. In other words, training of the 3D head model can be completed using only one camera. The efficiency of processing one video stream is far greater than processing 16 or hundreds of video streams simultaneously. Therefore, this embodiment can significantly improve the generation efficiency of the 3D head model.

[0061] This application provides a conference system; please refer to [link / reference]. Figure 1The conference system 100 includes an electronic device 200 and a target device 300, wherein the electronic device 200 and the target device 300 establish a conference connection. The electronic device 200 and the target device 300 can join the same remote conference to participate in the meeting. The conference interface of the electronic device 200 can display the head image of user A operating the target device 300, and similarly, the conference interface of the target device 300 can display the head image of user B operating the electronic device 200.

[0062] The head images presented by the electronic device 200 and the head images presented by the target device 300 are 3D head images. Since 3D head images are more three-dimensional than 2D head images, they are easier to create an immersive experience for users, making it easier for users to immerse themselves in the meeting atmosphere.

[0063] The 3D head image can be synthesized from multiple head rendering images by electronic device 200 or target device 300, wherein the head rendering image can be generated by electronic device 200 or target device 300 using a corresponding 3D head model. It is understood that both electronic device 200 and target device 300 can train their respective 3D head models using the 3D head model training methods described in the various embodiments below.

[0064] Please refer to the following: Figure 2 and Figure 3 Both the electronic device 200 and the target device 300 include an N-channel camera module 41, an ISP module 42, a video decoding module 43, a 3D model generation module 44, a model transmission module 45, a 3D rendering module 46, a 3D compositing module 47, a 3D playback module 48, and a SOC processing module 49.

[0065] The camera module 41 is used to acquire video data of the user's head in different postures and send the video data to the ISP module 42. The ISP module 42 processes the video data to improve image clarity, color reproduction, dynamic range, etc., and enhance the image's resolution in low light conditions.

[0066] The video decoding module 43 is used to restore the processed video data to obtain the original video image and audio signal. The 3D model generation module 44 is used to train a 3D head model according to the training method of the 3D head model described in the following embodiments. The model transmission module 45 is used to send the 3D head model to a designated device, which can be the device of the other party's participant or a server.

[0067] The 3D rendering module 46 processes user feature information using a 3D head model to output a rendered head image. The 3D compositing module 47 combines multiple rendered head images into a stereoscopic head image. The 3D playback module 48 combines the stereoscopic head image and audio signal into a video stream. The SOC processing module 49 processes the video stream to present the stereoscopic head image and audio signal.

[0068] It is understood that the camera module 41 can be any type of camera, the ISP module 42 can be a separate ISP chip, or the ISP module 42 and the video decoding module 43 can be integrated on the same video processing chip. The 3D model generation module 44 can be a 3D graphics card processor, etc. The model transmission module 45 can be a wired communication module or a wireless communication module. The 3D rendering module 46, the 3D compositing module 47, the 3D playback module 48, and the 3D playback module 49 can be various types of processors, such as DSP processors. The SOC processing module 49 can be a SOC processing chip.

[0069] In some embodiments, the electronic device 200 establishes an end-to-end communication connection with the target device 300. Based on this end-to-end communication connection, the electronic device 200 and the target device 300 exchange 3D head models for local storage. That is, the electronic device 200 can send its local 3D head model to the target device 300 for local storage via the end-to-end communication connection, and the target device 300 can send its local 3D head model to the electronic device 200 for local storage via the same connection.

[0070] For example, please see Figure 4 User A joins remote conference A using electronic device 200, and User B joins remote conference A using target device 300. Electronic device 200 trains to obtain a 3D head model J1 for User A, and target device 300 trains to obtain a 3D head model J2 for User B. Electronic device 200 and target device 300 can communicate end-to-end. Electronic device 200 sends User A's 3D head model J1 to target device 300, and target device 300 stores User A's 3D head model J1 locally. Target device 300 sends User B's 3D head model J2 to electronic device 200, and electronic device 200 stores User B's 3D head model J2 locally.

[0071] During the meeting, electronic device 200 captures real-time images of user A's head using a camera, obtaining a real-time head image X1. Electronic device 200 then determines user A's head control information based on the real-time head image X1 and sends this information to target device 300. Target device 300 uses a 3D head model J1 to process user A's head control information, thereby obtaining a baseline head rendering image of user A. Target device 300 generates multiple parallax head rendering images based on the baseline head rendering image, and then combines the baseline head rendering image and the multiple parallax head rendering images to create a 3D head image. Finally, target device 300 can display the 3D head image of user A on its local meeting interface.

[0072] Similarly, target device 300 captures real-time images of user B's head using a camera, obtaining a real-time head image X2. Then, target device 300 determines user B's head control information based on the real-time head image X2 and sends this information to electronic device 200. Electronic device 200 uses 3D head model J2 to process user B's head control information, thereby obtaining a baseline head rendering image of user B. Finally, electronic device 200 generates a 3D head image of user B according to the above procedure and displays the 3D head image of user B on the local conference interface.

[0073] In related technologies, to display a 3D head image of a target device 300 on an electronic device 200, the technology directly controls the target device 300 to transmit N video streams to the electronic device 200. The electronic device 200 then processes the N video streams into a 3D head image for display. However, based on the transmission requirements of a video stream resolution of 1920*1080 and a frame rate of 30fps, the bandwidth required to transmit one video stream is: 1920*1080*3*8 / 1024 / 1024 / 1024*30 frames*1 stream = 1.39Gbps. If N=4, meaning the target device 300 transmits four video streams to the electronic device 200, then the technology requires 5.56Gbps of bandwidth. Therefore, the technology requires a large bandwidth to achieve the purpose of displaying a 3D head image. However, many data transmission scenarios cannot support such a high bandwidth data transmission, which significantly limits the application scope of the technology.

[0074] In this embodiment, the amount of header control information is small, typically less than 1 Mbps. Therefore, the method provided in this embodiment occupies less bandwidth, which helps to broaden the application scope of the method provided in this embodiment.

[0075] In some embodiments, please refer to the following: Figure 5a and Figure 5bElectronic device 200 and target device 300 establish communication connections with server 400 respectively, and server 400 can forward the other party's 3D head model to the other party's device.

[0076] Please continue reading. Figure 5a and Figure 5b When electronic device 200 and target device 300 join the same remote conference, server 400 can send the other party's 3D head model to the other party's device.

[0077] Electronic device 200 sends user A's 3D head model J1 to server 400 for storage, and target device 300 sends user B's 3D head model J2 to server 400 for storage.

[0078] When server 400 detects that user A's user information and user B's user information appear on the same remote conference's conference list, where user A's user information is bound to electronic device 200 and user B's user information is bound to target device 300, server 400 sends 3D head model J1 to target device 300 and 3D head model J2 to electronic device 200.

[0079] During the meeting, electronic device 200 captures real-time images of user A's head using a camera, obtaining a real-time head image X1. Electronic device 200 then determines user A's head control information based on real-time head image X1 and sends this information to server 400. Similarly, target device 300 captures real-time images of user B's head using a camera, obtaining a real-time head image X2. Target device 300 then determines user B's head control information based on real-time head image X2 and sends this information to server 400.

[0080] Server 400 forwards user B's header control information to electronic device 200, and forwards user A's header control information to target device 300.

[0081] Target device 300 calls 3D head model J1 to process user A's head control information, synthesizes a 3D head image according to the above method, and displays the 3D head image of user A on the local conference interface. Similarly, electronic device 200 calls 3D head model J2 to process user B's head control information, generates a 3D head image of user B according to the above method, and displays the 3D head image of user B on the local conference interface.

[0082] It is understood that in some embodiments, the 3D head model can be generated before the meeting begins. As previously described, electronic device 200 generates a 3D head model J1 for user A in advance, and target device 300 generates a 3D head model J2 for user B in advance. Then, before the meeting begins, electronic device 200 sends 3D head model J1 to target device 300 for local storage, and target device 300 sends 3D head model J2 to electronic device 200 for local storage. Alternatively, before the meeting begins, electronic device 200 sends 3D head model J1 to server 400, target device 300 sends 3D head model J2 to server 400, server 400 forwards 3D head model J1 to target device 300 for local storage, and forwards 3D head model J2 to electronic device 200 for local storage.

[0083] It is also understood that, in some embodiments, the 3D head model can be generated at the start of the meeting. As mentioned earlier, at the start of the meeting, User A uses the camera of electronic device 200 to photograph User A's head and records a first video data of a preset duration. User B uses the camera of target device 300 to photograph User B's head and records a second video data of a preset duration. Electronic device 200 generates a 3D head model J1 based on the first video data and sends the 3D head model J1 directly to target device 300 for local storage or to server 400, which then forwards it to target device 300 for local storage. Simultaneously, target device 300 generates a 3D head model J2 based on the second video data and sends the 3D head model J2 directly to electronic device 200 for local storage or to server 400, which then forwards it to electronic device 200 for local storage.

[0084] In some embodiments, the video data used to generate the 3D head model may be acquired by a camera carried by the electronic device 200 or the target device 300 itself.

[0085] In some embodiments, the video data used to generate the 3D head model can be captured by cameras set up at the conference venue. See also Figure 6 The conference system 100 includes a first camera module 500 and a second camera module 600. The first camera module 500 includes N first cameras, which are communicatively connected to the electronic device 200. The second camera module 600 includes M second cameras, which are communicatively connected to the target device 300, where N and M are both positive integers.

[0086] User A and User B are conducting a remote video conference. When User A enters the conference room, the first camera module 500 captures video data of User A's head in different poses, and then transmits this video data to the electronic device 200. The electronic device 200 generates a 3D head model J1 based on the video data. Similarly, when User B enters the conference room, the second camera module 600 captures video data of User B's head in different poses, and then transmits this video data to the target device 300. The target device 300 generates a 3D head model J2 based on the video data.

[0087] As another aspect of this application, this application provides a method for training a 3D head model. Please refer to... Figure 7 The training method for a 3D head model includes the following steps:

[0088] S71: Acquire target video data, which includes multiple original head images of the target head in different poses. The 3D head model includes a Gaussian sphere learning model and a head rendering model.

[0089] In this step, the target head is the user's head portion from which a 3D head image needs to be generated, and the original head image is an image of the target head in the corresponding pose. Multiple original head images can be obtained from a single camera or from multiple cameras.

[0090] In some embodiments, a user may use a single-channel camera carried by an electronic device to record the head in order to obtain recording data of a preset duration. This recording data can be used as target video data, which includes multiple original head images representing the target head in different poses.

[0091] In some embodiments, when the recorded data contains images unrelated to the target head, the electronic device may select from the recorded data multiple original head images containing head region images and package them into target video data.

[0092] In some embodiments, a single-channel camera is provided at the meeting venue. When a user enters the meeting venue, the user's head can be moved in different postures in front of the single-channel camera so that the single-channel camera can capture raw head images in different postures.

[0093] In some embodiments, the meeting venue is equipped with multiple cameras. When a user enters the meeting venue, the multiple cameras in different positions can capture the user's head from different angles, thereby obtaining multiple original head images of the target head in different postures.

[0094] Understandably, when capturing raw head images using a single or multiple cameras, the operating parameters of each camera remain unchanged. This helps ensure the consistency and reliability of the parameters across all raw head images. Operating parameters include camera focal length, field of view, camera aperture, shutter speed, and ISO.

[0095] It is also understandable that the target video data can be acquired before the meeting begins, at the start of the meeting, or during the meeting.

[0096] S72: Determine the head control information corresponding to the target head image.

[0097] In this step, head control information is used to represent the head features of the target head corresponding to the target head image. These head features include the target head's pose, shape, and expression. The target head's pose can be represented by the camera's pose when capturing the target head, while the shape and expression can be represented by a combination of multiple head keypoints (i.e., head landmarks). Therefore, in some embodiments, the head control information includes the camera pose and the head landmarks of the target head image. The camera pose instructs the camera on the pose of the target head, and multiple head landmarks can effectively combine to represent the head's shape and expression. Please refer to [link to relevant documentation]. Figure 8 The human head FLAME model is configured with k head key points Q, which can effectively combine to represent the shape and expression of the head.

[0098] The target head image is the original head image in the training state. When the target head image is used to complete the training operation in this embodiment, the training state of the target head image is marked as trained.

[0099] Determining the head control information corresponding to the target head image includes the following steps: determining the camera pose and multiple head key points of the target head image based on a preset visual model, wherein the multiple head key points meet the requirements of the human head FLAME model. The preset visual model can be a Colmap model or a MediaPipe model. The Colmap model is used for 3D reconstruction from a series of 2D images, supporting features detection and matching, incremental SfM, multi-view stereo matching (MVS), and optimization functions. It can recover the geometric structure of a 3D scene and the camera pose corresponding to each image from an unordered or ordered set of 2D images. The MediaPipe model is a graph-based data processing pipeline used to build machine learning applications that use various forms of data sources (such as video, audio, sensor data, and any time-series data).

[0100] S73: Obtain the Gaussian sphere parameters output by the Gaussian sphere learning model when the target head image is used as the training input sample. The Gaussian sphere parameters are used to constrain the Gaussian sphere of the head rendering model.

[0101] In this step, the Gaussian sphere learning model is used to learn the parameters of the output Gaussian sphere. The Gaussian sphere learning model can employ any suitable model architecture, such as a neural network model or other custom learning model. The head rendering model is used to render the output head rendering image. The head rendering model is connected to the Gaussian sphere learning model, and the Gaussian sphere parameters output by the Gaussian sphere learning model can be used as an input to the head rendering model.

[0102] S74: Input the Gaussian sphere parameters and head control information into the head rendering model to obtain the head rendering image.

[0103] In this step, the 3D head model is used to generate the head rendering image, wherein the head rendering image is a 2D rendering image. This embodiment of the application can synthesize multiple head rendering images into a 3D head image. The 3D head model supports various human head model algorithms and various rendering algorithms, such as the human head model algorithm being the head FLAME algorithm, and the rendering algorithm being the 3D Gaussian Splatting rendering algorithm.

[0104] S75: Determine the image difference information between the rendered head image and the target head image.

[0105] In this step, image difference information is used to represent the degree of matching between the head rendering image and the target head image. Image difference information can be represented by loss value, histogram similarity, or feature matching degree.

[0106] Image difference information includes the total loss value. Determining the image difference information between the rendered head image and the target head image includes the following steps: determining the feature loss value, image similarity loss value, and image difference loss value between the rendered head image and the target head image; multiplying the feature loss value by a first weight coefficient to obtain a first weighted loss value; multiplying the image similarity loss value by a second weight coefficient to obtain a second weighted loss value; multiplying the image difference loss value by a third weight coefficient to obtain a third weighted loss value; and adding the first weighted loss value, the second weighted loss value, and the third weighted loss value together to obtain the total loss value.

[0107] For example, embodiments of this application calculate the total loss based on the following formula, where the formula is: L0=λ1L1+λ2L ssim +λ3L vgg , Where L0 is the total loss, L1 is the feature loss value, and L ssimThe image similarity loss value (SSIM) is L. vgg The image difference loss value (VGGLoss) is given by λ1, where λ1 is the first weighting coefficient, λ2 is the second weighting coefficient, and λ3 is the third weighting coefficient. t For the t-th original head image, render t This is the head rendering image corresponding to the t-th original head image.

[0108] S76: Train a 3D head model based on image difference information.

[0109] In this step, when the image difference information is the total loss value, this embodiment of the application optimizes the network parameters of the 3D head model using a preset gradient descent algorithm based on the total loss value to obtain an adjusted 3D head model.

[0110] After training the 3D head model based on image difference information according to this embodiment, the training state of the target head image is marked as trained. Next, this embodiment detects whether the 3D head model meets the training stop condition. If it does, the training operation is stopped; if not, one original head image is selected from at least one original head image in the untrained state as the new target head image, and the process returns to the step of determining the head control information corresponding to the target head image, so as to continue training the 3D head model.

[0111] To illustrate in detail the training method for the 3D head model provided in the embodiments of this application, the embodiments of this application combine... Figure 9 The following is a detailed explanation of this matter:

[0112] Please see Figure 9 In this embodiment, multiple raw head images (img) are acquired, and a pre-trained visual model is used to extract the camera poses (pose1, pose2, ..., pose) corresponding to each raw head image. t and satisfy the k facial key points G of the human head FLAME model t ∈R k×3 Among them, the original head image (img) j Head control information includes camera pose. j and facial key points G j ∈R k×3 .

[0113] During training, this embodiment of the application uses the original head image img1 as the target head image and obtains the Gaussian sphere parameters output by the Gaussian sphere learning model 91 when the original head image img1 is used as the training input sample. The camera pose (pose1) and facial key points (G1∈R) of the original head image (img1) are determined. k×3 and Gaussian sphere parameters The head rendering model 92 is input as a head rendering model. The head rendering model 92 transmits head rendering parameters to the 3D Gaussian rendering model 93 to obtain the head rendering image render1. Next, this embodiment of the application calculates the image difference information L between the head rendering image render1 and the original head image img1. 01 Based on image difference information L 01 Training a 3D head model. Due to image difference information L... 01 The model has not yet converged; therefore, this embodiment of the application requires continued training of the 3D head model.

[0114] Next, in this embodiment of the application, the original head image img2 is used as the target head image, and the Gaussian sphere parameters output by the Gaussian sphere learning model 91 when the original head image img2 is used as the training input sample are obtained. The camera pose (pose2) and facial key points (G2∈R) of the original head image (img2) are used to determine the camera pose (pose2) and facial key points (G2∈R). k×3 and Gaussian sphere parameters The input head rendering model is a human head FLAME model 92. The human head FLAME model 92 transmits head rendering parameters to the 3D Gaussian rendering model 93 to obtain the head rendering image render2. Next, this embodiment of the application calculates the image difference information L between the head rendering image render2 and the original head image img2. 02 Based on image difference information L 02 Training a 3D head model. Due to image difference information L... 02 The convergence has not yet occurred; therefore, this embodiment requires continued training of the 3D head model. The same logic applies, and will not be elaborated further here. When the image difference information L... 02 When the training converges to a preset threshold, the embodiment of this application stops training the 3D head model and outputs the trained 3D head model.

[0115] In summary, this application embodiment utilizes a Gaussian sphere learning model to automatically learn the output Gaussian sphere parameters, compensating for the deviations caused by head control information in image generation. This improves the performance of the 3D head model. Furthermore, by using the image difference information between the rendered head image and the target head image as supervisory information to train the 3D head model, it enhances the accuracy and reliability of the 3D head model, enabling it to output rendered head images with a high degree of matching to the original head image. Additionally, this application embodiment only requires target video data collected by a single camera to complete the training of the 3D head model, eliminating the need for multiple cameras and thus improving the generation efficiency and application scope of the 3D head model.

[0116] The Gaussian sphere parameters include Gaussian sphere rendering parameters and position offset parameters. The Gaussian sphere rendering parameters are used to represent the rendering properties of the Gaussian sphere in the 3D Gaussian rendering model, and the position offset parameters are used to indicate the position offset of the Gaussian sphere in the 3D Gaussian rendering model.

[0117] The Gaussian sphere learning model includes a neural network model, which is used to learn the rendering parameters and position offset parameters of the Gaussian sphere. This neural network model can be a Multi-Layer Perceptron (MLP) network, a Convolutional Neural Network, a Transformer Neural Network, etc. Understandably, the architecture and number of neural network models can be customized by the designer according to business needs.

[0118] The head rendering model includes a human head FLAME model and a 3D Gaussian rendering model. The human head FLAME model is used to construct the head model, and the 3D Gaussian rendering model is used to render the head model to output the head rendering image. The Gaussian sphere rendering parameters include the spherical harmonic function color, opacity ο, rotation matrix R, and scale vector S. The position offset parameters include the offset vector Δμ of the center point of the Gaussian sphere and the weight values ​​W of the vertices associated with the pose of the target head.

[0119] Image difference information includes the total loss value. Training a 3D head model based on the image difference information includes: optimizing the network parameters of the Gaussian sphere learning model using a preset gradient descent algorithm based on the total loss value to obtain an adjusted 3D head model.

[0120] Obtaining the Gaussian sphere parameters output by the Gaussian sphere learning model when the target head image is used as the training input sample includes the following steps:

[0121] S731: When the target head image is used as a training input sample, determine the vertex features in the human head FLAME model used to associate the vertices with the pose of the target head.

[0122] S732: Input the vertex features of the vertices into the neural network model to obtain the Gaussian sphere rendering parameters and position offset parameters.

[0123] In S731, vertices are the location points in the head FLAME model used to associate with the pose of the target head, such as... Figure 8As shown in the left-hand head image, the human head FLAME model is divided into multiple meshes, each with vertices. A standard human head FLAME model contains 5023 vertices, each assigned a weight value W to represent the degree of influence of each joint on that vertex. During deformation, based on the vertex positions and corresponding weight values ​​before deformation, this embodiment can calculate the deformed vertex positions. Vertex features are used to characterize vertices; it is understood that this embodiment can use any feature representation method to generate vertex features.

[0124] Determining the vertex features of vertices in a head FLAME model used to associate with the pose of a target head includes the following steps: obtaining the vertex positions of the vertices in the head FLAME model used to associate with the pose of the target head; performing feature sampling operations in a preset sampling space based on the vertex positions to obtain the vertex features. For example, in this embodiment, based on the vertex positions, a random sampling algorithm, a distance-based sampling algorithm, a distribution-based sampling algorithm, or a sampling algorithm based on an adversarial network model can be used to randomly sample in the preset sampling space to obtain the vertex features.

[0125] For example, a human head FLAME model is configured with a three-dimensional coordinate system {x c y c , z c The position of the i-th vertex in the three-dimensional coordinate system is (x... i y i , z i In this embodiment of the application, the vertex position (x) of the i-th vertex is used as the basis for the calculation. i y i , z i Perform feature sampling operations in the preset sampling space to obtain the vertex feature f of the i-th vertex. i =[f x i f y i f z i ].

[0126] In S732, in some embodiments, please refer to Figure 10 The neural network model is a multi-layer MLP neural network. A multi-layer MLP neural network can automatically learn and output the spherical harmonic function color, opacity o, rotation matrix R, scale vector S, offset vector Δμ, and weight values ​​W. For example, the expression for a multi-layer MLP neural network is:

[0127] W, color, o, △u, R, S = mlp(f).

[0128] In some embodiments, please refer to Figure 11 The neural network model consists of six multi-layer MLP neural networks. These six MLP neural networks can automatically learn to output the spherical harmonic function color, opacity o, rotation matrix R, scale vector S, offset vector Δμ, and weight values ​​W, respectively. For example, the expression for the six multi-layer MLP neural networks is:

[0129] W = mlp1(f)

[0130] color = mlp2(f)

[0131] o = mlp3(f)

[0132] △u=mlp4(f)

[0133] R = mlp5(f)

[0134] S = mlp6(f)

[0135] This application embodiment uses a human head FLAME model as the baseline human head model, and then integrates Gaussian sphere rendering parameters and position offset parameters to create a head rendering image with a higher degree of matching. Furthermore, this application embodiment automatically learns and outputs Gaussian sphere parameters through a neural network model. This not only reduces the dependence on hardware—for example, while related technologies require 16 or hundreds of cameras—this application embodiment only requires one or two cameras, or a small number of cameras, to achieve the goal of training a high-performance 3D head model. Moreover, the Gaussian sphere parameters automatically learned and output by the neural network model can compensate for the bias caused by manual annotation, decoupling human factors and improving the performance of the 3D head model. The trained 3D head model has higher robustness.

[0136] The steps to obtain the head rendering image by inputting the Gaussian sphere parameters and head control information into the head rendering model include:

[0137] S741: Determine the target position of the target vertex after the target head changes based on the position offset parameters and head control information. The target vertex is the vertex in the target head image.

[0138] S742: Input the target position and Gaussian sphere rendering parameters of each target vertex into the human head FLAME model to obtain the head rendering parameters corresponding to each target vertex.

[0139] S743: Input the head rendering parameters corresponding to each target vertex into the 3D Gaussian rendering model to obtain the head rendering image.

[0140] In S741, the target vertex matches the requirements of the human head FLAME model. The process of determining the target position of the target vertex after the target head changes based on the position offset parameter and head control information includes: updating the vertex position of the target vertex according to the position offset parameter to obtain the static position of the target vertex before the target head changes, and moving the static position according to the head control information to obtain the target position of the target vertex after the target head changes.

[0141] In some embodiments, after obtaining the static position of the target vertex before the change in the target head, this application embodiment can input the static position of the target vertex before the change in the target head into a 3D Gaussian rendering model to obtain an intermediate rendered image of the target head in a static state. The trainer can compare the intermediate rendered image with the target head image to verify the correctness of the 3D head model during training.

[0142] The position offset parameter includes the offset vector of the center point of the Gaussian sphere. The vertex position of the target vertex is updated according to the position offset parameter to obtain the static position of the target vertex before the target head changes. This includes adding the vertex position of the target vertex to the offset vector of the center point of the Gaussian sphere to obtain the static position of the target vertex before the target head changes.

[0143] For example, in this embodiment of the application, the static position of the target vertex before the change in the target head is calculated according to the following formula:

[0144] (x' t y' t , z' t )=(x t y t , z t )+△u

[0145] Among them, (x' t y' t , z' t (x) represents the static position of the target vertex before the target head changes. t y t , z t ) represents the vertex position of the target vertex, and △u represents the offset vector of the center point of the Gaussian sphere.

[0146] The head control information includes the camera pose and head key points of the target head image. The camera pose is used to indicate the pose of the target head captured by the camera. The position offset parameter includes the weight values ​​of the vertices associated with the pose of the target head. Moving the static position according to the head control information to obtain the target position of the target vertex after the target head changes includes: updating the position of the target vertex using the static position, camera pose, head key points and vertex weights according to a preset linear hybrid skinning algorithm to obtain the target position of the target vertex after the target head changes.

[0147] For example, in this embodiment of the application, the target position after the target vertex changes at the target head is calculated according to the following formula:

[0148] (x″ t ,y″ t , z″ t ) = LBS(G t W t , {x' t y' t , z' t})

[0149] Among them, (x″ t ,y″ t , z″ t ) represents the target position after the target vertex changes within the target head, LBS() is the LBS (linear blend skinning) function of the preset linear blend skinning algorithm, and G t For facial key points corresponding to the target vertex, W t The weight value of the target vertex.

[0150] In S742, the linear blending skin component of the human head FLAME model can use the LBS algorithm to process the target position and Gaussian sphere rendering parameters of each target vertex, thereby obtaining the head rendering parameters corresponding to each target vertex, where each target vertex corresponds to a Gaussian sphere.

[0151] In S743, the 3D Gaussian rendering model processes the head rendering parameters corresponding to each target vertex based on the 3D Gaussian Splatting algorithm to obtain the head rendering image.

[0152] This application embodiment first determines the static position of the target vertex before the target head changes, and then determines the target position of the target vertex after the target head changes based on the static position, thus obtaining the head rendering image efficiently.

[0153] It should be noted that in the above embodiments, there is no necessarily a certain order between the steps. Those skilled in the art can understand from the description of the embodiments of this application that the above steps may have different execution orders in different embodiments, that is, they may be executed in parallel or in turn, etc.

[0154] As another aspect of the embodiments of this application, this application provides a training device for a 3D head model. The training device for the 3D head model can be a software module, which includes several instructions stored in a memory. A processor can access the memory, call the instructions, and execute them to complete the 3D head model training method described in the above embodiments.

[0155] As another aspect of this application, this application provides a method for rendering a 3D head, applied to an electronic device. Please refer to... Figure 12 The method for rendering a 3D head includes the following steps:

[0156] S121: When the electronic device establishes a conference connection with the target device, it receives real-time header control information of the target user sent by the target device. The target user is the user participating in the conference through the target device.

[0157] In this step, the electronic device and the target device can be a mobile phone, a conference tablet, a desktop computer, a tablet computer, or other similar devices.

[0158] S122: Input real-time head control information into the 3D head model corresponding to the target user, so that the 3D head model outputs a real-time head rendering image.

[0159] In this step, the 3D head model is the 3D head model obtained by training using the methods described in the above embodiments.

[0160] S123: Generate a 3D head image based on the real-time head rendering image.

[0161] In this step, the embodiments of this application use the real-time head rendering image as a reference to generate multiple reference head rendering images, and then combine the multiple reference head rendering images and the real-time head rendering image into a 3D head image.

[0162] S124: Presents a 3D head image of the target user.

[0163] In this step, the electronic device can display a 3D head image of the target user on the conference interface.

[0164] The embodiments of this application can utilize high-performance 3D head models to output accurate and reliable real-time head rendering images, so as to synthesize 3D head images, which helps to improve the immersion of the meeting and thus enhance the user experience.

[0165] As another aspect of the embodiments of this application, this application provides a training device for a 3D head model. In some embodiments, the 3D head model training device can also be constructed from hardware devices. For example, the 3D head model training device can be constructed from one or more chips, which can work in coordination to complete the 3D head model training method described in the various embodiments above. Furthermore, the 3D head model training device can also be constructed from various logic devices, such as general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), microcontrollers, ARM (Acorn RISC Machine) or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or any combination of these components.

[0166] Please see Figure 13 The training device 130 for the 3D head model includes a data acquisition unit 131, an information determination unit 132, a parameter learning unit 133, a head rendering unit 134, a difference determination unit 135, and a model training unit 136.

[0167] The data acquisition unit 131 acquires target video data, including multiple original head images of the target head in different poses. The 3D head model includes a Gaussian sphere learning model and a head rendering model. The information determination unit 132 determines the head control information corresponding to the target head image, which is an original head image in a training state. The parameter learning unit 133 acquires the Gaussian sphere parameters output by the Gaussian sphere learning model when the target head image is used as a training input sample. The Gaussian sphere parameters are used to constrain the Gaussian sphere of the head rendering model. The head rendering unit 134 inputs the Gaussian sphere parameters and head control information into the head rendering model to obtain a head rendering image. The difference determination unit 135 determines the image difference information between the head rendering image and the target head image. The model training unit 136 trains the 3D head model based on the image difference information.

[0168] This application embodiment utilizes a Gaussian sphere learning model to automatically learn and output Gaussian sphere parameters, compensating for the deviation caused by head control information in image generation. This can improve the performance of the 3D head model and help the 3D head model output a head rendering image with a high degree of matching with the original head image.

[0169] In some embodiments, the Gaussian sphere parameters include Gaussian sphere rendering parameters and position offset parameters. The Gaussian sphere learning model includes a neural network model, and the head rendering model includes a human head FLAME model and a 3D Gaussian rendering model. The parameter learning unit 133 is specifically used to: when the target head image is used as a training input sample, determine the vertex features of the vertices in the human head FLAME model that are associated with the pose of the target head, input the vertex features of the vertices into the neural network model, and obtain the Gaussian sphere rendering parameters and position offset parameters. The Gaussian sphere rendering parameters are used to represent the rendering attributes of the Gaussian sphere of the 3D Gaussian rendering model, and the position offset parameters are used to indicate the position offset of the Gaussian sphere of the 3D Gaussian rendering model.

[0170] In some embodiments, the Gaussian sphere rendering parameters include spherical harmonic functions, opacity, rotation matrix, and scale vector, and the position offset parameters include the offset vector of the center point of the Gaussian sphere and the weight values ​​of the vertices associated with the pose of the target head.

[0171] In some embodiments, the head rendering unit 134 is specifically used to: determine the target position of the target vertex after the target head changes according to the position offset parameters and head control information, wherein the target vertex is a vertex in the target head image; input the target position of each target vertex and the Gaussian sphere rendering parameters into the human head FLAME model to obtain the head rendering parameters corresponding to each target vertex; input the head rendering parameters corresponding to each target vertex into the 3D Gaussian rendering model to obtain the head rendering image.

[0172] In some embodiments, the head rendering unit 134 is specifically used to: update the vertex position of the target vertex according to the position offset parameter to obtain the static position of the target vertex before the target head changes, and move the static position according to the head control information to obtain the target position of the target vertex after the target head changes.

[0173] In some embodiments, the position offset parameter includes the offset vector of the center point of the Gaussian sphere, and the head rendering unit 134 is specifically used to: add the vertex position of the target vertex to the offset vector of the center point of the Gaussian sphere to obtain the static position of the target vertex before the target head changes.

[0174] In some embodiments, the head control information includes camera pose and head key points of the target head image. The camera pose is used to indicate the pose of the camera capturing the target head. The position offset parameter includes the weight values ​​of the vertices associated with the pose of the target head. The head rendering unit 134 is specifically used to: update the position of the target vertex using the static position, camera pose, head key points and vertex weights according to a preset linear blending skinning algorithm, so as to obtain the target position of the target vertex after the target head changes.

[0175] In some embodiments, the parameter learning unit 133 is specifically used to: obtain the vertex positions of the vertices in the human head FLAME model that are associated with the pose of the target head, and perform feature sampling operations in a preset sampling space based on the vertex positions of the vertices to obtain the vertex features of the vertices.

[0176] It should be noted that the above-mentioned 3D head model training device can execute the 3D head model training method provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects of the method. Technical details not described in detail in the embodiments of the 3D head model training device can be found in the 3D head model training method provided in the embodiments of this application.

[0177] As another aspect of this application, this application provides a 3D head rendering device applied to an electronic device. Please refer to... Figure 14 The 3D head presentation device 140 includes an information receiving unit 141, an information input unit 142, an image generation unit 143, and an image presentation unit 144.

[0178] The information receiving unit 141 is used to receive real-time head control information of the target user sent by the target device when the electronic device establishes a conference connection with the target device. The target user is the user participating in the conference through the target device. The information input unit 142 is used to input the real-time head control information into the 3D head model corresponding to the target user, so that the 3D head model outputs a real-time head rendering image. The 3D head model is the 3D head model obtained after training in the above embodiments. The image generation unit 143 is used to generate a 3D head image based on the real-time head rendering image. The image presentation unit 144 is used to present the 3D head image of the target user.

[0179] As another aspect of this application, this application provides an electronic device. Please refer to... Figure 15 , Figure 15 This is a schematic diagram of an electronic device provided in an embodiment of this application. The electronic device 150 includes one or more processors 151 and a memory 152. The memory 152 is connected to one or more processors 151, for example, via a bus.

[0180] Processor 151 is configured to support the electronic device in performing the corresponding functions in the methods described in the above method embodiments. Processor 151 may be a central processing unit (CPU), a network processor (NP), a hardware chip, or any combination thereof. The aforementioned hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The aforementioned PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0181] Memory 152 is used to store program code, etc. Memory may include volatile memory (VM), such as random access memory (RAM); memory may also include non-volatile memory (NVM), such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); memory may also include combinations of the above types of memory.

[0182] The memory 152 can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the 3D head model training method or the 3D head rendering method in the embodiments of this application. The processor executes various functional applications and data processing of the 3D head model training method or the 3D head rendering method by running the non-volatile software programs, instructions, and modules stored in the memory, thereby realizing the functions of the 3D head model training method or the 3D head rendering method provided in the above method embodiments.

[0183] The memory 152 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and applications required for at least one function. The data storage area may store data created based on the use of the training device for the 3D head model. In some embodiments, the memory may optionally include memory remotely located relative to the processor, which can be connected to the training device for the 3D head model via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0184] The one or more modules are stored in the memory. When executed by the one or more processors, they perform the 3D head model training method or 3D head rendering method in any of the above method embodiments. For example, they perform the method steps described in the above method embodiments to realize the functions of the modules described in the above device embodiments.

[0185] This application also provides a computer-readable storage medium storing a computer program, the computer program including program instructions, which, when executed by a computer, cause the computer to perform the method described in the foregoing embodiments.

[0186] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0187] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A training method for a 3D head model, characterized in that, include: Acquire target video data, which includes multiple original head images of the target head in different poses, and the 3D head model includes a Gaussian sphere learning model and a head rendering model. Determine the head control information corresponding to the target head image, wherein the target head image is the original head image in the training state; Obtain the Gaussian sphere parameters output by the Gaussian sphere learning model when the target head image is used as a training input sample. The Gaussian sphere parameters are used to constrain the Gaussian sphere of the head rendering model. The Gaussian sphere parameters and the head control information are input into the head rendering model to obtain the head rendering image; Determine the image difference information between the rendered head image and the target head image; The 3D head model is trained based on the image difference information.

2. The training method according to claim 1, characterized in that, The Gaussian sphere parameters include Gaussian sphere rendering parameters and position offset parameters. The Gaussian sphere learning model includes a neural network model. The head rendering model includes a human head FLAME model and a 3D Gaussian rendering model. Obtaining the Gaussian sphere parameters output by the Gaussian sphere learning model when the target head image is used as a training input sample includes: When the target head image is used as a training input sample, the vertex features of the vertices in the human head FLAME model used to associate with the pose of the target head are determined. The vertex features of the vertex are input into the neural network model to obtain Gaussian sphere rendering parameters and position offset parameters. The Gaussian sphere rendering parameters are used to represent the rendering properties of the Gaussian sphere in the 3D Gaussian rendering model, and the position offset parameters are used to indicate the position offset of the Gaussian sphere in the 3D Gaussian rendering model.

3. The training method according to claim 2, characterized in that, The Gaussian sphere rendering parameters include spherical harmonic functions, opacity, rotation matrix, and scale vector. The position offset parameters include the offset vector of the center point of the Gaussian sphere and the weight values ​​of the vertices associated with the pose of the target head.

4. The training method according to claim 2, characterized in that, The step of inputting the Gaussian sphere parameters and the head control information into the head rendering model to obtain the head rendering image includes: The target position of the target vertex after the target head changes is determined based on the position offset parameter and the head control information, wherein the target vertex is a vertex in the target head image; Input the target position and Gaussian sphere rendering parameters of each target vertex into the human head FLAME model to obtain the head rendering parameters corresponding to each target vertex; The head rendering parameters corresponding to each target vertex are input into the 3D Gaussian rendering model to obtain the head rendering image.

5. The training method according to claim 4, characterized in that, Determining the target position of the target vertex after the target head changes, based on the position offset parameter and the head control information, includes: The vertex position of the target vertex is updated according to the position offset parameter to obtain the static position of the target vertex before the target head changes. The static position is moved according to the head control information to obtain the target position of the target vertex after the target head changes.

6. The training method according to claim 5, characterized in that, The position offset parameter includes the offset vector of the center point of the Gaussian sphere. Updating the vertex position of the target vertex according to the position offset parameter to obtain the static position of the target vertex before the change in the target head includes: The position of the target vertex is added to the offset vector of the center point of the Gaussian sphere to obtain the static position of the target vertex before the target head changes.

7. The training method according to claim 5, characterized in that, The head control information includes camera pose and head key points of the target head image. The camera pose is used to instruct the camera to capture the pose of the target head. The position offset parameter includes the weight values ​​of vertices associated with the pose of the target head. Moving the static position according to the head control information to obtain the target position of the target vertex after the target head changes includes: According to the preset linear hybrid skinning algorithm, the target vertex is updated using the static position, the camera pose, the head key points and the vertex weights to obtain the target position of the target vertex after the target head changes.

8. The training method according to any one of claims 2 to 7, characterized in that, The vertex features used to associate the vertices in the head FLAME model with the pose of the target head include: Obtain the vertex positions in the human head FLAME model used to associate the vertices with the pose of the target head; Based on the vertex position, a feature sampling operation is performed in a preset sampling space to obtain the vertex features.

9. A method for rendering a 3D head, applied to an electronic device, characterized in that, include: When the electronic device establishes a conference connection with the target device, it receives real-time head control information of the target user sent by the target device. The target user is the user participating in the conference through the target device. The real-time head control information is input into the 3D head model corresponding to the target user, so that the 3D head model outputs a real-time head rendering image, wherein the 3D head model is the 3D head model obtained by training according to any one of claims 1 to 8. Generate a 3D head image based on the real-time head rendering image; Present a 3D head image of the target user.

10. An electronic device, characterized in that, The device includes a memory and a processor, the memory being connected to the processor, the processor being configured to execute one or more computer programs stored in the memory, the processor, when executing the one or more computer programs, causing the electronic device to implement the training method for a 3D head model as described in any one of claims 1-8 or the rendering method for a 3D head as described in claim 9.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the training method for the 3D head model as described in any one of claims 1-8.

Images