Method, system, medium and product for generating a video by fusing a digital human with a 3D scene

Abstract

This invention relates to a method, system, medium, and product for fusing digital humans with 3D scenes to generate videos, and pertains to the field of data processing technology. The method first acquires digital human image and 3D scene data, and normalizes the image data using a preset resolution and scene type. Based on spatial coordinates and perspective parameters, the fusion position is calculated, and the image and scene are spatially fused to generate a camera motion trajectory. The input text is semantically parsed and pauses are identified to generate audio, and phoneme sequences are extracted to generate matching lip-sync data. Finally, the timestamps of the audio, lip-sync, and camera motion are mapped to a unified timeline for synchronization, and the final digital human video file is output through frame-by-frame rendering. This method achieves a high degree of coordination between the digital human image, scene, voice, and lip-sync, generating natural and smooth digital human video content. Implementing the technical solution provided in this application can improve the accuracy of video generation through the fusion of digital humans and 3D scenes.

Description

Technical Field

[0001] This application relates to the field of data processing technology, specifically to a method, system, medium, and product for generating videos by fusing digital humans with 3D scenes. Background Technology

[0002] With the rapid development of virtual reality and artificial intelligence technologies, digital humans, as a new generation of human-computer interaction interfaces, are widely used in fields such as news broadcasting, commercial marketing, and cultural display. Digital human technology uses virtual avatars to achieve intelligent information transmission, providing users with a more intuitive and immersive interactive experience.

[0003] Currently, scene integration of digital humans mainly relies on preset action templates and 2D scene overlay technology. This method, based on pre-made animation, can meet basic information display needs and is widely used in fixed-scene broadcasting applications.

[0004] However, with the increasing diversity of application scenarios, the demand for interaction between digital humans and 3D scenes is constantly rising. In practical applications, the implementation method based on pre-made animations struggles to accurately grasp the spatial positioning and motion continuity of digital humans in complex 3D scenes, and the realism and coordination of visual effects are difficult to guarantee. In particular, when digital humans need to provide dynamic explanations within a scene, the lack of effective spatial fusion and synchronization control mechanisms can easily lead to inconsistencies between the digital human's performance and the scene environment, thereby reducing the accuracy of the videos generated by fusing digital humans with 3D scenes. Summary of the Invention

[0005] This application provides a method, system, medium, and product for fusing digital humans with 3D scenes to generate videos, which can improve the accuracy of videos generated by fusing digital humans with 3D scenes.

[0006] The first aspect of this application provides a method for generating video by fusing digital humans with 3D scenes, comprising: Acquire the image data and 3D virtual scene data of the target digital human; The size parameters corresponding to the image data are determined according to the preset resolution mapping relationship, the posture parameters are determined according to the preset broadcast scene type, and the image data is normalized based on the size parameters and the posture parameters to obtain the target image data. Calculate the fusion position coordinates based on the spatial coordinates of the image data and the perspective parameters of the three-dimensional virtual scene data, and spatially fuse the target image data and the three-dimensional virtual scene data according to the fusion position coordinates to generate camera motion trajectory data. The system receives input text, performs semantic parsing and pause recognition on the input text to generate audio data, extracts the phoneme sequence of the input text, and generates lip-sync data matching the target digital human based on the phoneme sequence and the timestamp information of the audio data. The timestamps of the audio data, the lip-sync data, and the camera motion trajectory data are mapped to preset time axis coordinates and synchronized. Based on the synchronized timestamps, the camera motion trajectory data, lip-sync data, and audio data are rendered and fused frame by frame to output a digital human video file.

[0007] By employing the aforementioned technical solutions, image data and 3D virtual scene data of the target digital human are acquired. The image data is then normalized based on a preset resolution mapping relationship and a preset broadcast scene type, ensuring that the digital human maintains appropriate size proportions and posture in different scenes. Spatial fusion is achieved by calculating the fusion position coordinates of the image data and the 3D virtual scene data, generating camera motion trajectory data, thus realizing precise positioning and dynamic display of the digital human in the 3D scene. Combining the audio data generated from semantic parsing and pause recognition of the input text, as well as the matching lip-sync data generated based on phoneme sequences and audio timestamps, further enhances the naturalness of the digital human's performance. Finally, by mapping the timestamps of the audio data, lip-sync data, and camera motion trajectory data to a unified timeline and performing synchronous alignment, the temporal consistency of each component is ensured during frame-by-frame rendering and fusion, thereby improving the accuracy of the video generated by fusing the digital human with the 3D scene. This enables the digital human to achieve smooth and natural dynamic narration performance in complex 3D scenes.

[0008] Optionally, the target output resolution is extracted from the image data, and the corresponding scaling factor is found in the preset resolution mapping relationship according to the target output resolution; the height pixel value and width pixel value of the image data are calculated according to the scaling factor to obtain the size parameter; a corresponding posture template is matched according to the preset broadcast scene type, the posture template includes multiple bone parameters, the bone parameters are standing posture bone parameters, sitting posture bone parameters or dynamic bone parameters; the bone node coordinates in the image data are mapped to the bone parameters of the posture template to complete the posture transformation; the transformed image data is normalized to make the coordinate range of the image data normalized to the preset rendering space coordinate system to obtain the target image data.

[0009] Optionally, virtual camera parameters are extracted from the 3D virtual scene data, the virtual camera parameters including a viewpoint matrix and a projection matrix; perspective parameters are calculated based on the viewpoint matrix and the projection matrix, and the 2D coordinates of the image data are converted into 3D spatial coordinates based on the perspective parameters; the axial coordinates of the target image data in the 3D virtual scene are calculated based on the scene depth information of the 3D virtual scene data; the fusion position coordinates are determined based on the 3D spatial coordinates and the axial coordinates, and the target image data is embedded into the lens position corresponding to the 3D virtual scene data; lens motion parameters of the lens position are obtained, the lens motion parameters including motion duration, starting focal length, and ending focal length; the total number of frames of lens motion is determined based on the motion duration, and the focal length change is calculated based on the starting focal length and the ending focal length; based on the total number of frames and the focal length change, the focal length value corresponding to each frame is calculated using a preset interpolation algorithm to generate the lens motion trajectory data.

[0010] Optionally, the following steps are taken: acquiring large-screen material data, including image data or video data; extracting position and size parameters of the virtual large-screen display area from the target 3D virtual scene data; scaling and transforming the large-screen material data according to the position and size parameters to generate adapted large-screen material data; mapping the adapted large-screen material data as a texture onto the virtual large-screen display area; acquiring scene prop data, including 3D model data and material data; determining placement coordinates based on the position coordinates of the target digital human image data; and loading the scene prop data to the placement coordinates of the target 3D virtual scene data.

[0011] Optionally, the input text is segmented into words, sentence boundaries are identified, and pause markers are inserted at the pause positions corresponding to the sentence boundaries to obtain processed text information; the processed text information is converted into speech to generate audio data and timestamp information corresponding to the audio data; the input text is decomposed into phonemes, the phoneme sequence is extracted, and the corresponding lip shape deformation parameters are matched according to the phoneme sequence; the lip shape switching time point corresponding to each phoneme is calculated according to the timestamp information of the audio data and the lip shape deformation parameters; the lip shape data is generated based on the lip shape switching time points, and the lip shape data includes a lip shape deformation parameter sequence and a corresponding timestamp sequence.

[0012] Optionally, the system iterates frame by frame according to the preset time axis coordinates, obtains the corresponding lens focal length and position parameters from the lens motion trajectory data based on the timestamp of the current frame, and obtains the corresponding lip shape deformation parameters from the lip shape data; updates the focal length and spatial position of the virtual camera based on the lens focal length and position parameters to adjust the perspective and depth of field of the 3D virtual scene; drives the facial bone nodes of the target digital human model according to the lip shape deformation parameters to complete the lip shape deformation; renders the target digital human model and the 3D virtual scene after the lip shape deformation is completed to generate the image frame data of the current frame; encodes all image frame data in chronological order and encapsulates them with the audio data to output the digital human video file.

[0013] Optionally, pre-rendering can be performed based on the timestamps of the synchronized audio data, lip-sync data, and camera motion trajectory data. Synchronization detection is performed on the audio data and lip-sync data in the pre-rendered results, and the time deviation between the pronunciation time of the audio data and the switching time of the lip-sync data is calculated. When the time deviation is greater than a preset threshold, the target phoneme sequence corresponding to the time interval in which the deviation occurred is extracted. Calculate the target lip-sync switching time point based on the target phoneme sequence, and update the timestamp sequence of the lip-sync data according to the target lip-sync switching time point; The updated lip-sync data is remapped to the preset time axis coordinates to generate corrected synchronization alignment data. Secondly, embodiments of this application provide a system for fusing digital humans with 3D scenes to generate videos. The system includes one or more processors and a memory. The memory is coupled to the one or more processors and is used to store computer program code, which includes computer instructions. The one or more processors call the computer instructions to cause the system for fusing digital humans with 3D scenes to perform the methods described in the first aspect and any possible implementation thereof.

[0014] Thirdly, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on a video generation system that fuses a digital human with a 3D scene, cause the video generation system to perform the method described in the first aspect and any possible implementation thereof.

[0015] Fourthly, embodiments of this application provide a computer program product containing instructions that, when the computer program product is run on a video generation system that fuses digital humans and 3D scenes, causes the video generation system to perform the method described in the first aspect and any possible implementation thereof.

[0016] In summary, one or more technical solutions provided in this application have at least the following technical effects or advantages: By employing the aforementioned technical solutions, image data and 3D virtual scene data of the target digital human are acquired. The image data is then normalized based on a preset resolution mapping relationship and a preset broadcast scene type, ensuring that the digital human maintains appropriate size proportions and posture in different scenes. Spatial fusion is achieved by calculating the fusion position coordinates of the image data and the 3D virtual scene data, generating camera motion trajectory data, thus realizing precise positioning and dynamic display of the digital human in the 3D scene. Combining the audio data generated from semantic parsing and pause recognition of the input text, as well as the matching lip-sync data generated based on phoneme sequences and audio timestamps, further enhances the naturalness of the digital human's performance. Finally, by mapping the timestamps of the audio data, lip-sync data, and camera motion trajectory data to a unified timeline and performing synchronous alignment, the temporal consistency of each component is ensured during frame-by-frame rendering and fusion, thereby improving the accuracy of the video generated by fusing the digital human with the 3D scene. This enables the digital human to achieve smooth and natural dynamic narration performance in complex 3D scenes. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating a method for generating video by fusing digital humans with 3D scenes, as disclosed in an embodiment of this application. Figure 2 This is another flowchart illustrating a method for generating video by fusing digital humans with 3D scenes, as disclosed in an embodiment of this application. Figure 3 This is a schematic diagram of the structure of a system provided in an embodiment of this application.

[0018] Explanation of reference numerals in the attached drawings: 301, Central Processing Unit; 302, Read-Only Memory; 303, Random Access Memory; 304, Bus; 305, Input / Output Interface; 306, Input Section; 307, Output Section; 308, Storage Section; 309, Communication Section; 310, Driver; 311, Removable Media. Detailed Implementation

[0019] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0020] In the description of the embodiments of this application, the words "for example" or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design that is described as "for example" or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design options. Rather, the use of the words "for example" or "for instance" is intended to present the relevant concepts in a specific manner.

[0021] In the description of the embodiments of this application, the term "multiple" means two or more. For example, multiple systems means two or more systems, and multiple screen terminals means two or more screen terminals. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the indicated technical features. Thus, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature. The terms "comprising," "including," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0022] This application provides a method for acquiring data in cable channels, referring to... Figure 1 , Figure 1 This is a flowchart illustrating a method for fusing a digital human with a 3D scene to generate a video, as provided in an embodiment of this application. The method is applied to a system, which refers to a hardware and software integrated platform capable of executing a program for fusing a digital human with a 3D scene to generate a video. The system can execute a program for fusing a digital human with a 3D scene to generate a video. The method includes steps 101 to 106, as follows: Step 101: Obtain the image data and 3D virtual scene data of the target digital human.

[0023] Target digital human image data refers to the digital human model data created based on 3D modeling technology, including geometric mesh data, skeletal rigging data, material mapping data, and animation data. Geometric mesh data defines the digital human's outline and surface details; skeletal rigging data defines the digital human's motion structure and deformation control points; material mapping data includes surface attributes such as skin, hair, and clothing; and animation data includes preset expressions and motion sequences. 3D virtual scene data refers to the collection of 3D models used to construct a virtual environment, including scene model data, lighting data, and camera data. Scene model data defines the shape and position of various objects in the environment; lighting data defines the scene's lighting effects; and camera data defines the viewing angle and rendering parameters.

[0024] Specifically, the methods for acquiring digital human avatar data include: reading existing digital human model files from a pre-set digital human resource library; creating and exporting a new digital human model using 3D modeling software such as Maya or 3ds Max; or acquiring and reconstructing real human data using a 3D scanning device. The data format uses the common FBX or glTF format, ensuring complete mesh data, skeleton data, material data, and animation data. The methods for acquiring 3D virtual scene data include: selecting a pre-made virtual scene model from a scene resource library; creating and exporting a custom scene using modeling software; or dynamically creating a scene using procedural generation techniques. Scene data also uses FBX or glTF format, ensuring the integrity of lighting, shadow, and material information. During data acquisition, it is necessary to verify the integrity and validity of the data, check the correctness of the geometric structure, material mapping, and skeleton hierarchy, and perform necessary data repair and optimization to lay the foundation for subsequent processing.

[0025] Step 102: Determine the size parameters corresponding to the image data according to the preset resolution mapping relationship, determine the posture parameters according to the preset broadcast scene type, and normalize the image data based on the size parameters and posture parameters to obtain the target image data.

[0026] The preset resolution mapping relationship refers to a pre-established correspondence table between output resolution and scaling factor, used to determine the display size of the digital human model at different resolutions. For example, 1920x1080 resolution corresponds to a scaling factor of 1.0, and 3840x2160 resolution corresponds to a scaling factor of 2.0. Size parameters include the height and width pixel values ​​of the digital human model, calculated using the scaling factor. Preset broadcast scene types refer to the standard posture types of the digital human in different application scenarios, such as standing broadcast, sitting broadcast, and walking explanation. Posture parameters include the spatial position and rotation angle of skeletal nodes, used to control the digital human's standing position and movements. Normalization processing refers to standardizing the size and position of the digital human model to a preset coordinate range to ensure a consistent proportional relationship in different scenarios.

[0027] Specifically, the target output resolution, such as 1920x1080, is first read from the image data, and the corresponding scaling factor of 1.0 is found in the preset resolution mapping table. The original pixel size of the digital human model is multiplied by the scaling factor to calculate the actual display size, such as a height of 1800 pixels and a width of 600 pixels. Next, the corresponding skeletal parameter template is loaded according to the scene type (such as standing posture broadcast). This template defines the position and angle values ​​of each skeletal node under the standard standing posture. The skeletal hierarchy of the digital human model is mapped to the template, and the Transform matrix of each skeletal node is adjusted to conform to the standard standing posture requirements. Then, the adjusted digital human model is placed into a normalized rendering space coordinate system, and the model's position, rotation, and scaling values ​​are mapped to the standard range of [-1, 1]. Through matrix transformation, the model's vertex coordinates, UV coordinates, and skeletal weights are normalized accordingly to generate standardized target image data. This data achieves a unified appearance under different resolutions and scenes while maintaining the original appearance characteristics.

[0028] In one possible implementation, the size parameters corresponding to the image data are determined according to a preset resolution mapping relationship, the posture parameters are determined according to a preset broadcast scene type, and the image data is normalized based on the size parameters and posture parameters to obtain the target image data. Specifically, this includes steps 1021-1025, as follows: Step 1021: Extract the target output resolution from the image data, and find the corresponding scaling factor in the preset resolution mapping relationship based on the target output resolution.

[0029] Specifically, the target output resolution refers to the output size of the video image, expressed as the horizontal pixel count multiplied by the vertical pixel count. Common resolutions include 1920x1080 (FHD) and 3840x2160 (4K). The resolution information in the image data is stored in the metadata section of the model file, used to specify the target size for the rendered output. The preset resolution mapping relationship is a pre-configured data table that records the standard scaling factors corresponding to different resolutions, used to maintain the correct proportions of the digital human image at different resolutions. The scaling factor is a floating-point value representing the magnification or reduction factor relative to the base resolution; for example, 1920x1080 corresponds to a scaling factor of 1.0, 3840x2160 corresponds to a scaling factor of 2.0, and 1280x720 corresponds to a scaling factor of 0.67.

[0030] Specifically, the process first parses the header information of the image data file, reading the target output resolution value stored in the metadata field. For FBX format files, the resolution attribute is obtained through the metadata interface of the FBX SDK; for glTF format files, custom resolution information is read from the extras field. The extracted resolution value includes the number of pixels in both horizontal and vertical dimensions, such as 1920 and 1080. Then, a search is performed in a pre-defined resolution mapping table, which is stored in key-value pairs. The key is the resolution string (e.g., "1920x1080"), and the value is the corresponding scaling factor. The search process uses an exact match method, combining the extracted resolution values ​​into a search key to retrieve the corresponding scaling factor in the mapping table. If a match is found, the corresponding scaling factor value is returned; if no exact match is found, the nearest matching principle is used, selecting the scaling factor corresponding to the closest resolution. For example, when the target output resolution is 1920x1080, the search returns a scaling factor of 1.0; when the resolution is 3840x2160, it returns a scaling factor of 2.0. This scaling factor will be used for subsequent size parameter calculations.

[0031] Step 1022: Calculate the height and width pixel values ​​of the image data based on the scaling factor to obtain the size parameters.

[0032] The scaling factor is a numerical value used to adjust the displayed size of a digitized human model, representing the magnification or reduction factor relative to a reference size. The height and width pixel values ​​of the avatar data refer to the actual display size of the digitized human model in the rendered view, in pixels. The height pixel value defines the vertical distance from the top of the digitized human's head to its feet, and the width pixel value defines the horizontal distance between the digitized human's side edges. The size parameter is a two-dimensional vector composed of the height and width pixel values, used to control the display ratio of the digitized human in the scene. The reference size is a predefined standard display size, such as a height of 1800 pixels and a width of 600 pixels, used as a reference value for scaling calculations.

[0033] Specifically, the baseline dimensions of the digital human model are first obtained, including the baseline height H_base and the baseline width W_base. The values ​​of the baseline height and width are usually preset in the configuration file, such as H_base = 1800 pixels and W_base = 600 pixels. Then, the baseline dimensions are multiplied by a scaling factor to calculate the actual display size. The calculation formula is: Height (in pixels) H = H_base × scaling factor, Width (in pixels) W = W_base × scaling factor. For example, when the scaling factor is 1.0, the actual display size is 1800 × 600 pixels; when the scaling factor is 2.0, the actual display size is 3600 × 1200 pixels. The calculated height and width pixel values ​​constitute the size parameters, denoted as size_params = (H, W). In practical applications, these size parameters are used to set the size of the rendering viewport, adjust the display ratio of the model, and ensure correct display effects at different resolutions. The calculation process requires ensuring that pixel values ​​are integers and rounding the calculation results. At the same time, it is also necessary to verify whether the calculated size is within the valid range to avoid display sizes that are too large or too small.

[0034] Step 1023: Match the corresponding posture template according to the preset broadcast scene type. The posture template includes multiple skeletal parameters, which can be standing skeletal parameters, sitting skeletal parameters, or dynamic skeletal parameters.

[0035] Preset broadcast scene types refer to the standard performance forms of the digital human in different scenarios, including basic types such as standing broadcast, sitting broadcast, and walking explanation. Posture templates are predefined standard motion posture datasets, containing skeletal hierarchical structures and joint parameters. Skeletal parameters describe the spatial position, rotation angle, and scaling ratio of skeletal nodes, stored in the form of Transform matrices. Standing posture skeletal parameters define the parameter values ​​of each skeletal node in a standard standing state, such as a posture with the spine vertical and arms hanging naturally. Sitting posture skeletal parameters define the skeletal configuration in a standard sitting state, such as a posture with hip flexion and knee flexion. Dynamic skeletal parameters define the sequence of skeletal changes during movement, used to achieve continuous movements such as walking and turning.

[0036] Specifically, the broadcast scene type identifier is first read from the scene configuration, such as "standing_broadcast" indicating standing broadcast and "seated_broadcast" indicating seated broadcast. Based on the scene type identifier, the corresponding template data is retrieved from the pose template library. The pose templates are stored using a hierarchical data structure, with each template containing a complete skeletal hierarchy tree and parameter set. The skeletal hierarchy tree defines the hierarchical relationship from the root node (usually the pelvis) to the end nodes (such as fingers or toes). For the standing pose template, the root node is located at the vertical projection point on the ground, the spine remains vertical, the arms hang naturally, and the head is slightly tilted forward by 15 degrees; for the seated pose template, the root node is located on the seat surface, the hip joint is bent at 90 degrees, the knee joint is bent at 90 degrees, and the arms are placed on the armrests or knees. The parameters of each skeletal node include: a position vector P(x, y, z), a rotation quaternion R(x, y, z, w), and a scaling vector S(x, y, z). These parameters form the Transform matrix T = [P|R|S], used to determine the precise position and pose of the bones in three-dimensional space. The dynamic template additionally includes a sequence of keyframes, each recording the skeletal parameters at a specific moment. Continuous motion changes are calculated through interpolation. After matching is complete, the selected pose template data is loaded into memory to prepare for subsequent skeletal mapping.

[0037] Step 1024: Map the coordinates of the skeletal nodes in the image data to the skeletal parameters of the pose template to complete the pose transformation.

[0038] Skeletal node coordinates are the positional information of skeletal joints in 3D space as defined in the image data. Represented using a local coordinate system, they include three components: position, rotation, and scaling. Each skeletal node forms a hierarchical structure through parent-child relationships, with child nodes' transformations influenced by their parent nodes. The skeletal parameters of the pose template are the target transformation data for skeletal nodes under standard pose conditions, also containing position, rotation, and scaling information. Pose transformation refers to the process of adjusting the current skeletal pose to a target pose, involving coordinate system transformation and interpolation calculations.

[0039] Specifically, the correspondence between source and target bone nodes is first established by matching bone names or IDs. For each matched bone node pair, the local transformation matrix Ms=[Ps|Rs|Ss] of the source bone node and the transformation matrix Mt=[Pt|Rt|St] of the target bone parameters are extracted, where P is the position vector, R is the rotation quaternion, and S is the scaling vector. The transformation difference matrix Md=Mt×Ms^(-1) is calculated, and then vector addition P'=Ps+(Pt-Ps) is performed on the position component, spherical linear interpolation R'=slerp(Rs, Rt, 1.0) is used on the rotation component, and linear interpolation S'=Ss+(St-Ss) is performed on the scaling component. Finally, a new transformation matrix M'=[P'|R'|S'] is constructed. The transformation calculation follows the skeletal hierarchy, processing each child node sequentially starting from the root node. For a parent node p and a child node c, the world coordinates of the child node are calculated using the formula Wc = Wp × Mc, where Wc is the world transformation matrix of the child node, Wp is the world transformation matrix of the parent node, and Mc is the local transformation matrix of the child node. After transforming all skeletal nodes, the positions of the mesh vertices are updated using Linear Blend Skinning (LBS). Each vertex is influenced by multiple skeletal nodes, and the influence weights are stored in the vertex attributes. The final position of the vertex is calculated using the formula V' = ∑(Wi × Mi × V), where V is the original vertex position, Wi is the bone weight, Mi is the world transformation matrix of the bone, and V' is the transformed vertex position. This mapping process achieves a smooth transition of the digital human model from the current pose to the target pose, while maintaining the integrity of the skeletal structure and the continuity of motion.

[0040] Step 1025: Normalize the transformed image data to bring the coordinate range of the image data to a preset rendering space coordinate system, thus obtaining the target image data.

[0041] The transformed image data is the digital human model data after pose transformation, containing updated vertex coordinates, bone parameters, and material information. Normalization is a mathematical transformation process that maps data to a standardized range, ensuring that the data is distributed within a fixed interval. The rendering space coordinate system is the standard coordinate system used for 3D graphics rendering, typically limiting the coordinate range to the interval [-1, 1] or [0, 1]. The target image data is the normalized digital human model data, suitable for direct use in the rendering pipeline.

[0042] Specifically, the normalization process begins by calculating the bounding box of the current image data. This involves iterating through all vertex coordinates to find the minimum (min_x), minimum (min_y), and maximum (max_x), maximum (max_y), and maximum (max_z) values ​​along the x, y, and z axes. The center point of the bounding box is calculated as center = ((max_x + min_x) / 2, (max_y + min_y) / 2, (max_z + min_z) / 2), and its size is calculated as size = (max_x - min_x, max_y - min_y, max_z - min_z). Based on the bounding box size, a normalization scaling factor is calculated as scale = 2.0 / max(size.x, size.y, size.z), ensuring that the maximum side length is mapped to the interval [-1, 1]. A normalization transformation is performed on all vertex coordinates using the formula: normalized_position = (original_position - center) * scale. The same normalization transformation is then performed on the position coordinates of the skeletal nodes to ensure a consistent proportional relationship between the skeletal system and the mesh model. Simultaneously, the normal and tangent vectors are updated, but only rotation transformations are performed without translation or scaling, maintaining the unit length of the vectors. The UV coordinates are already within the range [0, 1], requiring no additional normalization. The sum of the weight data remains 1, also requiring no normalization. Finally, the normalized data is encapsulated into target image data, including standardized vertex buffers, bone data, material parameters, and transformation matrices. This data is further transformed during rendering using view and projection matrices and projected onto screen space. Normalization ensures that the digital human model maintains consistent display proportions and positional relationships across different rendering environments.

[0043] Step 103: Calculate the fusion position coordinates based on the spatial coordinates of the image data and the perspective parameters of the 3D virtual scene data. Spatially fuse the target image data and the 3D virtual scene data according to the fusion position coordinates to generate camera motion trajectory data.

[0044] Spatial coordinates of the image data refer to the positional information of the digital human model in three-dimensional space, represented using the world coordinate system. Perspective parameters of the 3D virtual scene data include the virtual camera's viewpoint matrix and projection matrix, determining the scene's viewing angle and perspective effect. Blended position coordinates are the precise placement of the digital human model in the virtual scene, calculated through coordinate transformation. Lens motion trajectory data describes the virtual camera's movement path in the scene, including a sequence of changes in parameters such as camera position, orientation, and focal length over time. Spatial fusion is the process of integrating the digital human model into the virtual scene according to its calculated position coordinates.

[0045] Specifically, spatial fusion is achieved by first extracting camera parameters from the 3D virtual scene data, including the viewpoint position eye(x, y, z), the observed target target(x, y, z), and the up direction vector up(x, y, z). Based on these parameters, a view matrix View = lookAt(eye, target, up) is constructed. Simultaneously, the camera's field of view fov, aspect ratio aspect, near plane near, and far plane far are obtained, and a projection matrix Projection = perspective(fov, aspect, near, far) is constructed. These are combined to obtain the view projection matrix VP = Projection × View, used to convert world coordinates to clip space coordinates. Depth buffer information is read from the scene data to obtain the depth values ​​of scene objects. The world coordinates of the digital human model are multiplied by the view projection matrix to obtain the clip space coordinates clip_pos = VP × world_pos. The clip space coordinates are then converted to NDC (Normalized Device Coordinates) space, and the screen space coordinates screen_pos = (clip_pos.xy / clip_pos.w+1) * 0.5 are calculated. The depth value `scene_depth` at a given location in the scene is obtained by sampling from the depth buffer based on screen space coordinates. The depth value `model_depth` of the digital human model is compared with the scene depth value `scene_depth` to ensure correct occlusion relationships. The z-coordinate of the digital human model is adjusted based on the depth comparison results to complete precise spatial positioning calibration. For camera movement, the initial camera state `state0 = {position0, rotation0, fov0}` and the target camera state `state1 = {position1, rotation1, fov1}` are recorded. The motion duration and frame rate `fps` are set, and the total number of frames `frames = duration × fps` is calculated. Spherical linear interpolation (SLERP) is used to calculate rotation interpolation, and linear interpolation is used to calculate position and focal length interpolation. For each frame `time_t`, the camera state is calculated as: `position_t = lerp(position0, position1, t)`, `rotation_t = slerp(rotation0, rotation1, t)`, `fov_t = lerp(fov0, fov1, t)`, where `t = time_t / duration`. The interpolated camera state sequence is saved as lens motion trajectory data for subsequent rendering.

[0046] Step 104: Receive input text, perform semantic parsing and pause recognition on the input text to generate audio data, extract the phoneme sequence of the input text, and generate lip-sync data that matches the target digital human based on the phoneme sequence and the timestamp information of the audio data.

[0047] The input text is the text content to be broadcast. Semantic parsing is the process of analyzing the grammatical structure of the text and performing semantic annotation, including word segmentation,词性标注 (not sure what this should be in English, maybe "pos tagging"), syntactic analysis, etc. Pause recognition is the process of determining the pause positions and durations in the text, judged by punctuation marks and grammatical structures. Audio data is the sound information generated by text-to-speech, including waveform data and time information. The phoneme sequence is the sequence of basic pronunciation units in speech, recording the type and duration of each phoneme. The timestamp information marks the start and end time points of each phoneme. The lip movement data describes the sequence of lip movements during speech, including the position information of lip key points.

[0048] Specifically, first perform natural language processing on the input text. Segment the text into a sequence of words through a word segmenter, and use a pos tagger to tag the词性 of each word. Determine the grammatical structure based on the syntactic analysis tree and identify components such as the subject, predicate, and object. Determine the pause positions according to punctuation marks and grammatical structures, insert a 1000ms pause at the end of a sentence, a 500ms pause at a comma, and a 200ms pause between words. Input the processed text into a speech synthesis engine to generate audio data. The audio data is in PCM format with a sampling rate of 16kHz and a sampling depth of 16bit, and at the same time record the timestamp of each phoneme. Convert the text into a phoneme sequence through a phoneme recognizer, for example, "你好" is converted to " / n / i / h / ao / ". Create a mapping table from phonemes to lip shapes, defining the standard lip shape parameters corresponding to each phoneme. The lip shape parameters include: upper lip height, lower lip height, lip corner distance, and lip roundness. According to the phoneme sequence and timestamp, assign the corresponding lip shape parameters to each phoneme. Based on the lip shape parameters of adjacent phonemes and the time interval, use cubic spline interpolation to calculate the lip shape parameters of the intermediate frames to ensure the smoothness of lip movement changes. During pauses, gradually transition the lip shape to the natural closed state. The generated lip movement data sequence records the lip shape parameters and corresponding timestamps of each frame, which are used to control the mouth deformation of the digital human model. The lip shape parameters are applied to the model mesh through skeletal animation or deformers to achieve dynamic lip movement changes. At the same time, ensure that the lip movement changes are synchronized with the audio playback, and the lip shape parameters at the start moment of the phoneme accurately correspond to the pronunciation requirements.

[0049] In a possible implementation, perform semantic parsing and pause recognition on the input text to generate audio data, extract the phoneme sequence of the input text, and generate lip movement data matching the target digital human based on the phoneme sequence and the timestamp information of the audio data, specifically including steps 1041 - step 1044. The above steps are as follows: Step 1041: Perform word segmentation on the input text, identify the sentence boundaries, and insert pause markers at the pause positions corresponding to the sentence boundaries to obtain the processed text information.

[0050] Note: The "词性标注" in the original text needs to be replaced with the correct English term for "pos tagging" if there is a specific one. Also, "你好" remains in Chinese as it's not clear if there is a specific English equivalent provided in the context.The sentence boundary is the starting and ending positions of sentences in the text, which are determined by punctuation marks and grammatical features. The pause marker is a special symbol used to indicate where pauses need to be inserted in speech synthesis, and is represented using a specific format such as [PAU=duration]. The processed text information includes word segmentation results, sentence boundary markers, and pause markers. The text information is stored in a structured format, recording the position, type, and pause information of each word.

[0051] Specifically, the word segmentation process first converts the input text into Unicode encoding to ensure correct processing of multi-language texts such as Chinese and English. The maximum matching algorithm is used to segment the text, and a trie structure is maintained to store dictionary information. Starting from the beginning of the sentence, the longest matching word is searched for in the trie. For out-of-vocabulary words, the probability of binary character combinations and a conditional random field model are used for identification. Example of word segmentation: "今天天气真好" is segmented into "今天 / 天气 / 真 / 好". Sentence boundary recognition is based on punctuation marks and grammar rules. Periods, question marks, and exclamation marks mark the end of sentences, while commas and semicolons mark the boundaries of clauses. For each recognized boundary position, pause markers of different durations are inserted according to the boundary type. [PAU=1000] is inserted at the end of the sentence to indicate a 1000-millisecond pause, [PAU=500] is inserted at the comma to indicate a 500-millisecond pause, and [PAU=200] is inserted between words to indicate a 200-millisecond natural pause. The processed text information is organized into structured data, and each word record contains: word text (text), starting position (start_pos), ending position (end_pos),词性标注 (pos_tag), boundary type (boundary_type), and pause duration (pause_duration). Example output: "今天[PAU=200]天气[PAU=500]真[PAU=200]好[PAU=1000]". This structured text information facilitates subsequent speech synthesis and lip movement generation processing, ensuring that the generated speech has a natural tone and rhythm. At the same time, the position and duration information of the pause markers are also used to coordinate the synchronization of audio and lip animation.

[0052] Step 1042: Perform speech conversion on the processed text information to generate audio data and timestamp information corresponding to the audio data.

[0053] The processed text information is structured text data that has undergone word segmentation and pause marking, containing word sequences and pause markers. Speech conversion is the process of converting text into speech signals, implemented based on acoustic models and vocoders. Audio data is a digital audio signal in PCM format, recording a sequence of sampled sound waveform values. Timestamp information records the start time, end time, and duration of each phoneme and pause in the audio. A phoneme is the smallest unit of speech, and each phoneme has its specific acoustic features and duration parameters. A vocoder is a module that converts acoustic features into audio waveforms, synthesizing natural speech through acoustic parameters.

[0054] Specifically, the text sequence is first input into the acoustic model. The acoustic model employs a Transformer architecture, using a self-attention mechanism to capture long-range dependencies in the text. The acoustic model outputs a sequence of acoustic features, including a Mel spectrogram, fundamental frequency (F0), phoneme duration, and sound intensity. The sampling rate for the acoustic features is 100Hz, or one frame every 10 milliseconds. For each phoneme, its acoustic parameters are calculated: the spectral envelope describes the duct resonance characteristics, the fundamental frequency describes pitch variation, and the energy envelope describes loudness variation. The acoustic feature sequence is then input into a vocoder, using a neural network model such as WaveNet or HiFi-GAN to convert the acoustic features into a 16kHz sampling rate, 16-bit deep PCM audio waveform. During audio generation, the timestamp information for each phoneme is recorded synchronously. The timestamp data structure includes: phoneme ID, start time (in milliseconds), end time (in milliseconds), and duration (in milliseconds). For example, for the pronunciation of "today," a timestamp sequence is generated: {phoneme: "j", start: 0, end: 120}, {phoneme: "in", start: 120, end: 250}. For pause markers [PAU=duration], a silent segment of corresponding duration is inserted into the audio and recorded as a special phoneme type in the timestamp. The final generated audio data and timestamp information correspond one-to-one, providing a precise timing control basis for subsequent lip-sync generation. The timestamp precision is controlled at the 10-millisecond level to ensure accurate positioning of phoneme boundaries.

[0055] Step 1043: Decompose the input text into phonemes, extract the phoneme sequence, and match the corresponding mouth shape deformation parameters according to the phoneme sequence.

[0056] Phoneme decomposition is the process of converting characters in text into pronunciation units, mapping text to phoneme symbols based on a phoneme rule library. A phoneme sequence is a string of phoneme symbols arranged in chronological order, where each phoneme represents a basic pronunciation unit. Lip shape deformation parameters describe the shape changes of the mouth and lips during pronunciation, including numerical parameters such as lip opening degree, lip corner stretching degree, and jaw position. The deformation parameters control the mouth shape changes of a 3D model through vertex weights or bone animations. The phoneme mapping table defines the correspondence between phonemes and mouth shape parameters, storing the mouth shape configuration during the standard pronunciation of each phoneme.

[0057] Specifically, the specific implementation of phoneme decomposition first loads the phoneme rule library, which contains the mapping rules from characters to phonemes and context-related rules. For each character in the input text, its corresponding phoneme representation is found. For Chinese characters, the pinyin is obtained through a pinyin dictionary and then converted into phonemes. For example, "你" is converted into the phonemes "n-i". For English, it is directly converted into phonemes according to the phonetic rules. The generated phoneme sequence is arranged in chronological order, and each phoneme contains a type identifier and duration information. A standard mapping table from phonemes to mouth shapes is established, where each phoneme corresponds to a set of lip shape deformation parameters. The mouth shape parameters include: upper lip height (upper_lip_height, value range 0 - 1), lower lip height (lower_lip_height, value range 0 - 1), lip corner distance (lip_corner_distance, value range 0 - 1), and jaw rotation angle (jaw_rotation, value range -30 to 0 degrees). For example, the parameter combination corresponding to the phoneme "a" is {upper_lip_height: 0.8, lower_lip_height: 0.7, lip_corner_distance: 0.5, jaw_rotation: -20}. Parameter interpolation is performed between consecutive phonemes, using cubic spline interpolation to ensure the smoothness of mouth shape changes. Interpolation calculation formula: P(t)=(1 - t)³P0 + 3t(1 - t)²P1 + 3t²(1 - t)P2 + t³P3, where t is the time parameter, and P0 to P3 are control points. The generated sequence of lip shape deformation parameters is time-aligned with the phoneme sequence, and the complete set of mouth shape parameters is recorded at each time point. The parameter sequence is stored in the form of key frames, and the transition between key frames is interpolated by the real-time rendering system. For the pause segment, the mouth shape parameters smoothly transition to the natural closed state {upper_lip_height: 0.2, lower_lip_height: 0.2, lip_corner_distance: 0.3, jaw_rotation: -5}.

[0058] Step 1044: Calculate the lip-shape switching time point corresponding to each phoneme based on the timestamp information and lip-shape deformation parameters of the audio data; generate lip-shape data based on the lip-shape switching time points, the lip-shape data containing the lip-shape deformation parameter sequence and the corresponding timestamp sequence.

[0059] The timestamp information in the audio data records the start time and duration of each phoneme. Lip shape deformation parameters describe the standard lip shape configuration when a single phoneme is pronounced. Lip shape transition time points are the critical moments of lip shape changes between adjacent phonemes, determining the speed and rhythm of the lip shape transition. Lip shape data is lip shape animation information arranged in chronological order, containing the complete lip shape change process. The deformation parameter sequence records the specific lip shape parameter values ​​at each time point. The timestamp sequence records the sampling time points of the deformation parameters.

[0060] Specifically, a timeline is first constructed based on audio timestamps. Each phoneme occupies a time interval [t_start, t_end] on the timeline. To ensure the naturalness of lip-sync transitions, transition intervals are set between adjacent phonemes. The length of the transition interval depends on the phoneme duration, typically set to 20% of the phoneme duration. For a phoneme with a duration of T, its preceding transition interval is [t_start, t_start+0.2T], and its following transition interval is [t_end-0.2T, t_end]. Lip-sync parameters are interpolated within the transition intervals. Cubic Hermite interpolation is used to ensure the continuity and smoothness of parameter changes. The interpolation function is: P(t) = h00(t)P0 + h10(t)m0 + h01(t)P1 + h11(t)m1, where P0 and P1 are the lip-sync parameters of adjacent phonemes, m0 and m1 are tangent vectors, and h00, h10, h01, and h11 are Hermite basis functions. For each time point t, the lip shape parameters at that moment are calculated: all lip shape control parameters (upper lip height, lower lip height, lip corner distance, jaw angle, etc.) are traversed, and interpolation calculations are performed for each parameter. The sampling interval is set to 10 milliseconds, generating a uniformly sampled deformation parameter sequence. For audio data of length N, lip shape data of N / 10 sampling points are generated. Each sampling point records the complete set of lip shape parameters and the corresponding timestamp. For example, the record at t=100ms is: {timestamp: 100, params: {upper_lip_height: 0.6, lower_lip_height: 0.5, lip_corner_distance: 0.4, jaw_rotation: -15}}. For pauses, the lip shape parameters are gradually transitioned to a natural closed state, with the transition time set to 30% of the pause duration. The final generated lip shape data is sorted by timestamp to form a complete lip shape animation sequence. Each data frame in the sequence contains a timestamp and a complete set of lip shape parameters, used by the real-time rendering system to control the lip shape deformation of the digital human model.

[0061] Step 105: Map the timestamps of the audio data, lip movement data, and camera motion trajectory data to preset time axis coordinates and perform synchronization alignment.

[0062] The timestamps on the audio data record the temporal information of the audio sampling points, in milliseconds. The timestamps on the lip-sync data mark key moments of lip-sync changes, including the time points of parameter changes. The timestamps on the camera motion trajectory data describe the temporal information of the camera motion, recording the time points of changes in position and direction. The preset timeline coordinates are a unified time reference system used to align the temporal sequence of different data streams. Synchronization alignment processing ensures the precise correspondence of each data stream on the timeline.

[0063] Specifically, a unified time reference coordinate system is first established. The start time of the audio data is chosen as the zero point (t=0), and a time axis with millisecond precision is established. The timestamps of all data streams are converted to offset values ​​relative to the zero point. The audio data uses a fixed sampling rate (16kHz), and the timestamp for each sample point is calculated as t_audio = sample_index × (1000 / 16000) milliseconds. The lip-sync data sampling interval is 10 milliseconds, and the timestamp is calculated as t_viseme = frame_index × 10 milliseconds. Camera motion data has its timestamp calculated based on the frame rate (e.g., 30fps), t_camera = frame_index × (1000 / 30) milliseconds. A time mapping table is established to record the correspondence between the three data streams at each key time point. The data structure of the mapping table is: {timestamp: t, audio_index: i_audio, viseme_index: i_viseme, camera_index: i_camera}. For any time t, the corresponding data index is calculated using linear interpolation: i = i0 + (t - t0) / (t1 - t0) × (i1 - i0), where t0 and t1 are adjacent timestamps, and i0 and i1 are the corresponding data indices. Synchronization discrepancies between audio and lip-sync data are addressed by introducing time offset compensation. Lip-sync data typically needs to be 5-10 milliseconds ahead of audio, compensating for processing delays in lip-sync animation rendering. For camera movement, keyframe timestamps are aligned to the nearest audio frame boundary to ensure smooth scene transitions. A synchronization control signal sequence is generated, recorded in the format: {master_time: t, audio_offset: δt_audio, vision_offset: δt_viseme, camera_offset: δt_camera}. The rendering system calculates the playback position of each data stream in real time based on the master clock (master_time) and offset values, ensuring precise synchronization of audio playback, lip-sync changes, and camera movement.

[0064] In one possible implementation, after mapping the timestamps of the audio data, lip-sync data, and camera movement trajectory data to preset time axis coordinates and performing synchronization alignment, the process further includes steps 1051-1053, as follows: Step 1051: Perform pre-rendering based on the timestamps of the synchronized audio data, lip-sync data, and camera motion trajectory data; perform synchronization detection on the audio data and lip-sync data in the pre-rendered results, and calculate the time deviation between the pronunciation time of the audio data and the switching time of the lip-sync data.

[0065] Pre-rendering is a test rendering process before the official rendering. The pronunciation moment of audio data is the starting point of speech in the sound waveform. The switching moment of lip-sync data is the point in time when the lip shape changes. Timing deviation is the temporal difference between audio and lip-sync animation. Synchronization detection is the analytical process that evaluates the degree of audio-video synchronization. A timestamp is a time marker in a data sequence.

[0066] Specifically, first, a time axis buffer is constructed with a time precision of 1 millisecond. Waveform analysis is performed on the audio data: a short-time Fourier transform (STFT) is used to calculate the spectrogram with a window size of 512 samples and an overlap rate of 50%. The start point of a speech segment is detected: the pronunciation time is identified based on the energy curve E(t) = ∑|X(f,t)|² and the zero-crossing rate ZCR(t). Speech detection thresholds are set: energy threshold E_threshold = mean(E) + 2×std(E), and zero-crossing rate threshold ZCR_threshold = 100Hz. Keyframe analysis is performed on the lip-sync data: changes in lip-sync parameters are extracted, and the parameter derivatives dp / dt are calculated. A switching moment is marked when |dp / dt|>0.1. An audio-lip-sync correspondence table is established: {audio_time, viseme_time, phoneme_type}. The time deviation is calculated: for each phoneme, the deviation value δt = viseme_time - audio_time. Statistical deviation distribution: Calculate the mean (δt) and standard deviation (std(δt)) and generate a deviation histogram.

[0067] Step 1052: When the time deviation is greater than the preset threshold, extract the target phoneme sequence corresponding to the time interval in which the deviation occurs; calculate the target lip-sync switching time point based on the target phoneme sequence, and update the timestamp sequence of the lip-sync data according to the target lip-sync switching time point.

[0068] Time deviation is the time difference between audio and lip-sync switching. The preset threshold is the maximum allowable time deviation. The target phoneme sequence is the phoneme segment that needs correction. The target lip-sync switching time point is the corrected lip-sync change moment. The time interval is the data segment where synchronization problems occur. The timestamp sequence is the temporal information of the lip-sync data.

[0069] Specifically, the time deviation correction first sets the synchronization threshold to threshold = 50 milliseconds. When |δt|>threshold is detected, the time interval [t_start, t_end] is marked. The phoneme sequence phonemes[i] and its duration duration[i] of this interval are extracted. The lip-sync switching time is recalculated: for each phoneme i, the start time t_start[i] = t_start[i-1] + duration[i-1], and the end time t_end[i] = t_start[i] + duration[i]. The transition interval is generated: transition_time = min(duration[i] × 0.2, 30ms). The keyframe timestamps of the lip-sync parameters are updated: key_frame_time = t_start[i] + transition_time. The lip-sync interpolation parameters are recalculated: cubic Hermite interpolation is used, and control points P0 and P1 are set at the boundary of the transition interval. The timestamp sequence of the lip-sync data is updated: lip-sync parameters are sampled at 33.33ms intervals to generate a new keyframe sequence.

[0070] Step 1053: Remap the updated lip shape data to the preset time axis coordinates to generate corrected synchronization alignment data.

[0071] The default time axis coordinates use a unified time reference system. The corrected, synchronized data is a calibrated multimodal data sequence. Mapping is the process of aligning data to the time reference system. Timestamps are time markers for data points.

[0072] Specifically, the timeline remapping first establishes a unified time base: using the video frame rate (30fps) as the basic time unit. The updated lip-sync data is quantized by frame time: frame_time = floor(timestamp / 33.33) × 33.33. A keyframe index table is constructed: frame_index = {timestamp, param_index}. For each time point t, the two nearest keyframes k1 and k2 are retrieved. The interpolation coefficient α = (t - t1) / (t2 - t1) is calculated. The intermediate frame parameters are calculated using linear interpolation: param(t) = param(t1) + α × (param(t2) - param(t1)). A new data structure is generated: aligned_data = {timestamp, audio_data, vision_data, camera_data}. Synchronization accuracy is verified: the time deviation is recalculated, ensuring all deviation values ​​are less than a threshold. The corrected synchronization data sequence is output for subsequent rendering processing.

[0073] Step 106: Based on the synchronized timestamps, render and fuse the camera motion trajectory data, lip movement data, and audio data frame by frame to output a digital human video file.

[0074] The synchronized timestamps are timing control information on a unified timeline, recording the precise playback time points of each data stream. Lens motion data includes camera position, orientation, and field-of-view parameters. Lip-sync data records the time series of lip-sync deformation parameters. Audio data is PCM format sound waveform data. Frame-by-frame rendering is the process of generating each frame sequentially according to the video frame rate. Render blending combines the 3D scene, digital human model, and special effects into the final image. The digital human video file is a multimedia container file containing both video and audio streams.

[0075] Specifically, first, the output video specifications are determined: resolution is set to 1920×1080 pixels, frame rate is set to 30fps, video encoding is H.264 / AVC, and audio encoding is AAC. The time interval per frame is calculated based on the frame rate: Δt = 1000 / 30 ≈ 33.33 milliseconds. For each time step t, the following rendering steps are performed: The current camera state {position(x, y, z), rotation(pitch, yaw, roll), fov} is read from the camera motion trajectory data, and the view matrix and projection matrix are constructed. The lip-sync parameters {upper_lip_height, lower_lip_height, lip_corner_distance, jaw_rotation} are read from the lip-sync data, and the lip-sync deformation of the digital human model is updated through the vertex shader. Material and lighting parameters are applied, including diffuse maps, normal maps, and specular maps. The ambient light intensity {ambient_intensity} and the main light source parameters {light_direction, light_color, light_intensity} are set. Perform shadow mapping calculations to generate a depth map (depth_map) for real-time shadow rendering. Perform frustum culling on all objects in the scene to remove objects outside the field of view. Using deferred rendering, first render the geometry information to a G-Buffer: position_buffer, normal_buffer, and material_buffer. During the lighting stage, calculate the final color of each pixel using the G-Buffer. Apply post-processing effects: global illumination (SSAO), depth of field (DOF), and dynamic exposure adjustment. Write the rendering results to a frame buffer (frame_buffer). Read the audio sampling points for the current time period [t, t+Δt] from the audio data. Write the video frames and audio data to the encoder. The video frames are encoded using H.264 with the following parameters: bitrate 8Mbps, keyframe interval 60 frames, and B-frames 2. The audio data is encoded using AAC with the following parameters: bitrate 192kbps and sampling rate 48kHz. Use a video container to write the encoded video and audio streams to an MP4 container file. Maintain timestamp information during encoding to ensure synchronization of video frames and audio data. In the final output digital human video file, the video stream and audio stream are precisely synchronized based on PTS (PresentationTimeStamp).

[0076] In one possible implementation, based on the synchronized and aligned timestamps, the camera motion trajectory data, lip-sync data, and audio data are rendered and fused frame by frame to output a digital human video file. Specifically, this includes steps 1061-1064, as follows: Step 1061: Traverse frame by frame according to the time order of the preset time axis coordinates, obtain the corresponding lens focal length and position parameters from the lens motion trajectory data based on the timestamp of the current frame, and obtain the corresponding lip shape deformation parameters from the lip shape data.

[0077] The preset timeline coordinates use a unified time reference system measured in milliseconds. The current frame is the video frame being processed during the rendering process. The lens focal length parameter controls the field of view, determining the visual range of the image. The position parameter includes the camera's position coordinates and orientation angle in 3D space. The lip shape deformation parameter describes the deformation state of the lip shape mesh vertices, including the displacement of each control point. Frame-by-frame traversal is the process of processing each frame sequentially in chronological order. The corresponding parameter is the precise parameter value calculated through interpolation at the current timestamp.

[0078] Specifically, the frame interval Δt = 33.33 milliseconds is calculated based on the video frame rate (30fps). The timestamp corresponding to the current frame number frame_index is calculated as t_current = frame_index × Δt milliseconds. For camera motion data, the current time interval [t0, t1] is located in the timestamp array using a binary search. Camera parameters at the interval endpoints are extracted: position0(x0, y0, z0) and position1(x1, y1, z1) represent positions, rotation0(pitch0, yaw0, roll0) and rotation1(pitch1, yaw1, roll1) represent orientations, and fov0 and fov1 represent field of view angles. The time interpolation coefficient α = (t_current - t0) / (t1 - t0) is calculated. Linear interpolation is performed on the position coordinates: position = position0 + α(position1 - position0). Spherical linear interpolation (SLERP) is used on the rotation angle: rotation = slerp(rotation0, rotation1, α). Linear interpolation is performed on the field of view: fov = fov0 + α(fov1 - fov0). Similarly, parametric interpolation is performed on the lip-sync data. The lip-sync keyframe interval [t_v0, t_v1] corresponding to the current time is located. The lip-sync parameters at the interval endpoints are extracted: params0{upper_lip_height0, lower_lip_height0, lip_corner_distance0, jaw_rotation0} and params1{upper_lip_height1, lower_lip_height1, lip_corner_distance1, jaw_rotation1}. The temporal interpolation coefficient β = (t_current - t_v0) / (t_v1 - t_v0). Linear interpolation is performed on all lip-sync parameters: param = param0 + β(param1 - param0). The interpolation calculation ensures the smoothness of parameter changes and avoids abrupt changes between keyframes. The calculated camera and lip-sync parameters are cached in the rendering context of the current frame for use in subsequent rendering processes.

[0079] Step 1062: Update the focal length and spatial position of the virtual camera according to the lens focal length and position parameters to adjust the perspective and depth of field of the 3D virtual scene; drive the facial bone nodes of the target digital human model according to the lip shape deformation parameters to complete the lip shape deformation.

[0080] Lens focal length is an optical parameter of the virtual camera that controls the size of the field of view. Position parameters define the camera's position and orientation in the world coordinate system. The virtual camera is the viewpoint in the 3D scene, determining the scene's viewing angle. Depth of field is the range of sharp images before and after the focal point, related to focal length and aperture value. Facial bone nodes are the skeletal hierarchy that controls the deformation of the digital human's face, containing multiple independently controllable joints. Lip deformation is the facial mesh deformation process achieved by adjusting bone nodes. The target digital human model is a collection of mesh and skeletal data for a 3D human character.

[0081] Specifically, the camera position is set according to the position parameters position(x, y, z), and a rotation matrix R = Rz(roll) × Rx(pitch) × Ry(yaw) is constructed according to the orientation parameters rotation(pitch, yaw, roll). These are combined to obtain the camera view matrix View = R × Translate(-x, -y, -z). A projection matrix is ​​constructed based on the focal length parameter fov, with the field of view θ = 2 × arctan(sensor_height / (2 × focal_length)). The perspective projection matrix Projection = perspective(θ, aspect_ratio, near_plane, far_plane) is used. Depth-of-field parameters are set: focal distance = ||target_position - camera_position||, and depth range = 2 × focal_distance × f_number / (focal_length × circle_of_confusion). Lip shape distortion is implemented starting from the facial skeletal system. The facial skeleton includes: upper lip bone (upper_lip_bone), lower lip bone (lower_lip_bone), lip corner bones (lip_corner_bone_left / right), and jaw bone (jaw_bone). Each bone defines a local coordinate system and parent-child relationships. Bone transformations are set according to lip shape parameters: upper lip height controls the Y-axis displacement of upper_lip_bone, with a parameter range of [0, 1] mapped to a displacement range of [0, max_lip_height]. Lower lip height controls the Y-axis displacement of lower_lip_bone and the rotation angle of jaw bone, with a displacement range of [0, max_lip_height] and a rotation range of [-30°, 0°]. Lip corner distance controls the X-axis displacement of lip corner bone, with a range of [0, max_corner_distance]. The global transformation matrix of the bones is calculated as: Global = Parent_Global × Local, where Local = Translate × Rotate × Scale. Vertex deformation is calculated using LinearBlendSkinning: v' = ∑(w_i × M_i × v), where w_i is the bone weight, M_i is the bone transformation matrix, and v is the initial vertex position. After applying the deformation, the mesh vertex buffer is updated, and vertex normals and tangents are recalculated. Finally, the mesh bounding box is updated for frustum culling and shadow calculation.

[0082] Step 1063: Render the target digital human model and the 3D virtual scene after the lip-shape deformation is completed, and generate the image frame data of the current frame.

[0083] Lip-sync distortion is the result of facial mesh deformation based on skeletal animation. The target digital human model includes a geometric mesh, material maps, and a skeletal system. The 3D virtual scene includes an environment model, lighting, and special effects. Rendering is the process of converting a 3D scene into a 2D image. Image frame data is the pixel data of a complete rendered frame. The current frame is the image corresponding to the current point in time in the rendering sequence.

[0084] Specifically, the geometry processing phase begins with rendering the G-Buffer: a position buffer (RGB32F format) stores world-space position, a normal buffer (RGB16F format) stores surface normals, and a material buffer (RGBA8 format) stores diffuse color and roughness. Shadow mapping is then performed, rendering a depth map (R32F format) from the light source's perspective. In the lighting calculation phase, for each pixel, the following calculations are performed: ambient lighting (ambient = ambient_color × ambient_intensity), diffuse lighting (diffuse = max(dot(N,L),0) × light_color × diffuse_color), and specular lighting (specular = pow(max(dot(R,V),0), shininess) × light_color × specular_color). Physically based rendering (PBR) is then calculated: using a metallic-roughness workflow, the BRDF is calculated based on the material's metallicity and roughness. Ambient Occlusion (SSAO) is applied: the occlusion factor is calculated using the sampling depth buffer as ∑(sample_weight × step(sample_depth, current_depth)). Depth of Field (DOF) effect: the blur radius is calculated based on the pixel depth value as blur_radius = abs(pixel_depth - focus_distance) / focal_range. Color grading is performed: a Look-Up Table (LUT) is applied to adjust the colors. Finally, anti-aliasing (FXAA) is performed: pixel edge patterns are analyzed, and high-contrast edges are smoothed. The output is a 1920×1080 RGB image with 32 bits of color depth per pixel.

[0085] Step 1064: Encode all image frame data in chronological order, and encapsulate them with audio data to output a digital human video file.

[0086] Image frame data consists of the raw RGB pixel data for each frame, stored at a resolution of 1920×1080. The temporal order is based on frame indices along a preset timeline. Video encoding is the process of converting the image sequence into a compressed bitstream, employing inter-frame prediction and entropy coding techniques. Audio data is a 16-bit PCM format waveform sampling sequence. Audio / video encapsulation is the process of combining the encoded video and audio streams into a single container file. A digital human video file is a multimedia file conforming to a specific container format specification. Encoding parameters include compression configurations such as bitrate, GOP structure, and prediction mode.

[0087] Specifically, first, initialize the H.264 encoder, setting the encoding parameters: Profile=High, Level=4.2, and a fixed encoding bitrate of 8Mbps. Set the color space parameters: YUV420P format, chroma half-sampling 4:2:0. Configure the GOP (Group of Pictures) structure: keyframe interval set to 60 frames, number of B-frames 2, reference frame buffer size 4. Motion estimation parameters: search range ±32 pixels, minimum block size 4×4, block partitioning modes support 16×16 to 4×4. Enable adaptive intra-frame prediction: use 9 directional prediction modes for I-frames, selecting the optimal mode based on rate-distortion optimization. Perform motion estimation for P-frames and B-frames, supporting multi-reference frame prediction, with motion vector precision set to 1 / 4 pixel. Transform coding uses integer DCT: luma components support 8×8 and 4×4 transforms, chroma components use 4×4 transforms. Entropy coding adopts CABAC, adaptively updating the context probability model. Audio encoding uses an AAC-LC encoder: 48kHz sampling rate, dual-channel, 192kbps bitrate, and 1024 sample points per frame. Audio and video encapsulation uses the MP4 container format: two media tracks are created, with a video track timescale of 30000 and an audio track timescale of 48000. An interleaving storage strategy is employed: every 2 seconds of audio and video data forms an interleaving unit. Synchronization timestamps are set: video PTS is calculated based on frame number × frame_duration, and audio PTS is calculated based on sample point number × sample_duration. MP4 metadata is written: encoder information, resolution, frame rate, sampling parameters, and duration information. The file layout is optimized using MOOV: metadata blocks are moved to the beginning of the file for faster indexing. The final generated MP4 file has the following characteristics: supports drag-and-drop playback, supports streaming, and audio / video synchronization error is controlled within ±1 frame. The output file is named "output_timestamp.mp4", where timestamp is the generation timestamp.

[0088] In the above embodiments, basic digital human scene fusion functionality was achieved through virtual camera parameter extraction and spatial coordinate transformation. To further enhance the realism and interactivity of scene synthesis, this application also provides a digital human scene fusion method. This method analyzes scene depth information, camera perspective parameters, and motion trajectory features to construct a unified spatial transformation framework and performs real-time focal length adjustment, enabling the system to more naturally handle digital human fusion needs in complex scene environments. The following section combines... Figure 2 Another method for generating video by fusing digital humans with 3D scenes in the embodiments of this application is described below: Please see Figure 2 This is a flowchart illustrating a method for generating video by fusing digital humans with 3D scenes in an embodiment of this application.

[0089] Step 201: Extract virtual camera parameters from the 3D virtual scene data. The virtual camera parameters include the view matrix and the projection matrix.

[0090] 3D virtual scene data is a dataset containing scene geometry, materials, and camera information. Virtual camera parameters describe the position and projection characteristics of the viewpoint. The view matrix defines the camera's position and orientation transformation. The projection matrix defines the projection transformation from camera space to clip space. Camera parameters control how the scene is viewed and its visual effects. Extraction is the process of obtaining specific parameters from the scene data.

[0091] Specifically, obtain the camera position vector (position(x, y, z) and orientation vector (rotation(pitch, yaw, roll)). Calculate the camera's forward vector (front), up vector (up), and right vector (right): front = normalize(direction), right = normalize(cross(front, world_up)), up = normalize(cross(right, front)). Construct the view matrix View = [right.xright.yright.z-dot(right, eye); up.xup.yup.z-dot(up, eye); -front.x-front.y-front.z-dot(front, eye); 0001]. Obtain the projection parameters: field of view fov = 60°, aspect ratio = width / height, near plane near = 0.1, far plane far = 1000.0. Construct the projection matrix: For perspective projection, f = 1 / tan(fov / 2), Projection = [f / aspect000; 0f00; 00(far+near) / (near-far)2farnear / (near-far); 00-10]. Store the view matrix and projection matrix as 4×4 floating-point matrices for subsequent coordinate transformation calculations.

[0092] Step 202: Calculate the perspective parameters based on the viewpoint matrix and projection matrix, and convert the two-dimensional coordinates of the image data into three-dimensional spatial coordinates based on the perspective parameters; calculate the axis coordinates of the target image data in the three-dimensional virtual scene based on the scene depth information of the three-dimensional virtual scene data.

[0093] Perspective parameters control the projection transformation from 3D to 2D. 2D coordinates are pixel coordinates in screen space. 3D coordinates are position coordinates in world space. Scene depth information records the distance from each point in the scene to the camera. Axis coordinates are the position components of an object in 3D space. Target image data is the character model data to be placed in the scene.

[0094] Specifically, first, the perspective transformation parameters are calculated. Parameters are extracted from the projection matrix: fov_y = 2 * atan(1 / Projection[1][1]), aspect = Projection[0][0] / Projection[1][1]. The perspective transformation equation is constructed: x_world = (2x_screen / width-1) * z * tan(fov_x / 2) * aspect, y_world = (1-2y_screen / height) * z * tan(fov_y / 2), where z is the depth value. The coordinate transformation from two dimensions to three dimensions is performed: for the input screen coordinates (x_screen, y_screen), the depth value of the point is first obtained from the depth buffer. The world space coordinates are calculated: x_world and y_world are calculated using the perspective transformation equation, z_world = depth. The transformation from view space to world space is performed by applying the inverse of the view matrix: world_pos = inverse(View) * vec4(x_world, y_world, z_world, 1.0). Scene Depth Calculation: Perform depth rendering on the scene to generate a depth map (depth_map). Obtain the depth value of the target location from the depth map: depth = sample_depth_map(x_screen, y_screen). Calculate Axis Coordinates: Based on the world space position (world_pos), extract the x, y, and z components as axis coordinates. Verify Coordinate Validity: Check if the depth value lies between the near and far planes to ensure the transformed coordinates are within the scene bounding box. Output the final 3D coordinates and axial components for subsequent scene compositing.

[0095] Step 203: Determine the fusion position coordinates based on the three-dimensional spatial coordinates and axis coordinates, and embed the target image data into the corresponding camera position of the three-dimensional virtual scene data.

[0096] Three-dimensional spatial coordinates represent the position of the target image in world space. Axis coordinates are the components of the position along each coordinate axis. The merged position coordinates determine the final placement location. Target image data is the character model data to be embedded. Camera position is the virtual camera's observation position within the scene. Embedding is the process of integrating the image data into the scene. 3D virtual scene data includes scene elements such as environment and lighting.

[0097] Specifically, check if the 3D spatial coordinates (x, y, z) are within the scene boundary: boundary_check = (min_bound ≤ position ≤ max_bound). Fine-tune the position using axis coordinates: apply an offset vector offset = (δx, δy, δz) to ensure the image doesn't intersect with scene objects. Calculate the merged position: final_position = position + offset. Perform collision detection: construct the image's bounding box and check for intersections with scene objects. Use an octree to accelerate collision detection: divide the scene space into 8³ = 512 subspaces and only detect objects in relevant subspaces. Calculate the contact surface normal: contact_normal = normalize(cross(edge1, edge2)). Adjust the image rotation: rotation = quaternion_from_vectors(up_vector, contact_normal). Update the image transformation matrix: transform = translate(final_position) × rotate(rotation) × scale(1, 1, 1). Configure image rendering properties: Enable depth testing, set depth_bias=0.001 to avoid z-fighting. Configure shadow casting: Update shadow map, calculate light_space_matrix.

[0098] Step 204: Obtain the lens motion parameters of the lens position, including motion duration, starting focal length, and ending focal length; determine the total number of frames of lens motion based on the motion duration, and calculate the focal length change based on the starting focal length and ending focal length.

[0099] Camera motion parameters describe the characteristics of camera movement. Motion duration is the duration of the animation. Initial focal length is the initial focal length value. Ending focal length is the final focal length value. Total frames are the number of frames in the animation. Focal length change is the total change in focal length. Camera position defines the viewpoint's location within the scene.

[0100] Specifically, based on a frame rate of frame_rate=30fps, the total number of frames is calculated as total_frames = duration × frame_rate. Focal length parameters are read: starting focal_start (in millimeters), ending focal_end (in millimeters). The focal length change is calculated as: focal_delta = focal_end - focal_start. The focal length is converted to the field of view: fov_start = 2 × arctan(sensor_height / (2 × focal_start)), fov_end = 2 × arctan(sensor_height / (2 × focal_end)). The focal length increment per frame is calculated as: focal_step = focal_delta / total_frames. The timeline array is constructed as: time_array = linspace(0, duration, total_frames). The keyframe data structure is created as: keyframe = {frame_index, position, rotation, focal_length}. Set motion boundary conditions: initial state {t0, p0, r0, f0} and ending state {t1, p1, r1, f1}. Generate a frame index table: frame_table = {frame_index, timestamp, parameters}.

[0101] Step 205: Based on the total number of frames and the change in focal length, calculate the focal length value corresponding to each frame using a preset interpolation algorithm to generate lens motion trajectory data.

[0102] Total frames refers to the number of frames in an animation sequence. Focal length change is the magnitude of the change in focal length. Preset interpolation algorithms are mathematical methods used to calculate intermediate values. Focal length value is the lens focal length for each frame. Lens motion trajectory data records the camera's movement path. A frame is a single shot in an animation sequence. Interpolation is the process of calculating intermediate states.

[0103] Specifically, trajectory generation employs a cubic spline interpolation algorithm. The time series is created as follows: t = linspace(0, total_frames-1, total_frames). Focal control points are constructed: control_points = {(0, focal_start), (total_frames / 3, focal_1), (2×total_frames / 3, focal_2), (total_frames-1, focal_end)}. Spline coefficients are calculated for each interval [tᵢ, t...]. ᵢ₊1Solve the system of equations to obtain the coefficient matrix `coefficient_matrix`. The spline function is defined as: `focal(t) = aᵢ(t-tᵢ)³ + bᵢ(t-tᵢ)² + cᵢ(t-tᵢ) + dᵢ`. For each frame, perform interpolation: `frame_focal[i] = focal(i)`. Apply smoothness constraints: ensure the first and second derivatives of adjacent intervals are continuous. Velocity control: calculate the focal length change rate `v(t) = df / dt`, limiting the maximum rate `|v(t)| ≤ v_max`. Acceleration control: calculate the focal length change acceleration `a(t) = d²f / dt²`, limiting the maximum acceleration `|a(t)| ≤ a_max`. Generate the trajectory data structure: `trajectory = {frame_index, position(x, y, z), rotation(pitch, yaw, roll), focal_length}`. Output trajectory file: Saved as JSON format, containing parameters such as frame index, position, rotation, and focal length.

[0104] In one possible implementation, after generating the lens motion trajectory data, the process further includes steps 2051-2053, as follows: Step 2051: Obtain large screen material data, which includes image data or video data; extract the position and size parameters of the virtual large screen display area from the target 3D virtual scene data.

[0105] Large-screen content data refers to the media content used for virtual large-screen display, including image or video formats. Image data is a static pixel matrix. Video data is a continuous sequence of images. Target 3D virtual scene data is a complete 3D environment description. The virtual large-screen display area is the planar area in the scene used to display content. Position parameters define the spatial location of the display area. Size parameters define the size of the display area.

[0106] Specifically, for image formats, read the resolution (resolution(width, height)) and color channels. For video formats, additionally read the frame rate (fps) and total duration. Check the integrity of the footage: verify the file header (header_check) and data checksum. Extract display area parameters from the scene data: plane center point (position(x, y, z), plane normal vector (normal(nx, ny, nz), plane dimensions (dimensions(width, height)). Calculate the transformation matrices of the display area: translation matrix T = translate(position), rotation matrix R = rotation_matrix(normal, up_vector), scaling matrix S = scale(width, height, 1). Construct the bounding box of the display area: bounds = {min_point, max_point}. Generate UV coordinate mapping: uv_coords = generate_planar_mapping(bounds). Set the rendering properties of the display area: double-sided display (double_sided = true), self-illuminating material (emissive = true). Output parameter data structure: screen_params={position, normal, dimensions, transform, uv_mapping}.

[0107] Step 2052: Scale and transform the large-screen material data according to the position and size parameters to generate adapted large-screen material data; map the adapted large-screen material data as a texture to the virtual large-screen display area.

[0108] Position parameters define the display position. Size parameters define the display size. Large-screen content data is the displayed content. Scaling is the process of adjusting the content size. Position transformation is the process of adjusting the content position. Adapted large-screen content is the processed display content. Texture mapping is a technique that applies two-dimensional images to three-dimensional surfaces. The virtual large-screen display area is the display plane.

[0109] Specifically, first, calculate the scaling ratio: scale_x = display_width / content_width, scale_y = display_height / content_height. Select the scaling mode: maintain the aspect ratio, aspect_ratio = min(scale_x, scale_y). Perform image resampling: use bicubic interpolation, kernel_size = 4×4. Process the video frame sequence: apply the same scaling transformation to each frame. Calculate the position offset: offset_x = (display_width - scaled_width) / 2, offset_y = (display_height - scaled_height) / 2. Generate the transformation matrix: transform = translate(offset) × scale(aspect_ratio). Create a texture object: set the texture parameters format = RGBA8, wrap_mode = clamp_to_edge, filter_mode = linear. For video textures: create a frame buffer, frame_buffer, with a size of scaled_dimensions. Upload texture data: use pixel buffer (PBO) to optimize transmission. Configure UV coordinates: Calculate the texture coordinate range tex_coords = {u_min, v_min, u_max, v_max}. Apply texture mapping: Set the texture matrix texture_matrix = scale(tex_scale) × translate(tex_offset).

[0110] Step 2053: Obtain scene prop data, which includes 3D model data and material data; determine the placement coordinates based on the position coordinates of the target digital human image data, and load the scene prop data to the placement coordinates of the target 3D virtual scene data.

[0111] Scene prop data refers to the data of decorative objects in the scene. 3D model data describes the geometry of objects. Material data defines the surface properties of objects. Target digitized human character data is the character model information. Position coordinates are the spatial position of the character. Placement coordinates are the target position of the prop. Target 3D virtual scene data is the environmental scene information.

[0112] Specifically, the process begins by parsing the prop data: reading the model format (model_format) and mesh data (mesh_data). It then loads the geometry data: vertex array (vertex_array), index array (index_array), UV coordinates (uv_array), and normal array (normal_array). Next, it processes the material data: diffuse map (diffuse_map), normal map (normal_map), and roughness map (roughness_map). Placement parameters are extracted from the character's position: base_point = character_position, direction vector = character_forward. The prop placement position is calculated: prop_position = base_point + offset_vector. Position constraints are applied: collision check (collision_check) and ground alignment (ground_alignment). The prop transformation is set: transform = translate(prop_position) × rotate(prop_rotation) × scale(prop_scale). Rendering properties are configured: cast_shadows = true, receive_shadows = true. Finally, physical properties are initialized: a collision shape is created, and physical parameters (physical_params) are set. Integrate prop data into the scene: Update the scene graph (scene_graph) and establish node hierarchies. Output scene update data: scene_update={new_objects, transform_updates, material_updates}.

[0113] In one optional embodiment, when a real-life digital human is a 2D digital human, the method for generating video by fusing the digital human with a 3D scene further includes spatial fusion and lip-syncing processing tailored to 2D characteristics, specifically including the following steps: Step A: Acquire 2D digital human image data and establish a planar mapping relationship with the 3D virtual scene.

[0114] 2D digital human avatar data is typically a video sequence or image sequence with an alpha channel. Unlike the mesh model of a 3D digital human, a 2D digital human is essentially a two-dimensional texture plane. After acquiring the data, a quadrilateral mesh is first created in the 3D virtual scene as the carrier plane, and the initial image of the 2D digital human is mapped onto this quadrilateral mesh as a texture map. According to the preset resolution mapping relationship, the aspect ratio of the quadrilateral mesh is adjusted to match the original pixel ratio of the 2D digital human. For example, if the resolution of the 2D digital human is 1080x1920, the aspect ratio of the quadrilateral mesh is set to 9:16.

[0115] Step B: Perform dynamic spatial redirection (Billboard processing) on ​​the 2D digital human based on eye tracking, according to the camera motion trajectory data.

[0116] This is the core difference between 2D and 3D digital humans in terms of spatial positioning algorithms. 3D digital humans have volume and depth, and naturally present their side or back when the camera rotates; while 2D digital humans, if not processed, will appear "paper-thin" or invisible when the camera moves to the side. Therefore, it is necessary to calculate the facing angle of 2D digital humans in real time.

[0117] Specifically, for each frame in the camera's motion trajectory, the world coordinate position P of the virtual camera is obtained. camera (x c ,y c ,z c ) and the center coordinates P of the 2D digital human bearing plane human (x h ,y h ,z h ).

[0118] Calculate the gaze vector V from the digital human to the camera. view V view =P camera -P human。

[0119] Normalize the gaze vector to obtain N view Construct the rotation matrix R of the 2D digital human. billboard The normal vector of the bearing plane is such that the normal vector N of the bearing plane is such that... plane Always with the line of sight vector N view Parallel. The constrained-axis Billboard algorithm is typically used, locking the Y-axis (vertical axis) and calculating the rotation angle only in the XZ plane. The formula is: θ = arctan2(x... c -xh ,z c -z h The calculated rotation matrix is ​​applied to the quadrilateral grid to ensure that the 2D digital human always faces the camera during camera switching, movement, or rotation, maintaining visual stereoscopic effect and interactivity.

[0120] Step C: Generate 2D digital morphological data based on a deep learning model using audio features and facial data.

[0121] This is the core difference between 2D and 3D digital humans in terms of lip-syncing algorithms. 3D algorithms drive skeletal parameters, while 2D algorithms generate or deform image pixels.

[0122] Specifically, this step includes: Facial data extraction and localization: First, perform facial landmark detection on the baseline image of the 2D digital human, extracting the coordinates of 68 or more landmarks. Construct a mask based on the landmark coordinates to accurately locate the lip-shape region (Region of Interest, ROI), and then normalize and crop this region to prepare for subsequent lip-shape inference.

[0123] Audio Feature Extraction: The input audio data is preprocessed, including noise reduction and pre-emphasis. Then, the core features of the audio are extracted using an acoustic model. Specifically, the audio stream is framed and windowed, subjected to Fast Fourier Transform (FFT), and Mel-Frequency Cepstral Coefficients (MFCCs) are calculated. The extracted feature vectors contain pitch, phoneme, and energy distribution information, serving as the acoustic basis for driving lip movements.

[0124] Real-time Inference and Generation: An end-to-end inference model based on Generative Adversarial Networks (GANs) or Convolutional Neural Networks (CNNs) is constructed. The extracted MFCC audio feature sequences and preprocessed facial data (including baseline lip-shape images and facial contour features) are used as model inputs. The model maps the audio features to a latent space through an encoder, and then, through a decoder, combines facial context information to generate corresponding lip-shape region image data frame by frame.

[0125] Image fusion: The lip-sync data obtained for each frame is not skeletal parameters, but rather pixel-level image patches. The inferred lip-sync images are then fitted back to their original facial positions in real-time using Poisson blending or multi-resolution spline fusion techniques, resulting in complete 2D digital human image data corresponding to each frame of audio. This model undergoes lightweight pruning and quantization processing, enabling low-latency real-time inference for real-time interactive effects.

[0126] Step D: The generated 2D digital human image sequence is used as a dynamic texture and updated frame by frame to the carrying plane in the 3D scene, and then rendered with lighting fusion with the scene.

[0127] In the rendering stage, unlike the geometric rendering of 3D models, the rendering of 2D digital humans involves updating textures. The image sequence with matching lip movements generated in step C is updated frame by frame to the quadrilateral mesh that has undergone spatial relocation in step B, according to the synchronized and aligned timestamps. At the same time, based on the lighting data of the 3D scene, the ambient light intensity received by the quadrilateral mesh is calculated, and the brightness and color temperature of the texture are adjusted to ensure that the 2D digital human and the 3D background are consistent in tone, ultimately outputting a blended video.

[0128] The following describes a video generation system for fusing digital humans and 3D scenes from the perspective of hardware processing in an embodiment of this invention. Please refer to [link / reference]. Figure 3 This is a schematic diagram of the structure of a video generation system that fuses a digital human with a 3D scene according to an embodiment of this application.

[0129] It should be noted that, Figure 3 The structure of the digital human and 3D scene fusion video generation system shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0130] like Figure 3 As shown, a video generation system that fuses a digital human with a 3D scene includes a Central Processing Unit (CPU) 301, which can perform various appropriate actions and processes based on a program stored in a Read-Only Memory (ROM) 302 or a program loaded from a storage section 308 into a Random Access Memory (RAM) 303, such as the method described in the above embodiment. The RAM 303 also stores various programs and data required for system operation. The CPU 301, ROM 302, and RAM 303 are interconnected via a bus 304. An Input / Output (I / O) interface 305 is also connected to the bus 304.

[0131] The following components are connected to I / O interface 305: input section 306 including audio input devices, push-button switches, etc.; output section 307 including a liquid crystal display (LCD) and audio output devices, indicator lights, etc.; storage section 308 including a hard disk, etc.; and communication section 309 including a network interface card such as a LAN (Local Area Network) card, modem, etc. Communication section 309 performs communication processing via a network such as the Internet. Drive 310 is also connected to I / O interface 305 as needed. Removable media 311, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 310 as needed so that computer programs read from them can be installed into storage section 308 as needed.

[0132] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing computer programs for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 309, and / or installed from removable medium 311. When the computer program is executed by central processing unit (CPU) 301, it performs the various functions defined in the present invention.

[0133] It should be noted that specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0134] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those shown in the drawings.

[0135] Specifically, the digital human and 3D scene fusion video generation system of this embodiment includes a processor and a memory. The memory stores a computer program. When the computer program is executed by the processor, it implements the digital human and 3D scene fusion video generation method provided in the above embodiment.

[0136] In another aspect, the present invention also provides a computer-readable storage medium, which may be included in a digital human and 3D scene fusion video generation system described in the above embodiments; or it may exist independently and not assembled into the digital human and 3D scene fusion video generation system. The storage medium carries one or more computer programs, which, when executed by a processor of the digital human and 3D scene fusion video generation system, cause the digital human and 3D scene fusion video generation system to implement the digital human and 3D scene fusion video generation method based on IoT data encryption transmission provided in the above embodiments.

Claims

1. A method for generating video by fusing digital humans with 3D scenes, characterized in that, The method includes: Acquire the image data and 3D virtual scene data of the target digital human; The size parameters corresponding to the image data are determined according to the preset resolution mapping relationship, the posture parameters are determined according to the preset broadcast scene type, and the image data is normalized based on the size parameters and the posture parameters to obtain the target image data. Calculate the fusion position coordinates based on the spatial coordinates of the image data and the perspective parameters of the three-dimensional virtual scene data, and spatially fuse the target image data and the three-dimensional virtual scene data according to the fusion position coordinates to generate camera motion trajectory data. The system receives input text, performs semantic parsing and pause recognition on the input text to generate audio data, extracts the phoneme sequence of the input text, and generates lip-sync data matching the target digital human based on the phoneme sequence and the timestamp information of the audio data. The timestamps of the audio data, the lip-sync data, and the camera motion trajectory data are mapped to preset time axis coordinates and synchronized. Based on the synchronized timestamps, the camera motion trajectory data, lip-sync data, and audio data are rendered and fused frame by frame to output a digital human video file.

2. The method according to claim 1, characterized in that, The process involves determining the size parameters corresponding to the image data based on a preset resolution mapping relationship, determining the posture parameters based on a preset broadcast scene type, and normalizing the image data based on the size parameters and the posture parameters to obtain the target image data, including: Extract the target output resolution from the image data, and find the corresponding scaling factor in the preset resolution mapping relationship based on the target output resolution; The height and width pixel values ​​of the image data are calculated based on the scaling factor to obtain the size parameters; Match the corresponding posture template according to the preset broadcast scene type. The posture template includes multiple skeletal parameters, which are standing skeletal parameters, sitting skeletal parameters, or dynamic skeletal parameters. The coordinates of the skeletal nodes in the image data are mapped to the skeletal parameters of the posture template to complete the posture transformation; The transformed image data is normalized so that the coordinate range of the image data is normalized to the preset rendering space coordinate system, thus obtaining the target image data.

3. The method according to claim 1, characterized in that, The step of calculating the fusion position coordinates based on the spatial coordinates of the image data and the perspective parameters of the 3D virtual scene data, and spatially fusing the target image data and the 3D virtual scene data according to the fusion position coordinates to generate camera motion trajectory data includes: Virtual camera parameters are extracted from the 3D virtual scene data, and the virtual camera parameters include a view matrix and a projection matrix; The perspective parameters are calculated based on the viewpoint matrix and the projection matrix, and the two-dimensional coordinates of the image data are converted into three-dimensional spatial coordinates based on the perspective parameters. Based on the scene depth information of the three-dimensional virtual scene data, the axis coordinates of the target image data in the three-dimensional virtual scene are calculated; The fusion position coordinates are determined based on the three-dimensional spatial coordinates and the axial coordinates, and the target image data is embedded into the camera position corresponding to the three-dimensional virtual scene data. Obtain lens motion parameters for the lens position, including motion duration, initial focal length, and final focal length; The total number of frames of lens movement is determined based on the movement duration, and the focal length change is calculated based on the initial focal length and the final focal length. Based on the total number of frames and the focal length change, the focal length value corresponding to each frame is calculated using a preset interpolation algorithm to generate the lens motion trajectory data.

4. The method according to claim 3, characterized in that, After spatially fusing the target image data and the 3D virtual scene data according to the fusion position coordinates, the method further includes: Acquire large-screen content data, which includes image data or video data; Extract the position and size parameters of the virtual large screen display area from the target 3D virtual scene data; The large-screen material data is scaled and repositioned according to the position parameters and the size parameters to generate adapted large-screen material data. The adapted large-screen material data is mapped as a texture onto the virtual large-screen display area; Acquire scene prop data, which includes 3D model data and material data; The placement coordinates are determined based on the position coordinates of the target digital human image data, and the scene prop data is loaded into the placement coordinates of the target three-dimensional virtual scene data.

5. The method according to claim 1, characterized in that, The process of semantically parsing and pausing recognition of the input text to generate audio data, extracting the phoneme sequence of the input text, and generating lip-sync data matching the target digital human based on the phoneme sequence and the timestamp information of the audio data includes: The input text is segmented into words, sentence boundaries are identified, and pause markers are inserted at the pause positions corresponding to the sentence boundaries to obtain the processed text information. The processed text information is converted into speech to generate the audio data and the timestamp information corresponding to the audio data. The input text is decomposed into phonemes, the phoneme sequence is extracted, and the corresponding lip shape deformation parameters are matched according to the phoneme sequence. Based on the timestamp information of the audio data and the lip shape deformation parameters, calculate the lip shape switching time point corresponding to each phoneme; The lip shape data is generated based on the lip shape switching time point, and the lip shape data includes a sequence of lip shape deformation parameters and a corresponding timestamp sequence.

6. The method according to claim 1, characterized in that, The step of rendering and fusing the camera motion trajectory data, lip-sync data, and audio data frame-by-frame based on the synchronized and aligned timestamps to output a digital human video file includes: The system iterates frame by frame according to the time sequence of the preset time axis coordinates, and obtains the corresponding lens focal length and position parameters from the lens motion trajectory data based on the timestamp of the current frame, and obtains the corresponding lip shape deformation parameters from the lip shape data. The focal length and spatial position of the virtual camera are updated based on the lens focal length and position parameters to adjust the viewpoint and depth of field of the three-dimensional virtual scene. The facial bone nodes of the target digital human model are driven according to the mouth shape deformation parameters to complete the mouth shape deformation; Render the target digital human model and the 3D virtual scene after the lip-shape deformation is completed, and generate the image frame data of the current frame; All image frame data are encoded in chronological order and then encapsulated with the audio data to output the digital human video file.

7. The method according to claim 1, characterized in that, After mapping the timestamps of the audio data, the lip-sync data, and the camera movement trajectory data to preset time axis coordinates and performing synchronization alignment, the process further includes: Pre-rendering is performed based on the timestamps of synchronized and aligned audio data, lip-sync data, and camera motion trajectory data. Synchronization detection is performed on the audio data and lip-sync data in the pre-rendered results, and the time deviation between the pronunciation time of the audio data and the switching time of the lip-sync data is calculated. When the time deviation is greater than a preset threshold, the target phoneme sequence corresponding to the time interval in which the deviation occurred is extracted. Calculate the target lip-sync switching time point based on the target phoneme sequence, and update the timestamp sequence of the lip-sync data according to the target lip-sync switching time point; The updated lip-sync data is remapped to the preset time axis coordinates to generate corrected synchronization alignment data.

8. A video generation system that fuses digital humans with 3D scenes, characterized in that, The digital human-3D scene fusion video generation system includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to cause the digital human-3D scene fusion video generation system to perform the method as described in any one of claims 1-7.

9. A computer-readable storage medium comprising instructions, characterized in that, When the instructions are executed on the video generation system that fuses digital humans with 3D scenes, the video generation system that fuses digital humans with 3D scenes performs the method as described in any one of claims 1-7.

10. A computer program product, characterized in that, When the computer program product is run on the video generation system that fuses digital humans with 3D scenes, the video generation system that fuses digital humans with 3D scenes performs the method as described in any one of claims 1-7.

Images