A fully intelligent, automated interaction method and system for virtual humans

Abstract

This application provides a fully intelligent, automated interaction method and system for virtual humans. The method includes: calling the corresponding live streaming process template from a template library according to live streaming task instructions; parsing the user-uploaded modeling file according to the live streaming process template to obtain the virtual human model's driver calibration file, and generating a live streaming execution plan by combining the configuration rules for the live streaming scene, camera movement, and special effects triggering; collecting the virtual human model's driver data according to the live streaming execution plan, synchronizing the driver data to the virtual human model, and generating a live audio and video stream by combining the virtual live streaming scene and audio data; pushing the live audio and video stream to the target live streaming platform, collecting audience interaction data from the target live streaming platform, and triggering a real-time response from the virtual human live stream by parsing the audience interaction data to complete the live stream. This application solves the problem of fragmented stages in virtual human live streaming and achieves a fully automated closed-loop process driven by virtual humans.

Description

Technical Field

[0001] This application relates to the field of live streaming technology, and in particular to a fully intelligent, automated interaction method and system for virtual humans. Background Technology

[0002] With the rapid development of virtual human technology and the live streaming industry, virtual human live streaming has been widely used in various scenarios such as e-commerce sales, entertainment live streaming, corporate presentations, online education, and government announcements. However, current virtual human live streaming solutions generally suffer from the following core technical defects and business pain points: In the prior art, Chinese invention patent application with publication number CN115208872A discloses a virtual human live streaming method and system. This solution realizes the basic functions of virtual human image driving and live streaming, but it only covers a single link of virtual human driving and cannot realize the whole process closed loop of "image creation - scene building - live streaming - interactive marketing - content production - data operation". Each link of live streaming belongs to multiple independent tools, and the data and process cannot be linked, resulting in a very low degree of automation. Summary of the Invention

[0003] This application provides a fully intelligent, automated interaction method and system for virtual humans, which addresses the problem that existing virtual human live streaming solutions in related technologies suffer from a lack of data and process linkage and extremely low automation.

[0004] The first aspect of this application provides a fully intelligent, automated interaction method for virtual humans, the fully intelligent, automated interaction method for virtual humans comprising: Based on the scene type identifier in the live streaming task instruction, the corresponding live streaming process template is retrieved from the template library; The modeling files uploaded by users are parsed according to the image creation rules in the live streaming process template to obtain the driving calibration file of the virtual human model. Combined with the configuration rules of live streaming scene, camera position, and special effects triggering, a live streaming execution plan is generated. According to the live streaming execution plan, the driving data of the virtual human model is collected, the driving data is synchronized to the virtual human model, and the live audio and video stream is generated by combining the virtual live streaming scene and audio data. The live audio and video stream is pushed to the target live streaming platform, and the audience interaction data of the target live streaming platform is collected. The virtual human live stream is triggered to respond in real time by parsing the audience interaction data, so as to complete the live stream.

[0005] Optionally, in the first implementation of the first aspect of this application, the step of parsing the user-uploaded modeling file according to the image creation rules in the live streaming process template to obtain the driving calibration file of the virtual human model includes: Based on the image creation rules in the live streaming process template, the skeletal hierarchy in the user-uploaded modeling file is traversed to obtain the parent node index and child node index of each skeletal node. Based on the connection relationship between the parent node index and the child node index, an initial virtual human model containing a skeletal topology linked list is generated; The initial position coordinates and initial rotation angle of each skeletal node in the initial virtual human model are zeroed out to generate a calibrated virtual human model in a standard standing posture. The node name of each skeletal node in the calibrated virtual human model is associated with its position coordinates and rotation angle under the standard standing posture and stored to generate a drive calibration file containing skeletal node identifiers and standard posture parameters.

[0006] Optionally, in the second implementation of the first aspect of this application, the steps for generating a live streaming execution plan, combining the configuration rules of the live streaming scenario, camera movement, and special effects triggering, include: According to the scene configuration rules in the live streaming process template, the mesh vertex coordinates and texture coordinates of the 3D scene file corresponding to the scene template identifier are read and sampled to generate a virtual live streaming scene. According to the camera movement rules in the live streaming process template, keyframe interpolation is performed on the position coordinates and viewing direction of the virtual camera on the time axis to obtain a camera switching sequence with multiple preset camera positions. Then, parameter values ​​are assigned to the transition duration between adjacent camera positions in the camera switching sequence to generate a camera movement scheduling file. Based on the special effects triggering rules in the live streaming process template, condition action pairing is performed on the type identifier of the triggering condition and the instruction code of the execution action to generate a special effects triggering rule table; By associating the virtual live streaming scene, the camera movement scheduling file, and the special effects triggering rule table, and establishing a mapping relationship between the associated data and the skeletal node identifiers in the driver calibration file, a live streaming execution plan is generated.

[0007] Optionally, in the third implementation of the first aspect of this application, the steps of collecting driving data of the virtual human model according to the live streaming execution scheme, synchronizing the driving data to the virtual human model, and generating a live audio and video stream by combining the virtual live streaming scene and audio data include: The original bone data of the current frame is obtained based on the bone node identifier. Optimized bone data is generated by smoothing the original bone data and historical bone data. By mapping the optimized skeletal data with the standard posture parameters of the driving calibration file, the target position data of each skeletal segment is obtained, and the target position data is synchronized to the corresponding skeletal nodes in the virtual human model to generate the driven virtual human model. Audio features are extracted from the audio frame data in the input audio stream to obtain the lip-sync driving parameters corresponding to the current audio frame. The lip-sync driving parameters are then synchronized to the face control node in the driven virtual human model to generate a lip-sync synchronized virtual human model. By rendering the lip-synced virtual human model and scene elements in the virtual live streaming scene frame by frame, the image data of the current frame is obtained. The image data and the corresponding audio frame data are then time-stamp aligned to generate a live audio and video stream.

[0008] Optionally, in the fourth implementation of the first aspect of this application, the steps of pushing the live audio and video stream to the target live streaming platform, collecting audience interaction data from the target live streaming platform, and triggering a real-time response from the virtual human live stream by parsing the audience interaction data include: According to the platform address list in the live streaming execution scheme, the live audio and video streams are simultaneously encapsulated into transmission protocol packets adapted to multiple target platforms, and the encapsulated transmission protocol packets are pushed to the corresponding streaming platform addresses respectively. The system collects bullet screen text and reward records in real time from the interactive data interfaces of various target live streaming platforms. By performing keyword hierarchical matching on the bullet screen text, it generates trigger keywords and their corresponding priority levels. Based on the comparison results of the reward amount in the reward record with a preset threshold, it generates a reward trigger identifier. Based on the trigger keyword, the priority level, and the reward trigger identifier, the corresponding execution action instruction is called from the special effects trigger rule table in the live streaming execution plan, and the corresponding lip-sync driving parameters are retrieved from the driver calibration file according to the execution action instruction to generate trigger response data; Based on the execution action instructions in the trigger response data, data is injected into the audio track, video track of the live audio and video stream or the bullet screen sending interface of the target live streaming platform to generate an updated live audio and video stream or bullet screen reply message.

[0009] Optionally, in the fifth implementation of the first aspect of this application, the method further includes: Based on the live streaming topic and live streaming duration parameters in the live streaming task instruction, the text data input by the user is parsed in a structured manner to generate a live streaming script sequence arranged in timestamp order. Based on the timestamp of each speech in the live speech sequence, generate a lip-sync driving parameter sequence for the corresponding time point; By combining the live-streamed speech sequence with the voiceprint feature vector extracted from the audio samples uploaded by the user, the timbre features of each speech text in the live-streamed speech sequence are fused to generate a speech speech sequence with a unique timbre. The speech-to-speech sequence is transcribed to generate a subtitle text sequence with timestamp information. The subtitle text sequence is then aligned and bound to the lip-sync driving parameter sequence based on the timestamp information to generate a synchronized subtitle track.

[0010] A second aspect of this application provides a fully intelligent virtual human automated interaction system, wherein the fully intelligent virtual human automated interaction system is used to implement a fully intelligent virtual human automated interaction method, and the fully intelligent virtual human automated interaction system includes: The acquisition module is used to retrieve the corresponding live streaming process template from the template library based on the scene type identifier in the live streaming task instruction; The live streaming configuration module is used to parse the user-uploaded modeling file according to the image creation rules in the live streaming process template, obtain the virtual human model's driving calibration file, and generate a live streaming execution plan by combining the configuration rules of live streaming scene, camera position, and special effects triggering. The live streaming engine module is used to collect driving data of the virtual human model according to the live streaming execution plan, synchronize the driving data to the virtual human model, and generate live audio and video streams by combining the virtual live streaming scene and audio data. The push-stream interaction module is used to push the live audio and video stream to the target live streaming platform, collect the audience interaction data of the target live streaming platform, and trigger the virtual human live streaming real-time response by parsing the audience interaction data to complete the live streaming.

[0011] Optionally, in the first embodiment of the second aspect of this application, the fully intelligent virtual human automated interaction system further includes: The AI ​​content production module is used to perform structured parsing of the text data input by the user according to the live broadcast topic and live broadcast duration parameters in the live broadcast task instruction, and generate a live broadcast script sequence arranged in timestamp order. Based on the timestamp of each speech in the live speech sequence, generate a lip-sync driving parameter sequence for the corresponding time point; By combining the live-streamed speech sequence with the voiceprint feature vector extracted from the audio samples uploaded by the user, the timbre features of each speech text in the live-streamed speech sequence are fused to generate a speech speech sequence with a unique timbre. The speech-to-speech sequence is transcribed to generate a subtitle text sequence with timestamp information. The subtitle text sequence is then aligned and bound to the lip-sync driving parameter sequence based on the timestamp information to generate a synchronized subtitle track.

[0012] A third aspect of this application provides an electronic device, including a memory and a processor, wherein the processor is configured to execute a computer program stored in the memory, and when the processor executes the computer program, it implements each step of the fully intelligent-driven virtual human automated interaction method provided in the first aspect of this application.

[0013] The fourth aspect of this application provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the steps of the fully intelligent-driven virtual human automated interaction method provided in the first aspect of this application.

[0014] In summary, the fully intelligent, automated interaction method and system for virtual humans provided in this application involves: calling the corresponding live streaming process template from the template library based on the scene type identifier in the live streaming task instruction; parsing the user-uploaded modeling file according to the image creation rules in the live streaming process template to obtain the virtual human model's driver calibration file; generating a live streaming execution plan by combining the configuration rules for the live streaming scene, camera movement, and special effects triggering; collecting the virtual human model's driver data according to the live streaming execution plan; synchronizing the driver data to the virtual human model; generating a live audio and video stream by combining the virtual live streaming scene and audio data; pushing the live audio and video stream to the target live streaming platform; collecting audience interaction data from the target live streaming platform; and triggering a real-time response from the virtual human live stream by parsing the audience interaction data to complete the live stream. This application solves the problem of fragmented stages in virtual human live streaming by generating execution plans through template invocation, synchronizing driver data to the virtual human model, pushing the stream, and parsing interaction data to trigger responses, thus achieving a fully automated closed-loop process driven by virtual humans. Attached Figure Description

[0015] Figure 1 A flowchart illustrating the fully intelligent-driven automated interaction method for virtual humans provided in this application embodiment; Figure 2 A schematic diagram of the program modules of the fully intelligent-driven virtual human automated interaction system provided in the embodiments of this application; Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0016] To make the inventive objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0017] To address the issues of data and process incompatibility and extremely low automation in existing virtual human live streaming solutions, this application provides a fully intelligent, end-to-end automated interaction method for virtual humans. Figure 1 This is a flowchart illustrating the fully intelligent virtual human automated interaction method provided in this embodiment. The fully intelligent virtual human automated interaction method includes the following steps: Step 210: Based on the scene type identifier in the live streaming task instruction, call the corresponding live streaming process template from the template library.

[0018] Specifically, the scene type field in the live streaming task instruction is parsed, and the parsed identifier is matched with the pre-established template index in the template library to determine the target live streaming process template. Then, various preset configuration data in the template are read, including image creation rules, scene configuration parameters, and interaction logic parameters, and the integrity of the parameters in the template is verified. After the verification is passed, the various configurations in the template are loaded into the runtime environment to form the initialization configuration set of the current live streaming task, and the key parameters are formatted in a unified manner so that each link can directly call them, thereby completing the calling and initialization of the live streaming process template.

[0019] Step 220: Parse the user-uploaded modeling file according to the image creation rules in the live streaming process template to obtain the virtual human model's driver calibration file, and generate a live streaming execution plan by combining the configuration rules of live streaming scene, camera position, and special effects triggering.

[0020] Specifically, the modeling file is structurally analyzed to extract bone hierarchy information, mesh data, and node names. Based on the character creation rules, the bone nodes are standardized to generate a unified bone topology. On this basis, the initial position and posture parameters of each bone node are calibrated and recorded as standard posture parameters, forming a driving calibration file. Subsequently, the scene configuration data in the template is loaded into a virtual live streaming scene, the camera movement configuration is converted into time-series camera switching data, and the special effects triggering rules are organized into a mapping relationship between conditions and execution actions. By unifying and integrating the above data and establishing the association between bone node identifiers and various configuration data, a complete live streaming execution plan is generated.

[0021] Step 230: Collect the driving data of the virtual human model according to the live broadcast execution plan, synchronize the driving data to the virtual human model, and generate live audio and video streams by combining the virtual live broadcast scene and audio data.

[0022] Specifically, the skeletal drive data of the current frame is obtained according to the drive configuration in the live streaming execution plan, and the data is smoothed between consecutive frames to obtain stable skeletal data. Then, the processed skeletal data is mapped with the standard pose parameters in the drive calibration file to calculate the target position and pose of each skeletal node, and synchronized to the virtual human model to update the model state. At the same time, the input audio data is processed by frame segmentation and the corresponding lip-sync parameters are extracted and synchronized to the face control node of the virtual human model. Based on this, the updated virtual human model and the elements in the virtual live streaming scene are drawn in the same frame to obtain the current frame image data, and the image data is time-aligned with the corresponding audio frame to generate a continuous live audio and video stream.

[0023] Step 240: Push the live audio and video stream to the target live streaming platform and collect the audience interaction data of the target live streaming platform. Trigger the virtual human live streaming real-time response by parsing the audience interaction data to complete the live streaming.

[0024] Specifically, based on the target platform addresses configured in the live streaming execution plan, the generated live audio and video streams are encapsulated into data packets of the corresponding transmission protocol and sent to each target platform for synchronous streaming. Simultaneously, the system retrieves real-time viewer-sent comments and donation data from each platform's interface, performs keyword matching on the comments to determine triggering keywords and their priorities, and performs numerical evaluation on the donation data to generate corresponding trigger identifiers. Subsequently, based on the triggering keywords, priorities, and donation trigger identifiers, the system selects corresponding execution action instructions from the special effects triggering rules in the live streaming execution plan and generates response data. Finally, the system updates the audio and video streams or platform interaction interfaces based on the response data, enabling virtual human actions, voice, or comment feedback, thereby completing real-time interactive responses during the live stream.

[0025] In one optional implementation, the step of parsing the user-uploaded modeling file according to the image creation rules in the live streaming process template to obtain the driving calibration file of the virtual human model includes: traversing the skeletal hierarchy in the user-uploaded modeling file according to the image creation rules in the live streaming process template, obtaining the parent node index and child node index of each skeletal node; generating an initial virtual human model containing a skeletal topology linked list based on the connection relationship between the parent node index and the child node index; zeroing the initial position coordinates and initial rotation angle of each skeletal node in the initial virtual human model to generate a calibrated virtual human model in a standard standing posture; associating and storing the node name of each skeletal node in the calibrated virtual human model with the position coordinates and rotation angle in the standard standing posture to generate a driving calibration file containing skeletal node identifiers and standard posture parameters.

[0026] In this embodiment, after receiving the modeling file uploaded by the user, the skeletal hierarchy is parsed according to the image creation rules. The skeletal hierarchy can be understood as a hierarchical network of nodes, each representing a joint in the virtual human body, such as the shoulder, elbow, or knee. Nodes are connected through parent-child relationships to form a complete human structure. By traversing the node information recorded in the file, the unique identifier of each node can be read one by one, and its corresponding parent node index and all directly connected child node indices can be determined. The parent node index indicates the node's superior joint in the hierarchy, while the child node index indicates its subordinate joint position. Taking an e-commerce virtual anchor as an example, the parent node of the shoulder node might be the spine node, and its child nodes might include upper arm nodes. This index relationship allows for the complete reconstruction of the human skeleton's connection logic. After obtaining the parent-child relationships of all nodes, the connections between nodes are organized to generate a skeletal topology linked list structure. The skeletal topology linked list can be understood as a structural description method arranged according to connection order, where each node not only records its own information but also the relationships between its directly connected nodes, thus forming a sequentially accessible set of connections. The construction process of this structure involves sequentially connecting each node from parent to child nodes, allowing any node to quickly find its adjacent nodes through this chain-like relationship. For example, when a virtual anchor demonstrates gestures, starting from the shoulder node, one can sequentially access the upper arm, forearm, and finally the hand node. This chain-like relationship provides a continuous data foundation for subsequent action driving, thus forming an initial virtual human structure containing complete skeletal connections. Based on this, zero-point realignment is performed on each skeletal node in the initial virtual human structure. Zero-point realignment means uniformly adjusting the position and posture of all nodes to a preset standard reference state to eliminate the offset effects caused by differences in the modeling file. The position coordinates represent the specific location of the node in 3D space, and the rotation angle represents the node's orientation relative to its parent node. By adjusting the root node's position to the origin and progressively adjusting the relative positions of child nodes, the overall structure presents a standard standing posture, such as feet apart and arms hanging naturally. Simultaneously, the rotation angles of each node are unified to a standard direction, thus forming a consistent reference posture. Taking a virtual anchor standing and explaining a product as an example, after zero-point realignment, the character's body will not tilt or rotate. After completing the posture calibration, the node name of each bone node is associated with its position coordinates and rotation angle in the standard standing posture and stored. The node name is used to uniquely identify the specific joint position, such as "left wrist" or "right knee", while the position coordinates and rotation angle are used to describe the spatial information of the node in the standard state.By establishing a one-to-one correspondence between node identifiers and corresponding posture parameters, a complete drive calibration file can be formed. This file is essentially a structured dataset, where each record contains a node name and its corresponding standard posture parameters. Taking a virtual anchor demonstrating products as an example, when a hand-raising motion needs to be driven, the standard posture parameters corresponding to the "right wrist" can be found, and the target position can be calculated by combining real-time drive data. This ensures consistent motion performance in different scenarios and avoids joint misalignment or abnormal motion.

[0027] In one optional implementation, the steps of generating a live streaming execution plan by combining the configuration rules of the live streaming scene, camera movement, and special effects triggering include: according to the scene configuration rules in the live streaming process template, reading the mesh vertex coordinates and sampling the texture coordinates of the 3D scene file corresponding to the scene template identifier to generate a virtual live streaming scene; according to the camera movement rules in the live streaming process template, performing keyframe interpolation on the position coordinates and viewpoint direction of the virtual camera on the time axis to obtain a camera switching sequence with multiple preset camera positions, and assigning parameter values ​​to the transition duration between adjacent camera positions in the camera switching sequence to generate a camera movement scheduling file; according to the special effects triggering rules in the live streaming process template, performing conditional action pairing on the type identifier of the triggering condition and the instruction code of the execution action to generate a special effects triggering rule table; and generating a live streaming execution plan by associating the virtual live streaming scene, the camera movement scheduling file, and the special effects triggering rule table, and establishing a mapping relationship between the associated data and the bone node identifiers in the driver calibration file.

[0028] In this embodiment, when reading the 3D scene file corresponding to the scene template identifier, the path of the scene resources to be loaded is determined according to the scene configuration rules, and the mesh data in the 3D scene file is parsed. The mesh can be understood as a spatial structure formed by connecting multiple vertices. The vertex coordinates represent the specific position of each point in 3D space. By connecting multiple vertices in a preset order, a surface structure for expressing the shape of an object can be formed. During the parsing process, the coordinate data of each vertex is read one by one, and the complete spatial structure is restored according to the connection relationship recorded in the file. At the same time, the texture coordinates are sampled. The texture coordinates are used to describe how a 2D image is mapped to a 3D surface. Essentially, it maps the pixel positions in the image to the vertices of the mesh surface, thereby giving the scene color and detail. By binding the vertex coordinates and texture coordinates, and combining the lighting parameters and environmental information in the scene configuration rules, a virtual live streaming scene with spatial structure and visual performance can be constructed. For example, in an e-commerce live streaming scene, the product display stand, background wall, and lighting positions are all generated through this process and form a stable layout in space. After the virtual scene is constructed, the spatial parameters of the virtual camera are processed according to the camera movement rules. The virtual camera can be understood as a viewing device used to determine the observation position and direction of the image; its position coordinates represent the camera's position in the scene, and its viewing direction represents the direction the camera is facing. By reading keyframe data on the timeline—keyframes being camera state information recorded at specific points in time, including position and orientation—interpolation is performed between adjacent keyframes in chronological order. This means gradually transitioning between two states according to the time ratio, making the camera movement appear as a continuous change. For example, during a product display, switching from a front close-up to a side view can be achieved through a continuous transition between keyframes, allowing the image to move smoothly rather than abruptly. After generating multiple preset camera positions, these positions are arranged in chronological order to form a camera switching sequence, and the transition duration between adjacent camera positions is assigned a value, giving each perspective switch a clear duration. This forms a complete camera movement scheduling file, which records the camera's movement trajectory and switching rhythm throughout the live broadcast. Based on this, triggering conditions and execution actions are paired according to the special effects triggering rules. The type identifier of the triggering condition describes what kind of event can activate the special effect, such as a viewer sending a specific keyword or making a donation. The instruction code of the execution action defines the specific manifestation after being triggered, such as an animation effect on the screen or a virtual person performing a specified action. By classifying and identifying each type of triggering condition and assigning it a corresponding execution instruction, a one-to-one correspondence is established between the two, thus forming a special effects triggering rule table. Taking e-commerce live streaming as an example, when a viewer sends the keyword "place an order," a product highlighting effect can be triggered, and when the donation amount reaches a certain value, a celebration animation can be triggered. The rule table can quickly locate the corresponding execution instruction and complete the invocation.After completing the construction of the virtual scene, camera movement scheduling file, and special effect trigger rule table, the three are unified and associated, and a mapping relationship is established with the bone node identifiers in the drive calibration file to generate a complete live broadcast execution plan. The core of the association process is to integrate spatial information, time series, and trigger logic, enabling each part to work together within the same operating framework. The scene data provides the spatial basis, the camera movement scheduling file provides time-varying information, the special effect trigger rule table provides an event response mechanism, and the bone node identifiers are used to determine the action positions of the virtual human. For example, during a product demonstration, when the "display product" instruction is triggered, the hand node can be located through the bone node identifier, the camera perspective can be adjusted in combination with the camera movement scheduling file, and a highlight special effect can be superimposed on the scene, thus realizing the linkage between actions, images, and special effects, and finally forming a directly executable live broadcast execution plan.

[0029] In an optional implementation manner, the steps of collecting the drive data of the virtual human model according to the live broadcast execution plan, synchronizing the drive data to the virtual human model, and generating a live broadcast audio-video stream in combination with the virtual live broadcast scene and audio data include: obtaining the original bone data of the current frame according to the bone node identifier, generating optimized bone data by performing data smoothing on the original bone data and historical bone data; performing bone position mapping on the optimized bone data and the standard pose parameters of the drive calibration file to obtain the target position data of each bone segment, and synchronizing the target position data to the corresponding bone nodes in the virtual human model to generate a driven virtual human model; extracting audio features from the audio frame data in the input audio stream to obtain the lip synchronization drive parameters corresponding to the current audio frame, and synchronizing the lip synchronization drive parameters to the facial control nodes in the driven virtual human model to generate a lip-synchronized virtual human model; performing same-frame rendering on the lip-synchronized virtual human model and the scene elements in the virtual live broadcast scene to obtain the image data of the current frame, and performing timestamp alignment processing on the image data and the corresponding audio frame data to generate a live broadcast audio-video stream.

[0030] In this embodiment, when acquiring the skeletal data of the current frame, the position and pose information of the corresponding node at that moment are read from the driving data source based on the skeletal node identifier. The original skeletal data can be understood as a set of spatial parameters that change over time, where each node contains position coordinates and rotation information. However, due to slight fluctuations in the acquisition process, jumps may occur between adjacent frames. Therefore, it is necessary to introduce historical skeletal data for smoothing. The smoothing process is implemented as follows: when the current frame data arrives, the data of the corresponding nodes in several consecutive historical frames are retrieved simultaneously. The data of each frame are weighted and fused in chronological order so that the result of the current frame reflects both the latest state and retains the historical trend, thereby eliminating the jitter caused by sudden changes. For example, when an e-commerce virtual anchor explains products with gestures, the hand nodes may experience slight tremors if the original data is used directly. However, after smoothing, the hand movement trajectory shows a continuous change effect, thus obtaining stable optimized skeletal data. After obtaining the optimized skeletal data, it is mapped to the standard pose parameters in the drive calibration file. The standard pose parameters provide the position and orientation information of each node in the reference state. By comparing the difference between the current data and the standard pose, the target position and pose that each skeletal node needs to be adjusted to can be calculated. The mapping process is based on the skeletal hierarchy, passing down level by level from the root node, superimposing the changes of the current node onto the child nodes, thereby ensuring the consistency of the overall structure and finally obtaining the target position data of each skeletal segment. Then, this target data is synchronized to the corresponding skeletal nodes in the virtual human structure, so that the virtual human presents a movement state consistent with the drive data. For example, when displaying a product, the action of raising the arm will be passed down level by level to the wrist and fingers through node mapping, so that the action remains visually natural and coherent, thus forming the driven virtual human state. After the action drive is completed, the input audio stream is processed. The audio stream consists of continuous audio frames, each representing a sound signal within a short time. By extracting features from the audio frames, parameter information reflecting changes in pronunciation can be obtained, such as changes in sound intensity and the degree of opening and closing of the mouth. This information can be used to describe the changing trend of mouth shape. By converting the features of the current audio frame into corresponding lip-sync parameters and synchronizing these parameters to the control nodes of the virtual human's face, the facial nodes adjust their shape according to audio changes, thus achieving a correspondence between speech and lip movements. For example, when a virtual anchor explains a product name, open vowels in the audio drive the mouth to open, while closed vowels cause the mouth to close, generating a lip-synced virtual human state. After synchronizing the actions and lip movements, the virtual human state and various elements in the virtual live-streaming scene are rendered in the same frame. Frame-sync rendering refers to calculating and generating a single frame of image data for all objects to be displayed at the same point in time, including the virtual human, background environment, and any possible special effects elements. During the rendering process, these elements are combined according to their spatial position and hierarchical relationship to form a complete image.Subsequently, the generated image data is timestamped and aligned with the audio frame data of the corresponding time period. A timestamp can be understood as an identifier marking the time position of the data. By aligning the time stamps of the image and audio, the changes in the visuals are kept consistent with the sound playback, avoiding visual delays or lip-sync issues. For example, when a virtual anchor introduces the price of a product, the sound output and lip movements can be presented at the same time, while the gestures and camera transitions in the visuals are also synchronized, ultimately forming a continuous live audio and video stream.

[0031] It should be noted that after the motion capture device outputs each frame of skeletal data, each skeletal node contains its position value in three-dimensional space and its rotation information relative to its parent node. However, numerical fluctuations often occur between adjacent frames. For example, the position of the same wrist node in three consecutive frames may be (1.20, 0.50, 0.30), (1.35, 0.48, 0.31), and (1.18, 0.52, 0.29), respectively. The change in the second frame deviates significantly from the previous trend. Therefore, before the data enters the driving stage, a set of continuous frame state records needs to be established to constrain the current frame data. In specific processing, the output result and the change range of the previous frame are first saved. When the new frame data arrives, the difference between the current frame and the previous frame is calculated and compared with the historical change range. If the current change range is significantly greater than the historical change range, the change is reduced to bring it closer to the historical change trend, thereby eliminating sudden jitter while retaining the overall directional change of the real movement. This ensures a smoother transition between consecutive frames. After stabilizing continuous frame data, structural constraints need to be corrected for each node in the skeletal chain to ensure that bone length and connection relationships remain consistent. Specifically, based on a skeletal topology linked list, the process backtracks from the end node to the parent node, calculating the actual distance between the current node and its parent node and comparing it to the standard distance recorded in the calibration file. If the current distance is too large, the child node is pulled inward along the parent node's direction; if the distance is too small, it is adjusted outward in the opposite direction. For example, in the arm structure, the standard distance between the shoulder and elbow is 0.30 units. When the calculated current distance is 0.36, the elbow position is moved 0.06 units towards the shoulder to restore it to a reasonable range. Then, the adjustment result is passed from the root node to the child nodes, ensuring the continuity of the entire skeletal chain after adjustment. Through this back-and-forth correction, the bone length is ensured to remain within a stable range while preserving the overall shape of the original movement as much as possible. After adjusting the skeletal position, it's necessary to detect and correct any potential node overlaps in the space. To do this, a spatial enclosure can be set for key areas. For example, a spatial enclosure with a radius of 0.25 units can be established centered on the chest node. When a hand node enters this enclosure, overlap is considered to have occurred. Upon detection, the depth to which the hand node enters the enclosure is calculated, and it is then displaced in the opposite direction from the chest to the hand, pushing the hand node out of the enclosure boundary. For instance, if the hand node enters to a depth of 0.05 units, it is moved outward by 0.05 units in that direction, placing it precisely at the boundary. Simultaneously, the connection between the wrist and forearm is maintained during this movement to prevent the visual problem of the hand appearing to pass through the body.After the above processing is completed, the stabilization results and structural correction results are fused. Specifically, the data after continuous frame constraints is used as the base position, and then the displacement after bone length correction and spatial conflict correction is superimposed to obtain the final bone data used for driving. For example, when an e-commerce virtual anchor displays products, the arm is raised from a hanging state to the display position. The original data may have a fluctuation of ±0.05 units. After continuous frame constraints, the fluctuation is compressed to ±0.01 units. At the same time, the length of the upper arm and forearm is kept constant through bone length correction, and the spatial range is restricted to avoid crossing when the hand is close to the body. The final output motion trajectory maintains the original raising trend and is numerically continuous and stable, thus presenting a natural and smooth motion effect visually.

[0032] In one optional implementation, to avoid overlap or penetration between different parts of the virtual human during motion control, a dual-layer collision detection mechanism is introduced, comprising two consecutive processing stages: a coarse-grained collision detection layer and a fine-grained collision correction layer. The core of the coarse-grained collision detection layer is establishing a spatial bounding area for key skeletal nodes. This bounding area describes the area occupied by the node in three-dimensional space and can be represented by a bounding volume with a fixed radius centered on the node. For example, a bounding area with a radius of 0.25 units is constructed centered on the chest node, and a bounding area with a radius of 0.10 units is constructed centered on the hand node. After the driving data is updated, the spatial distance between each node is calculated frame by frame and compared with the corresponding bounding area. When the actual distance between any two nodes is less than the sum of their bounding areas, a primary collision is determined to have occurred. Taking an e-commerce virtual anchor bringing their arm close to their body while showcasing products as an example, when the distance between the hand node and the chest node gradually decreases from 0.40 to 0.28, and further to 0.20, since this distance is less than the sum of their enclosing ranges (0.35), it is marked as having a potential clipping risk in the coarse-grained collision detection layer, and corresponding collision identification information is generated. This information records the node pair that collided and the current distance deviation value. After completing the coarse-grained collision detection, the information with collision identification is input to the fine-grained collision correction layer for further processing. The goal of the fine-grained collision correction layer is to adjust the position of the colliding nodes without disrupting the overall motion continuity. Specifically, it calculates the direction vector between the two nodes based on the collision identification. This direction vector describes the spatial direction from one node to another. Then, based on the current distance deviation value, it determines the amount of displacement that needs to be adjusted, and corrects the position of one of the nodes along the opposite direction. For example, if the actual distance between the hand node and the chest node is 0.20 units, while the safe distance is 0.35 units, then a spacing of 0.15 units needs to be added. This can be achieved by moving the hand node 0.15 units away from the chest to restore it to the safe range. Simultaneously, to ensure the continuity of the skeletal structure, after moving the hand node, corresponding positional compensation is needed for its superior nodes (such as the forearm node). This is done by distributing displacement along the skeletal chain to prevent joint breaks or unnatural bending during the adjustment process. After each correction, the result needs to be re-checked to confirm complete escape from the collision state. If some overlap remains, directional adjustment and displacement compensation continue until the distance between all relevant nodes meets the safe range requirements. During this process, to avoid abrupt changes in movement due to excessively large corrections, an upper limit can be set for the displacement amount of each adjustment, for example, a maximum single adjustment of no more than 0.08 units. Corrections are then gradually completed through multiple iterations, resulting in a visually continuous change rather than abrupt changes.For example, when a virtual anchor explains using gestures, if the hand is detected crossing the body as it moves from the front to the side, multiple small displacements are used to gradually move the hand out of the body's range, while maintaining the smoothness of the overall arm movement path. After fine-grained collision correction, the correction results are fused with the original driving data to ensure that the final output satisfies spatial constraints while closely approximating the original motion trajectory. During the fusion process, the original driving trajectory is used as a basis, and the displacement correction is superimposed on the nodes that collide, while the original data of the nodes that do not collide remains unchanged, thus generating the final skeletal driving data. For example, during product demonstrations, when the virtual anchor makes a gesture of holding a product, a dual-layer collision detection and automatic clipping correction mechanism ensures that the hand is always outside the body and maintains a reasonable relationship with the product's position, preventing the hand from crossing the body or objects, thus presenting a stable and natural interactive effect.

[0033] It should be noted that while completing the positional repair, it is necessary to introduce skeletal physiological constraint rules to limit the adjustment results, so as to ensure that the movement visually conforms to the laws of human movement. Skeletal physiological constraints can be understood as restrictions on the range of motion of each joint and the length of the bones. For example, the elbow can only bend within a certain angle range, and the distance between the shoulder and the elbow is kept within a fixed range. The specific thresholds are shown in Table 1. During the repair process, when a node is moved, it is necessary to recalculate its distance and relative direction with its parent node. If the distance is found to exceed the preset range, it needs to be pulled back along the connection direction to restore it to the allowable range. For example, if the standard length of the upper arm is 0.30 units, and it becomes 0.34 units after adjustment, then the elbow is pulled back 0.04 units along the shoulder direction. At the same time, it is also necessary to check whether the rotation direction of the joint exceeds the limit. For example, the elbow is not allowed to bend in the opposite direction. When an abnormal direction is detected, the node position is readjusted along the allowable direction to restore it to a reasonable posture.

[0034] Table 1 In one optional implementation, during the speech-driven lip-syncing process, the continuous audio data is first divided into multiple temporally continuous audio segments, each corresponding to a short vocalization, and each segment is marked with a precise time position. Then, the articulation structure of each segment is analyzed, breaking down the speech content into finer-grained articulation units. An articulation unit can be understood as the smallest vocal element constituting speech; for example, "ma" can be broken down into a continuous change from closed to open mouth. After decomposition, a specific time interval is assigned to each articulation unit, giving it a clear start and end position on the timeline, thus forming a temporally continuous articulation sequence. After obtaining the articulation sequence, each articulation unit needs to be converted into corresponding lip-syncing parameters. These parameters describe the specific state of facial control nodes, such as the degree of mouth opening and closing, the direction of mouth corner stretching, and the degree of lip protrusion. To achieve this conversion, a set of correspondences between articulation units and lip shapes is pre-established, with each articulation unit corresponding to a basic lip shape and containing multiple control parameters. For example, closed-mouth pronunciation corresponds to a completely closed mouth, open-mouth pronunciation corresponds to a mouth that is open to a certain extent, and rounded-lip pronunciation corresponds to lips that are pulled forward. In actual processing, each pronunciation unit is mapped to a corresponding basic mouth shape parameter, and these parameters are arranged in chronological order to form an initial mouth shape sequence. Since there are transitional changes between adjacent pronunciation units during continuous pronunciation, directly using discrete mouth shapes would cause abrupt changes in mouth movements. Therefore, it is necessary to smoothly connect the mouth shape parameters between adjacent time segments. Specifically, at the time boundary between two adjacent pronunciation units, the two mouth shape states are gradually transitioned according to the time ratio, so that the previous mouth shape gradually changes to the next, thus forming a continuous change process. For example, when pronouncing "ba," the mouth closes first and then quickly opens. On the time axis, the closed state can be gradually transitioned to the open state, rather than switching instantaneously, making the mouth shape change more natural. After completing the construction of the basic mouth shape sequence, it is also necessary to dynamically adjust the mouth shape amplitude based on the intensity changes in the audio signal. Audio intensity can be understood as the loudness of the sound, and its numerical changes reflect the strength of the airflow during pronunciation. In the specific processing, the intensity value of each audio segment is associated with the corresponding lip-sync parameters. When the intensity is high, the mouth opening and closing amplitude is appropriately increased; when the intensity is low, the opening and closing amplitude is decreased, thus ensuring that the lip-sync changes are consistent with the actual vocal intensity. For example, when an e-commerce virtual anchor emphasizes the price of a product, the vocal intensity increases, and the corresponding mouth opening and closing amplitude also increases, making the expression more expressive. After generating a complete lip-sync sequence, this sequence is bound to facial control nodes. Facial control nodes can be understood as a set of key points used to drive facial shape changes, with each node corresponding to a specific location on the lips, chin, or cheek. By passing the lip-sync parameters frame by frame to the corresponding nodes, the facial shape at each time point is updated according to the sequence, thereby achieving real-time synchronization between speech and lip-sync.During time synchronization, the audio time markers are aligned with the lip-sync sequence time markers to ensure that the lip-sync changes in each frame are consistent with the corresponding audio segment. For example, when a virtual anchor introduces a product name, the pronunciation of each syllable in the audio is reflected in the mouth shape at the same time, so that the lip-sync changes seen by the audience perfectly match the spoken content, thus creating a coherent and natural expression.

[0035] In one optional implementation, the steps of pushing the live audio and video stream to the target live streaming platform and collecting audience interaction data from the target live streaming platform, and triggering a virtual human's real-time live response by parsing the audience interaction data, include: encapsulating the live audio and video stream into transmission protocol packets adapted to multiple target platforms according to the platform address list in the live streaming execution plan, and pushing the encapsulated transmission protocol packets to the corresponding push platform addresses respectively; collecting bullet screen text and reward records in real time from the interaction data interfaces of each target live streaming platform, generating trigger keywords and their corresponding priority levels by performing keyword hierarchical matching on the bullet screen text, and generating a reward trigger identifier based on the comparison result of the reward amount in the reward record with a preset threshold; calling the corresponding execution action instruction from the special effects trigger rule table in the live streaming execution plan according to the trigger keyword, priority level, and reward trigger identifier, and retrieving the corresponding lip-sync driving parameters from the driver calibration file according to the execution action instruction to generate trigger response data; and injecting data into the audio track, video track, or bullet screen sending interface of the live audio and video stream or the target live streaming platform according to the execution action instruction in the trigger response data to generate an updated live audio and video stream or bullet screen reply message.

[0036] In this embodiment, after generating the live audio and video streams, the data is distributed according to the platform address list in the live execution plan. The platform address list records the push address corresponding to each target platform and the required data transmission format. Before sending, the audio and video streams need to be encapsulated. Encapsulation can be understood as combining continuous audio and image data according to a certain data structure, adding time stamps and control information, so that it can be correctly parsed by different platforms. Since different platforms have different data formats and transmission protocols, corresponding data packets need to be generated according to the requirements of each platform. For example, the same audio and video content may be organized according to different encapsulation structures so that the data received by each platform conforms to its parsing specifications. After encapsulation, each data packet is sent to the corresponding push address through the network interface, thereby achieving simultaneous live streaming on multiple platforms. For example, when an e-commerce virtual anchor displays products on multiple platforms simultaneously, the same screen content will be organized into multiple platform data packets with the same format but compatible structure, and pushed to different platform entry points respectively. While the stream is running, audience behavior information is obtained from the interactive data interfaces provided by various target platforms. These interfaces can be understood as real-time data channels provided by the platforms, used to return information such as the bullet screen text sent by viewers and donation records. The bullet screen text consists of real-time comments entered by viewers, which need to be parsed and processed. By scanning the text character by character and comparing it with a pre-defined set of keywords, it can be identified whether it contains trigger commands. Keywords are then categorized according to their importance, with priority levels used to distinguish the order in which different keywords trigger responses. For example, during product demonstrations, words like "purchase" and "discount" can be identified as high-priority trigger content, while ordinary comments are of lower priority. Simultaneously, the monetary information in the donation records is read and compared with pre-defined numerical limits. When the amount reaches a specified condition, a corresponding trigger flag is generated, indicating that a higher-level response action is required. After obtaining the trigger keywords, priority levels, and donation trigger flags, matching execution action commands are searched from the special effects trigger rule table according to the corresponding relationships. Execution action commands can be understood as specific descriptions of virtual human behavior or visual effects, such as performing a gesture, playing a specific voice, or triggering a visual effect. After the instruction is determined, the lip-sync driving parameters related to the action are retrieved from the driver calibration file. These parameters control the changes in facial nodes during speech production, ensuring consistency between the speech output and facial expression. By combining the action instruction with the corresponding lip-sync parameters, complete trigger response data can be generated. This data includes both action execution information and speech and lip-sync control information. For example, when a viewer sends "place order" along with a tip, response data can be generated showing the virtual human nodding and saying thank you, with the lip-sync changes matching the speech content.After generating the trigger response data, the current live stream content is updated according to the execution action instructions contained therein. This update includes writing data to the audio track, video track, and platform interaction interface. The audio track can be understood as the transmission channel for sound data in the live stream; by inserting new voice data into it, the virtual character can respond in real time. The video track is the output channel for visual data; by overlaying action changes or special effects, the visuals change accordingly. The platform's bullet screen interface is used to send text-based replies, allowing viewers to see feedback information on the screen. By injecting trigger response data into these channels, real-time interaction can be achieved without affecting the overall live stream flow. For example, during a product live stream, when a viewer triggers a high-priority interaction, the virtual character on screen will immediately execute the corresponding action and speak, while simultaneously sending a thank-you message in the bullet screen area, thus generating an updated live stream audio and video stream or interactive reply content, ensuring that the interactive effect remains synchronized with the live stream content.

[0037] In one optional implementation, the user-input text data is structured and parsed according to the live broadcast topic and duration parameters in the live broadcast task instruction to generate a live broadcast script sequence arranged in timestamp order; a lip-sync driving parameter sequence for the corresponding time point is generated based on the timestamp of each script in the live broadcast script sequence; timbre feature fusion is performed on each script text in the live broadcast script sequence using the live broadcast script sequence and the voiceprint feature vector extracted from the audio samples uploaded by the user to generate a script speech sequence with a unique timbre; the script speech sequence is transcribed into speech to generate a subtitle text sequence with timestamp information, and the subtitle text sequence is aligned and bound with the lip-sync driving parameter sequence according to the timestamp information to generate a synchronized subtitle track.

[0038] In this embodiment, upon receiving the live streaming task instruction, the text data input by the user is processed using structured parsing based on the live streaming theme and duration parameters. The text data is first semantically broken down into several independent sentence units. Then, combined with the live streaming duration parameter, the overall time is divided into multiple consecutive time segments, and each sentence unit is sequentially assigned to its corresponding time segment. The timeline can be understood as a sequence structure that marks the entire live streaming process chronologically. Each point in time corresponds to a specific content expression position. By binding sentence units to their positions on the timeline, clear timestamp information can be generated for each segment of dialogue, thus forming a live streaming dialogue sequence arranged chronologically. For example, in e-commerce live streaming, the first 30 seconds are dedicated to product introduction, the middle stage to function explanation, and the remaining time to promotional explanation; each sentence has a clear sequential relationship on the timeline. After generating the speech sequence, a lip-sync parameter sequence is generated based on the timestamp information corresponding to each speech. The lip-sync parameters describe the opening and closing changes of facial features during phonation. Their generation process relies on decomposing the speech text into a sequence of phonological units, assigning duration to each unit based on its position on the timeline. Subsequently, each phonological unit is mapped to a corresponding lip-sync state, and these are arranged chronologically to form a continuous sequence of changes, thus obtaining a lip-sync parameter sequence consistent with the timeline. For example, in the pronunciation of a product name, different syllables correspond to different mouth shapes. Time allocation ensures that lip-sync changes are consistent with the pronunciation rhythm, providing a synchronization basis for subsequent audio playback. After generating the lip-sync parameters, the live speech sequence is fused with user-uploaded audio samples. These audio samples contain the speaker's voiceprint features, which can be understood as a digital representation of voice characteristics, reflecting the speaker's timbre, pitch, and vocal habits. By processing each sentence unit in the speech text segment by segment, the text content is combined with the voiceprint feature vector, allowing the generated speech to maintain the semantics of the text while presenting a specific timbre effect. In the specific implementation process, text is converted into basic speech data, and then the frequency distribution and timbre details of the sound are adjusted according to voiceprint characteristics to form a speech sequence with a consistent voice style. For example, when a virtual anchor explains a product, all voices maintain a uniform timbre, making the overall expression coherent. After generating the speech sequence, the speech data is transcribed to generate a subtitle text sequence. Speech transcription can be understood as the process of converting sound signals into corresponding text content. In the conversion process, it is necessary not only to recognize the speech content, but also to record the time position corresponding to each piece of text, thereby adding timestamp information to each subtitle. The generated subtitle text sequence maintains a temporal correspondence with the speech, ensuring that each piece of text can accurately match the speech playback position.Subsequently, the subtitle text sequence is aligned and bound to the aforementioned lip-sync driving parameter sequence. By comparing the timestamp information of the two, the subtitle content and lip-sync changes within the same time range are associated, ensuring that the subtitle display, audio playback, and lip-sync changes are completely synchronized in time. For example, when a virtual anchor introduces the price of a product, the audio output, mouth movements, and on-screen subtitles will appear at the same time, thereby generating a synchronized subtitle track and ensuring the overall coordination and consistency of the expression.

[0039] In one optional implementation, based on the resource requirement parameters in the live streaming execution plan, the available computing resources are subjected to status detection and load assessment. The rendering computation, audio and video processing, and interactive processing in the live streaming task are divided into tasks. The tasks are allocated according to the processing capacity and current load of each computing resource, generating computing power scheduling results. Based on the computing power scheduling results, dynamic resource allocation and task migration control are performed on each computing node to ensure the continuity and stability of the live streaming process.

[0040] Specifically, the system's computing resources are monitored based on the resource requirement parameters recorded in the live streaming execution plan. These parameters describe the computing power consumption of various processing tasks during the live stream, such as the graphics processing requirements for rendering, the signal processing requirements for audio, and the response speed requirements for interactive processing. Simultaneously, the operating status of each computing node is collected. A computing node can be understood as a processing unit undertaking specific computing tasks, and its status information includes current load, available processing power, and operational stability. Summarizing the status of each node forms a complete resource status description. Subsequently, the various processing components in the live streaming task are broken down, dividing rendering, audio / video processing, and interactive responses into multiple independently executable task units. This allows different tasks to be executed in parallel on different computing nodes, thereby improving overall processing efficiency. After task decomposition, each task unit is matched with the processing power of the computing nodes. Based on the node's current load and remaining processing power, a suitable execution location is assigned to each task unit. High-load tasks are preferentially assigned to nodes with stronger processing power and lower current load, while avoiding multiple high-load tasks concentrated on the same node, thus generating a computing power scheduling result. The computing power scheduling result can be understood as a mapping between a set of tasks and computing nodes, clearly indicating the execution location and resource consumption of each task. For example, when an e-commerce virtual anchor displays high-resolution images, image rendering tasks are assigned to nodes with strong graphics processing capabilities, while bullet screen parsing tasks are assigned to nodes with fast response times, thus achieving a reasonable distribution of different tasks. After task allocation, dynamic resource control is performed on each computing node based on the computing power scheduling result. During operation, the node status is continuously monitored. When the load on a node continues to increase or its processing capacity decreases, some tasks on that node are migrated to other idle nodes. Task migration can be understood as transferring the ongoing data processing process between different nodes while ensuring the continuity of the data state, so that the migration process does not cause interruption or loss of processing results. For example, if the graphics processing node experiences excessive load during a live broadcast, some rendering tasks can be transferred to a backup node to avoid screen lag; at the same time, resource allocation is adjusted in real time so that new tasks can obtain processing resources in a timely manner. Through the above process, the balanced use of computing resources can be maintained throughout the live broadcast, ensuring stable operation of audio and video generation, interactive response, and image rendering, thereby guaranteeing the continuity and smoothness of the live broadcast output.

[0041] According to the fully intelligent virtual human automated interaction method provided in this application, the corresponding live streaming process template is called from the template library based on the scene type identifier in the live streaming task instruction; the user-uploaded modeling file is parsed according to the image creation rules in the live streaming process template to obtain the virtual human model's driving calibration file, and a live streaming execution plan is generated by combining the configuration rules of the live streaming scene, camera position, and special effects triggering; the driving data of the virtual human model is collected according to the live streaming execution plan, and the driving data is synchronized to the virtual human model; a live audio and video stream is generated by combining the virtual live streaming scene and audio data; the live audio and video stream is pushed to the target live streaming platform, and the audience interaction data of the target live streaming platform is collected; the virtual human live streaming real-time response is triggered by parsing the audience interaction data to complete the live streaming. This application solves the problem of fragmented links in virtual human live streaming by generating execution plans through template calling, synchronizing driving data to the virtual human model, pushing the stream and parsing interaction data to trigger responses, and realizes a fully automated closed loop of virtual human-driven process.

[0042] Figure 2 This application provides a fully intelligent, automated virtual human interaction system, which can be used to implement the fully intelligent, automated virtual human interaction method described in the foregoing embodiments. For example... Figure 2 As shown, this fully intelligent, automated virtual human interaction system mainly includes: The acquisition module 10 is used to retrieve the corresponding live streaming process template from the template library based on the scene type identifier in the live streaming task instruction. The live streaming configuration module 20 is used to parse the modeling file uploaded by the user according to the image creation rules in the live streaming process template, obtain the driving calibration file of the virtual human model, and generate a live streaming execution plan by combining the configuration rules of live streaming scene, camera position, and special effects triggering. The live streaming engine module 30 is used to collect driving data of the virtual human model according to the live streaming execution plan, synchronize the driving data to the virtual human model, and generate live audio and video streams by combining the virtual live streaming scene and audio data. The push-stream interaction module 40 is used to push the live audio and video stream to the target live streaming platform and collect the audience interaction data of the target live streaming platform. By parsing the audience interaction data, it triggers the virtual human live streaming real-time response to complete the live stream.

[0043] In one optional implementation, the fully intelligent virtual human automated interaction system further includes an AI content production module. The AI ​​content production module is used to perform structured parsing of the user-input text data based on the live broadcast topic and duration parameters in the live broadcast task instructions, generating a live broadcast script sequence arranged in timestamp order; generating a lip-sync driving parameter sequence for the corresponding time point based on the timestamp of each script in the live broadcast script sequence; fusing timbre features of each script text in the live broadcast script sequence with the voiceprint feature vector extracted from user-uploaded audio samples, generating a script speech sequence with a unique timbre; transcribing the script speech sequence to generate a subtitle text sequence with timestamp information, and aligning and binding the subtitle text sequence with the lip-sync driving parameter sequence based on the timestamp information to generate a synchronized subtitle track.

[0044] In one optional implementation, the live streaming configuration module is specifically used to: traverse the skeletal hierarchy in the user-uploaded modeling file according to the image creation rules in the live streaming process template, and obtain the parent node index and child node index of each skeletal node; generate an initial virtual human model containing a skeletal topology linked list based on the connection relationship between the parent node index and the child node index; perform zero-point repositioning on the initial position coordinates and initial rotation angle of each skeletal node in the initial virtual human model to generate a calibrated virtual human model in a standard standing posture; and associate and store the node name of each skeletal node in the calibrated virtual human model with the position coordinates and rotation angle in the standard standing posture to generate a driver calibration file containing skeletal node identifiers and standard posture parameters.

[0045] In an optional implementation, the live streaming configuration module is further configured to: read the mesh vertex coordinates and sample the texture coordinates of the 3D scene file corresponding to the scene template identifier according to the scene configuration rules in the live streaming process template, and generate a virtual live streaming scene; perform keyframe interpolation on the position coordinates and viewpoint direction of the virtual camera on the time axis according to the camera movement rules in the live streaming process template, obtain a camera switching sequence of multiple preset camera positions, and assign parameter values ​​to the transition duration between adjacent camera positions in the camera switching sequence, and generate a camera movement scheduling file; perform condition action pairing on the type identifier of the trigger condition and the instruction code of the execution action according to the special effects triggering rules in the live streaming process template, and generate a special effects triggering rule table; and generate a live streaming execution plan by associating the virtual live streaming scene, the camera movement scheduling file, and the special effects triggering rule table, and establishing a mapping relationship between the associated data and the bone node identifier in the driver calibration file.

[0046] In one optional implementation, the live streaming engine module is specifically used for: collecting driving data of the virtual human model according to the live streaming execution plan, synchronizing the driving data to the virtual human model, and generating a live audio and video stream by combining the virtual live streaming scene and audio data. This includes: obtaining the original bone data of the current frame based on the bone node identifier; generating optimized bone data by smoothing the original bone data and historical bone data; mapping the optimized bone data to the standard pose parameters in the driving calibration file to obtain the target position data of each bone segment, and synchronizing the target position data to the corresponding bone nodes in the virtual human model to generate the driven virtual human model; extracting audio features from the audio frame data in the input audio stream to obtain the lip-sync driving parameters corresponding to the current audio frame, synchronizing the lip-sync driving parameters to the facial control nodes in the driven virtual human model to generate a lip-sync synchronized virtual human model; and obtaining the image data of the current frame by rendering the lip-sync synchronized virtual human model and scene elements in the virtual live streaming scene in the same frame, and aligning the image data with the corresponding audio frame data using timestamps to generate a live audio and video stream.

[0047] In one optional implementation, the push-stream interaction module is specifically used for: encapsulating the live audio and video streams into transmission protocol packets adapted to multiple target platforms based on the platform address list in the live execution plan, and pushing the encapsulated transmission protocol packets to the corresponding push-stream platform addresses respectively; collecting bullet screen text and reward records in real time from the interactive data interfaces of each target live streaming platform, generating trigger keywords and their corresponding priority levels by performing keyword hierarchical matching on the bullet screen text, and generating a reward trigger identifier based on the comparison result of the reward amount in the reward record with a preset threshold; calling the corresponding execution action instruction from the special effects trigger rule table in the live execution plan based on the trigger keyword, priority level, and reward trigger identifier, and retrieving the corresponding lip-sync driving parameters from the driver calibration file based on the execution action instruction to generate trigger response data; and injecting data into the audio track, video track, or bullet screen sending interface of the target live streaming platform based on the execution action instruction in the trigger response data to generate an updated live audio and video stream or bullet screen reply message.

[0048] The present application provides a fully intelligent, automated interactive system for virtual humans. Based on the scene type identifier in the live streaming task instruction, it retrieves the corresponding live streaming process template from a template library. It then parses the user-uploaded modeling file according to the image creation rules in the live streaming process template to obtain the virtual human model's driver calibration file. Combining this with the configuration rules for the live streaming scene, camera movement, and special effects triggering, it generates a live streaming execution plan. Based on the live streaming execution plan, it collects the virtual human model's driver data, synchronizes the driver data to the virtual human model, and generates a live audio and video stream by combining the virtual live streaming scene and audio data. The live audio and video stream is pushed to the target live streaming platform, and audience interaction data from the target live streaming platform is collected. By parsing the audience interaction data, it triggers a real-time response from the virtual human to complete the live stream. This application solves the problem of fragmented stages in virtual human live streaming by using template-based execution plan generation, driver data synchronization to the virtual human model, and streaming and parsing interaction data to trigger responses, achieving a fully automated closed-loop process driven by virtual humans.

[0049] According to the scheme provided in this application Figure 3 An electronic device is provided as an embodiment of this application. This electronic device can be used to implement the fully intelligently driven virtual human automated interaction method in the foregoing embodiments, mainly including: The system includes a memory 301, a processor 302, and a computer program 303 stored on the memory 301 and executable on the processor 302. The memory 301 and the processor 302 are connected via communication. When the processor 302 executes the computer program 303, it implements the fully intelligent virtual human automated interaction method described in the foregoing embodiments. The number of processors can be one or more.

[0050] The memory 301 can be a high-speed random access memory (RAM) or a non-volatile memory, such as a disk storage device. The memory 301 is used to store executable program code, and the processor 302 is coupled to the memory 301.

[0051] Furthermore, embodiments of this application also provide a computer-readable storage medium, which may be disposed in the electronic device described in the above embodiments, and the computer-readable storage medium may be as described above. Figure 3 The memory in the illustrated embodiment.

[0052] The computer-readable storage medium stores a computer program that, when executed by a processor, implements the fully intelligent virtual human automated interaction method described in the foregoing embodiments. Furthermore, the computer-readable storage medium can also be a USB flash drive, external hard drive, read-only memory (ROM), RAM, magnetic disk, or optical disk, or any other medium capable of storing program code.

[0053] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0054] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0055] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A full-process intelligent driving virtual human automatic interaction method, characterized in that, The method comprises the following steps: According to the scene type identifier in the live broadcast task instruction, the corresponding live broadcast process template is called from the template library; According to the image creation rule in the live broadcast process template, the modeling file uploaded by the user is parsed to obtain the driving calibration file of the virtual human model, and a live broadcast execution scheme is generated in combination with the configuration rules of the live broadcast scene, camera angle, and special effect triggering; According to the live broadcast execution scheme, the driving data of the virtual human model is collected, the driving data is synchronized to the virtual human model, and a live broadcast audio and video stream is generated in combination with the virtual live broadcast scene and audio data; The live broadcast audio and video stream is pushed to the target live broadcast platform, and the audience interaction data of the target live broadcast platform is collected, the virtual human live broadcast real-time response is triggered by analyzing the audience interaction data, and the live broadcast is completed.

2. The full-process intelligent driving virtual human automated interaction method of claim 1, wherein, The step of parsing the modeling file uploaded by the user according to the image creation rule in the live broadcast process template to obtain the driving calibration file of the virtual human model comprises the following steps: According to the image creation rule in the live broadcast process template, the node traversal is performed on the bone hierarchy structure in the modeling file uploaded by the user, the parent node index and the child node index of each bone node are obtained; According to the connection relationship between the parent node index and the child node index, an initial virtual human model containing a bone topology linked list is generated; The initial position coordinates and the initial rotation angle of each bone node in the initial virtual human model are zero-point homing, and a calibrated virtual human model in a standard standing posture is generated; The node name, position coordinates and rotation angle of each bone node in the calibrated virtual human model are associated and stored, and a driving calibration file containing bone node identifiers and standard posture parameters is generated.

3. The full-process intelligent driving virtual human automated interaction method of claim 2, wherein, The step of generating a live broadcast execution scheme in combination with the configuration rules of the live broadcast scene, camera angle, and special effect triggering comprises the following steps: According to the scene configuration rule in the live broadcast process template, the grid vertex coordinates and the texture coordinate sampling of the three-dimensional scene file corresponding to the scene template identifier are read to generate a virtual live broadcast scene; According to the camera angle and motion rules in the live broadcast process template, the position coordinates and view direction of the virtual camera on the time axis are key frame interpolated to obtain a camera angle switching sequence of multiple preset camera angles, and the transition time between adjacent camera angles in the camera angle switching sequence is parameterized to generate a motion scheduling file; According to the special effect triggering rule in the live broadcast process template, the type identifier of the triggering condition and the instruction code of the execution action are condition-action paired to generate a special effect triggering rule table; The virtual live broadcast scene, the motion scheduling file, and the special effect triggering rule table are associated, and the associated data and the bone node identifiers in the driving calibration file are mapped to generate a live broadcast execution scheme.

4. The full-process intelligent driving virtual human automated interaction method of claim 2, wherein, The step of collecting the driving data of the virtual human model according to the live broadcast execution scheme, synchronizing the driving data to the virtual human model, and generating a live broadcast audio and video stream in combination with the virtual live broadcast scene and audio data comprises the following steps: The original bone data of the current frame is obtained based on the bone node identifier. Optimized bone data is generated by smoothing the original bone data and historical bone data. By mapping the optimized skeletal data with the standard posture parameters of the driving calibration file, the target position data of each skeletal segment is obtained, and the target position data is synchronized to the corresponding skeletal nodes in the virtual human model to generate the driven virtual human model. Audio features are extracted from the audio frame data in the input audio stream to obtain the lip-sync driving parameters corresponding to the current audio frame. The lip-sync driving parameters are then synchronized to the face control node in the driven virtual human model to generate a lip-sync synchronized virtual human model. By rendering the lip-synced virtual human model and scene elements in the virtual live streaming scene frame by frame, the image data of the current frame is obtained. The image data and the corresponding audio frame data are then time-stamp aligned to generate a live audio and video stream.

5. The full-process, intelligently-driven virtual human automated interaction method of claim 1, wherein, The steps of pushing the live audio and video stream to the target live streaming platform, collecting audience interaction data from the target live streaming platform, and triggering a real-time response from the virtual human live stream by parsing the audience interaction data include: According to the platform address list in the live streaming execution scheme, the live audio and video streams are simultaneously encapsulated into transmission protocol packets adapted to multiple target platforms, and the encapsulated transmission protocol packets are pushed to the corresponding streaming platform addresses respectively. The system collects bullet screen text and reward records in real time from the interactive data interfaces of various target live streaming platforms. By performing keyword hierarchical matching on the bullet screen text, it generates trigger keywords and their corresponding priority levels. Based on the comparison results of the reward amount in the reward record with a preset threshold, it generates a reward trigger identifier. Based on the trigger keyword, the priority level, and the reward trigger identifier, the corresponding execution action instruction is called from the special effects trigger rule table in the live streaming execution plan, and the corresponding lip-sync driving parameters are retrieved from the driver calibration file according to the execution action instruction to generate trigger response data; Based on the execution action instructions in the trigger response data, data is injected into the audio track, video track of the live audio and video stream or the bullet screen sending interface of the target live streaming platform to generate an updated live audio and video stream or bullet screen reply message.

6. The full-process, intelligently-driven virtual human automated interaction method of claim 1, wherein, The method further includes: Based on the live streaming topic and live streaming duration parameters in the live streaming task instruction, the text data input by the user is parsed in a structured manner to generate a live streaming script sequence arranged in timestamp order. Based on the timestamp of each speech in the live speech sequence, generate a lip-sync driving parameter sequence for the corresponding time point; By combining the live-streamed speech sequence with the voiceprint feature vector extracted from the audio samples uploaded by the user, the timbre features of each speech text in the live-streamed speech sequence are fused to generate a speech speech sequence with a unique timbre. The speech-to-speech sequence is transcribed to generate a subtitle text sequence with timestamp information. The subtitle text sequence is then aligned and bound to the lip-sync driving parameter sequence based on the timestamp information to generate a synchronized subtitle track.

7. A full-process intelligent driving virtual human automatic interaction system, characterized in that, The fully intelligent-driven virtual human automated interaction system is used to implement the fully intelligent-driven virtual human automated interaction method according to claim 1, wherein the fully intelligent-driven virtual human automated interaction system includes: The acquisition module is used to retrieve the corresponding live streaming process template from the template library based on the scene type identifier in the live streaming task instruction; The live streaming configuration module is used to parse the user-uploaded modeling file according to the image creation rules in the live streaming process template, obtain the virtual human model's driving calibration file, and generate a live streaming execution plan by combining the configuration rules of live streaming scene, camera position, and special effects triggering. The live streaming engine module is used to collect driving data of the virtual human model according to the live streaming execution plan, synchronize the driving data to the virtual human model, and generate live audio and video streams by combining the virtual live streaming scene and audio data. The push-stream interaction module is used to push the live audio and video stream to the target live streaming platform, collect the audience interaction data of the target live streaming platform, and trigger the virtual human live streaming real-time response by parsing the audience interaction data to complete the live streaming.

8. The full-process intelligent driving virtual human automated interaction system of claim 7, wherein, The fully intelligent, automated virtual human interaction system also includes: The AI ​​content production module is used to perform structured parsing of the text data input by the user according to the live broadcast topic and live broadcast duration parameters in the live broadcast task instruction, and generate a live broadcast script sequence arranged in timestamp order. Based on the timestamp of each speech in the live speech sequence, generate a lip-sync driving parameter sequence for the corresponding time point; By combining the live-streamed speech sequence with the voiceprint feature vector extracted from the audio samples uploaded by the user, the timbre features of each speech text in the live-streamed speech sequence are fused to generate a speech speech sequence with a unique timbre. The speech-to-speech sequence is transcribed to generate a subtitle text sequence with timestamp information. The subtitle text sequence is then aligned and bound to the lip-sync driving parameter sequence based on the timestamp information to generate a synchronized subtitle track.

9. An electronic device, comprising: Includes memory and processor, of which: The processor is used to execute computer programs stored in the memory; When the processor executes the computer program, it implements the steps in the fully intelligent virtual human automated interaction method according to any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps in the fully intelligent virtual human automated interaction method according to any one of claims 1 to 7.

Images