Digital avatar real-time broadcast action arrangement method, system and program product

Abstract

The invention belongs to the technical field of man-machine interaction, and particularly discloses a digital avatar real-time broadcast action arrangement method, a digital avatar real-time broadcast action arrangement system and a digital avatar real-time broadcast action arrangement program product. And the aspects of expression correlation, visual continuity, audio and video synchronism, real-time performance and the like of digital avatar interaction can be effectively improved. According to the method, the coupling degree of digital duplicate actions and semantics can be effectively improved, the visual abrupt and frame skipping rate of digital duplicate action switching can be reduced, the audio and video synchronization precision and the time consistency can be improved, the real-time performance can be ensured, and the retrieval and calling delay can be shortened.

Description

Technical Field

[0001] This invention belongs to the field of human-computer interaction technology, specifically relating to a method, system, and program product for arranging real-time broadcasting actions of a digital clone. Background Technology

[0002] In recent years, with the development of artificial intelligence, speech synthesis, and computer vision technologies, digital avatars (or digital humans, virtual humans) have been gradually applied in scenarios such as news broadcasting, intelligent customer service, online education, and interactive entertainment. Existing digital avatar interaction systems mainly rely on pre-recorded video content, using speech synthesis technology combined with lip-sync technology to achieve "voice-lip movement" judgment, lacking real-time personalized action interaction of digital avatars.

[0003] In real-time interactive scenarios, existing technologies often fail to dynamically change facial expressions and gestures based on different emotional keywords in the broadcast content, and the generated actions lack editability and have limited variability. Furthermore, directly splicing pre-recorded video footage can easily cause stuttering, abrupt movements, or frame skipping during transitions, lacking smooth and natural continuity. In summary, existing digital avatar interaction technologies based on pre-recorded video content still have the following problems and shortcomings: 1. Disconnect between action and semantics: Most existing technologies use fixed or predefined action scripts, lacking in-depth analysis of text semantics, and cannot dynamically trigger appropriate actions based on the semantic content of the input text. During the broadcast, body movements and language content often do not correspond. For example, when containing emphasis, question, or emotional words, the digital avatar still outputs neutral or repetitive actions, resulting in a disconnect between language and action performance, making it difficult to convey complete interactive information.

[0004] 2. Unnatural motion splicing: Most existing technologies generate motion by directly splicing together clips from recorded live video. Due to insufficient edge processing, frame skipping, stuttering, or abrupt motion often occur at the transitions between clips. This problem reduces the naturalness and immersion of the digital avatar, and may even cause viewing discomfort for the user.

[0005] 3. Lack of flexible action matching mechanism: Most current solutions rely on simple rule matching for action scheduling, which cannot identify keywords, emotional words or semantic context in the text to trigger corresponding gestures or facial expressions. They lack the ability to schedule actions differently based on text semantics and context, making digital clones appear monotonous when expressing complex contexts and failing to reflect the key points or emotions of the language.

[0006] 4. Audio and video synchronization and latency issues: In real-time interactive scenarios, audio streams and motion video streams often exhibit asynchrony, misalignment between lip movements and speech, and inconsistencies between speech rate and action rhythm, affecting naturalness. Especially in low-latency RTC (Real-Time Communication) streaming scenarios, ensuring simultaneous action matching, video continuity, and real-time transmission is a major challenge.

[0007] 5. The conflict between action library size and retrieval efficiency: To improve expressiveness, current technologies require recording and trimming a large number of action clips. However, an excessively large clip library can lead to reduced retrieval efficiency, while a limited library size cannot meet the diverse needs of various scenarios. Finding a balance between action library scalability and real-time invocation efficiency remains an unresolved issue. Summary of the Invention

[0008] The purpose of this invention is to provide a method, system, and program product for arranging real-time broadcast actions of digital clones, in order to solve the above-mentioned problems existing in the prior art.

[0009] To achieve the above objectives, the present invention adopts the following technical solution: Firstly, a method for arranging real-time broadcast actions for digital clones is provided, including: Receive interactive text transmitted from the user terminal, call the large language model to perform semantic analysis on the interactive text, and generate the corresponding response text; The response text is subjected to structured parsing and keyword extraction to obtain a set of hot words; The large language model is called to perform vector encoding on each hot word in the hot word set to obtain the hot word vector of each hot word; The similarity analysis is performed between each hot word vector and the action semantic vector associated with each action segment in the action segment library. Then, each action segment whose similarity with the corresponding hot word vector meets the set similarity condition is retrieved to form the initial action segment candidate set for the corresponding hot word vector. The union of the initial action fragment candidate sets for each hot word vector is used to obtain the candidate action set; The action segments in the candidate action set are sequentially arranged and transition frames are processed to obtain the action segment sequence; The response text is processed into text-to-speech to generate a response audio stream, and then the response audio stream is aligned and mapped with the action segment sequence. The motion segment sequence after being aligned and mapped with the response audio stream is spliced ​​and rendered to obtain a smooth motion sequence. The smooth motion sequence is then used to drive the motion of a preset digital clone model to generate a digital clone video stream. The response audio stream and the digital clone video stream are encapsulated and processed to obtain the corresponding audio and video media streams, which are then encoded and pushed to the user terminal.

[0010] In one possible design, the method further includes: It receives interactive voice transmitted from the user terminal, performs speech recognition processing on the interactive voice to obtain interactive text, and calls a large language model to perform semantic analysis on the interactive text to generate corresponding response text.

[0011] In one possible design, before performing similarity analysis between the hot word vectors and the action semantic vectors associated with each action segment in the action segment library, the method further includes: Construct an action fragment library, which contains several action fragments and the action semantic vectors, semantic tags, keyframe sequences and metadata associated with each action fragment.

[0012] In one possible design, the similarity analysis is performed between each hot word vector and the action semantic vector associated with each action segment in the action segment library. Then, action segments whose similarity to the corresponding hot word vector satisfies a set of predefined similarity conditions are retrieved, forming an initial candidate set of action segments for the corresponding hot word vector. This includes: Calculate the cosine similarity between each hot word vector and the action semantic vector associated with each action segment in the action segment library; The action segments whose cosine similarity with the corresponding hot word vector exceeds a set similarity threshold are selected to form the initial action segment candidate set for the corresponding hot word vector.

[0013] In one possible design, the sequential arrangement and transition frame processing of each action segment in the candidate action set to obtain an action segment sequence includes: A weighted directed graph-based action arrangement method is used to sequentially arrange the action segments in the candidate action set to obtain the optimal action sequence; A frame transition algorithm based on motion interpolation is used to generate a transition frame sequence between the end frame and the start frame of two adjacent action segments in the optimal action sequence, thus obtaining the action segment sequence.

[0014] In one possible design, the step of splicing and rendering the sequence of motion segments aligned and mapped to the response audio stream to obtain a smooth motion sequence, and using the smooth motion sequence to drive the actions of a preset digital clone model to generate a digital clone video stream, includes: The motion segments in the motion segment sequence after being aligned and mapped with the response audio stream are spliced ​​together in sequence, and the transition frame sequence in each motion segment is rendered to obtain a smooth motion sequence. Apply a smooth motion sequence to a preset digital clone model to drive the digital clone model's movements and generate a digital clone video stream.

[0015] In one possible design, the step of performing audio-video synchronization encapsulation processing on the response audio stream and the digital clone video stream to obtain the corresponding audio-video media stream, and then encoding the audio-video media stream and pushing it to the user terminal includes: The response audio stream and the digital clone video stream are synchronously encapsulated according to a unified timeline to obtain the corresponding audio and video media streams; Audio and video media streams are pushed to the user's device after being encoded with low latency.

[0016] Secondly, a real-time broadcasting action arrangement system for digital avatars is provided, including an information receiving unit, a hot word extraction unit, a hot word encoding unit, an action candidate unit, an action combination unit, an arrangement transition unit, an audio generation unit, a video generation unit, and a packaging and push unit, wherein: The information receiving unit is used to receive interactive text transmitted by the user terminal, and call the large language model to perform semantic analysis on the interactive text to generate corresponding response text. The hot word extraction unit is used to perform structured parsing and keyword extraction on the response text to obtain a set of hot words; The hot word encoding unit is used to call the large language model to perform vectorization encoding on each hot word in the hot word set, and obtain the hot word vector of each hot word. The action candidate unit is used to perform similarity analysis between each hot word vector and the action semantic vector associated with each action segment in the action segment library, and to retrieve each action segment whose similarity with the corresponding hot word vector meets the set similarity conditions, forming the initial action segment candidate set of the corresponding hot word vector. The action combination unit is used to take the union of the initial action fragment candidate sets of each hot word vector to obtain the candidate action set; The transition unit is used to sequentially arrange and process transition frames for each action segment in the candidate action set to obtain an action segment sequence. The audio generation unit is used to perform text-to-speech processing on the response text, generate the response audio stream, and perform alignment mapping processing between the response audio stream and the action segment sequence; The video generation unit is used to splice and render the sequence of motion segments aligned and mapped with the response audio stream to obtain a smooth motion sequence, and use the smooth motion sequence to drive the motion of the preset digital clone model to generate a digital clone video stream. The encapsulation and push unit is used to encapsulate the response audio stream and the digital clone video stream in a synchronized audio-video manner to obtain the corresponding audio and video media streams, and then encodes and pushes the audio and video media streams to the user terminal.

[0017] Thirdly, a real-time broadcasting motion arrangement system for digital clones is provided, including: Memory, used to store instructions; The processor is configured to read instructions stored in the memory and execute any one of the digital clone real-time broadcasting action arrangement methods described in the first aspect above, according to the instructions.

[0018] Fourthly, a computer-readable storage medium is provided, on which instructions are stored, which, when executed on a computer, cause the computer to perform any one of the digital clone real-time broadcasting action arrangement methods described in the first aspect. Simultaneously, a computer program product is also provided, which, when executed on a computer, performs any one of the digital clone real-time broadcasting action arrangement methods described in the first aspect.

[0019] Beneficial effects: This invention organically combines text semantic analysis, action segment normalization, action segment transition processing, and audio-visual two-way synchronization technologies, effectively improving the expressive relevance, visual continuity, audio-visual synchronization, and real-time performance of digital avatar interaction. Specifically, it includes: 1. Improve the coupling between digital avatar actions and semantics. By semantically parsing interactive text and mapping the parsing results to corresponding action sequences, the triggered actions can be more aligned with the text intent, thereby reducing mismatches between actions and semantics and enhancing the completeness and accuracy of information delivery.

[0020] 2. Reduce visual abruptness and frame skipping rate in digital clone action transitions. By standardizing the action sequence and performing necessary frame interpolation motion compensation on the action segments, continuous actions are visually seamless, significantly reducing abruptness and the sense of frame loss, thereby improving visual continuity and smoothness of the viewing experience.

[0021] 3. Improve audio and video synchronization accuracy and time consistency. By aligning and mapping the response audio stream with the action sequence and performing audio and video synchronization encapsulation processing on the response audio stream and the digital clone video stream, bidirectional synchronization can reduce lip-syncing misalignment and motion latency in the digital clone, maintaining overall consistency even in network jitter or low-latency streaming scenarios.

[0022] 4. Ensure real-time performance and reduce retrieval and retrieval latency. By establishing an efficient indexing mechanism for action clips, the retrieval efficiency and action retrieval response speed are significantly improved while ensuring the coverage of the action clip library. In real-time interaction, the latency of action selection and loading can be controlled within an acceptable range, thereby avoiding playback interruptions or significant delays caused by retrieval waiting. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart illustrating the method in Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of the system configuration in Embodiment 2 of the present invention; Figure 3 This is a schematic diagram of the system configuration in Embodiment 3 of the present invention. Detailed Implementation

[0025] It should be noted that the descriptions of these embodiments are intended to aid in understanding the invention and do not constitute a limitation thereof. The specific structural and functional details disclosed herein are merely for describing exemplary embodiments of the invention. However, the invention may be embodied in many alternative forms and should not be construed as being limited to the embodiments described herein.

[0026] It should be understood that, unless otherwise explicitly specified and limited, the corresponding terms should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be a connection within two components. Those skilled in the art can understand the specific meaning of the above terms in the embodiments according to the specific circumstances.

[0027] Specific details are provided in the following description to provide a complete understanding of the exemplary embodiments. However, those skilled in the art will understand that the exemplary embodiments can be implemented without these specific details. For example, apparatus may be shown in block diagrams to avoid obscuring the examples with unnecessary details. In other embodiments, well-known processes, structures, and techniques may be omitted with non-essential details to avoid obscuring the embodiments.

[0028] Example 1: This embodiment provides a method for arranging real-time broadcast actions for digital clones, which can be applied to corresponding interactive service systems, such as... Figure 1 As shown, the method includes the following steps: S1. Receive the interactive text transmitted by the user, and call the large language model to perform semantic analysis on the interactive text to generate the corresponding response text.

[0029] In practice, users can input interactive text through a front-end interface on their client devices. The system receives the interactive text transmitted from the client and then uses a large language model (such as ChatGPT or Deepseek) to perform semantic analysis on the interactive text, generating the corresponding response text. Alternatively, users can input interactive voice through a front-end interface on their client devices. The system receives the interactive voice transmitted from the client devices, performs speech recognition processing on the interactive voice (such as using open-source speech recognition frameworks like Whisper or PaddleSpeech), obtains the interactive text, and then uses a large language model to perform semantic analysis on the interactive text, generating the corresponding response text.

[0030] The system invokes a large language model to perform deep semantic analysis on the interactive text, aiming to extract the structured semantic information of the interactive text, mainly including: Word segmentation and part-of-speech tagging: Identifying the basic lexical units and grammatical roles of text.

[0031] Dependency parsing: analyzing logical relationships such as modification, subject-verb-object, etc. between words.

[0032] Semantic role labeling: Analyze the predicate-argument structure in a sentence to identify semantic components such as core action, agent, and patient.

[0033] Based on the results of deep semantic analysis, the large language model can generate corresponding response text, that is, generate a structured semantic representation that can formally express the corresponding semantic information.

[0034] S2. Perform structured parsing and keyword extraction on the response text to obtain a set of hot words.

[0035] In practice, the system can perform structured parsing and keyword extraction on the structured response text to form a hot word set K={k1, k2, ..., k n The hot words in the hot word set are keywords or semantic units (such as "greeting", "refusal", "emphasis") identified based on semantic analysis that can be used to trigger the digital clone to perform specific actions. The extraction logic can be expressed as follows: for each semantic unit in the structured semantic representation, if its semantic label matches the predefined action trigger library, it is included in the hot word set K.

[0036] S3. Call the large language model to perform vectorization encoding on each hot word in the hot word set to obtain the hot word vector of each hot word.

[0037] In practice, the system can call a large language model to perform vectorization encoding on each hot word in the hot word set, obtaining a high-dimensional vector for each hot word, i.e., the hot word vector E. keyword (k) i ).

[0038] S4. Perform similarity analysis between each hot word vector and the action semantic vector associated with each action segment in the action segment library, and retrieve each action segment whose similarity with the corresponding hot word vector meets the set similarity conditions to form the initial action segment candidate set of the corresponding hot word vector.

[0039] In practice, the system pre-builds an action fragment library, which contains several action fragments {V1, V2, ..., V...}. m} and the action semantic vectors, semantic tags, keyframe sequences, and metadata associated with each action segment, where: Semantic tags: Describe the intention expressed by the action in text form, such as "waving", "nodding in affirmation", "shaking head in refusal".

[0040] Action semantic vector: the action semantic vector E corresponding to the action segment. action (V) j ).

[0041] Keyframe sequence: includes start frame, end frame, and key intermediate pose frames.

[0042] Metadata: such as the expected duration of an action, the intensity of emotion, etc.

[0043] The system can calculate the vector E of each hot word. keyword (k) i The action semantic vector E associated with each action segment in the action segment library action (V) j The cosine similarity between the two is used to retrieve their associated action semantic vector E. action (V) j ) and corresponding hot word vector E keyword (k) i Action segments whose cosine similarity to each other exceeds a set similarity threshold constitute the initial action segment candidate set for the corresponding hot word vectors. S5. Take the union of the initial action fragment candidate sets of each hot word vector to obtain the candidate action set.

[0044] In practice, the system takes the union of the initial action fragment candidate sets of each hot word vector to form the final candidate action set A = {A1, A2, ..., A...} p}

[0045] S6. Arrange the action segments in the candidate action set in sequence and process the transition frames to obtain the action segment sequence.

[0046] In practical implementation, the system can use a weighted directed graph-based action orchestration method to sequentially orchestrate each action segment in the candidate action set, obtaining a natural and coherent optimal action sequence S=(A1→A2→…A…). p Weighted directed graph-based motion orchestration is an optimization method that models a sequence of tasks or actions as a graph structure, where nodes represent states or actions, edges represent state transitions or action executions, and weights represent execution costs, time, or resource consumption. This method is widely used in fields such as robot motion design, automated process design, and game AI decision-making.

[0047] Meanwhile, to ensure natural and smooth action transitions in the optimal action sequence S, the system can employ a motion interpolation-based frame transition algorithm to generate a transition frame sequence between the end and start frames of two adjacent action segments in the optimal action sequence, thus obtaining the action segment sequence. Specifically, in action segment A... i End frame F end (A) i ) and action segment A i+1 The starting frame F strat (A) i+1 Between ), a transition frame sequence is generated, and the transition frame F in the transition frame sequence is... trans (t) is calculated using the following interpolation function: F trans (t) = α(t) F end (A) i )+[1 α(t)] F strat (A) i+1 ) Here, α(t) is a time function that smoothly transitions from 1 to 0, and F represents the pose parameters of the corresponding frame, a vectorized dataset used to accurately describe the skeletal drive parameters or morphological features of the digital clone in that frame. By generating a sequence of transition frames, the system can effectively avoid mechanical jumps between actions.

[0048] S7. Perform text-to-speech processing on the response text to generate a response audio stream, and perform alignment mapping processing on the response audio stream and the action segment sequence.

[0049] In practice, the system performs text-to-speech processing on the response text (e.g., using open-source speech synthesis frameworks such as PaddleSpeech) to generate a response audio stream A with rich prosodic features. u (t). Then, the system extracts A. uThe prosodic boundaries (pauses, stresses, speech rate) of (t) are aligned and mapped with the sequence of action segments. This process can be achieved through a corresponding alignment mapping function, which associates audio with the action sequence, for each action segment A. i Assign a specific start timestamp and duration Δt. i , forming a mapping set T→{(A i , △t i This ensures that the actions and speech are accurately matched on the timeline.

[0050] S8. The motion segment sequence after being aligned and mapped with the response audio stream is spliced ​​and rendered to obtain a smooth motion sequence. The smooth motion sequence is then used to drive the preset digital clone model to generate a digital clone video stream.

[0051] In practice, the system can allocate Δt based on the duration of each action segment in the action segment sequence obtained from the aforementioned steps. i The process involves sequentially splicing together the action segments in the action sequence and rendering transition frames within each action segment to achieve a visually seamless transition, resulting in a smooth action sequence. This smooth action sequence is then applied to a pre-set digital avatar model, driving the model's movements and generating the corresponding digital avatar video stream V. out (t).

[0052] S9. Perform audio-video synchronization encapsulation processing on the response audio stream and the digital clone video stream to obtain the corresponding audio-video media stream, and then push the encoded audio-video media stream to the user terminal.

[0053] In practical implementation, the system will use digital clone video stream V out (t) and the response audio stream A u (t) Synchronous encapsulation is performed according to a unified timeline to ensure that the lip movements, actions, and audio of the video are precisely synchronized in every frame, ultimately synthesizing an audio and video media stream. The system then encodes the audio and video media stream using a low-latency encoding protocol, and subsequently pushes the audio and video media stream to the user terminal via WebRTC or a similar real-time communication protocol. After receiving the encoded audio and video media stream, the user terminal decodes and renders it in real time, allowing the user to watch a real-time interactive, natural-looking, and synchronized digital avatar broadcast.

[0054] This method organically combines text semantic analysis, action segment normalization, action segment transition processing, and audio-visual two-way synchronization techniques to effectively improve the expressive relevance, visual continuity, audio-visual synchronization, and real-time performance of digital avatar interactions. It can effectively enhance the coupling between digital avatar actions and semantics, reduce visual abruptness and frame skipping rates during action transitions, improve audio-visual synchronization accuracy and temporal consistency, ensure real-time performance, and shorten retrieval and recall latency.

[0055] Example 2: This embodiment provides a digital clone real-time broadcasting action arrangement system, such as... Figure 2 As shown, it includes an information receiving unit, a hot word extraction unit, a hot word encoding unit, an action candidate unit, an action combination unit, an arrangement and transition unit, an audio generation unit, a video generation unit, and a packaging and push unit, wherein: The information receiving unit is used to receive interactive text transmitted by the user terminal, and call the large language model to perform semantic analysis on the interactive text to generate corresponding response text. The hot word extraction unit is used to perform structured parsing and keyword extraction on the response text to obtain a set of hot words; The hot word encoding unit is used to call the large language model to perform vectorization encoding on each hot word in the hot word set, and obtain the hot word vector of each hot word. The action candidate unit is used to perform similarity analysis between each hot word vector and the action semantic vector associated with each action segment in the action segment library, and to retrieve each action segment whose similarity with the corresponding hot word vector meets the set similarity conditions, forming the initial action segment candidate set of the corresponding hot word vector. The action combination unit is used to take the union of the initial action fragment candidate sets of each hot word vector to obtain the candidate action set; The transition unit is used to sequentially arrange and process transition frames for each action segment in the candidate action set to obtain an action segment sequence. The audio generation unit is used to perform text-to-speech processing on the response text, generate the response audio stream, and perform alignment mapping processing between the response audio stream and the action segment sequence; The video generation unit is used to splice and render the sequence of motion segments aligned and mapped with the response audio stream to obtain a smooth motion sequence, and use the smooth motion sequence to drive the motion of the preset digital clone model to generate a digital clone video stream. The encapsulation and push unit is used to encapsulate the response audio stream and the digital clone video stream in a synchronized audio-video manner to obtain the corresponding audio and video media streams, and then encodes and pushes the audio and video media streams to the user terminal.

[0056] Example 3: This embodiment provides a digital clone real-time broadcasting action arrangement system, such as... Figure 3 As shown, at the hardware level, it includes: The data interface is used to establish data communication between the processor and the user terminal; Memory, used to store instructions; The processor is used to read the instructions stored in the memory and execute the digital clone real-time broadcasting action arrangement method in Embodiment 1 according to the instructions.

[0057] Optionally, the system also includes an internal bus, through which the processor, memory, and data interface can be interconnected. This internal bus can be a PCIe (Peripheral Component Interconnect Eexpress) bus, which can be divided into an address bus, a data bus, a control bus, etc. The memory can include, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Flash Memory, First Input First Output (FIFO), and / or First In Last Out (FILO). The processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0058] Example 4: This embodiment provides a computer-readable storage medium storing instructions. When these instructions are executed on a computer, the computer performs the digital clone real-time broadcasting action arrangement method described in Embodiment 1. The computer-readable storage medium refers to a data storage medium, which may include, but is not limited to, floppy disks, optical disks, hard disks, flash memory, USB flash drives, and / or Memory Sticks. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0059] This embodiment also provides a computer program product that, when run on a computer, executes the digital clone real-time broadcasting action arrangement method in Embodiment 1. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0060] Finally, it should be noted that the above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for arranging actions for real-time broadcasting of digital clones, characterized in that, include: Receive interactive text transmitted from the user terminal, call the large language model to perform semantic analysis on the interactive text, and generate the corresponding response text; The response text is subjected to structured parsing and keyword extraction to obtain a set of hot words; The large language model is called to perform vector encoding on each hot word in the hot word set to obtain the hot word vector of each hot word; The similarity analysis is performed between each hot word vector and the action semantic vector associated with each action segment in the action segment library. Then, each action segment whose similarity with the corresponding hot word vector meets the set similarity condition is retrieved to form the initial action segment candidate set for the corresponding hot word vector. The union of the initial action fragment candidate sets for each hot word vector is used to obtain the candidate action set; The action segments in the candidate action set are sequentially arranged and transition frames are processed to obtain the action segment sequence; The response text is processed into text-to-speech to generate a response audio stream, and then the response audio stream is aligned and mapped with the action segment sequence. The motion segment sequence after being aligned and mapped with the response audio stream is spliced ​​and rendered to obtain a smooth motion sequence. The smooth motion sequence is then used to drive the motion of a preset digital clone model to generate a digital clone video stream. The response audio stream and the digital clone video stream are encapsulated and processed to obtain the corresponding audio and video media streams, which are then encoded and pushed to the user terminal.

2. The method for arranging actions for real-time broadcasting of digital clones according to claim 1, characterized in that, The method further includes: It receives interactive voice transmitted from the user terminal, performs speech recognition processing on the interactive voice to obtain interactive text, and calls a large language model to perform semantic analysis on the interactive text to generate corresponding response text.

3. The method for arranging real-time broadcast actions for digital clones according to claim 1, characterized in that, Before performing similarity analysis between each hot word vector and the action semantic vectors associated with each action segment in the action segment library, the method further includes: Construct an action fragment library, which contains several action fragments and the action semantic vectors, semantic tags, keyframe sequences and metadata associated with each action fragment.

4. The method for arranging actions for real-time broadcasting of digital clones according to claim 1, characterized in that, The step involves performing similarity analysis between each hot word vector and the action semantic vector associated with each action segment in the action segment library, and retrieving action segments whose similarity with the corresponding hot word vector meets the set similarity conditions to form an initial action segment candidate set for the corresponding hot word vector, including: Calculate the cosine similarity between each hot word vector and the action semantic vector associated with each action segment in the action segment library; The action segments whose cosine similarity with the corresponding hot word vector exceeds a set similarity threshold are selected to form the initial action segment candidate set for the corresponding hot word vector.

5. The method for arranging actions for real-time broadcasting of digital clones according to claim 1, characterized in that, The process of sequentially arranging and processing transition frames for each action segment in the candidate action set to obtain an action segment sequence includes: A weighted directed graph-based action arrangement method is used to sequentially arrange the action segments in the candidate action set to obtain the optimal action sequence; A frame transition algorithm based on motion interpolation is used to generate a transition frame sequence between the end frame and the start frame of two adjacent action segments in the optimal action sequence, thus obtaining the action segment sequence.

6. The method for arranging actions for real-time broadcasting of digital clones according to claim 5, characterized in that, The process of splicing and rendering the motion segment sequence aligned and mapped with the response audio stream to obtain a smooth motion sequence, and using the smooth motion sequence to drive the actions of a preset digital clone model to generate a digital clone video stream, includes: The motion segments in the motion segment sequence after being aligned and mapped with the response audio stream are spliced ​​together in sequence, and the transition frame sequence in each motion segment is rendered to obtain a smooth motion sequence. Apply a smooth motion sequence to a preset digital clone model to drive the digital clone model's movements and generate a digital clone video stream.

7. The method for arranging real-time broadcast actions for digital clones according to claim 1, characterized in that, The step of performing audio-video synchronization encapsulation processing on the response audio stream and the digital clone video stream to obtain the corresponding audio-video media stream, and then encoding the audio-video media stream and pushing it to the user terminal includes: The response audio stream and the digital clone video stream are synchronously encapsulated according to a unified timeline to obtain the corresponding audio and video media streams; Audio and video media streams are pushed to the user's device after being encoded with low latency.

8. A digital clone real-time broadcasting motion choreography system, characterized in that, It includes an information receiving unit, a hot word extraction unit, a hot word encoding unit, an action candidate unit, an action combination unit, an arrangement and transition unit, an audio generation unit, a video generation unit, and a packaging and push unit, wherein: The information receiving unit is used to receive interactive text transmitted by the user terminal, and call the large language model to perform semantic analysis on the interactive text to generate corresponding response text. The hot word extraction unit is used to perform structured parsing and keyword extraction on the response text to obtain a set of hot words; The hot word encoding unit is used to call the large language model to perform vectorization encoding on each hot word in the hot word set, and obtain the hot word vector of each hot word. The action candidate unit is used to perform similarity analysis between each hot word vector and the action semantic vector associated with each action segment in the action segment library, and to retrieve each action segment whose similarity with the corresponding hot word vector meets the set similarity conditions, forming the initial action segment candidate set of the corresponding hot word vector. The action combination unit is used to take the union of the initial action fragment candidate sets of each hot word vector to obtain the candidate action set; The transition unit is used to sequentially arrange and process transition frames for each action segment in the candidate action set to obtain an action segment sequence. The audio generation unit is used to perform text-to-speech processing on the response text, generate the response audio stream, and align and map the response audio stream with the action segment sequence. The video generation unit is used to splice and render the sequence of motion segments aligned and mapped with the response audio stream to obtain a smooth motion sequence, and use the smooth motion sequence to drive the motion of the preset digital clone model to generate a digital clone video stream. The encapsulation and push unit is used to encapsulate the response audio stream and the digital clone video stream in a synchronized audio-video manner to obtain the corresponding audio and video media streams, and then encodes and pushes the audio and video media streams to the user terminal.

9. A digital clone real-time broadcasting motion choreography system, characterized in that, include: Memory, used to store instructions; A processor is configured to read instructions stored in the memory and execute the digital clone real-time broadcasting action arrangement method according to any one of claims 1-7.

10. A computer program product, characterized in that, When the computer program product is run on a computer, it executes the digital clone real-time broadcasting action arrangement method according to any one of claims 1-7.

Images