Video generation method and device based on multi-flow cross attention, equipment and medium

Abstract

The application relates to the technical field of image processing, and discloses a video generation method and device based on multi-flow cross attention, equipment and a medium, which comprises the following steps: acquiring multiple entity images, multiple audio streams and a text prompt, generating an entity image region sequence, a region mask sequence, a text feature vector and a mapping relationship; extracting and aligning entity features, audio features and speaking state features; generating local driving features through cross attention, speaking state adjustment and region mask constraint; and fusing the local driving features and combining the text features to generate a multi-entity interaction video. The application can be applied to business scenes such as financial technology and medical health, the entities, audio, speaking state and region mask are kept corresponding through the mapping relationship and sequence alignment, the audio driving range is limited through cross attention and region mask, so that driving mismatch, abnormal motion of non-speaking entities and region interference are reduced.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a video generation method, apparatus, device, and medium based on multi-stream cross-attention. Background Technology

[0002] Existing multi-person speech video generation technologies face a key challenge when processing multiple entity images, multiple audio streams, and text prompts: the correspondence between multi-channel audio driving information and different entity image regions is prone to misalignment. Current methods typically rely on fixed identity markers, preset audio channels, or static binding for multi-stream fusion. When the speaker switches, different audio stream states change continuously, or entity image regions are spatially adjacent, it becomes difficult to maintain a consistent correspondence between entity image regions, audio streams, speaking states, and region masks. This leads to audio driving incorrectly applying to non-corresponding entity image regions, resulting in lip-syncing errors, abnormal movement of non-speaking objects, and interference from adjacent entity region driving mechanisms.

[0003] In the fintech business sector, robo-advisory presentations, online wealth management consultations, financial product roadshows, risk disclosure videos, remote client manager services, and compliance training videos typically involve multiple participants, including financial advisors, clients, and auditors. Existing methods for generating multi-person spoken videos in such scenarios often fail to reliably maintain the relationship between different financial business participants and their corresponding audio streams. This can lead to issues such as client voices driving financial advisor lip movements, risk warning audio corresponding to the wrong role, and abnormal lip movements by non-speakers, resulting in inconsistencies in role representation, risk disclosure, and compliance explanations within the financial business videos.

[0004] In the healthcare field, remote consultations, online consultations, health education, rehabilitation guidance, and multi-expert case discussions typically involve multi-party communication among doctors, patients, nurses, or experts. Existing methods for generating multi-person spoken video, when multiple entity image regions and multiple audio streams are input simultaneously, can easily lead to mismatches between the speaker, the speaking state, and the entity image region if they cannot maintain a stable correspondence. This can result in inaccurate lip-sync, abnormal responses from non-speaking individuals, and interference at entity region boundaries, ultimately affecting the clarity of doctor-patient communication and health guidance content in healthcare videos. Summary of the Invention

[0005] The main objective of this invention is to provide a video generation method, apparatus, device, and storage medium based on multi-stream cross-attention, aiming to solve the technical problem in the prior art that when multiple entity images and multiple audio streams are input together, it is difficult to maintain stable alignment between entity image regions, audio streams, speech state features, and binary space masks of entity image regions, which leads to audio drivers easily acting on incorrect entity regions.

[0006] To achieve the above objectives, the present invention provides a video generation method based on multi-stream cross-attention, comprising: Acquire multiple entity images, multiple audio streams, and text prompts; generate entity image region sequences and entity image region binary space mask sequences based on the multiple entity images; establish a mapping relationship between the entity image region sequences and the multiple audio streams; and extract text feature vectors from the text prompts. Feature extraction is performed on the entity image region sequence to generate an entity feature vector sequence. Temporal encoding and speech state extraction are performed on the multiple audio streams to generate an audio feature vector sequence and a speech state feature sequence. Based on the mapping relationship, the entity feature vector sequence, the audio feature vector sequence, the speech state feature sequence, and the entity image region binary space mask sequence are aligned with each other. Traverse the entity feature vector sequence, and for each current entity feature vector obtained in each traversal, read the current audio feature vector, the current speaking state feature, and the current entity image region binary space mask that are aligned with the same sequence position; The current entity feature vector is used as the query feature, and the current audio feature vector is used as the key feature and value feature to perform cross-attention operation to generate basic driving features; Based on the current speaking state features, the driving weights of the basic driving features are adjusted to generate the current entity audio driving features. The current entity image region binary space mask is used to constrain the current entity audio driving features to generate the current local region driving features. After the entity feature vector sequence is traversed, the current local region driving features generated in each traversal are normalized and fused to generate fused multi-stream features. Based on the text feature vector and the fused multi-stream features, the video latent representation is generated and decoded to obtain a multi-entity interactive video.

[0007] Furthermore, to achieve the above objectives, the present invention provides a video generation apparatus based on multi-stream cross-attention, comprising: The input mapping preprocessing module is used to acquire multiple entity images, multiple audio streams and text prompts, generate entity image region sequences and entity image region binary space mask sequences based on the multiple entity images, establish the mapping relationship between the entity image region sequences and the multiple audio streams, and extract text feature vectors from the text prompts; The multi-stream feature alignment module is used to extract features from the entity image region sequence to generate an entity feature vector sequence, perform temporal encoding and speech state extraction on the multiple audio streams to generate an audio feature vector sequence and a speech state feature sequence, and align the entity feature vector sequence, the audio feature vector sequence, the speech state feature sequence, and the entity image region binary space mask sequence with each other based on the mapping relationship. The alignment position reading module is used to traverse the entity feature vector sequence and, for each traversal, read the current audio feature vector, the current speaking state feature, and the current entity image region binary space mask that are aligned at the same sequence position. The cross-attention driven module is used to perform cross-attention operations on the current entity feature vector as a query feature and the current audio feature vector as a key feature and value feature to generate basic driving features; The state weight constraint module is used to adjust the driving weight of the basic driving feature based on the current speaking state feature, generate the current entity audio driving feature, and constrain the current entity audio driving feature using the binary space mask of the current entity image region to generate the current local region driving feature. The fusion decoding generation module is used to normalize and fuse the current local region driving features generated in each traversal after the entity feature vector sequence is traversed, generate fused multi-stream features, generate and decode video latent representations based on the text feature vectors and the fused multi-stream features, and obtain multi-entity interactive video.

[0008] Furthermore, to achieve the above objectives, the present invention also provides a computer device, the computer device including a memory, a processor, and a video generation program based on multi-stream cross-attention stored in the memory and executable on the processor, wherein when the video generation program based on multi-stream cross-attention is executed by the processor, it implements the steps of the video generation method based on multi-stream cross-attention as described above.

[0009] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing a video generation program based on multi-stream cross-attention, wherein the video generation program based on multi-stream cross-attention, when executed by a processor, implements the steps of the video generation method based on multi-stream cross-attention as described above.

[0010] Beneficial Effects: This invention relates to the field of image processing technology and discloses a video generation method, apparatus, device, and medium based on multi-stream cross-attention. The method includes: acquiring multiple entity images, multiple audio streams, and text prompts; generating entity image region sequences, entity image region binary space mask sequences, and text feature vectors; establishing a mapping relationship between the entity image region sequences and multiple audio streams; extracting features from the entity image region sequences and multiple audio streams to generate entity feature vector sequences, audio feature vector sequences, and speech state feature sequences, and aligning them based on the mapping relationship; traversing the entity feature vector sequences and reading the current audio feature vector, current speech state feature, and current entity image region binary space mask at the alignment position of the same sequence; generating basic driving features through cross-attention operations, and generating current local region driving features through speech state adjustment and mask constraints; normalizing and fusing the current local region driving features generated in each traversal, and combining them with text feature vectors to generate a multi-entity interactive video. This invention can be applied to business scenarios such as fintech and healthcare. By mapping relationships and sequence alignment, entity features, audio features, speaking states, and region masks are kept in correspondence. Through cross-attention operations, speaking state adjustment, and mask constraints, audio driving is applied to the corresponding entity regions. By normalizing and fusing the results of each local driving, audio driving mismatch, abnormal movement of non-speaking entities, and driving interference from adjacent entity regions are reduced. Attached Figure Description

[0011] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings: Figure 1 This is a schematic diagram of an application environment for a video generation method based on multi-stream cross-attention in one embodiment of the present invention; Figure 2 This is a flowchart illustrating an embodiment of the video generation method based on multi-stream cross-attention according to the present invention; Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the video generation device based on multi-stream cross-attention of the present invention; Figure 4 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 5 This is another structural schematic diagram of a computer device according to one embodiment of the present invention; Figure 6 This is a flowchart illustrating a video generation method based on multi-stream cross-attention in one embodiment of the present invention. Detailed Implementation

[0012] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0013] The video generation method based on multi-stream cross-attention provided in this invention can be applied to, for example... Figure 1 In this application environment, the client communicates with the server via a network. The server can obtain multiple entity images, multiple audio streams, and text prompts from the client, generate entity image region sequences, entity image region binary space mask sequences, and text feature vectors, and establish a mapping relationship between the entity image region sequences and multiple audio streams; perform feature extraction on the entity image region sequences and multiple audio streams to generate entity feature vector sequences, audio feature vector sequences, and speech state feature sequences, and complete mutual alignment based on the mapping relationship; traverse the entity feature vector sequences, read the current audio feature vector, current speech state feature, and current entity image region binary space mask at the alignment position of the same sequence; generate basic driving features through cross-attention operation, and generate current local region driving features through speech state adjustment and mask constraints; normalize and fuse the current local region driving features generated in each traversal, and combine them with text feature vectors to generate multi-entity interactive video. This invention can be applied to business scenarios such as fintech and healthcare. Through mapping relationships and sequence alignment, it maintains the correspondence between entity features, audio features, speaking states, and region masks. Through cross-attention operations, speaking state adjustment, and mask constraints, it ensures that audio drivers act on corresponding entity regions. By normalizing and fusing the results of each local driver, it reduces audio driver mismatch, abnormal movement of non-speaking entities, and interference from adjacent entity region drivers. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will be described in detail below through specific embodiments.

[0014] Please see Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of the video generation method based on multi-stream cross-attention provided by the present invention. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0015] like Figure 2 As shown, the video generation method based on multi-stream cross-attention proposed in this invention includes the following steps: S10: Acquire multiple entity images, multiple audio streams, and text prompts; generate entity image region sequences and entity image region binary space mask sequences based on the multiple entity images; establish a mapping relationship between the entity image region sequences and the multiple audio streams; and extract text feature vectors from the text prompts. In this embodiment, multiple entity images represent the source of the object appearance in the image to be generated. These images can originate from single-frame images, video frame extractions, virtual character images, or object images from the business interaction interface. After multiple entity images are input, they need to undergo size normalization, channel format adjustment, sharpness filtering, and duplicate region exclusion to prevent low-quality images from participating in subsequent region generation. When generating an entity image region sequence based on multiple entity images, target region detection, key point localization, and region segmentation can be used to jointly determine the entity image regions. An entity image region represents the spatial range that can participate in audio-driven and image generation, and can include the movable areas of the face, head, upper body, or virtual objects. The entity image region sequence represents the arrangement of multiple entity image regions according to the input order, screen spatial position, object reference relationship in text prompts, or business role order. In implementation, visual detection is performed on multiple entity images to output candidate regions. Key point localization is performed on the candidate regions to output the positions of movable parts. Region segmentation is performed based on the candidate regions and key point ranges to form entity image regions. Then, multiple entity image regions are arranged into an entity image region sequence according to the input order, screen spatial position, object reference relationship in text prompts, or business role order.

[0016] The entity image region binary space mask sequence originates from the entity image region sequence. The entity image region binary space mask represents the effective driving range of the entity image region in the image space. It can be in matrix form consistent with the image size or in compressed region labeling form. The binary representation distinguishes between driving and non-driving regions, and the spatial representation indicates a positional relationship between the mask and image coordinates. In implementation, the entity image region binary space mask can be generated based on the contour boundaries, keypoint enclosure range, and segmented foreground region of the entity image region. Valid labels are assigned to the interior of the entity image region, while invalid labels are assigned to the exterior and adjacent object regions. Multiple entity image region binary space masks are arranged according to the entity image region sequence to form an entity image region binary space mask sequence, ensuring that each entity image region obtains spatial constraint data of the same order in subsequent processing.

[0017] Multiple audio streams provide sound sources for different objects, which can come from independent audio tracks, multi-channel speech, separated speaker speech, business script text-synthesized speech, or real-time acquired speech. The multiple audio streams may not have the same order as the entity image region sequence at input; therefore, a mapping relationship needs to be established between the entity image region sequence and the multiple audio streams. This mapping relationship can be formed by role tags, channel tags, object reference segments in text prompts, audio stream names, sound source direction markers, or pairing information in the input interface. In implementation, entity region tags can be generated for each entity image region in the entity image region sequence, and audio stream tags can be generated for the multiple audio streams. Then, pairing records are generated based on role name consistency, business object number, audio stream channel tags, or object reference information in text prompts. Pairing records can be stored using index tables, pairing lists, key-value structures, or matrix tags to ensure consistency of entity features, audio features, speaking status, and region masks in the same location later.

[0018] Text prompts provide a global source of semantics and generation conditions, and can include scene descriptions, object identity descriptions, dialogue relationship descriptions, business content descriptions, and visual style descriptions. When extracting text feature vectors from text prompts, text cleaning, word segmentation, fragment division, and role identification can be performed first, followed by encoding the text prompts to generate vector representations. The text encoding process can include lexical embedding, context encoding, role relationship fusion, and pooling. Lexical embedding transforms text fragments into vectors, context encoding extracts semantic dependencies, role relationship fusion integrates object identity, audio references, and interaction relationships, and pooling outputs text feature vectors. These text feature vectors can carry financial product descriptions, risk disclosure tone, customer consultation relationships, health guidance themes, rehabilitation instruction tone, or multi-party communication relationships, providing overall semantic conditions for subsequent video latent representation generation.

[0019] In one implementation, visual detection is performed on multiple entity images, outputting candidate bounding boxes and keypoint coordinates. Entity image regions are generated based on the candidate bounding boxes, keypoint coordinates, and foreground segmentation results. The input image order, the lateral position of the entity image regions, and the object names in the text prompts are read to arrange the entity image regions into a sequence. During mask generation, a binary space mask for the entity image regions is generated based on their boundaries, and a binary space mask sequence for the entity image regions is formed according to the arrangement of the entity image region sequence. After multiple audio streams are input, channel markers and speaking object markers are read, and a mapping relationship is generated between the speaking object markers and the object names in the text prompts. After text encoding, the text prompts are output as text feature vectors. This implementation is suitable for business video generation environments with a small number of input objects and audio streams with clear character markers.

[0020] In another implementation, multiple entity images are processed through multi-scale visual encoding to extract coarse-grained regions and fine-grained boundaries. The entity image region is formed by fusing these coarse-grained regions and fine-grained boundaries. The binary space mask of the entity image region is formed by the foreground region, the region near the mouth, the head contour region, and the boundary transition region. Multiple audio streams are read from the sound source direction and parsed from the audio channels to extract sound source direction markers, channel markers, and speaker labels. During mapping, the sound source direction markers, channel markers, speaker labels, and spatial positions in the entity image region sequence are merged to generate a mapping relationship between the entity image region sequence and the multiple audio streams. The text is processed using a long text encoding layer for financial compliance instructions, remote health guidance scripts, or multi-party question-and-answer scripts, outputting a text feature vector containing scene semantics and interaction relationships. This implementation is suitable for generation environments with a large number of multiple objects and complex speaking relationships.

[0021] It can also be implemented in a lightweight manner in low-latency environments. Entity image regions are directly generated by single-stage detection, and the binary space mask of the entity image region is formed by rectangular regions, keypoint-enclosed regions, or fast segmentation regions. Multiple audio streams establish a mapping relationship with the entity image region sequence through input channel binding information, and text prompts extract text feature vectors through a small text encoding structure. This implementation reduces the number of candidate region screenings and text encoding layers, making it suitable for online customer management explanations, quick explanations of wealth management products, and short risk disclosure videos in fintech businesses, as well as for generating health education videos, rehabilitation exercise instructions, and remote health consultation segments in healthcare businesses.

[0022] In the fintech business, when generating robo-advisory consultation videos, the input includes images of financial advisors, clients, and risk control personnel, along with audio explanations, consultations, risk warnings, and text prompts containing product descriptions, risk disclosures, and question-and-answer relationships. Visual inspection is performed on these three types of images to generate a sequence of entity image regions and a sequence of binary masks for these regions. Based on the object references (financial advisor, client, risk control personnel, etc.) in the text prompts, and the annotations for explanations, consultations, and risk warnings in the audio streams, a mapping relationship is established between the entity image region sequence and multiple audio streams. The text prompts are encoded, and text feature vectors are extracted to enable subsequent generation processes to utilize financial product descriptions, compliant tone, and role interaction relationships.

[0023] In the healthcare field, the generation of remote health consultation videos involves inputting images of doctors, patients, and health managers, along with audio recordings of doctor explanations, patient questions, health guidance, and text prompts containing health topics, question-and-answer relationships, and guidance tone. The system locates the corresponding entity image regions for doctors, patients, and health managers from the images and generates a binary space mask sequence for these entity image regions. Based on the referential information for doctors, patients, and health managers in the text prompts and the speaker object labels from multiple audio streams, a mapping relationship is established between the entity image region sequence and the multiple audio streams. The text prompts are encoded to extract health topics, question-and-answer relationships, and guidance tone, generating text feature vectors. This helps subsequent processing reduce instances where patient speech corresponds to the wrong doctor image region or health guidance audio corresponds to the wrong object.

[0024] This embodiment generates entity image region sequences and entity image region binary space mask sequences from multiple entity images, forming spatial range and spatial constraint data for each entity object. By establishing a mapping relationship between the entity image region sequences and multiple audio streams, object-audio stream mismatches can be reduced in subsequent feature generation and driving generation. By extracting text feature vectors from text prompts, scene semantics, interaction relationships, and generation style can be converted into conditional expressions usable in subsequent video generation. Therefore, subsequent processing can organize image regions, region masks, audio streams, and text conditions around the same entity object, reducing audio-driven object errors, interference between adjacent entity regions, and inconsistencies in scene semantics in multi-object spoken videos.

[0025] S20, perform feature extraction on the entity image region sequence to generate an entity feature vector sequence, perform temporal encoding and speech state extraction on the multiple audio streams to generate an audio feature vector sequence and a speech state feature sequence, and align the entity feature vector sequence, the audio feature vector sequence, the speech state feature sequence and the entity image region binary space mask sequence with each other based on the mapping relationship; In this embodiment, the entity image region sequence is formed by arranging valid object regions from multiple entity images. When extracting features from the entity image region sequence, the processing object is each entity image region in the sequence, not the entire entity image. In implementation, each entity image region undergoes size normalization, boundary padding, and local texture preservation, followed by the extraction of appearance texture, contour structure, local motion reference points, and region location representation. Appearance texture can originate from color distribution, local texture, and identity appearance information within the entity image region; contour structure can originate from entity image region boundaries, head outlines, or boundaries of movable parts; local motion reference points can originate from the mouth, eyes, head direction, or other drivable positions; and region location representation can originate from the arrangement position of the entity image region within the entity image region sequence. Image encoding can be performed sequentially, consisting of region cropping, local convolution, attention encoding, and vector convergence. Region cropping outputs region image blocks, local convolution extracts texture and boundaries, attention encoding merges information from different positions within the region, and vector convergence outputs the entity feature vector corresponding to each entity image region. Multiple entity feature vectors are combined according to the arrangement result of the entity image region sequence to form an entity feature vector sequence.

[0026] Temporal coding of multiple audio streams is used to convert sound segments into audio representations that can be paired with entity features. In implementation, each audio stream undergoes sampling, silence segment marking, segmentation, and spectral representation extraction, before being processed by a temporal coding layer to generate audio feature vectors. The temporal coding layer can employ convolutional temporal coding, recurrent temporal coding, attention-based temporal coding, or multi-scale segment coding. Audio feature vectors can contain information on speech rhythm, phoneme variations, energy fluctuations, and articulation continuity. Multiple audio streams, after temporal coding, form a sequence of audio feature vectors. To avoid inconsistencies between the order of the input audio streams and the order of the entity image region sequence, the generated audio feature vector sequence needs to retain audio stream markers, segment position markers, or mapping position markers for subsequent alignment.

[0027] Speech state extraction is used to obtain a speech state feature sequence from multiple audio streams. Speech state features represent the speech intensity, pauses, transitions, and non-speech response tendencies in different segments of the current audio stream. In implementation, energy changes, speech activity intervals, pause boundaries, phoneme transition positions, and sustained vocal segments can be extracted from the audio stream, and this information is then encoded into speech state features. The speech state feature sequence and the audio feature vector sequence originate from the same batch of multiple audio streams; therefore, the same audio stream labels and segment position labels must be used during generation. In fintech, speech state features can distinguish speech segments from financial advisor explanations, customer questions, and risk control personnel prompts. In healthcare, speech state features can distinguish speech segments from health manager explanations, patient questions, and doctor responses.

[0028] The mapping relationship is used to transform the correspondence information between entity image region sequences and multiple audio streams into a basis for sequence arrangement. When aligning with each other based on the mapping relationship, sequence alignment position markers can be generated. These markers identify the corresponding positions in the entity feature vector sequence, audio feature vector sequence, speech state feature sequence, and entity image region binary space mask sequence. In implementation, the entity feature vector sequence retains the arrangement position of the entity image region sequence, the audio feature vector sequence and speech state feature sequence are rearranged according to the mapping relationship, and the entity image region binary space mask sequence retains the same arrangement position as the entity image region sequence. After rearrangement, entity feature vectors, audio feature vectors, speech state features, and entity image region binary space masks at the same sequence position point to the same entity object and corresponding audio stream. This reduces misalignment between multiple audio streams and multiple entity image regions in subsequent processing.

[0029] In one implementation, the entity image region sequence is sequentially processed by region cropping, local texture encoding, and position encoding. Region cropping extracts region image blocks from the entity image region sequence; local texture encoding extracts appearance texture and boundary structure; position encoding generates position representations based on the arrangement of entity image regions within the entity image region sequence; and vector aggregation synthesizes the appearance texture, boundary structure, and position representations into an entity feature vector sequence. Multiple audio streams are sequentially processed by segmentation, spectral encoding, and temporal aggregation. The segmentation layer divides multiple audio streams according to speech activity intervals; the spectral encoding layer extracts audio segment representations; and the temporal aggregation layer generates an audio feature vector sequence. A speech state feature sequence is generated based on energy changes, pause boundaries, and vocal duration segments. During sequence alignment, the mapping relationship is read, and the audio feature vector sequence and speech state feature sequence are rearranged to their corresponding positions in the entity feature vector sequence, while preserving the same-order arrangement of the entity image region binary space mask sequence.

[0030] In another implementation, multi-scale visual encoding and region attention processing are performed on the entity image region. Multi-scale visual encoding processes the overall contour and local movable positions within the entity image region, while region attention processing generates entity feature vector sequences based on region boundaries and movable positions. Multiple audio streams are encoded using multi-scale audio segment encoding, separately encoding pronunciation changes in short segments and speech rhythm in longer segments, resulting in audio feature vector sequences. A speech state feature sequence is generated based on the probability of speech activity, pause boundaries, and phoneme changes in the audio streams. During sequence alignment, an alignment index table is generated using mapping relationships, and the positions of the audio feature vector sequence, speech state feature sequence, and entity image region binary space mask sequence are adjusted according to the alignment index table. This implementation is suitable for generation environments with more intensive multi-user interaction and a larger number of audio streams.

[0031] In the low-latency implementation, lightweight region encoding can be performed on entity image region sequences, directly extracting compressed entity feature vector sequences from these sequences. Short-window temporal encoding can be performed on multiple audio streams to generate audio feature vector sequences, and speech state feature sequences can be generated by speech activity detection. During sequence alignment, the positions are rearranged directly based on the entity region markers and audio stream markers in the mapping relationship. This implementation is suitable for quick financial explanation videos and short risk warning videos in fintech businesses, as well as health education segments and remote health consultation segments in healthcare businesses.

[0032] In the fintech business, online wealth management consultation videos include image regions for wealth management advisors, clients, and risk control personnel, as well as audio for explanations, consultations, and risk warnings. Entity feature vector sequences are extracted from each entity's image region, and audio feature vector sequences and speaking status feature sequences are generated from multiple audio streams. A mapping relationship maps the explanation audio to the wealth management advisor's image region, the consultation audio to the client's image region, and the risk warning audio to the risk control personnel's image region. After mutual alignment, the same sequence position can simultaneously obtain the binary space mask of entity features, audio features, speaking status, and entity image region, reducing the possibility of client question audio being mapped to the wealth management advisor's image region or risk warning audio being mapped to the wrong object.

[0033] In the healthcare field, remote health consultation videos include image regions for doctors, patients, and health managers, as well as audio sequences for doctor explanations, patient questions, and health guidance. Entity feature vector sequences are generated from the doctor, patient, and health manager image regions, and audio feature vector sequences and speaking state feature sequences are generated from the doctor's explanation audio, patient's question audio, and health guidance audio. A mapping relationship maps the doctor's explanation audio to the doctor's image region, the patient's question audio to the patient's image region, and the health guidance audio to the health manager's image region. After mutual alignment, subsequent processing can read the matching image region representation, audio representation, speaking state representation, and region mask at the same sequence position, reducing the possibility of patient audio driving doctor region or health guidance audio driving patient region.

[0034] This embodiment extracts region-level features from entity image region sequences, allowing the entity feature vector sequence to retain the object's appearance, boundary structure, and arrangement. By performing temporal encoding and speech state extraction on multiple audio streams, the audio feature vector sequence and speech state feature sequence can carry speech-driven information and speech state information, respectively. Through mapping relationships, the entity feature vector sequence, audio feature vector sequence, speech state feature sequence, and entity image region binary space mask sequence are aligned, ensuring that image regions, audio streams, speech states, and region masks at the same sequence position point to the same object, thereby reducing audio feature mismatches, speech state mismatches, and region mask mismatches during subsequent generation.

[0035] S30, traverse the entity feature vector sequence, and for each traversal of the current entity feature vector, read the current audio feature vector, the current speaking state feature, and the current entity image region binary space mask that are aligned with the same sequence. In this embodiment, the entity image region sequence undergoes size unification, boundary padding, and channel adjustment before region-level encoding, enabling region images from different sources to enter the same image encoding process. The image encoding process may include region cropping, local texture encoding, boundary structure encoding, and vector convergence. Region cropping extracts region image blocks one by one according to the entity image region sequence. Local texture encoding extracts color distribution, local texture, and identity appearance information. Boundary structure encoding extracts region contours, head boundaries, contours near the mouth, and boundaries of movable parts. Vector convergence compresses the encoding results at different locations within the region into entity feature vectors. Multiple entity feature vectors are combined according to the arrangement order of the entity image region sequence to form an entity feature vector sequence. Each position in the entity feature vector sequence retains the appearance representation, local structure representation, and sequence position information of the entity image region.

[0036] Before entering temporal encoding, multiple audio streams undergo sampling rate unification, amplitude normalization, silence segment marking, and segment division. The audio encoding process can include spectral transformation, short-window temporal encoding, long-window contextual encoding, and vector convergence. Spectral transformation converts audio segments into spectral or acoustic representations; short-window temporal encoding extracts phoneme variations, energy fluctuations, and articulation boundaries; long-window contextual encoding extracts continuous vocal rhythms and pause distributions; and vector convergence generates audio feature vectors. Multiple audio streams undergoing the same encoding process form a sequence of audio feature vectors. This sequence of audio feature vectors represents the driving content of each audio stream in the temporal dimension.

[0037] Speech state extraction revolves around the vocalization states across multiple audio streams. Speech state recognition involves reading energy changes, speech activity intervals, pause boundaries, phoneme transition positions, and sustained vocal segments to form speech state features. These features express speech intensity, pause states, transition states, and non-speech response tendencies. Speech state features from multiple audio streams are combined according to their source to form a speech state feature sequence. While the speech state feature sequence and the audio feature vector sequence share the same source, they carry different content. The audio feature vector sequence is biased towards acoustically driven content, while the speech state feature sequence is biased towards whether the current subject is speaking and changes in their speech state.

[0038] The mapping relationship is used to transform the pairing results of entity image region sequences and multiple audio streams into a sequence arrangement basis. In implementation, a region ordinal marker is retained for each entity feature vector in the entity feature vector sequence, and an audio stream marker is retained for the audio feature vector sequence and the speech state feature sequence. An alignment index is then generated based on the mapping relationship. The alignment index is used to adjust the arrangement of the audio feature vector sequence and the speech state feature sequence, ensuring that the audio feature vectors, speech state features, and the same entity image region are in the same sequence position. The entity image region binary space mask sequence itself is arranged along with the entity image region sequence and, after verification by the alignment index, maintains a co-position with the entity feature vector sequence. After mutual alignment, entity feature vectors, audio feature vectors, speech state features, and entity image region binary space masks at the same sequence position point to the same entity object and the paired audio stream.

[0039] In one implementation, image encoding employs region cropping, convolutional coding, and position embedding. The region cropping layer extracts fixed-size regions from the entity image region sequence; the convolutional coding layer extracts region texture and boundary morphology; the position embedding layer writes the arrangement position of the entity image region within the entity image region sequence; and the vector convergence layer outputs a sequence of entity feature vectors. Audio encoding employs spectral transformation and temporal attention coding. The spectral transformation layer generates audio segment representations, and the temporal attention coding layer extracts pronunciation continuity and rhythmic variations, outputting a sequence of audio feature vectors. Speech state recognition outputs a speech state feature sequence based on speech activity intervals, energy peaks, and pause boundaries. During sequence alignment, the mapping relationship is read, and the audio feature vector sequence and the speech state feature sequence are rearranged to positions consistent with the entity feature vector sequence, while simultaneously verifying the arrangement position of the entity image region binary space mask sequence.

[0040] In another implementation, image encoding employs multi-scale region encoding. Multi-scale region encoding extracts the overall contour, local mouth region, head boundary, and region texture, which are then fused by a fusion layer to generate an entity feature vector sequence. Audio encoding uses short-segment encoding and long-segment encoding. Short-segment encoding extracts phoneme variations, while long-segment encoding extracts speech rhythm and pause states; these are fused to generate an audio feature vector sequence. Speech state extraction generates a speech state feature sequence based on speech activity probability, pause boundaries, and vocalization duration. Sequence alignment uses a mapping relationship to generate a sequence position table, which is used to rearrange the audio feature vector sequence and the speech state feature sequence, ensuring consistency across the four types of sequences at the same position.

[0041] In the low-latency implementation, image encoding uses a lightweight region encoder to output a compressed sequence of entity feature vectors. Audio encoding uses a short-window encoder to output an audio feature vector sequence, and the speech state extraction process uses speech activity detection and energy thresholding to segment and output a speech state feature sequence. Sequence alignment directly reads the entity region markers and audio stream markers from the mapping relationship to complete the sequence position rearrangement.

[0042] In the fintech business, online wealth management consultation videos include image areas for the wealth management advisor, the client, and the risk control personnel, as well as audio recordings of explanations, consultations, and risk warnings. Image encoding generates entity feature vector sequences, audio encoding generates audio feature vector sequences, and speech status recognition generates speech status feature sequences. A mapping relationship places the explanation audio in the same sequence position within the wealth management advisor's image area, the consultation audio in the same sequence position within the client's image area, and the risk warning audio in the same sequence position within the risk control personnel's image area. This minimizes the chance of client audio being mistakenly placed in the wealth management advisor's position or risk warning audio being mistakenly placed in the client's position during subsequent readings.

[0043] In the healthcare field, remote health consultation videos include image regions for doctors, patients, and health managers, as well as audio recordings of doctor explanations, patient questions, and health guidance. Image encoding generates entity feature vector sequences, audio encoding generates audio feature vector sequences, and speech state extraction generates speech state feature sequences. A mapping relationship ensures that the audio recordings of doctor explanations, patient questions, and health guidance are aligned with their corresponding image regions, and the binary space mask sequences of the entity image regions are synchronously aligned, enabling subsequent readings to obtain matching entity representations, audio representations, speech states, and spatial masks.

[0044] This embodiment uses regional-level encoding of entity image region sequences to preserve entity appearance, local structure, and arrangement position of entity feature vector sequences. Through temporal encoding of multiple audio streams and extraction of speaking states, audio feature vector sequences and speaking state feature sequences can respectively carry acoustically driven content and speaking state changes. By mapping relationships, the four types of sequences are aligned with each other, and the same sequence position can simultaneously point to entity image regions, paired audio streams, speaking states, and spatial masks, thereby reducing audio stream mismatches, speaking state mismatches, and region mask mismatches in multi-object generation.

[0045] S40, the current entity feature vector is used as the query feature, and the current audio feature vector is used as the key feature and value feature to perform cross-attention operation to generate basic driving features; In this embodiment, the current entity feature vector carries the appearance, local structure, and sequence position information of the current entity image region. When the current entity feature vector is used as a query feature, the image-side information is converted into a vector representation for retrieving audio-driven content. In implementation, the current entity feature vector undergoes vector projection, normalization, channel transformation, and dimensionality adjustment to obtain the query feature. Normalization reduces amplitude differences between different entity image regions, channel transformation maps the entity appearance and local structure representations to the space required for cross-attention operations, and dimensionality adjustment enables the query feature to perform similarity matching with the key feature. The query feature preserves the appearance boundary, local structure, and sequence position of the current entity object, allowing subsequent audio-driven content to unfold around the current entity object.

[0046] The current audio feature vector carries information about rhythm, articulation variations, and energy fluctuations in the current audio stream. When the current audio feature vector serves as both key and value features, the audio-side information is decomposed into two categories: matching criteria and driving content. In implementation, the current audio feature vector undergoes two-way mapping: one path generates key features, and the other generates value features. Key features primarily retain the temporal position, rhythm variations, and articulation boundaries of the audio segment, used to form matching weights with query features; value features primarily retain phoneme variations, energy intensity, and driving content, used to form an audio-driven response under the influence of attention weights. Key and value features originate from the same source, but different parameter sets are used in the two audio projections, allowing the matching process and response extraction process to be separated.

[0047] Cross-attention operations utilize the matching results between query features and key features to extract audio-driven content relevant to the current entity from value features. In implementation, similarity calculations and weight normalization are performed on query and key features to obtain cross-modal attention weights. These weights are then applied to value features to form audio-driven responses. These responses are then fused with structural information from the current entity's feature vector to generate basic driving features. These basic driving features contain both the spatial and structural constraints of the current entity object and the driving content provided by the current audio stream, enabling subsequent speech state adjustments.

[0048] In one implementation, the cross-attention operation may include entity projection, audio key projection, audio value projection, attention weight generation, and driver fusion. Entity projection receives the current entity feature vector and outputs query features; audio key projection receives the current audio feature vector and outputs key features; and audio value projection receives the current audio feature vector and outputs value features. Attention weight generation generates cross-modal attention weights based on the query features and key features. Driver fusion generates an audio driver response based on the cross-modal attention weights and value features, and fuses the audio driver response with the structural representation in the current entity feature vector as the base driver feature.

[0049] In another implementation, cross-attention can be achieved using segmented window matching. Entity projection maintains a current entity query representation, while audio projection divides the current audio feature vector into multiple time-segment representations. Each time-segment representation generates key and value features. Segmented attention weights are generated within the time-segment range, and the driving responses of each time segment are synthesized into basic driving features.

[0050] Multi-head cross-attention computation can also be used. Multiple attention heads process local mouth-driven, head micro-motion-driven, and global rhythm-driven functions respectively. Each attention head receives the same query features, key features, and value features, but uses different projection parameters. The outputs of multiple attention heads are concatenated and compressed to form the basic driving features.

[0051] In the fintech business, the current entity feature vector corresponding to the financial advisor's image region is converted into query features, and the current audio feature vector corresponding to the financial advisor's audio explanation is converted into key and value features. Cross-attention operations extract audio-driven responses based on the matching relationship between the financial advisor's image region and the audio explanation, generating basic driving features. This ensures that subsequent content is organized around the financial advisor's objectives, reducing the likelihood of incorrect driving of the financial advisor's image region by customer consultation audio.

[0052] In the healthcare business, the current entity feature vector corresponding to the patient's image region is converted into query features, and the current audio feature vector corresponding to the patient's question audio is converted into key and value features. Cross-attention operation extracts the audio-driven response matching the patient's image region from the patient's question audio, generating basic driving features. This enables subsequent generated content to form question-related driving expressions around the patient object, reducing the impact of doctor's explanation audio or health guidance audio on the patient's image region.

[0053] This embodiment converts the current entity feature vector into query features and the current audio feature vector into key features and value features respectively. The current entity object on the image side can serve as the retrieval basis, while the current audio stream on the audio side can simultaneously provide matching criteria and driving content. By extracting the audio driving response matching the query features from the value features through cross-attention operations, and then fusing the structural information of the current entity object to generate basic driving features, the likelihood of audio content being directly applied to incorrect entities can be reduced, and the driving consistency between the current entity region and the current audio stream can be improved.

[0054] S50, adjust the driving weight of the basic driving feature based on the current speaking state feature, generate the current entity audio driving feature, and use the binary space mask of the current entity image region to constrain the current entity audio driving feature to generate the current local region driving feature. In this embodiment, the current speaking state feature is used to characterize the vocal state changes of the object corresponding to the current audio segment, including information such as vocal intensity, pause positions, state transition positions, and non-speaking response tendencies. In implementation, the current speaking state feature is parsed to output speaking activation information, state transition information, and response tendency information. Speaking activation information is used to determine that the audio driver should focus more on lip-sync related expressions; state transition information is used to determine the transition position from speaking to pausing or from pausing to speaking; and response tendency information is used to determine whether non-speaking objects need to produce subtle head movements, changes in gaze, or slight changes in facial expressions. State parsing can employ fully connected processing, gating processing, and normalization processing, using the current speaking state feature as input and outputting the required state vector as the driving weights.

[0055] The basic driving features are derived from the cross-attention operation and already include the driving response between the current entity and the current audio. Driving weights are used to adjust the intensity of different driving components within the basic driving features. In implementation, the state vector output from state resolution is projected and normalized through channels to obtain lip movement weights, head micro-motion weights, and listening response weights. Lip movement weights control components related to mouth shape changes in the basic driving features, head micro-motion weights control components related to subtle changes in head posture, and listening response weights control micro-expressions, eye contact, or slight responses in non-speaking states. The driving weights and basic driving features are adjusted channel-by-channel or group-by-group to obtain the current entity's audio driving features.

[0056] The current entity audio driving features represent the driving expression after speech state adjustment. When adjusting features, the basic driving features can be decomposed into lip-shape driving sub-features, head micro-motion sub-features, and listening response sub-features, and then multiplied by lip-shape motion weights, head micro-motion weights, and listening response weights, respectively. Alternatively, the driving weights can be mapped to gating vectors, and the gating vectors can be used to directly adjust the channel response of the basic driving features. If the current speech state features indicate high speech intensity, the lip-shape motion weight is increased, and the proportion of the lip-shape driving component in the current entity audio driving features increases. If the current speech state features indicate a pause or listening state, the head micro-motion weight and listening response weight are increased, and the current entity audio driving features are more biased towards micro-motion expression.

[0057] A binary space mask of the current entity image region is used to limit the spatial scope of the current entity's audio driving features. When applying mask constraints, the binary space mask of the current entity image region and the current entity's audio driving features are read. The mask matrix is ​​adjusted to the same spatial size as the current entity's audio driving features. Then, driving content within the effective mask region is preserved through positional multiplication, region sampling, or spatial gating, while driving content within the invalid mask region is suppressed. The binary space mask of the current entity image region can preserve the driving response within the entity region, reducing the diffusion of the current entity's audio driving features into adjacent entity regions or background regions.

[0058] The current local region driving features are formed from the current entity audio driving features after mask constraints. When applying mask constraints, a transition region can be set at the boundary of the binary space mask of the current entity image region to smoothly attenuate the driving response at the boundary according to spatial distance; alternatively, a binary mask can be directly used to retain the effective region and mask the invalid region. The current local region driving features include the audio driving content corresponding to the current entity, the driving strength after adjusting the speaking state, and the spatial constraint results within the current entity region, for subsequent multi-entity driving fusion.

[0059] In one implementation, a gated fully connected layer is used to process the current speaking state features. The gated fully connected layer outputs lip movement weights, head micro-motion weights, and listening response weights. During feature adjustment, the basic driving features are divided into lip movement driving channel groups, head micro-motion channel groups, and listening response channel groups, and channel scaling is performed using the corresponding weights to obtain the current entity audio driving features. When performing mask constraints, the binary space mask of the current entity image region is upsampled to the spatial resolution of the current entity audio driving features. Then, the upsampled mask is used to constrain the current entity audio driving features positionally, generating the current local region driving features.

[0060] In another implementation, temporal state encoding is used to process the current speaking state features during state parsing. Temporal state encoding preserves short-term changes near speaking transitions. Continuously changing driving weights are generated based on the temporal state encoding output, and these weights are used to adjust the basic driving features, reducing abrupt changes at speaking transitions. During mask constraint, the current entity image region is decomposed into a core region and a boundary transition region using a binary mask. The core region retains the complete driving response, while the boundary transition region generates attenuation coefficients based on distance from the boundary. The current entity audio driving features are then synthesized into the current local region driving features after being constrained by both the core region and the boundary transition region.

[0061] Alternatively, a partitioned adjustment approach can be used. During state parsing, facial region weights, head region weights, and response region weights are generated based on the current speaking state features. During feature adjustment, the responses of different spatial regions in the basic driving features are adjusted according to the region weights. When applying mask constraints, a binary space mask of the current entity image region is used to limit the overall region, and then the local response intensity is adjusted within the mask according to the region weights to generate the current local region driving features.

[0062] In the fintech business, during online financial consultation videos, the current speaking state features corresponding to the customer's audio questions indicate that the customer is speaking. Based on these features, the weight of lip movements in the customer's image region is increased, while the lip movement intensity in the financial advisor's image region is reduced during the same time period. The current entity audio driving features corresponding to the customer's image region are constrained by a binary space mask of the current entity image region, forming current local region driving features that only affect the customer's image region. This processing reduces the likelihood of the customer's audio questions causing changes in the financial advisor's lip movements or abnormal actions in the risk control personnel's area.

[0063] In the healthcare field, during remote health consultation videos, the current speaking state characteristics corresponding to the health manager's audio explanation indicate that the health manager is speaking, and after state analysis, a higher weight is generated for lip movements. The patient's image region is in a listening state during adjacent time periods, resulting in a higher weight for listening responses and a lower weight for lip movements. A binary space mask of the current entity's image region restricts each driving response to its corresponding object region, ensuring that the health manager's audio explanation primarily affects the health manager's image region, while maintaining a slight listening response in the patient's image region, thus reducing driving interference between objects.

[0064] This embodiment generates driving weights based on the current speaking state features. Different driving components in the basic driving features can adjust their intensity according to speaking, pausing, and listening states. The current entity audio driving features are constrained by the binary space mask of the current entity image region, and the driving response is limited to the spatial range corresponding to the current entity image region. By forming the current local region driving features, the local driving results that have been adjusted by state and constrained by region can be used in subsequent fusion, thereby reducing abnormal movement of non-speaking objects, incorrect driving of adjacent entity regions, and abrupt changes in action at the speaking switch.

[0065] S60, after the entity feature vector sequence is traversed, the current local region driving features generated in each traversal are normalized and fused to generate fused multi-stream features. Based on the text feature vector and the fused multi-stream features, a video latent representation is generated and decoded to obtain a multi-entity interactive video.

[0066] In this embodiment, after the entity feature vector sequence is traversed, the current local region driving features generated by each traversal correspond to different entity regions. The current local region driving features contain the local driving content of the entity object, as well as the spatial range after regional constraints. Normalization fusion requires first organizing the current local region driving features generated by each traversal into driving expressions of the same scale, the same channel dimension, and the same time length to avoid the fusion result being biased towards a certain entity due to different entity region areas, different audio driving intensities, or different local response amplitudes. In implementation, each current local region driving feature can be written into a local driving cache, which records the driving source, spatial range, amplitude distribution, and time segment marker. Then, fusion weights are generated based on the amplitude distribution and spatial range. Normalization fusion can use amplitude normalization, region area normalization, time segment normalization, or channel response normalization to form fused multi-stream features under the same fusion scale for multiple local driving expressions.

[0067] Multi-stream feature fusion is used to carry a unified driving expression for multiple entity objects. During fusion, it's necessary to preserve the driving differences between different entity regions to avoid excessively strong local responses or mutual interference between adjacent regions caused by simple superposition. In implementation, the current local region driving features in the local driving cache can be written into the spatial feature plane according to the entity region's location, and then normalized weights can be applied to overlapping or adjacent regions. If a certain entity region has a large area, area normalization can reduce its excessive influence on the overall fusion result. If a certain audio segment has strong energy, amplitude normalization can limit the amplitude of that local driving response in the multi-stream feature fusion. In fintech scenarios such as multi-party investment advisory explanations, risk warnings, and customer consultations, multi-stream feature fusion can simultaneously retain the local driving content corresponding to financial advisors, customers, and risk control personnel. In healthcare scenarios such as health consultations, rehabilitation instructions, and multi-party communication, multi-stream feature fusion can simultaneously retain the local driving content corresponding to doctors, patients, and health managers.

[0068] Text feature vectors provide semantic conditions for video latent representation generation. These vectors can contain scene semantics, object relationships, interaction intent, and visual style. In implementation, text feature vectors can be jointly encoded with fused multi-stream features through conditional fusion. Conditional fusion can employ gated fusion, cross-attention, or conditional normalization. Gated fusion generates textual conditional weights based on the text feature vectors and adjusts the channel responses of the fused multi-stream features. Cross-attention fusion uses the text feature vectors as conditional input, enabling the fused multi-stream features to carry scene semantics and interaction relationships when generating video latent representations. The conditional normalization layer adjusts the mean and magnitude of the fused multi-stream features based on the text feature vectors, ensuring the output video latent representation carries semantic constraints consistent with the text cues.

[0069] Video latent representation is a compressed video representation before decoding, containing temporal variations, entity region drivers, scene conditions, and frame structure information. In implementation, during latent mapping, multi-stream features and text feature vectors are input into the temporal generation layer to obtain video latent representations covering multiple time segments. The video latent representation is then decoded, a process that may include a latent frame reconstruction layer, a texture reconstruction layer, a region consistency correction layer, and a temporal smoothing layer. The latent frame reconstruction layer converts the compressed representation into an initial frame representation; the texture reconstruction layer completes the entity appearance and background details; the region consistency correction layer reduces entity region boundary misalignment; and the temporal smoothing layer reduces jitter between adjacent frames, ultimately yielding a multi-entity interactive video.

[0070] In one implementation, during normalized fusion, the current local region driving features generated in each traversal are received. Each current local region driving feature is written into a local driving cache according to its source entity region, and amplitude statistics, region occupancy, and time segment markers are generated for the local driving cache. Fusion weights are generated based on the amplitude statistics and region occupancy, and then the fusion weights are used to adjust each current local region driving feature to obtain fused multi-stream features. During conditional fusion, text feature vectors are mapped to text conditional weights, and the text conditional weights are used to adjust the fused multi-stream features. Then, the adjusted fused multi-stream features are latently mapped to convert them into a video latent representation. Frame restoration, texture restoration, and boundary correction are performed on the video latent representation to obtain a multi-entity interactive video.

[0071] In another implementation, during normalization fusion, a spatial writing approach is used to process the driving features of each current local region. Each current local region driving feature is written into a unified feature plane according to its regional spatial range. Overlapping regions are normalized using regional weights, while non-overlapping regions retain the local driving expressions of the corresponding entities. During conditional fusion, a cross-attention layer is used to process text feature vectors and fuse multi-stream features. Interaction relationships, scene semantics, and visual style are incorporated into the video latent representation. The video latent representation is temporally decoded to generate an initial video frame sequence, and then a region consistency correction layer is used to adjust entity boundaries and adjacent object regions.

[0072] Low-latency implementation is also possible. Normalization fusion uses channel amplitude normalization and temporal segment normalization to reduce the computational cost of spatial writing. Conditional fusion uses a lightweight gating layer to process text feature vectors, and latent mapping uses a short-window temporal layer to generate the video latent representation. Decoding uses a lightweight frame recovery layer to output multi-entity interactive video.

[0073] In the fintech business, robo-advisory videos involve three entities: financial advisors, clients, and risk control personnel. The driving features of each local region generated during each traversal are written into a local driving cache. During normalization fusion, fusion weights are generated based on the area of ​​each entity's region and the driving amplitude, forming fused multi-stream features. Text feature vectors carry the product description, risk disclosure tone, and client consultation relationship. During conditional fusion, text feature vectors are used to adjust the fused multi-stream features. During latent mapping, a latent video representation is generated, and after decoding, a multi-entity interactive video is output. In the generated result, the financial advisor's explanation, client questions, and risk control prompts can be expressed within their respective regions, reducing the possibility of incorrect driving in the client or risk control personnel regions.

[0074] In the healthcare field, remote health consultation videos involve three entity objects: doctors, patients, and health managers. During normalized fusion, the driving features of the current local regions corresponding to the doctor's explanations, patient questions, and health guidance are fused, preserving the driving differences within each object region while fusing multi-stream features. Text feature vectors carry health topics, question-and-answer relationships, and guidance tone; these are incorporated into the video latent representation during conditional fusion. During decoding, multi-entity interactive videos are generated based on the video latent representation, ensuring that the doctor's explanations, patient responses, and health manager guidance maintain object correspondences within the frame and reducing the mixing of multi-object region driving features.

[0075] This embodiment reduces fusion bias caused by different entity region areas, different driving amplitudes, and different audio intensities by normalizing and fusing the driving features of each current local region after traversing the entity feature vector sequence. This allows the fused multi-stream features to retain the local driving content of multiple entity regions. Through joint encoding of text feature vectors and fused multi-stream features, scene semantics, object relationships, and visual style can be incorporated into the video latent representation. By decoding the video latent representation, the normalized, fused, and text-constrained multi-stream expression can be converted into a multi-entity interactive video, thereby reducing excessive local driving superposition, interference from adjacent regions, and semantic deviation in the visuals.

[0076] In one embodiment, step S10 above includes: S101, acquire multiple entity images, multiple audio streams and text prompts, extract entity reference fragments, audio reference fragments, interaction relationship fragments, scene semantic fragments and generation style fragments from the text prompts, and extract entity candidate regions from the multiple entity images, and generate a set of entity structure key points, entity outline boundary and entity region confidence for each entity candidate region; S102, based on the set of key points of the entity structure, the entity contour boundary and the entity region confidence, target entity regions are selected from the entity candidate regions, and the initial arrangement number of the target entity regions is determined based on the entity image input order. The reference correction number of the target entity regions is determined based on the spatial position of the target entity regions in the image and the entity reference fragment. The entity region arrangement number is generated according to the initial arrangement number and the reference correction number. The target entity regions are arranged according to the entity region arrangement number to generate an entity image region sequence. S103, for each entity image region in the entity image region sequence, determine the driveable core region and the boundary transition region, generate an entity image region binary space mask corresponding to each entity image region based on the driveable core region and the boundary transition region, and arrange the entity image region binary space masks into an entity image region binary space mask sequence according to the arrangement order of the entity image region sequence. S104, extract audio stream channel markers and audio subject indication information from the multiple audio streams, and establish a mapping relationship between the entity image region sequence and the multiple audio streams based on the entity reference segment, the audio reference segment, the audio stream channel markers, the audio subject indication information and the entity image region sequence; S105, the scene semantic segment, the interaction relationship segment, and the generation style segment are encoded and fused to generate a text feature vector.

[0077] In this embodiment, multiple entity images are formatted, resolution-adjusted, and effective region detected before entity candidate regions are extracted. Entity candidate regions are derived from areas in the image that have a drivable appearance, stable contour, and locatable local structures. When extracting entity candidate regions, region detection, foreground segmentation, and keypoint localization can be combined. Region detection provides candidate bounding boxes, foreground segmentation provides candidate foreground ranges, and keypoint localization provides the positions of movable parts. Each entity candidate region further generates a set of entity structure keypoints, entity contour boundaries, and entity region confidence. The entity structure keypoint set originates from local geometric locations within the candidate region, such as the mouth, eyes, head edges, or drivable part positions. The entity contour boundaries originate from the outer contour of the candidate region and the foreground segmentation boundaries. The entity region confidence is derived from the fusion result of region detection output, keypoint integrity, contour closure degree, and image quality score, used to determine whether the candidate region is suitable for forming subsequent entity image regions.

[0078] After text parsing, the text prompts are broken down into entity reference fragments, audio reference fragments, interaction relationship fragments, scene semantic fragments, and generative style fragments. Entity reference fragments are derived from the character names, object numbers, location descriptions, or appearance descriptions of image objects in the text, used to assist in determining the arrangement of candidate entity regions. Audio reference fragments are derived from the character names, audio track names, descriptions of speaking persons, or speech order descriptions of sound sources in the text, used to assist in establishing pairings between audio streams and entity image regions. Interaction relationship fragments are derived from question-and-answer relationships, speaking relationships, listening relationships, response relationships, or multi-party communication relationships in the text. Scene semantic fragments are derived from the task scene, theme content, and visual semantics in the text. Generative style fragments are derived from the visual style, restraint of actions, tone of voice, and visual presentation requirements in the text. Text parsing can include lexical embedding, fragment boundary recognition, character reference recognition, and semantic fragment aggregation, enabling different fragments to participate in subsequent processing in a structured form.

[0079] The target entity region is obtained by jointly filtering the entity structure keypoint set, entity contour boundary, and entity region confidence score. In implementation, entity candidate regions with entity region confidence scores below a set condition are first excluded. Then, based on the completeness of the entity structure keypoint set and the continuity of the entity contour boundary, candidate regions with strong dynamism are retained to form the target entity region. The initial arrangement sequence number of the target entity region is determined by the entity image input order, and the reference correction sequence number is determined by the spatial position of the target entity region in the image and the entity reference fragment. The spatial position can include horizontal position, vertical position, region center point, region area, and adjacent region relationships. The entity reference fragment can provide object name, role order, or position description. The initial arrangement sequence number and the reference correction sequence number are merged to generate an entity region arrangement sequence number. Then, the target entity regions are arranged according to the entity region arrangement sequence number to form an entity image region sequence. The resulting entity image region sequence can simultaneously reflect the input order, the spatial relationship of the image, and the object reference relationship in the text.

[0080] The binary space mask for an entity image region is generated from the drivable core region and the boundary transition region within the entity image region. The drivable core region originates from the foreground range within the entity image region that is related to local motion, and is typically determined by the area enclosed by keypoints, the foreground segmentation range, and the region inside the entity contour boundary. The boundary transition region originates from the extended range near the entity contour boundary and is used to preserve transition margins at spatial edges. The binary space mask for the entity image region is generated based on the drivable core region and the boundary transition region. The binary space mask can be in matrix form with the same dimensions as the image space or in region-marked form. Valid regions are used to preserve the range of subsequent driving influence, while invalid regions are used to exclude adjacent entity regions or background regions. The binary space masks for each entity image region are arranged in the order of the entity image region sequence to form a binary space mask sequence for each entity image region, ensuring that each entity image region has a corresponding mask in the sequence.

[0081] After audio parsing, multiple audio streams are processed to extract audio stream channel markers and audio subject indication information. Audio stream channel markers are derived from input track numbers, acquisition channel numbers, file track markers, or real-time input port markers. Audio subject indication information is derived from audio metadata, speaker tags, sound source direction information, speech script markers, or audio reference fragment matching results. A mapping relationship is established based on entity reference fragments, audio reference fragments, audio stream channel markers, audio subject indication information, and entity image region sequences. In implementation, region markers are generated for entity image regions in the entity image region sequence, audio markers are generated for multiple audio streams, and then pairing records are generated based on reference consistency, channel consistency, subject consistency, and sequence positional relationships. The mapping relationship can be stored as an index table, pairing matrix, region audio pairing list, or key-value structure to indicate the audio stream corresponding to each entity image region.

[0082] The text feature vector is obtained by encoding and fusing scene semantic fragments, interaction relationship fragments, and generative style fragments. During text encoding, scene semantic fragments are converted into scene vectors, interaction relationship fragments into relationship vectors, and generative style fragments into style vectors. Encoding fusion can employ attention fusion, gating fusion, or concatenation compression to compress the scene vectors, relationship vectors, and style vectors into a text feature vector. The text feature vector retains scene content, object relationships, and generative style, and can subsequently serve as conditional input in the video's latent expression generation process.

[0083] This embodiment filters target entity regions by using a set of key points of entity structure, entity contour boundaries, and entity region confidence scores, which can reduce invalid regions entering the entity image region sequence. It generates entity region arrangement numbers by combining initial arrangement numbers and referential correction numbers, ensuring the entity image region sequence considers input order, spatial location, and textual referential relationships. It generates a binary space mask sequence for entity image regions by driving core regions and boundary transition regions, providing spatial constraint data for each entity image region. It establishes mapping relationships by using audio stream channel markers, audio subject indication information, entity referential fragments, and audio referential fragments, reducing pairing errors between entity image regions and audio streams. Finally, it generates text feature vectors by using scene semantic fragments, interaction relationship fragments, and generation style fragments, converting scenes, relationships, and styles in the text into subsequently usable vector conditions, thereby reducing generation bias caused by chaotic region arrangement, audio object mismatch, and missing text conditions in multi-entity input.

[0084] In one embodiment, step S20 above includes: S201, based on the binary space mask sequence of the entity image region, determine the drivable region image block and contour neighborhood image block of each entity image region from the entity image region sequence, and determine the region position mark based on the arrangement position of each entity image region in the entity image region sequence, and extract the entity appearance features of the drivable region image block, the boundary structure features of the contour neighborhood image block and the spatial order features corresponding to the region position mark respectively. S202, the entity appearance features, boundary structure features and spatial order features are fused to generate an entity region feature vector corresponding to each entity image region, and the entity region feature vectors are arranged according to the order of the entity image region sequence to generate an entity feature vector sequence; S203, based on the mapping relationship, configure audio stream alignment marks for multiple audio streams, and perform segmentation, temporal acoustic coding, phoneme change mark extraction and pause boundary mark extraction on multiple audio streams according to the audio stream alignment marks, to generate audio segment feature set, phoneme change mark set and pause boundary mark set; S204, generate an audio feature vector sequence based on the audio segment feature set and the phoneme change marker set, and generate a speech state feature sequence based on the audio segment feature set, the phoneme change marker set and the pause boundary marker set; S205, generate sequence alignment position markers according to the mapping relationship, and configure the sequence alignment position markers for the entity feature vector sequence, the audio feature vector sequence, the speech state feature sequence and the entity image region binary space mask sequence, so that the entity feature vector sequence, the audio feature vector sequence, the speech state feature sequence and the entity image region binary space mask sequence are aligned with each other.

[0085] In this embodiment, the binary space mask sequence of entity image regions is used to determine the drivable region image block and the contour neighborhood image block from the entity image region sequence. The drivable region image block can be obtained by intersecting the effective mask region with the entity image region, and the region range covers the appearance region that can participate in the subsequent driving response. The contour neighborhood image block can be obtained by extending the mask boundary inward and outward, and the region range covers the edge of the entity image region, the edge of the movable part, and the transition region of the adjacent background. During image cropping, the entity image region in the entity image region sequence is read, and the mask data of the same position in the binary space mask sequence of the entity image region is read. The drivable region image block is cropped according to the effective mask region, and the contour neighborhood image block is cropped according to the extended range of the mask boundary. The two image blocks obtained in this way retain the internal appearance information and the boundary structure information respectively, avoiding the mixing of the main appearance and boundary changes in the same expression by a single full-region encoding.

[0086] Region location markers are derived from the position of each entity image region within the entity image region sequence. Region location markers can be represented using sequence indexes, normalized ordinal vectors, position embedding vectors, or region permutation encoding. Region location markers do not alter the content of the entity image regions themselves; rather, they provide sequence positions to the encoding process during feature extraction, enabling the subsequently generated entity feature vector sequence to retain the permutation relationships between multiple entities. During position encoding, the permutation position of the entity image region within the entity image region sequence is read, mapped to a fixed-dimensional positional representation, and then fed into the fusion layer along with image-side features.

[0087] Entity appearance features are extracted from drivable region image patches, mainly including color distribution, texture distribution, local appearance boundaries, and the appearance of movable parts. Image appearance encoding can include local convolution, region attention, and vector convergence. Local convolution extracts texture and local edges, region attention assigns weights based on the spatial response within the drivable region image patch, and vector convergence compresses the spatial response into entity appearance features. Boundary structure features are extracted from contour neighborhood image patches, mainly including entity contour shape, boundary curvature, local edge direction, and background transition information. Boundary structure encoding can employ edge response, contour aggregation, and boundary vector convergence to output boundary structure features. Spatial order features are obtained by encoding region location markers and can be generated through embedding lookup, linear projection, or location encoding layers.

[0088] Entity appearance features, boundary structure features, and spatial order features are fused during entity region feature generation. Fusion methods can include post-stitching projection, gated weighting, attention fusion, or grouped channel fusion. Post-stitching projection is suitable for processing with high requirements for dimensionality uniformity; gated weighting is suitable for adjusting the proportion of appearance information, boundary information, and positional representation among different entity regions; attention fusion is suitable for enhancing key local representations when there are significant differences in the internal structure of entity regions. After fusion, entity region feature vectors are generated for each entity image region. These entity region feature vectors are arranged according to the order of the entity image region sequence, forming an entity feature vector sequence that maintains the same order as the entity image region sequence.

[0089] Mapping relationships are used to configure audio stream alignment markers for multiple audio streams. Audio stream alignment markers can include audio stream source markers, entity region pairing markers, sequence target position markers, and time segment markers. During audio processing, multiple audio streams are read, and the sequence position each audio stream needs to enter is determined based on the audio stream alignment markers. During segmentation, multiple audio streams are segmented according to speech activity intervals, silence intervals, or fixed time windows. During temporal acoustic coding, the segmented audio segments undergo spectral transformation, short-time feature extraction, and context encoding to generate an audio segment feature set. During phoneme change marker extraction, a phoneme change marker set is generated based on articulation transitions, energy jumps, and spectral changes in the audio segments. During pause boundary marker extraction, a pause boundary marker set is generated based on silence intervals, speech activity termination points, and energy drop positions.

[0090] The audio segment feature set and the phoneme change marker set jointly generate an audio feature vector sequence. The audio segment feature set provides acoustic content, rhythm, and energy information, while the phoneme change marker set provides the location of articulation changes, enabling the audio feature vector sequence to retain the temporal variations required for audio driving. When generating the audio feature vector sequence, the audio segment feature set can be input into the temporal coding layer, and the phoneme change marker set can be used as position adjustment or gating information, outputting an audio feature vector sequence corresponding to multiple audio streams. The speech state feature sequence is jointly generated by the audio segment feature set, the phoneme change marker set, and the pause boundary marker set. During state extraction, based on the energy intensity, phoneme change density, and pause boundary location in the audio segment, a speech state feature sequence related to speech intensity, speech transitions, and pause states can be output.

[0091] Sequence alignment position markers are generated based on mapping relationships. These markers are used to assign the same sequence position to the entity feature vector sequence, audio feature vector sequence, speech state feature sequence, and entity image region binary space mask sequence. During alignment configuration, the pairing records between entity image regions and audio streams in the mapping relationship are read to determine the entity feature vector corresponding to the entity image region, the audio feature vector corresponding to the audio stream, the speech state feature corresponding to the audio stream, and the mask data corresponding to the entity image region. The same sequence alignment position marker is assigned to all four types of data, or the audio feature vector sequence and speech state feature sequence are rearranged according to the sequence alignment position markers. After configuration, the entity feature vector sequence, audio feature vector sequence, speech state feature sequence, and entity image region binary space mask sequence all point to the same entity image region and paired audio stream at the same sequence position.

[0092] Image-side coding, audio-side coding, state extraction, and alignment configuration can be trained separately and then jointly optimized. Training data for image-side coding can include entity region images, region masks, and region location annotations, used to learn stable representations of entity appearance features, boundary structure features, and spatial order features. Training data for audio-side coding can include audio segments, phoneme change annotations, and pause boundary annotations, used to learn audio segment feature sets, phoneme change marker sets, and pause boundary marker sets. Training data for state extraction can include speech intervals, pause intervals, and transition interval annotations, used to generate speech state feature sequences. Training data for alignment configuration can include entity region-audio stream pairing records, used to generate sequence alignment position markers and complete the mutual alignment of the four types of sequences.

[0093] This embodiment determines the driveable region image block and contour neighborhood image block through the binary space mask sequence of entity image regions. This allows entity appearance features and boundary structure features to be extracted from different spatial ranges, reducing the mixing of appearance and boundary transition information within the region. Spatial order features are generated through region location markers, and entity appearance features, boundary structure features, and spatial order features are fused into entity region feature vectors, allowing the entity feature vector sequence to simultaneously retain appearance, boundary, and arrangement position. Audio stream alignment markers guide segmentation, temporal acoustic coding, phoneme change marker extraction, and pause boundary marker extraction, enabling the audio feature vector sequence and speech state feature sequence to carry acoustically driven content and speech state changes, respectively. Sequence alignment position markers configure the entity feature vector sequence, audio feature vector sequence, speech state feature sequence, and entity image region binary space mask sequence, ensuring consistency in image region representation, audio representation, speech state representation, and spatial mask at the same sequence position, thereby reducing audio object mismatch, speech state mismatch, and spatial mask mismatch in multi-entity video generation.

[0094] In one embodiment, step S30 above includes: S301, Based on the mutually aligned entity feature vector sequence, audio feature vector sequence, speech state feature sequence and entity image region binary space mask sequence, generate a sequence alignment index table, the sequence alignment index table includes sequence alignment position, audio feature vector reading position, speech state feature reading position and entity image region binary space mask reading position; S302, Generate an entity traversal cursor set according to the arrangement order of the entity feature vector sequence, and bind each entity traversal cursor in the entity traversal cursor set to a sequence alignment position; S303, sequentially read the entity traversal cursors in the entity traversal cursor set, read the current entity feature vector in the entity feature vector sequence based on the entity traversal cursor, and determine the sequence alignment position bound to the entity traversal cursor as the current sequence alignment position; S304, locate the current index item in the sequence alignment index table according to the current sequence alignment position, and read the current audio feature vector, current speech state feature, and current entity image region binary space mask from the audio feature vector sequence, the speech state feature sequence, and the entity image region binary space mask sequence, respectively, based on the audio feature vector reading position, the speech state feature reading position, and the entity image region binary space mask reading position in the current index item.

[0095] In this embodiment, the mutually aligned entity feature vector sequence, audio feature vector sequence, speech state feature sequence, and entity image region binary space mask sequence already have comparable sequence position relationships. To avoid repeated full sequence matching when traversing the entity feature vector sequence, a sequence alignment index table can be generated based on the alignment results of the four types of sequences. The sequence alignment index table records the correspondence between each sequence alignment position and the audio feature vector reading position, speech state feature reading position, and entity image region binary space mask reading position. The sequence alignment position can be represented by a sequence position number, position embedding tag, entity region tag, or alignment tag after mapping relationship transformation. The audio feature vector reading position is used to point to the target element in the audio feature vector sequence, the speech state feature reading position is used to point to the target element in the speech state feature sequence, and the entity image region binary space mask reading position is used to point to the target element in the entity image region binary space mask sequence. The sequence alignment index table can be stored using a table structure, array structure, key-value structure, or sparse index structure, so that subsequent readings can directly locate the required three types of data from the current sequence alignment position.

[0096] The entity traversal cursor set controls the traversal range and order of the entity feature vector sequence. When generating the entity traversal cursor set according to the arrangement order of the entity feature vector sequence, each entity traversal cursor is positionally associated with an entity feature vector in the sequence and bound to a sequence alignment position. An entity traversal cursor can include a cursor number, entity feature vector sequence position, sequence alignment position, and read status flag. The cursor number distinguishes different traversed objects, the entity feature vector sequence position is used to read the current entity feature vector, the sequence alignment position is used to locate the current index entry in the sequence alignment index table, and the read status flag records whether the current cursor has completed reading. The arrangement order of the entity traversal cursor set is consistent with the arrangement order of the entity feature vector sequence, ensuring that the traversal process does not change the sequence relationships between entity objects.

[0097] When sequentially reading entity traversal cursors in the entity traversal cursor set, the current entity feature vector is read from the entity feature vector sequence based on the position of the entity feature vector sequence carried by the entity traversal cursor. The current entity feature vector represents the entity representation selected in this traversal, retaining the appearance, local structure, and sequence position information from the entity feature vector sequence. Simultaneously, the sequence alignment position bound to the entity traversal cursor is obtained and used as the current sequence alignment position. The current sequence alignment position is not derived from the current audio feature vector or the current speaking state feature, but is carried by the entity traversal cursor, ensuring that subsequent readings always use the current entity feature vector as the primary index object.

[0098] The current sequence alignment position is used to locate the current index entry in the sequence alignment index table. The current index entry contains three types of read positions that are at the same sequence alignment position as the current entity feature vector. Based on the audio feature vector read position in the current index entry, the current audio feature vector is read from the audio feature vector sequence; based on the speech state feature read position in the current index entry, the current speech state feature is read from the speech state feature sequence; based on the entity image region binary space mask read position in the current index entry, the current entity image region binary space mask is read from the entity image region binary space mask sequence. The current audio feature vector is used to provide the audio representation paired with the current entity feature vector; the current speech state feature is used to provide the speech state representation paired with the current entity feature vector; and the current entity image region binary space mask is used to provide the spatial range constraint data paired with the current entity feature vector.

[0099] A sequence alignment index table, an entity traversal cursor set, the current sequence alignment position, and the current index entry together form a traceable reading mechanism. The entity traversal cursor is responsible for determining the current entity feature vector, the current sequence alignment position is responsible for associating with the sequence alignment index table, and the current index entry is responsible for locating the current audio feature vector, the current speech state feature, and the current entity image region binary space mask. The reading results maintain the same sequence alignment position relationship, avoiding reading misalignment caused by differences in the arrangement of the audio feature vector sequence, speech state feature sequence, or entity image region binary space mask sequence during traversal. For inputs with varying entity numbers, the entity traversal cursor set can be generated according to the length of the entity feature vector sequence, and the sequence alignment index table can be updated according to the mapping relationship, thus maintaining stable reading rules even when the number of entities changes.

[0100] This embodiment records the correspondence between sequence alignment positions and three types of reading positions through a sequence alignment index table. This allows for direct location of the current audio feature vector, current speech state feature, and current entity image region binary space mask when traversing the entity feature vector sequence, reducing reading instability caused by repeated matching of multiple sequences. Reading according to the order of the entity feature vector sequence using the entity traversal cursor set ensures the source of the current entity feature vector is clear and maintains the sequence relationship between entity objects. Locating the current index item by the current sequence alignment position and then reading the corresponding data from the current index item ensures that the current entity feature vector, current audio feature vector, current speech state feature, and current entity image region binary space mask maintain the same sequence position relationship, thereby reducing the risk of audio object mismatch, speech state mismatch, and mask mismatch during subsequent driving generation.

[0101] In one embodiment, step S40 above includes: S401, Separate the entity identity preservation component, local structure component and sequence position component from the current entity feature vector, and generate query features based on the entity identity preservation component, the local structure component and the sequence position component; S402, Separate the audio rhythm component, phoneme transition component and acoustic energy component from the current audio feature vector, generate key features based on the audio rhythm component and the phoneme transition component, and generate value features based on the phoneme transition component and the acoustic energy component; S403, Generate a cross-modal attention weight sequence based on the query features and the key features, and impose positional constraints on the cross-modal attention weight sequence according to the sequence position components to generate a position-constrained attention weight sequence; S404, Extract audio-driven response features from the value features based on the position-constrained attention weight sequence; S405, the audio driving response feature, the entity identity preservation component, and the local structure component are fused to generate the basic driving feature.

[0102] In this embodiment, the current entity feature vector, after entity component separation processing, is split into an entity identity preservation component, a local structure component, and a sequence position component. The entity identity preservation component originates from the appearance texture, color distribution, stable contour, and object appearance information of the current entity image region, and is used to preserve the appearance consistency of the current entity when generating basic driving features. The local structure component originates from the movable parts, boundary morphology, and internal structure of the current entity image region, and is used to preserve the local morphology of the mouth, head, contour boundaries, or other movable parts. The sequence position component originates from the alignment position of the current entity feature vector in the entity feature vector sequence, and is used to mark the positional relationship of the current entity in the multi-entity sequence. Entity component separation processing can employ channel grouping projection, gated projection, or attention decomposition to separate the appearance information, local structure information, and sequence position information in the current entity feature vector into different vector components.

[0103] The query features are generated jointly by entity identity preservation components, local structure components, and sequence position components. During query projection, the dimensions of the three types of components are first unified, and then concatenated, compressed, gated, or weighted fusion is performed to output the query features. The entity identity preservation component ensures that the query features retain the appearance orientation of the current entity, the local structure component ensures that the query features retain the morphology of the drivable regions of the current entity, and the sequence position component ensures that the query features retain the position of the current entity in the alignment sequence. The query features are used as image-side retrieval vectors in cross-attention operations, so that subsequent audio response extraction revolves around the current entity.

[0104] After audio component separation processing, the current audio feature vector is divided into audio rhythm components, phoneme transition components, and acoustic energy components. The audio rhythm component originates from changes in speech rate, pause intervals, and segment duration distribution within the current audio stream. The phoneme transition component originates from changes in the position of articulatory units, articulation boundaries, and continuous articulation variations. The acoustic energy component originates from amplitude variations, energy peaks, and weak vocalization intervals within the audio segment. Audio component separation processing can employ a combination of short-window coding, long-window context coding, and channel projection to transform the rhythmic information, articulation variation information, and energy information in the current audio feature vector into components that can participate in matching or drive response extraction.

[0105] Key features are generated from audio rhythm components and phoneme transition components. During key projection, the temporal variation information in the audio rhythm components is fused with the articulation boundary information in the phoneme transition components, outputting key features used to match query features. These key features are biased towards representing the matching basis of audio segments in terms of temporal position and articulation variation. Value features are generated from phoneme transition components and acoustic energy components. During value projection, the articulation variation content is fused with energy intensity information, outputting value features used to extract audio driving responses. Key features and value features originate from the same current audio feature vector, but use different projection processing and different component combinations to separate the audio matching basis from the audio driving content.

[0106] Query features and key features are used to generate a cross-modal attention weight sequence. When generating the cross-modal attention weight sequence, vector matching is performed on the query features and key features, and the matching results are normalized to generate the cross-modal attention weight sequence. The cross-modal attention weight sequence represents the distribution of matching strength between the current entity-side query information and the current audio-side key features. Sequence position components are used for position constraint processing. Position constraint processing applies positional constraints to the cross-modal attention weight sequence based on the current entity's position in the alignment sequence, generating a position-constrained attention weight sequence. Position constraints can employ position gating, position offsetting, alignment position masking, or sequence range compression to concentrate attention weights more within the audio response range related to the same alignment position of the current entity.

[0107] A position-constrained attention weight sequence is used for response extraction and applied to the value features. During response extraction, the value features are weighted according to the position-constrained attention weight sequence to obtain audio-driven response features. These audio-driven response features contain pronunciation variations, energy levels, and rhythmic response information extracted from the current audio feature vector, while maintaining alignment with the current entity feature vector after position constraints. This process avoids directly using the complete current audio feature vector as the driving content input, making the driving response more focused on the audio segment expression related to the current entity.

[0108] The basic driving features are generated by fusing audio driving response features, entity identity preservation components, and local structural components. During fusion, methods such as splicing compression, residual fusion, gated fusion, or channel recalibration can be used to combine the phonological driving content from the audio driving response features with the appearance information from the entity identity preservation components and the morphology of movable regions from the local structural components. The fused basic driving features simultaneously contain the driving content provided by the current audio stream and the structural constraint information provided by the current entity region, used to represent the basic audio driving expression of the current entity before it has undergone speech state adjustment and spatial mask constraints.

[0109] This embodiment separates entity identity preservation components, local structure components, and sequence position components from the current entity feature vector, and generates query features from these three types of components. This allows the image-side retrieval vector to simultaneously retain the appearance, structure, and sequence position information of the current entity. By separating audio rhythm components, phoneme transition components, and acoustic energy components from the current audio feature vector, and generating key features and value features respectively, the audio-side matching criteria and driving content can be used separately in cross-attention operations. By extracting audio-driven response features through cross-modal attention weight sequences and position-constrained attention weight sequences, the deviation of audio responses from the current entity alignment position can be reduced. By fusing audio-driven response features, entity identity preservation components, and local structure components to generate basic driving features, the basic driving features can simultaneously carry audio-driven content and current entity structure information, thereby reducing the risk of audio-driven content acting on incorrect entities in multi-entity scenarios.

[0110] In one embodiment, step S50 above includes: S501, extract the speech activation component, speech switching component and listening response component from the current speech state features; S502, Separate the basic lip-shape driving feature, the basic head micro-motion driving feature, and the basic listening response driving feature from the basic driving features; S503, based on the speech activation component, the speech switching component and the listening response component, generate driving weights corresponding to the basic driving features, the driving weights including lip movement driving weights, head micro-movement driving weights and listening response driving weights; S504, adjust the basic mouth shape driving feature based on the mouth shape movement driving weight, adjust the basic head micro-motion driving feature based on the head micro-motion driving weight, and adjust the basic listening response driving feature based on the listening response driving weight to generate the current weighted mouth shape driving feature, the current weighted head micro-motion driving feature, and the current weighted listening response driving feature; S505, the current weighted lip-sync driving feature, the current weighted head micro-motion driving feature, and the current weighted listening response driving feature are integrated to generate the current entity audio driving feature; S506, Based on the binary space mask of the current entity image region, determine the current mask core region and the current mask boundary transition region, use the current mask core region to constrain the current entity audio driving feature to generate the current core region driving feature, and use the current mask boundary transition region to constrain the current entity audio driving feature to generate the current boundary transition driving feature. S507, the current core region driving features and the current boundary transition driving features are merged to generate the current local region driving features.

[0111] In this embodiment, the current speaking state features are analyzed to form a speaking activation component, a speaking transition component, and a listening response component. The speaking activation component represents the intensity and duration of the current entity's vocalization in the current audio segment; this component can be extracted by state analysis based on speech activity response, energy distribution, and vocalization duration intervals. The speaking transition component represents the position of change between speaking and pause states; this component can be extracted by state analysis based on energy decrease intervals, pause boundaries, and state changes in adjacent segments. The listening response component represents the current entity's response tendency in non-speaking segments; this component can be generated by state analysis based on weak vocalization intervals, speaking intervals, and the vocalization states of adjacent objects. These three components together describe whether the current entity should be lip-synced, transition-driven, or listening response-driven in the current segment.

[0112] After decomposition, the basic driving features are divided into basic lip-shape driving features, basic head micro-motion driving features, and basic listening response driving features. Basic lip-shape driving features carry driving responses related to changes in mouth shape, lip opening and closing, and articulation synchronization. Basic head micro-motion driving features carry driving responses related to head angle, slight rotation, and nodding responses. Basic listening response driving features carry driving responses related to slight facial expressions, gaze changes, and responsive actions in non-speaking states. Driving decomposition can employ channel grouping, spatial region grouping, or multi-branch projection, allowing different driving contents within the basic driving features to enter different branches, reducing the mixing of different action types within the same vector space.

[0113] The speech activation component, speech switching component, and listening response component are used to generate driving weights corresponding to the basic driving features. During driving weight generation, state fusion, weight projection, and normalization are performed sequentially. State fusion merges the three types of state components, weight projection outputs lip movement driving weights, head micro-motion driving weights, and listening response driving weights, and normalization adjusts the amplitude relationship between each weight. The lip movement driving weight corresponds to the basic lip movement driving feature, the head micro-motion driving weight corresponds to the basic head micro-motion driving feature, and the listening response driving weight corresponds to the basic listening response driving feature. When the speech activation component is enhanced, the lip movement driving weight can increase; when the speech switching component is enhanced, the head micro-motion driving weight can smoothly change with the switching position; when the listening response component is enhanced, the listening response driving weight can increase.

[0114] The basic lip-sync driving features, basic head micro-motion driving features, and basic listening response driving features are each adjusted according to their corresponding driving weights to form the current weighted lip-sync driving features, current weighted head micro-motion driving features, and current weighted listening response driving features. Adjustment methods can include channel-by-channel scaling, branch gating, region weighting, or channel recalibration. Channel-by-channel scaling is suitable for situations where driving features are already organized by channel; branch gating is suitable for situations where lip-sync, head micro-motion, and listening response are output by different branches; and region weighting is suitable for situations where driving features carry spatial distribution information. The three types of currently weighted driving features after adjustment are then fused. Driving fusion can employ splicing compression, weighted summation, residual fusion, or gated fusion to generate the current entity audio driving features. The current entity audio driving features retain the current audio driving content while reflecting the intensity allocation of different driving types based on the current speaking state.

[0115] After mask parsing, the binary space mask of the current entity image region forms the current mask core region and the current mask boundary transition region. The current mask core region is derived from the spatial range that stably covers the entity in the effective mask region, and is used to carry the main driving response. The current mask boundary transition region is derived from the extended range near the mask edge, and is used to control the attenuation of the driving response at the edge of the entity region. Mask parsing can generate two types of regions through boundary extraction, region shrinking, region expansion, and boundary smoothing. After spatial constraints are applied to the current mask core region and the current entity audio driving features, the current core region driving features are generated. After spatial constraints are applied to the current mask boundary transition region and the current entity audio driving features, the current boundary transition driving features are generated. Spatial constraints can be achieved through position-wise multiplication, region sampling, spatial gating, or mask weight adjustment, so that the effective mask region retains the driving response, while the ineffective mask region reduces the driving response.

[0116] The driving features of the current core region and the driving features of the current boundary transition are fused to generate the driving features of the current local region. During region fusion, the driving response of the current core region is retained at a high level, while the driving features of the current boundary transition are smoothly fused according to the boundary distance or transition weight. This fusion method concentrates the driving response within the current entity image region, while forming continuous changes at the boundary, reducing the possibility of the driving response crossing the boundary of the current entity image region and entering adjacent regions.

[0117] This embodiment extracts speech activation, speech switching, and listening response components from the current speech state features, and generates lip-sync motion driving weights, head micro-motion driving weights, and listening response driving weights accordingly. This allows for differentiated adjustment of the lip-sync, head micro-motion, and listening response content in the basic driving features based on the current speech state. By separating the basic driving features into basic lip-sync driving features, basic head micro-motion driving features, and basic listening response driving features, and generating current weighted lip-sync driving features, current weighted head micro-motion driving features, and current weighted listening response driving features respectively, mixed interference between different action types can be reduced. By determining the current mask core region and current mask boundary transition region through the binary space mask of the current entity image region, and generating current core region driving features and current boundary transition driving features respectively, the current entity audio driving features can be confined to the current entity image region and the region boundaries can be smoothed. The generated current local region driving features simultaneously possess speech state adjustment results and spatial region constraint results, thereby reducing the risk of abnormal movement of non-speaking objects, incorrect driving of adjacent entity regions, and abrupt driving changes at the boundaries.

[0118] In one embodiment, step S60 above includes: S601, after the entity feature vector sequence is traversed, the current local region driving features generated by each traversal are written into the local region driving feature set, and a local driving source marker, a local driving amplitude marker, and a local driving space occupancy marker are generated for each current local region driving feature in the local region driving feature set. S602, generate local fusion weights based on the local driving source marker, the local driving amplitude marker, and the local driving space occupancy marker, and adjust each current local region driving feature in the local region driving feature set based on the local fusion weights to generate a weighted local region driving feature set; S603, normalize and fuse the weighted local region-driven feature set to generate fused multi-stream features; S604, extract scene condition vector, interaction condition vector and style condition vector from the text feature vector, and generate text condition gating vector based on the scene condition vector, the interaction condition vector and the style condition vector; S605, adjust the fused multi-stream features based on the text conditional gating vector to generate text-constrained multi-stream features, and perform temporal latent mapping on the text-constrained multi-stream features to generate a video latent representation; S606, decode the latent representation of the video to generate an initial video frame sequence, and perform entity region consistency correction on the initial video frame sequence based on the local driving source marker and the local driving space occupancy marker to obtain a multi-entity interactive video.

[0119] In this embodiment, after the entity feature vector sequence is traversed, the current local region driving features generated in each traversal already carry the local driving content corresponding to different entity image regions. A local region driving feature set is used to centrally store these current local region driving features. Each item in the set can be stored using a tensor block, region feature unit, or frame segment feature unit. During writing, the source order, spatial range, and driving amplitude need to be preserved to avoid losing the correspondence between different entity image regions during subsequent fusion. The local driving source marker can be generated from the region position in the entity image region sequence, the traversal position of the current entity feature vector, or the entity region number, and is used to record which entity image region the current local region driving feature originates from. The local driving amplitude marker can be obtained by statistically analyzing the channel response intensity, time period response intensity, or spatial response intensity of the current local region driving feature, and is used to reflect the amplitude of the local driving content during fusion. The local driving space occupancy marker can be generated from the binary space mask of the current entity image region, the effective spatial range of the current local region driving feature, or the region boundary coordinates, and is used to record the scope of the local driving content in the image.

[0120] The local fusion weights are generated jointly by local driving source markers, local driving amplitude markers, and local driving spatial occupancy markers. Local driving source markers distinguish different entity image regions, local driving amplitude markers adjust the strength of different local driving contents, and local driving spatial occupancy markers reflect the spatial range of different entity image regions. When generating local fusion weights, these three types of markers can be encoded, and then a normalization layer is used to generate the local fusion weights. If the amplitude of a current local region driving feature is too large, the local fusion weights can reduce the proportion of that feature in the fusion. If an entity image region occupies a large space, the local fusion weights can normalize the area based on the spatial occupancy range. If multiple entity image regions are adjacent or have close boundaries, the local fusion weights can reduce the response intensity at the boundary overlap based on the local driving spatial occupancy markers. After the local fusion weights are applied to each current local region driving feature in the local region driving feature set, a weighted local region driving feature set is formed.

[0121] When normalizing and fusing weighted local region-driven feature sets, channel normalization, spatial normalization, amplitude normalization, or temporal segment normalization can be employed. Channel normalization unifies the channel response range of different local driving contents; spatial normalization balances the fusion differences caused by the area occupied by different entity image regions; amplitude normalization limits the excessive influence of high-energy driving contents on the fusion result; and temporal segment normalization ensures that the local driving contents remain continuously changing across different time periods. During fusion, the weighted local region-driven feature set can be written into a unified spatial feature plane according to the local driving spatial occupancy markers, or the weighted local region-driven feature set can be concatenated according to the entity region order and then compressed and mapped. After normalization and fusion, a fused multi-stream feature is formed. The fused multi-stream feature retains the driving contents corresponding to multiple entity image regions and forms a unified multi-entity driving representation.

[0122] The text feature vector contains scene, interaction, and style information extracted from the text prompts. Scene condition vectors can be extracted from channels in the text feature vectors related to the visual environment, business theme, or video semantics. Interaction condition vectors can be extracted from channels in the text feature vectors related to speech relationships, response relationships, question-and-answer relationships, or screen-sharing relationships between multiple entities. Style condition vectors can be extracted from channels in the text feature vectors related to visual style, action intensity, tone of voice, or visual presentation. When extracting condition vectors, channel segmentation, attention pooling, gating selection, or multi-branch projection methods can be used to split the text feature vectors into three types of condition vectors. The text conditional gating vector is obtained by fusing the scene condition vector, interaction condition vector, and style condition vector. Fusion methods can include splicing projection, gating weighting, or attention compression, allowing the text conditional gating vector to act on the channel response, temporal response, and spatial response of the fused multi-stream features.

[0123] When adjusting the text-conditional gating vector to fuse multi-stream features, text-constrained multi-stream features can be generated through channel-wise scaling, conditional normalization, or gated multiplication. Channel-wise scaling is suitable for situations where the fused multi-stream features already have stable channel semantics; conditional normalization is suitable for situations where the feature distribution needs to be changed according to text conditions; and gated multiplication is suitable for situations where specific interactive content needs to be suppressed or enhanced. The text-constrained multi-stream features are further converted into video latent representations during the temporal latent mapping process. The temporal latent mapping process can include a temporal location coding layer, a frame segment mapping layer, and a latent compression layer. The temporal location coding layer writes the frame order relationship, the frame segment mapping layer extends the fused multi-stream driven content to multiple time segments, and the latent compression layer converts the multi-entity driven expression and the text-conditional expression into a video latent representation. The video latent representation preserves the compressed video expression after multi-entity region driving, temporal variations, and text-conditional constraints.

[0124] The video latent representation is decoded to generate an initial video frame sequence. The decoding process may include a latent upsampling layer, a frame restoration layer, a texture completion layer, and a temporal smoothing layer. The latent upsampling layer expands the compressed representation to frame-level spatial resolution; the frame restoration layer generates the initial frame content; the texture completion layer supplements entity appearance and region details; and the temporal smoothing layer reduces abrupt changes between adjacent frames. Entity region consistency correction is based on local driving source markers and local driving space occupancy markers. Local driving source markers are used to identify the driving source corresponding to each entity region in the initial video frame sequence, and local driving space occupancy markers are used to identify the spatial extent of each entity region in the frame. During entity region consistency correction, the boundary positions, texture continuity, and motion responses of the same entity region in consecutive frames can be checked in the initial video frame sequence. Region responses with inconsistent sources or spatial out-of-bounds are weakened, backfilled, or smoothed to obtain multi-entity interactive video.

[0125] The fusion processing, temporal latent mapping processing, and decoding processing can obtain parameters through phased training. In the first training phase, the training data can include multi-entity images, multiple audio streams, text prompts, and their corresponding video clips. The training objectives include video reconstruction, audio synchronization, entity region consistency, and text conditional consistency. During training, multiple entity images undergo region generation processing to obtain entity image region sequences and entity image region binary space mask sequences. Multiple audio streams undergo audio encoding processing to obtain audio feature representations. Text prompts undergo text encoding processing to obtain text feature representations. The fusion processing learns to synthesize the local driving content of multiple entities into fused multi-stream features. The temporal latent mapping processing learns to convert the fused multi-stream features and text feature representations into video latent representations. The decoding processing learns to restore the video latent representations to the video frame content.

[0126] In the second training phase, the training data can consist of a smaller number of multi-entity spoken video clips with higher image quality, clearer object boundaries, and more explicit audio-object correspondences. This second training phase can reduce the update magnitude of the low-level visual coding parameters in the generation parameters, while increasing the update magnitude of parameters related to entity region consistency, lip-sync, boundary stability, and inter-frame continuity in the fusion, temporal latent mapping, and decoding processes. Training objectives can include entity region boundary stability, local driver consistency, audio lip-sync, and inter-frame smoothing. After completing the second training phase, the fusion process can maintain amplitude balance among multiple current local region-driven features, the temporal latent mapping process can generate video latent representations based on text feature vectors and fused multi-stream features, and the decoding process can generate multi-entity interactive videos from the video latent representations.

[0127] This embodiment preserves the source, intensity, and spatial extent of different entity regions before fusion by writing the current local region driving features generated in each traversal into a local region driving feature set and generating a local driving source marker, a local driving amplitude marker, and a local driving spatial occupancy marker for each current local region driving feature. Adjusting the current local region driving features with local fusion weights and performing normalized fusion reduces the fusion bias caused by differences in the area, driving amplitude, and spatial occupancy of different entity regions. Extracting scene condition vectors, interaction condition vectors, and style condition vectors from text feature vectors and generating text condition gating vectors transforms text conditions into adjustment information for fusing multi-stream features. Generating a video latent representation through temporal latent mapping and using local driving source markers and local driving spatial occupancy markers for entity region consistency correction after decoding reduces problems such as excessive local driving superposition, entity region misalignment, interference between adjacent regions, and discontinuity between frames in multi-entity videos.

[0128] In one embodiment, a video generation apparatus based on multi-stream cross-attention is provided, which corresponds one-to-one with the video generation method based on multi-stream cross-attention described in the above embodiments. (Refer to...) Figure 3 , Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the video generation device based on multi-stream cross-attention of the present invention. The modules include an input mapping preprocessing module 10, a multi-stream feature alignment module 20, an alignment position reading module 30, a cross-attention driving module 40, a state weight constraint module 50, and a fusion decoding generation module 60. Detailed descriptions of each functional module are as follows: The input mapping preprocessing module 10 is used to acquire multiple entity images, multiple audio streams and text prompts, generate entity image region sequences and entity image region binary space mask sequences based on the multiple entity images, establish the mapping relationship between the entity image region sequences and the multiple audio streams, and extract text feature vectors from the text prompts; The multi-stream feature alignment module 20 is used to extract features from the entity image region sequence to generate an entity feature vector sequence, perform temporal encoding and speech state extraction on the multiple audio streams to generate an audio feature vector sequence and a speech state feature sequence, and align the entity feature vector sequence, the audio feature vector sequence, the speech state feature sequence and the entity image region binary space mask sequence with each other based on the mapping relationship. The alignment position reading module 30 is used to traverse the entity feature vector sequence and, for each traversal of the current entity feature vector, read the current audio feature vector, the current speaking state feature, and the current entity image region binary space mask that are aligned at the same sequence position. The cross-attention driving module 40 is used to perform cross-attention operations on the current entity feature vector as a query feature and the current audio feature vector as a key feature and value feature to generate basic driving features; The state weight constraint module 50 is used to adjust the driving weight of the basic driving feature based on the current speaking state feature, generate the current entity audio driving feature, and constrain the current entity audio driving feature using the binary space mask of the current entity image region to generate the current local region driving feature. The fusion decoding generation module 60 is used to normalize and fuse the current local region driving features generated in each traversal after the entity feature vector sequence is traversed, generate fused multi-stream features, generate and decode video latent representations based on the text feature vectors and the fused multi-stream features, and obtain multi-entity interactive video.

[0129] In one embodiment, the input mapping preprocessing module 10 is specifically used for: Acquire multiple entity images, multiple audio streams, and text prompts; extract entity reference fragments, audio reference fragments, interaction relationship fragments, scene semantic fragments, and generation style fragments from the text prompts; extract entity candidate regions from the multiple entity images; and generate a set of entity structure key points, entity outline boundaries, and entity region confidence for each entity candidate region. Based on the set of key points of the entity structure, the entity contour boundary, and the entity region confidence, target entity regions are selected from the entity candidate regions. The initial arrangement number of the target entity regions is determined based on the entity image input order. The reference correction number of the target entity regions is determined based on the spatial position of the target entity regions in the image and the entity reference fragment. The entity region arrangement number is generated according to the initial arrangement number and the reference correction number. The target entity regions are arranged according to the entity region arrangement number to generate an entity image region sequence. For each entity image region in the entity image region sequence, a driveable core region and a boundary transition region are determined. A binary space mask for each entity image region is generated based on the driveable core region and the boundary transition region. The binary space masks for each entity image region are arranged into a binary space mask sequence according to the order of the entity image region sequence. Audio stream channel markers and audio subject indication information are extracted from the multiple audio streams. Based on the entity reference segment, the audio reference segment, the audio stream channel marker, the audio subject indication information, and the entity image region sequence, a mapping relationship between the entity image region sequence and the multiple audio streams is established. The scene semantic fragment, the interaction relationship fragment, and the generation style fragment are encoded and fused to generate a text feature vector.

[0130] In one embodiment, the multi-stream feature alignment module 20 is specifically used for: Based on the binary space mask sequence of the entity image region, the drivable region image block and contour neighborhood image block of each entity image region are determined from the entity image region sequence, and the region position mark is determined based on the arrangement position of each entity image region in the entity image region sequence. The entity appearance features of the drivable region image block, the boundary structure features of the contour neighborhood image block, and the spatial order features corresponding to the region position mark are extracted respectively. By fusing the entity appearance features, the boundary structure features, and the spatial order features, an entity region feature vector corresponding to each entity image region is generated, and the entity region feature vectors are arranged according to the order of the entity image region sequence to generate an entity feature vector sequence. Based on the mapping relationship, audio stream alignment marks are configured for multiple audio streams, and segmentation, temporal acoustic coding, phoneme change mark extraction and pause boundary mark extraction are performed on multiple audio streams according to the audio stream alignment marks, generating audio segment feature set, phoneme change mark set and pause boundary mark set; An audio feature vector sequence is generated based on the audio segment feature set and the phoneme change marker set, and a speech state feature sequence is generated based on the audio segment feature set, the phoneme change marker set, and the pause boundary marker set; Based on the mapping relationship, a sequence alignment position marker is generated, and the sequence alignment position marker is configured for the entity feature vector sequence, the audio feature vector sequence, the speech state feature sequence, and the entity image region binary space mask sequence, so that the entity feature vector sequence, the audio feature vector sequence, the speech state feature sequence, and the entity image region binary space mask sequence are aligned with each other.

[0131] In one embodiment, the alignment position reading module 30 is specifically used for: Based on the mutually aligned entity feature vector sequence, audio feature vector sequence, speech state feature sequence, and entity image region binary space mask sequence, a sequence alignment index table is generated. The sequence alignment index table includes sequence alignment position, audio feature vector reading position, speech state feature reading position, and entity image region binary space mask reading position. Generate an entity traversal cursor set according to the arrangement order of the entity feature vector sequence, and bind each entity traversal cursor in the entity traversal cursor set to a sequence alignment position; The entity traversal cursors in the entity traversal cursor set are read sequentially. Based on the entity traversal cursors, the current entity feature vector is read in the entity feature vector sequence, and the sequence alignment position bound to the entity traversal cursor is determined as the current sequence alignment position. Based on the current sequence alignment position, locate the current index item in the sequence alignment index table, and based on the audio feature vector reading position, speech state feature reading position, and entity image region binary space mask reading position in the current index item, read the current audio feature vector, the current speech state feature, and the current entity image region binary space mask from the audio feature vector sequence, the speech state feature sequence, and the entity image region binary space mask sequence, respectively.

[0132] In one embodiment, the cross-attention driving module 40 is specifically used for: Separate the entity identity preservation component, local structure component, and sequence position component from the current entity feature vector, and generate query features based on the entity identity preservation component, the local structure component, and the sequence position component; Separate the audio rhythm component, phoneme transition component, and acoustic energy component from the current audio feature vector; generate key features based on the audio rhythm component and the phoneme transition component; and generate value features based on the phoneme transition component and the acoustic energy component. A cross-modal attention weight sequence is generated based on the query features and the key features, and the position of the cross-modal attention weight sequence is constrained according to the positional components of the sequence to generate a position-constrained attention weight sequence. Based on the position-constrained attention weight sequence, audio-driven response features are extracted from the value features; The basic driving features are generated by fusing the audio driving response features, the entity identity preservation components, and the local structure components.

[0133] In one embodiment, the state weight constraint module 50 is specifically used for: Extract speech activation component, speech switching component, and listening response component from the current speech state characteristics; From the aforementioned basic driving features, we separate the basic lip-shape driving features, the basic head micro-motion driving features, and the basic listening response driving features; Based on the speech activation component, the speech switching component, and the listening response component, driving weights corresponding to the basic driving features are generated. The driving weights include lip movement driving weights, head micro-movement driving weights, and listening response driving weights. The basic mouth shape driving feature is adjusted based on the mouth shape movement driving weight, the basic head micro-motion driving feature is adjusted based on the head micro-motion driving weight, and the basic listening response driving feature is adjusted based on the listening response driving weight, thereby generating the current weighted mouth shape driving feature, the current weighted head micro-motion driving feature, and the current weighted listening response driving feature. The current weighted lip-sync driving features, the current weighted head micro-motion driving features, and the current weighted listening response driving features are combined to generate the current entity audio driving features; Based on the binary space mask of the current entity image region, determine the current mask core region and the current mask boundary transition region. Use the current mask core region to constrain the current entity audio driving features to generate the current core region driving features. Use the current mask boundary transition region to constrain the current entity audio driving features to generate the current boundary transition driving features. The current core region driving features and the current boundary transition driving features are combined to generate the current local region driving features.

[0134] In one embodiment, the fusion decoding generation module 60 is specifically used for: After the entity feature vector sequence is traversed, the current local region driving features generated in each traversal are written into the local region driving feature set, and a local driving source marker, a local driving amplitude marker, and a local driving space occupancy marker are generated for each current local region driving feature in the local region driving feature set. A local fusion weight is generated based on the local driving source marker, the local driving amplitude marker, and the local driving space occupancy marker. Based on the local fusion weight, each current local region driving feature in the local region driving feature set is adjusted to generate a weighted local region driving feature set. The weighted local region-driven feature set is normalized and fused to generate fused multi-stream features; Extract scene condition vector, interaction condition vector and style condition vector from the text feature vector, and generate text condition gating vector based on the scene condition vector, the interaction condition vector and the style condition vector; Based on the text conditional gating vector, the fused multi-stream features are adjusted to generate text-constrained multi-stream features, and the text-constrained multi-stream features are subjected to temporal latent mapping to generate a video latent representation. The latent representation of the video is decoded to generate an initial video frame sequence. Based on the local driving source marker and the local driving space occupancy marker, the initial video frame sequence is corrected for entity region consistency to obtain a multi-entity interactive video.

[0135] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides determination and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When executed by the processor, the computer program implements the functions or steps of a server-side video generation method based on multi-stream cross-attention.

[0136] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 5As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides determination and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements client-side functions or steps of a video generation method based on multi-stream cross-attention.

[0137] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: Acquire multiple entity images, multiple audio streams, and text prompts; generate entity image region sequences and entity image region binary space mask sequences based on the multiple entity images; establish a mapping relationship between the entity image region sequences and the multiple audio streams; and extract text feature vectors from the text prompts. Feature extraction is performed on the entity image region sequence to generate an entity feature vector sequence. Temporal encoding and speech state extraction are performed on the multiple audio streams to generate an audio feature vector sequence and a speech state feature sequence. Based on the mapping relationship, the entity feature vector sequence, the audio feature vector sequence, the speech state feature sequence, and the entity image region binary space mask sequence are aligned with each other. Traverse the entity feature vector sequence, and for each current entity feature vector obtained in each traversal, read the current audio feature vector, the current speaking state feature, and the current entity image region binary space mask that are aligned with the same sequence position; The current entity feature vector is used as the query feature, and the current audio feature vector is used as the key feature and value feature to perform cross-attention operation to generate basic driving features; Based on the current speaking state features, the driving weights of the basic driving features are adjusted to generate the current entity audio driving features. The current entity image region binary space mask is used to constrain the current entity audio driving features to generate the current local region driving features. After the entity feature vector sequence is traversed, the current local region driving features generated in each traversal are normalized and fused to generate fused multi-stream features. Based on the text feature vector and the fused multi-stream features, the video latent representation is generated and decoded to obtain a multi-entity interactive video.

[0138] In one embodiment, a computer-readable storage medium is provided, which may be non-volatile or volatile, and a computer program is stored thereon, which, when executed by a processor, performs the following steps: Acquire multiple entity images, multiple audio streams, and text prompts; generate entity image region sequences and entity image region binary space mask sequences based on the multiple entity images; establish a mapping relationship between the entity image region sequences and the multiple audio streams; and extract text feature vectors from the text prompts. Feature extraction is performed on the entity image region sequence to generate an entity feature vector sequence. Temporal encoding and speech state extraction are performed on the multiple audio streams to generate an audio feature vector sequence and a speech state feature sequence. Based on the mapping relationship, the entity feature vector sequence, the audio feature vector sequence, the speech state feature sequence, and the entity image region binary space mask sequence are aligned with each other. Traverse the entity feature vector sequence, and for each current entity feature vector obtained in each traversal, read the current audio feature vector, the current speaking state feature, and the current entity image region binary space mask that are aligned with the same sequence position; The current entity feature vector is used as the query feature, and the current audio feature vector is used as the key feature and value feature to perform cross-attention operation to generate basic driving features; Based on the current speaking state features, the driving weights of the basic driving features are adjusted to generate the current entity audio driving features. The current entity image region binary space mask is used to constrain the current entity audio driving features to generate the current local region driving features. After the entity feature vector sequence is traversed, the current local region driving features generated in each traversal are normalized and fused to generate fused multi-stream features. Based on the text feature vector and the fused multi-stream features, the video latent representation is generated and decoded to obtain a multi-entity interactive video.

[0139] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0140] In one embodiment, a flowchart of a video generation method based on multi-stream cross-attention is provided for reference. Figure 6 .like Figure 6As shown, Identity 1 image, Identity 2 image to Identity N images are input as multiple identity images, where N represents the number of identities and can be expanded. Multiple identity images are input to the CLIP visual encoder. CLIP stands for Contrastive Language-Image Pre-training, which can be understood as contrastive language image pre-training, used to encode visual features of identity images and generate a sequence of identity feature vectors. The identity feature vector sequence is used to represent the image-side features corresponding to multiple identity images, and in this embodiment, it can be used as a specific implementation of an entity feature vector sequence. Audio 1, Audio 2 to Audio N are input as multiple audio inputs. Multiple audio inputs are input to the Wav2Vec2 audio encoder. Wav2Vec2 stands for Wave to Vector 2.0, which can be called a speech vector representation model, used to encode audio features of multiple audios and generate a sequence of audio feature vectors. The audio feature vector sequence is used to represent the audio-side driving expressions corresponding to different audios. Text prompts are input to the T5 text encoder. T5 stands for Text-to-Text Transfer Transformer, which can be understood as a text-to-text transfer transformer, used to encode text prompts and generate text feature vectors. Text feature vectors provide contextual information such as scene semantics, interaction relationships, and generation style to subsequent generation processes. The identity feature vector sequence and audio feature vector sequence are input to the audio-face cross-attention module. The audio-face cross-attention module includes iterative processing of N pairs of identity-audio features, cross-attention calculation, spatial mask constraints, and accumulation to global features. Iterative processing of N pairs of identity-audio features means processing identity features and audio features group by group according to the number of identities. Cross-attention calculation means using the current identity feature and the current audio feature for cross-modal feature interaction. Spatial mask constraints mean using a spatial mask corresponding to the current identity image region to limit the scope of the audio-driven features. Accumulation to global features means incorporating the local driving results obtained from each processing into the global feature representation. The audio-face cross-attention module outputs fused image multi-stream features. Fused image multi-stream features represent a unified feature representation formed by accumulating or fusing the local driving results corresponding to multiple identity images, and can correspond to fused multi-stream features. Text feature vectors and fused image multi-stream features are jointly input to the diffusion Transformer block. The Transformer is used to generate and model text feature vectors and fused multi-stream image features to obtain a video latent representation. The video latent representation output from the Diffusion Transformer block is input to the VAE decoder. VAE stands for Variational Autoencoder, which decodes the video latent representation into video frame content and outputs the generated multi-person spoken video. Figure 6The right side illustrates the first and second stages of training. The first stage of training is used to acquire basic generation capabilities, while the second stage is used to further refine the generation results with limited real-person data. Both the first and second stages of training pertain to the model parameter acquisition process, used to train or fine-tune relevant parameters in the CLIP visual encoder, Wav2Vec2 audio encoder, T5 text encoder, audio-face cross-attention module, diffusion Transformer block, and VAE decoder.

[0141] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0142] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0143] It should be noted that any AI models, software tools, or components not belonging to this company appearing in the embodiments of this application are merely illustrative examples and do not represent actual use. All user personal information involved in the embodiments of this application has been authorized (with the knowledge and consent) by the relevant parties or has been fully authorized by all parties, and the executing entity may obtain it through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with relevant laws and regulations and do not violate public order and good morals.

[0144] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A video generation method based on multi-stream cross-attention, characterized in that, Includes the following steps: Acquire multiple entity images, multiple audio streams, and text prompts; generate entity image region sequences and entity image region binary space mask sequences based on the multiple entity images; establish a mapping relationship between the entity image region sequences and the multiple audio streams; and extract text feature vectors from the text prompts. Feature extraction is performed on the entity image region sequence to generate an entity feature vector sequence. Temporal encoding and speech state extraction are performed on the multiple audio streams to generate an audio feature vector sequence and a speech state feature sequence. Based on the mapping relationship, the entity feature vector sequence, the audio feature vector sequence, the speech state feature sequence, and the entity image region binary space mask sequence are aligned with each other. Traverse the entity feature vector sequence, and for each current entity feature vector obtained in each traversal, read the current audio feature vector, the current speaking state feature, and the current entity image region binary space mask that are aligned with the same sequence position; The current entity feature vector is used as the query feature, and the current audio feature vector is used as the key feature and value feature to perform cross-attention operation to generate basic driving features; Based on the current speaking state features, the driving weights of the basic driving features are adjusted to generate the current entity audio driving features. The current entity image region binary space mask is used to constrain the current entity audio driving features to generate the current local region driving features. After the entity feature vector sequence traversal is completed, the current local region driving features generated in each traversal are normalized and fused to generate fused multi-stream features. Based on the text feature vectors and the fused multi-stream features, a video latent representation is generated and decoded to obtain a multi-entity interactive video. This includes: after the entity feature vector sequence traversal is completed, the current local region driving features generated in each traversal are written into a local region driving feature set, and a local driving source marker, a local driving amplitude marker, and a local driving space occupancy marker are generated for each current local region driving feature in the local region driving feature set; a local fusion weight is generated based on the local driving source marker, the local driving amplitude marker, and the local driving space occupancy marker, and the local fusion weight is used to adjust the local region driving features in the local region driving feature set. Each current local region driving feature is used to generate a weighted local region driving feature set; the weighted local region driving feature set is normalized and fused to generate fused multi-stream features; scene condition vector, interaction condition vector, and style condition vector are extracted from the text feature vector, and a text condition gating vector is generated based on the scene condition vector, the interaction condition vector, and the style condition vector; the fused multi-stream features are adjusted based on the text condition gating vector to generate text-constrained multi-stream features, and temporal latent mapping is performed on the text-constrained multi-stream features to generate a video latent representation; the video latent representation is decoded to generate an initial video frame sequence, and entity region consistency correction is performed on the initial video frame sequence based on the local driving source marker and the local driving space occupancy marker to obtain a multi-entity interactive video.

2. The video generation method based on multi-stream cross-attention as described in claim 1, characterized in that, The process involves acquiring multiple entity images, multiple audio streams, and text prompts; generating entity image region sequences and entity image region binary space mask sequences based on the multiple entity images; establishing a mapping relationship between the entity image region sequences and the multiple audio streams; and extracting text feature vectors from the text prompts, including: Acquire multiple entity images, multiple audio streams, and text prompts; extract entity reference fragments, audio reference fragments, interaction relationship fragments, scene semantic fragments, and generation style fragments from the text prompts; extract entity candidate regions from the multiple entity images; and generate a set of entity structure key points, entity outline boundaries, and entity region confidence for each entity candidate region. Based on the set of key points of the entity structure, the entity contour boundary, and the entity region confidence, target entity regions are selected from the entity candidate regions. The initial arrangement number of the target entity regions is determined based on the entity image input order. The reference correction number of the target entity regions is determined based on the spatial position of the target entity regions in the image and the entity reference fragment. The entity region arrangement number is generated according to the initial arrangement number and the reference correction number. The target entity regions are arranged according to the entity region arrangement number to generate an entity image region sequence. For each entity image region in the entity image region sequence, a driveable core region and a boundary transition region are determined. A binary space mask for each entity image region is generated based on the driveable core region and the boundary transition region. The binary space masks for each entity image region are arranged into a binary space mask sequence according to the order of the entity image region sequence. Audio stream channel markers and audio subject indication information are extracted from the multiple audio streams. Based on the entity reference segment, the audio reference segment, the audio stream channel marker, the audio subject indication information, and the entity image region sequence, a mapping relationship between the entity image region sequence and the multiple audio streams is established. The scene semantic fragment, the interaction relationship fragment, and the generation style fragment are encoded and fused to generate a text feature vector.

3. The video generation method based on multi-stream cross-attention as described in claim 1, characterized in that, Feature extraction is performed on the entity image region sequence to generate an entity feature vector sequence. Temporal encoding and speech state extraction are performed on the multiple audio streams to generate an audio feature vector sequence and a speech state feature sequence. Based on the mapping relationship, the entity feature vector sequence, the audio feature vector sequence, the speech state feature sequence, and the entity image region binary space mask sequence are aligned with each other, including: Based on the binary space mask sequence of the entity image region, the drivable region image block and contour neighborhood image block of each entity image region are determined from the entity image region sequence, and the region position mark is determined based on the arrangement position of each entity image region in the entity image region sequence. The entity appearance features of the drivable region image block, the boundary structure features of the contour neighborhood image block, and the spatial order features corresponding to the region position mark are extracted respectively. By fusing the entity appearance features, the boundary structure features, and the spatial order features, an entity region feature vector corresponding to each entity image region is generated, and the entity region feature vectors are arranged according to the order of the entity image region sequence to generate an entity feature vector sequence. Based on the mapping relationship, audio stream alignment marks are configured for multiple audio streams, and segmentation, temporal acoustic coding, phoneme change mark extraction and pause boundary mark extraction are performed on multiple audio streams according to the audio stream alignment marks, generating audio segment feature set, phoneme change mark set and pause boundary mark set; An audio feature vector sequence is generated based on the audio segment feature set and the phoneme change marker set, and a speech state feature sequence is generated based on the audio segment feature set, the phoneme change marker set, and the pause boundary marker set; Based on the mapping relationship, a sequence alignment position marker is generated, and the sequence alignment position marker is configured for the entity feature vector sequence, the audio feature vector sequence, the speech state feature sequence, and the entity image region binary space mask sequence, so that the entity feature vector sequence, the audio feature vector sequence, the speech state feature sequence, and the entity image region binary space mask sequence are aligned with each other.

4. The video generation method based on multi-stream cross-attention as described in claim 1, characterized in that, Traverse the entity feature vector sequence, and for each entity feature vector obtained in each traversal, read the current audio feature vector, the current speaking state feature, and the current entity image region binary space mask that are aligned with the same sequence position, including: Based on the mutually aligned entity feature vector sequence, audio feature vector sequence, speech state feature sequence, and entity image region binary space mask sequence, a sequence alignment index table is generated. The sequence alignment index table includes sequence alignment position, audio feature vector reading position, speech state feature reading position, and entity image region binary space mask reading position. Generate an entity traversal cursor set according to the arrangement order of the entity feature vector sequence, and bind each entity traversal cursor in the entity traversal cursor set to a sequence alignment position; The entity traversal cursors in the entity traversal cursor set are read sequentially. Based on the entity traversal cursors, the current entity feature vector is read in the entity feature vector sequence, and the sequence alignment position bound to the entity traversal cursor is determined as the current sequence alignment position. Based on the current sequence alignment position, locate the current index item in the sequence alignment index table, and based on the audio feature vector reading position, speech state feature reading position, and entity image region binary space mask reading position in the current index item, read the current audio feature vector, the current speech state feature, and the current entity image region binary space mask from the audio feature vector sequence, the speech state feature sequence, and the entity image region binary space mask sequence, respectively.

5. The video generation method based on multi-stream cross-attention as described in claim 1, characterized in that, Using the current entity feature vector as the query feature and the current audio feature vector as both key and value features, a cross-attention operation is performed to generate basic driving features, including: Separate the entity identity preservation component, local structure component, and sequence position component from the current entity feature vector, and generate query features based on the entity identity preservation component, the local structure component, and the sequence position component; Separate the audio rhythm component, phoneme transition component, and acoustic energy component from the current audio feature vector; generate key features based on the audio rhythm component and the phoneme transition component; and generate value features based on the phoneme transition component and the acoustic energy component. A cross-modal attention weight sequence is generated based on the query features and the key features, and the position of the cross-modal attention weight sequence is constrained according to the positional components of the sequence to generate a position-constrained attention weight sequence. Based on the position-constrained attention weight sequence, audio-driven response features are extracted from the value features; The basic driving features are generated by fusing the audio driving response features, the entity identity preservation components, and the local structure components.

6. The video generation method based on multi-stream cross-attention as described in claim 1, characterized in that, Based on the current speaking state features, the driving weights of the basic driving features are adjusted to generate the current entity audio driving features. Then, the current entity audio driving features are constrained using the binary space mask of the current entity image region to generate the current local region driving features, including: Extract speech activation component, speech switching component, and listening response component from the current speech state characteristics; From the aforementioned basic driving features, we separate the basic lip-shape driving features, the basic head micro-motion driving features, and the basic listening response driving features; Based on the speech activation component, the speech switching component, and the listening response component, driving weights corresponding to the basic driving features are generated. The driving weights include lip movement driving weights, head micro-movement driving weights, and listening response driving weights. The basic mouth shape driving feature is adjusted based on the mouth shape movement driving weight, the basic head micro-motion driving feature is adjusted based on the head micro-motion driving weight, and the basic listening response driving feature is adjusted based on the listening response driving weight, thereby generating the current weighted mouth shape driving feature, the current weighted head micro-motion driving feature, and the current weighted listening response driving feature. The current weighted lip-sync driving features, the current weighted head micro-motion driving features, and the current weighted listening response driving features are combined to generate the current entity audio driving features; Based on the binary space mask of the current entity image region, determine the current mask core region and the current mask boundary transition region. Use the current mask core region to constrain the current entity audio driving features to generate the current core region driving features. Use the current mask boundary transition region to constrain the current entity audio driving features to generate the current boundary transition driving features. The current core region driving features and the current boundary transition driving features are combined to generate the current local region driving features.

7. A video generation device based on multi-stream cross-attention, characterized in that, The video generation device based on multi-stream cross-attention includes: The input mapping preprocessing module is used to acquire multiple entity images, multiple audio streams and text prompts, generate entity image region sequences and entity image region binary space mask sequences based on the multiple entity images, establish the mapping relationship between the entity image region sequences and the multiple audio streams, and extract text feature vectors from the text prompts; The multi-stream feature alignment module is used to extract features from the entity image region sequence to generate an entity feature vector sequence, perform temporal encoding and speech state extraction on the multiple audio streams to generate an audio feature vector sequence and a speech state feature sequence, and align the entity feature vector sequence, the audio feature vector sequence, the speech state feature sequence, and the entity image region binary space mask sequence with each other based on the mapping relationship. The alignment position reading module is used to traverse the entity feature vector sequence and, for each traversal, read the current audio feature vector, the current speaking state feature, and the current entity image region binary space mask that are aligned at the same sequence position. The cross-attention driven module is used to perform cross-attention operations on the current entity feature vector as a query feature and the current audio feature vector as a key feature and value feature to generate basic driving features; The state weight constraint module is used to adjust the driving weight of the basic driving feature based on the current speaking state feature, generate the current entity audio driving feature, and constrain the current entity audio driving feature using the binary space mask of the current entity image region to generate the current local region driving feature. The fusion decoding generation module is used to normalize and fuse the current local region driving features generated in each traversal after the entity feature vector sequence traversal is completed, generating fused multi-stream features, and generating and decoding video latent representations based on the text feature vectors and the fused multi-stream features to obtain multi-entity interactive video. This includes: after the entity feature vector sequence traversal is completed, writing the current local region driving features generated in each traversal into a local region driving feature set, and generating a local driving source marker, a local driving amplitude marker, and a local driving space occupancy marker for each current local region driving feature in the local region driving feature set; generating local fusion weights based on the local driving source markers, the local driving amplitude markers, and the local driving space occupancy markers, and adjusting the local region driving features based on the local fusion weights. Each current local region driving feature in the feature set is used to generate a weighted local region driving feature set; the weighted local region driving feature set is normalized and fused to generate fused multi-stream features; scene condition vector, interaction condition vector, and style condition vector are extracted from the text feature vector, and a text condition gating vector is generated based on the scene condition vector, the interaction condition vector, and the style condition vector; the fused multi-stream features are adjusted based on the text condition gating vector to generate text-constrained multi-stream features, and the text-constrained multi-stream features are temporally latently mapped to generate a video latent representation; the video latent representation is decoded to generate an initial video frame sequence, and the initial video frame sequence is corrected for entity region consistency based on the local driving source marker and the local driving space occupancy marker to obtain a multi-entity interactive video.

8. A computer device, characterized in that, The computer device includes a memory, a processor, and a multi-stream cross-attention-based video generation program stored in the memory and executable on the processor, wherein the multi-stream cross-attention-based video generation program, when executed by the processor, implements the steps of the multi-stream cross-attention-based video generation method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The storage medium stores a video generation program based on multi-stream cross-attention, which, when executed by a processor, implements the steps of the video generation method based on multi-stream cross-attention as described in any one of claims 1-6.

Images