A multimedia content-driven method and system for bidirectional conversion and generation of VR digital human scenes and AI short videos

By employing technologies such as Huber robust weighted temporal anchoring, shared basis with modal low-rank projection, and synchronous hypergraph propagation, the problem of semantic fragmentation between VR digital human short videos and 3D scenes with multiple source materials has been solved, achieving reversible consistency of character, action, camera, and scene tags, and improving conversion accuracy and efficiency.

CN123075239APending Publication Date: 2026-11-10BEIJING JINGYUAN CHANGXIANG DIGITAL TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611470852.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-09-21
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

In the bidirectional conversion between VR digital human short videos and 3D scenes, the semantic fragmentation of multi-source materials in existing technologies makes it impossible to achieve reversible consistency in character movements, shot timing, and scene spatial labels, resulting in conversion offset and low efficiency.

Method used

The Huber robust weighted time anchoring mechanism is used for multimodal event time alignment. A shared basis plus modal low-rank incremental projection and synchronous hypergraph propagation mechanism are used for cross-modal semantic alignment and label decoupling. Modal conditional reversible affine coupling and Softmax category routing quantization are combined to generate reversible semantic token sequences. Reversible encoding and decoding of 2D AI short videos and 3D VR digital human scenes are achieved through semantic-geometric joint optimization.

Benefits of technology

It improves cross-modal semantic alignment accuracy, ensures consistency of character, action, shot, and scene tags, enhances the reversible reconstruction reliability and conversion consistency of VR digital human short video mass production, and improves production efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123075239A_ABST
    Figure CN123075239A_ABST
Patent Text Reader

Abstract

This invention relates to the field of multimedia content generation and processing technology, and discloses a method and system for bidirectional conversion and generation of VR digital human scenes and AI short videos driven by multimedia content. The method includes: acquiring multi-source materials; performing cross-modal semantic alignment and tag decoupling based on a unified temporal anchor point using a shared basis plus modal low-rank incremental projection and synchronous hypergraph propagation mechanism; generating 2D AI short videos and 3D VR digital human scenes by jointly driving multiple decoders based on reversible semantic token sequences; extracting 2D AI short videos from the bimodal generated content set, and performing reverse parsing reconstruction using a semantic-geometric joint optimization mechanism to obtain 3D VR digital human interactive scenes. Because this application achieves bidirectional consistent conversion between 2D short videos and 3D VR scenes through the hierarchical connection of unified temporal anchor points, decoupled semantic vectors, and reversible semantic token sequences, it improves the consistency of generated content tags and the accuracy of scene reconstruction.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of multimedia content generation and processing technology, and in particular to a method and system for bidirectional conversion and generation of VR digital human scenes and AI short videos driven by multimedia content. Background Technology

[0002] Currently, in the mass production of VR digital human short videos and the construction of 3D VR scenes, multiple sources of materials such as text, audio, images, and video are typically input. Semantic features of the text, acoustic features of the audio, visual features of the images, and action and shot features of the video are extracted separately. Then, 2D AI short videos are generated through methods such as timestamp alignment, feature stitching, attention fusion, or rule-driven templates. Alternatively, they can be further converted into 3D VR digital human scenes using camera estimation, pose estimation, and depth estimation. In existing processing paths, character identity, character actions, shot sequence, scene space, and narrative tags are often scattered across different encoders, matching rules, and post-processing modules. Cross-modal synchronization often relies on shallow feature alignment or manual thresholding, lacking a unified, reversible, and retrievable semantic intermediate representation.

[0003] For example, when text describes character actions, audio contains beat accents, images provide key moments, and videos contain shot transitions, each modality differs significantly in sampling rate, temporal granularity, semantic level, and confidence. The features extracted separately fall into fragmented semantic spaces. Once the action time window and the shot transition time window are misaligned, or the audio accent does not coincide with the narrative center, character action labels, shot sequence labels, and scene space labels become difficult to reversibly align. When reconstructing a 3D VR digital human interaction scene from a 2D AI short video, due to the lack of a unified cross-modal semantic space and reversible semantic token constraints, camera trajectory, skeletal movements, spatial layout, and narrative rhythm will accumulate errors at each level in pose estimation, depth estimation, and scene reconstruction. This leads to character identity drift, inconsistencies between actions and shots, disordered scene spatial relationships, and narrative label shifts, amplifying the bidirectional conversion shifts caused by the semantic fragmentation of multi-source materials.

[0004] Therefore, there is an urgent need for a method that can still construct a cross-modal unified semantic space and reversibly encode and decode character actions, shot sequence, and scene spatial tags when the same material is converted between 2D short videos and 3D VR digital human scenes, driven by a mixture of text, audio, images, and video. This method would ensure that the characters, actions, shots, spaces, and narrative tags remain consistent when the same material is converted between 2D short videos and 3D VR scenes, thereby solving the problem of bidirectional conversion offset caused by semantic fragmentation of multi-source materials, improving cross-modal semantic alignment accuracy, reversible reconstruction reliability, bidirectional conversion consistency, and the efficiency of mass production of VR digital human short videos. Summary of the Invention

[0005] To address the aforementioned technical shortcomings, the present invention aims to propose a multimedia content-driven method and system for bidirectional conversion and generation of VR digital human scenes and AI short videos. This method and system aims to solve the problem of semantic fragmentation of multi-source materials in the existing technology, especially the technical problem of the inability to achieve reversible consistency and offset suppression of character movements, shot sequence and scene spatial labels under the condition of bidirectional conversion between VR digital human short videos and 3D scenes driven by a mixture of text, audio, images and videos.

[0006] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: The present invention provides a method and system for bidirectional conversion and generation of VR digital human scenes and AI short videos driven by multimedia content, wherein the method includes:

[0007] Step S10: Obtain multi-source materials, and based on the multi-source materials, use the Huber robust weighted time anchoring mechanism to perform multimodal event time alignment tasks and output a unified time anchor point;

[0008] Step S20: Based on the unified time anchor point, a shared basis plus modal low-rank incremental projection and synchronous hypergraph propagation mechanism is used to perform cross-modal semantic alignment and label decoupling tasks, and output a unified semantic vector of five types of decoupled labels;

[0009] Step S30: Based on the unified semantic vector decoupled from the five types of tags, reversible semantic encoding preprocessing is performed using modal conditional reversible affine coupling and Softmax category routing quantization mechanism to output a reversible semantic token sequence;

[0010] Step S40: Based on the reversible semantic token sequence, jointly drive multiple decoders to generate two-dimensional AI short videos and three-dimensional VR digital human scenes, and output a bimodal generated content set with consistent labels;

[0011] Step S50: Extract two-dimensional AI short videos from the bimodal content set with consistent tags, use the reversible semantic token sequence as a priori, and perform reverse parsing reconstruction task using a semantic-geometric joint optimization mechanism to output a three-dimensional VR digital human interaction scene.

[0012] Preferably, step S10 specifically includes:

[0013] Step S101: Extract event responses and quality scores from the multi-source materials, including text, audio, images, and video clips. Evaluate the semantic integrity of the text, the signal-to-noise ratio of the audio, the clarity of the image, and the visibility of the video pose to determine the quality score. Static images only provide the identity of the person and the attributes of the scene, and do not participate in the action time anchoring.

[0014] Step S102: Map the event response to a unified time base to obtain an initial time base; divide the mass score of each mode participating in time anchoring by a preset temperature coefficient and take the exponent, then divide by the sum of the corresponding exponents of each mode to obtain the mode weight, wherein the temperature coefficient is between 0.3 and 0.8;

[0015] Step S103: For each candidate time, calculate the multimodal average response from the event response, and calculate the Huber residual of each modal event response relative to the multimodal average response. Sum the results with the modal weights and add a penalty term for the squared time deviation relative to the initial time base. Take the candidate time with the smallest sum as the unified time anchor point. The coefficient of the penalty term is preset to be 0.2 to 0.8.

[0016] Preferably, step S20 specifically includes:

[0017] Step S201: Select modal segments of the multi-source material according to the unified time anchor point, encode them respectively to obtain modal coding features, obtain the preset shared semantic basis and modal low-rank adaptation term, add the two to form a projection matrix, which is used to project the modal coding features to the unified semantic space;

[0018] Step S202: Using the modal fragment as a node, construct a hyperedge by connecting the set of nodes under the same unified time anchor point or the same narrative event, establish a synchronous hypergraph, and propagate the projected semantic features between the nodes connected by the hyperedge; divide the unified semantic space into five subspaces: character identity, character action, shot sequence, scene space, and narrative tag; calculate the action shot conflict degree; and trigger reorthogonality when the action shot conflict degree is greater than a preset conflict threshold.

[0019] Step S203: Through cross-modal comparison, super Figure 1 The unified semantic space is trained by constraints of consistency, orthogonal decoupling of five subspaces and encoding-decoding cyclic reconstruction, and the label categories are maintained by five-class label-assisted classification constraints. The unified semantic vector of the five-class label decoupling is output.

[0020] Preferably, in step S202, the degree of conflict and reorthogonality of the action shots are determined in the following manner:

[0021] Dimensionless features of the same non-zero dimension are extracted from the character action and shot timing subspaces respectively. Combined with the unified time anchor point, action time anchor points, shot transition time anchor points, and corresponding time windows are determined from the modal segments. The action-shot conflict degree is calculated using the following formula:

[0022]

[0023] in, The degree of conflict in the aforementioned action shots; , These are features extracted from the character's actions and the shot sequence subspace, respectively. , These are the action time anchor points and the camera switching time anchor points, respectively, in seconds; The overlap of the corresponding time window, with a value ranging from 0 to 1; The cosine similarity function is used. For absolute values, It is a norm 2; the degree of conflict of the action shot is dimensionless, and the denominator is not less than 1;

[0024] The conflict threshold is preset to 0.35. When the conflict degree of the action shot exceeds the conflict threshold, the square of the excess is multiplied by a conflict penalty weight preset to 0.5 to 2.0 to obtain the reorthogonal loss. The unified semantic space is updated by backpropagation using the reorthogonal loss.

[0025] Preferably, step S30 specifically includes:

[0026] Step S301: Extract subspace features of character identity, character action, shot sequence, scene space and narrative tag from the unified semantic vector decoupled from the five types of tags, and set up codebooks for each type of subspace feature;

[0027] Step S302: The subspace features are split into a first part and a second part according to the channel. The narrative tags of the unified semantic vector decoupled from the five types of tags are extracted and the quality confidence of the multi-source material is obtained. The condition vector is formed with the unified time anchor point. The condition scaling parameter and the condition translation parameter are determined by the condition vector and the second part. The first part is multiplied element by element by the exponent of the condition scaling parameter and then the condition translation parameter is added to obtain the first coupling feature. The condition coupling parameter is determined by the condition vector and the first coupling feature and added to the second part to obtain the second coupling feature.

[0028] Step S303: Calculate the category routing probabilities of the first and second coupled features using a preset classification matrix and Softmax function. Select the codeword that minimizes the sum of the squared distance between the feature and the codeword, and the cross-entropy of the category routing probability and the candidate codeword index, weighted by a preset weight. Save the difference between the feature and the selected codeword as the quantization residual. Combine the various codewords, the quantization residual, and the unified time anchor to form the reversible semantic token sequence. Use the same condition vector to recover the coupled features by adding the codeword and the residual. First, subtract the condition coupling parameter from the second coupled feature to recover the second part. Then, subtract the condition translation parameter from the first coupled feature and multiply each element by the negative exponent of the condition scaling parameter to recover the first part.

[0029] Preferably, step S40 specifically includes:

[0030] Step S401: Extract character identity, character actions, shot sequence, scene space and narrative tokens from the reversible semantic token sequence, and jointly drive the action, shot and scene decoders to generate a two-dimensional AI short video and a three-dimensional VR digital human scene.

[0031] Step S402: Obtain the peak action time and shot switching time from the generated 2D AI short video and 3D VR digital human scene, and obtain the corresponding audio accent time; add the absolute value of the difference between the peak action time and the shot switching time and the absolute value of the difference between the peak action time and the audio accent time to obtain the action shot timing closure error; for the part exceeding the preset allowable closure error of 0.03 seconds, square it and multiply it by the preset closure penalty weight of 2.0 to constrain the generation timing of the decoder;

[0032] Step S403: Re-encode the temporally constrained 2D AI short video and 3D VR digital human scene into tokens, and constrain the decoder with the readback consistency between the re-encoded tokens and the reversible semantic token sequence, so that the 2D AI short video and 3D VR digital human scene are consistent in terms of characters, actions, shots and narrative tags, forming a bimodal generated content set with consistent tags.

[0033] Preferably, step S50 specifically includes:

[0034] Step S501: Extract two-dimensional AI short videos from the bimodal content set with consistent tags, extract frames and perform shot segmentation, camera estimation, human pose estimation and depth estimation, and combine speech recognition and optical character recognition to obtain initial tokens and sparse three-dimensional observations of character identity, character actions, shot sequence, scene space and narrative tags.

[0035] Step S502: Use the initial token as the initial value of the candidate token, and the reversible semantic token sequence as the prior; take the encoding difference between the two-dimensional short video generated by the candidate token and the two-dimensional AI short video, the encoding difference between the three-dimensional scene generated by the candidate token and the sparse three-dimensional observation, and the square norm of the deviation of the candidate token from the initial token as the objective, and perform semantic-geometric joint optimization to minimize the objective, so as to obtain the optimized token;

[0036] Step S503: Extract the shot timing token and character action token from the optimized token, and generate camera trajectory prior and skeletal joint angle prior respectively; obtain 3D points from the sparse 3D observations, obtain corresponding 2D observations and skeletal joint 2D observations from the 2D AI short video, and optimize the camera and pose by constraining the reprojection error and the deviation relative to the camera trajectory prior and skeletal joint angle prior, and output a 3D VR digital human interaction scene that is consistent with the 2D AI short video in terms of character, action, shot, scene space and narrative tags.

[0037] This invention also provides a multimedia content-driven VR digital human scene and AI short video bidirectional conversion and generation system, comprising:

[0038] The multi-source material access and Huber robust time anchoring module is used to acquire multi-source materials, and based on the multi-source materials, the Huber robust weighted time anchoring mechanism is used to perform multimodal event time alignment tasks and output a unified time anchor point;

[0039] The cross-modal unified semantic alignment module is used to perform cross-modal semantic alignment and label decoupling tasks based on the unified temporal anchor point using a shared basis plus modal low-rank incremental projection and synchronous hypergraph propagation mechanism, and outputs a unified semantic vector for decoupling five types of labels;

[0040] The reversible semantic encoding module is used to perform reversible semantic encoding preprocessing based on the unified semantic vector decoupled from the five types of tags, using modal conditional reversible affine coupling and Softmax category routing quantization mechanism, and outputs a reversible semantic token sequence.

[0041] The forward bidirectional conversion generation module is used to jointly drive multiple decoders to generate two-dimensional AI short videos and three-dimensional VR digital human scenes based on the reversible semantic token sequence, and output a bimodal generated content set with consistent labels.

[0042] The reverse reconstruction and closed-loop correction module is used to extract two-dimensional AI short videos from the bimodal generated content set with consistent labels, and to perform reverse parsing and reconstruction tasks using the reversible semantic token sequence as a priori and a semantic-geometric joint optimization mechanism to output a three-dimensional VR digital human interaction scene.

[0043] The present invention also provides a multimedia content-driven VR digital human scene and AI short video bidirectional conversion generation device, the multimedia content-driven VR digital human scene and AI short video bidirectional conversion generation device comprising: a memory, a processor, and a multimedia content-driven VR digital human scene and AI short video bidirectional conversion generation program stored in the memory and executable on the processor, wherein the multimedia content-driven VR digital human scene and AI short video bidirectional conversion generation program implements the above method when executed by the processor.

[0044] The present invention also provides a computer program product, which includes a multimedia content-driven VR digital human scene and AI short video bidirectional conversion generation program. When the multimedia content-driven VR digital human scene and AI short video bidirectional conversion generation program is executed by a processor, it implements the above-mentioned method.

[0045] The beneficial effects of this invention are as follows:

[0046] By employing Huber's robust weighted time anchoring mechanism, event responses from text, audio, images, and video are weighted by quality and penalized by time deviation. This suppresses the interference of low-quality modalities and abnormal time deviations on the unified time anchor point, enabling multimodal events to achieve stable alignment on a unified time base. This provides a consistent time reference for subsequent cross-modal semantic alignment, thereby reducing time misalignment caused by sampling rate differences and noise, and improving the robustness of time alignment for multi-source materials.

[0047] By employing a shared basis plus modal low-rank incremental projection and synchronous hypergraph propagation mechanism, features of each modality are mapped to a unified semantic space and propagated along the hyperedge. At the same time, through orthogonal decoupling of the five subspaces and re-orthogonal constraints on action shot conflict degree, character identity, character action, shot sequence, scene space and narrative tags remain independent and consistent in the unified semantic vector, reducing cross-modal semantic fragmentation, providing a decoupled and rereadable intermediate representation for reversible semantic encoding, and improving tag consistency and reversible reconstruction reliability in bidirectional conversion.

[0048] A reversible semantic token sequence is generated by using modal conditional reversible affine coupling and Softmax class routing quantization mechanism. This token sequence serves as both a joint driving condition for the forward generation of 2D AI short videos and 3D VR digital human scenes and a priori constraint for reverse parsing reconstruction. Through semantic-geometric joint optimization and readback consistency constraints, the 2D short videos and 3D VR scenes maintain closed-loop consistency in terms of characters, actions, camera angles, space, and narrative tags, thereby suppressing the accumulation of errors in pose estimation and depth estimation, improving bidirectional conversion consistency, and increasing the efficiency of batch production of VR digital human short videos. Attached Figure Description

[0049] Figure 1 This is a flowchart illustrating the first embodiment of a multimedia content-driven method and system for bidirectional conversion and generation of VR digital human scenes and AI short videos according to the present invention. Detailed Implementation

[0050] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0051] Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0052] Example 1: As Figure 1 The diagram shown is a flowchart of the first embodiment of a multimedia content-driven VR digital human scene and AI short video bidirectional conversion generation method and system according to the present invention.

[0053] In the first embodiment, the method for bidirectional conversion and generation of multimedia content-driven VR digital human scenes and AI short videos includes:

[0054] Step S10: Obtain multi-source materials, and based on the multi-source materials, use the Huber robust weighted time anchoring mechanism to perform multimodal event time alignment tasks and output a unified time anchor point;

[0055] In this step, multi-source materials refer to four types of original records: text, audio, images, and video, left from the same narrative event or the same shooting location. Text materials can come from scripts, subtitles, dialogue transcripts, or scene descriptions; audio materials include on-site recordings, dubbing, background noise, or character voices; image materials cover keyframes, stills, product images, or scene screenshots; and video materials correspond to continuous frame sequences, action clips, or camera movement records. After these materials enter step S10, the event detectors corresponding to each modality extract event responses. The event response indicates the intensity of semantic emphasis, sound stress, scene transition, or action peaks in a certain modality near the candidate time. At the same time, text semantic completeness, audio signal-to-noise ratio, image sharpness, and video pose visibility are evaluated as quality scores, which reflect the credibility of the modality in providing time information in the current segment. Static images only provide character identity and scene attributes and do not participate in action time anchoring. Therefore, image modalities are only used as scene consistency references in the time alignment task and are not used as sources of action peaks. A unified time anchor is a single time reference obtained by jointly optimizing event responses from text, audio, images, and video on a common timeline. It is not the original timestamp of a particular modality, but rather a comprehensive time position formed after robust weighting, time base constraints, confidence constraints, and narrative center constraints. The Huber robust weighted time anchoring mechanism includes five processing steps: event response mapping, modality weight calculation, robust residual weighting, time deviation penalty, and narrative center constraint. These steps work together on the candidate time set to ensure that the final output unified time anchor represents a stable center of multimodal events on the timeline.

[0056] In step S10, when jointly processing the event responses from multiple sources, the event responses of each modality are first mapped to a unified time base, and then modal weights are obtained by normalizing the quality scores using a temperature coefficient. The contribution of low-quality modalities in time anchoring is automatically reduced, while the contribution of high-quality modalities is relatively amplified. At candidate times, the difference between each modal event response and the multimodal average response is processed by the Huber robust residual function. When a modality exhibits an abnormal peak or significant deviation, the residual is limited to a preset range and will not dominate candidate time selection through quadratic growth, thus avoiding single-modal noise skewing the time anchor. The squared deviation of the candidate time from the initial time base is multiplied by a preset penalty coefficient and added to the total cost, ensuring that the final time anchor will not deviate from the initial time base due to local response fluctuations. The difference between the modal quality confidence and the average confidence is also included in the constraint, ensuring that the credibility of each modality remains relatively consistent during the time alignment process. The narrative center constraint pulls the candidate times toward the center window of the current narrative segment, making the time anchor more consistent with the position of the event in the narrative structure. The resulting unified time anchor point forms a stable intermediate state that aligns text descriptions, audio accents, image keyframes, and video motion peaks to the same position on the time axis. When organizing multimodal nodes and hyperedges under this time anchor point in step S20, semantic splicing errors and misalignment of actions and shot labels caused by temporal misalignment are reduced, ensuring a consistent temporal basis for subsequent cross-modal semantic alignment and label decoupling tasks.

[0057] Traditional processing methods typically extract timestamps for each modality separately and then align them using nearest neighbor matching, fixed windows, or manually set thresholds. In the mass production of VR digital human short videos, the sampling rates, event granularities, and confidence levels of text, audio, images, and videos vary significantly. Fixed thresholds can easily cause misalignment between motion peaks and shot transitions. For example, when video frame rate fluctuations cause motion peaks to lag behind audio accents by a few milliseconds, nearest neighbor matching may incorrectly associate the motion peak with an adjacent shot event, fixed windows may miss true correspondences due to mismatched window widths, and manual thresholds are even less adaptable to different noise levels in different materials. This step uses modal quality weights and Huber robust residuals to jointly suppress the influence of anomalous modalities. When a modality experiences noise, dropped frames, or beat shifts, the anomalous response of that modality will not dominate candidate time selection, while the stable response of high-quality modalities can pull the time anchor point back to the correct position. The time base deviation constraint prevents candidate times from deviating significantly from the initial time base due to local response fluctuations, the confidence consistency constraint maintains a relative balance in the credibility of each modality during the time alignment process, and the narrative center constraint confines the time anchor point to the center window of the current narrative segment. These processing relationships collectively change the failure path of alignment results that are abnormally pulled by a single modality in traditional methods, ensuring that the time anchor point still falls near the center of the narrative event even when there is conflict in multimodal evidence, providing stable input for the subsequent construction of a unified semantic space.

[0058] In a short VR digital human product demonstration video, the following were simultaneously captured: a text script, audio narration, keyframe images of the product, and video of the digital human's movements. The text event "the character turns and points to the screen" corresponds to a specific time position in the script; the audio accent appears the moment the narrator emphasizes the product name; the keyframe images record the digital human's posture before turning; and the peak of the video motion occurs when the digital human's arm completes its pointing motion to the screen. Due to fluctuations in the video frame rate, the peak of the video motion occurs several milliseconds later than the audio accent. If nearest neighbor matching is used directly, this peak of motion might be incorrectly associated with the next sentence's corresponding shot transition. After acquiring these four types of materials, this step first extracts the event response and quality score for each modality. The text semantic completeness and audio signal-to-noise ratio are high; the image clarity is normal, but static images are not used for motion time anchoring; and the video posture visibility is slightly lower due to frame rate fluctuations. When calculating modal weights, text and audio have relatively high weights, while video has relatively low weights. In terms of candidate times, the multimodal average response is formed by the event responses of text, audio, and video. The difference between the video event response and the average response is processed by the Huber robust residual function, and its abnormal peaks are limited to a preset range, so they do not dominate the candidate time selection. The squared deviation of the candidate time from the initial time base is multiplied by a preset penalty coefficient and added to the total cost, so that the time anchor point will not deviate significantly from the initial time base due to the lag of the video action peak. The narrative center constraint pulls the candidate time towards the center window of the current narration segment, making the time anchor point more consistent with the position of the narrative event "the character turns to point at the screen" in the narration segment. The final output unified time anchor point falls near the common center of the text event, audio accent, and video action peak. Step S20 organizes the character, action, shot, space, and narrative segment into the same hyperedge under this anchor point, reducing the time misalignment of action and shot labels.

[0059] Step S20: Based on the unified time anchor point, a shared basis plus modal low-rank incremental projection and synchronous hypergraph propagation mechanism is used to perform cross-modal semantic alignment and label decoupling tasks, and output a unified semantic vector of five types of decoupled labels;

[0060] In this step, the unified semantic vector refers to the vector representation formed by mapping text, audio, image, and video modal segments under the same unified temporal anchor point to the unified semantic space. This vector has independent semantic components in five subspaces: character identity, character action, shot sequence, scene space, and narrative tag. The shared basis is a pre-constructed projection basis that carries the common semantics of characters, actions, shots, spaces, and narratives. Modal low-rank incremental projection is an incremental projection formed by superimposing low-rank adaptation terms on the encoded features of each modality, used to preserve the difference information of each modality on the shared basis. The synchronization hypergraph uses each modal segment as a node and the set of nodes under the same unified temporal anchor point or the same narrative event as a hyperedge. The hyperedge connects multimodal nodes within the same event, allowing semantic information between nodes to propagate along the hyperedge. Five-category tag decoupling refers to dividing the unified semantic space into five subspaces: character identity, character action, shot sequence, scene space, and narrative tag, and suppressing the coupling between different subspaces through orthogonal constraints and re-orthogonal mechanisms. This step first selects modal segments from multi-source materials according to a unified time anchor point, and encodes them separately to obtain modal coding features. A pre-defined shared semantic basis and modal low-rank adaptation terms are obtained, and these are added together to form a projection matrix, which is used to project the modal coding features onto a unified semantic space. The projected semantic features enter a synchronous hypergraph and propagate between nodes connected by hyperedges, allowing multimodal features within the same event to aggregate in their respective subspaces. Simultaneously, the action-shot conflict degree is calculated; when this conflict degree exceeds a preset conflict threshold, re-orthogonalization is triggered, further widening the representation gap between the character action subspace and the shot temporal subspace. Finally, through cross-modal comparison and hyper-modal mapping... Figure 1 The unified semantic space is trained by constraints of consistency, orthogonal decoupling of five subspaces, and cyclic reconstruction of encoding and decoding. The label categories are maintained by five-class label auxiliary classification constraints, and the unified semantic vector of the five-class label decoupling is output.

[0061] A shared basis plus modal low-rank incremental projection maps the encoded features of different modalities to the same semantic space, enabling text, audio, images, and videos to obtain comparable semantic representations under a unified temporal anchor. The shared basis carries common semantics, while the modal low-rank adaptor preserves the differences between modalities. The projected features have both cross-modal consistency and do not lose modality-specific information. Synchronous hypergraph propagation utilizes hyperedge connections between multimodal nodes under the same narrative event, allowing features of different modalities to correct each other during propagation, reducing semantic bias caused by single-modal noise or temporal shifts. The division of five subspaces and orthogonal decoupling constraints keep character identity, character actions, shot sequence, scene space, and narrative labels independent in the unified semantic vector, avoiding mutual interference between easily confused labels such as actions and shots in the representation space. The calculation of action-shot conflict degree and the re-orthogonal mechanism further suppress the coupling between the character action subspace and the shot sequence subspace. When actions and shots are temporally close and semantically similar, the unified semantic space is updated through backpropagation of re-orthogonal loss, making the feature directions of the two subspaces more separated. The final output of the unified semantic vector decoupled from the five categories of labels provides a well-structured input for step S30, enabling step S30 to perform reversible semantic encoding and codebook routing quantization independently by category, thus avoiding inconsistencies between character actions and camera movements in the subsequently generated content due to label aliasing.

[0062] Traditional processing methods typically feed text, audio, and video features into separate networks, then fuse them through concatenation or global attention. The resulting semantic space lacks explicit category division, and character actions and camera movements are easily mixed in the same representation. During reverse reconstruction, action labels and camera labels interfere with each other, leading to inconsistencies between character poses and camera trajectories in the 3D scene. For example, when the action "picking up a product" and the camera "pushing in" are temporally close, traditional fusion methods may encode them as similar semantic vectors, causing coupling shifts between camera trajectory and arm movement during subsequent decoding. This step employs a shared basis plus modal low-rank incremental projection to preserve common semantics and modal differences in a unified semantic space, avoiding information redundancy and semantic confusion caused by simple concatenation. Synchronous hypergraph propagation utilizes hyperedge connections between multimodal nodes within the same event, enabling features from different modalities to mutually correct each other during propagation, reducing semantic bias caused by single-modal noise or temporal shifts. The division of five subspaces and orthogonal decoupling constraints ensure that character identity, character actions, shot sequence, scene space, and narrative tags remain independent within a unified semantic vector. The calculation of action-shot conflict degree and the re-orthogonal mechanism further suppress the coupling between action and shot subspaces. Compared to the fusion method without category decoupling, this processing reduces the risk of tag aliasing, provides a structurally clear unified semantic vector for subsequent reversible encoding, and enables step S30 to be routed to the action codebook and shot codebook respectively, thereby maintaining the consistency of character actions and shot movements in the generated content.

[0063] In a short VR digital human product demonstration video, text scripts, audio explanations, product keyframe images, and digital human motion videos were simultaneously captured on-site. The unified time anchor point output in step S10 falls near the common center of the text event "the character turns to face the screen," the audio accent, and the video motion peak. This step selects modal fragments of four types of materials according to this unified time anchor point and encodes them to obtain modal coding features. A shared semantic base carries the common semantics of characters, actions, shots, space, and narrative. The modal low-rank adaptation term preserves the differences between each modality. The two are added together to form a projection matrix, projecting the modal coding features onto a unified semantic space. Using each modal fragment as a node, the set of nodes under the same unified time anchor point forms a hyperedge, establishing a synchronous hypergraph. The projected semantic features are propagated between the nodes connected by the hyperedge, causing the text "the character turns to face the screen," the audio accent, the product close-up in the image, and the turning motion in the video to aggregate within their respective subspaces. The unified semantic space is divided into five subspaces: character identity, character actions, shot sequence, scene space, and narrative tags. The conflict degree between actions and shots is calculated, and when this conflict degree exceeds a preset conflict threshold, re-orthogonalization is triggered to further differentiate the representation of actions and shots. Finally, through cross-modal comparison and hyper-modal analysis... Figure 1 The unified semantic space is trained by constraining consistency, orthogonal decoupling of the five subspaces, and cyclic reconstruction of encoding and decoding. Five-class label-assisted classification constraints are used to maintain label categories, outputting a unified semantic vector with decoupled labels across the five classes. In this vector, the character action subspace corresponds to the action semantics of "turning to point at the screen," and the shot temporal subspace corresponds to the temporal semantics of "camera zooming in." These two subspaces are independent of each other, providing clear input for the reversible semantic encoding and codebook routing quantization in step S30.

[0064] Step S30: Based on the unified semantic vector decoupled from the five types of tags, reversible semantic encoding preprocessing is performed using modal conditional reversible affine coupling and Softmax category routing quantization mechanism to output a reversible semantic token sequence;

[0065] In this step, the reversible semantic token sequence refers to the compact intermediate representation formed by modal conditional reversible affine coupling and category routing quantization of the unified semantic vector decoupled from five types of labels. This includes character identity codewords, character action codewords, shot timing codewords, scene space codewords, narrative codewords, various quantization residuals, and a unified time anchor. The unified semantic vector decoupled from the five types of labels comes from step S20, and its five sub-space features—character identity, character action, shot timing, scene space, and narrative label—correspond to digital human identity, action, shot movement, scene layout, and narrative semantics, respectively. This step sets up an independent codebook for each sub-space feature, where the codewords represent common semantic patterns within that sub-space. During processing, each sub-space feature is first split into a first part and a second part by channel. Simultaneously, narrative labels are extracted from the unified semantic vector and combined with the quality confidence of multi-source materials and the unified time anchor to form a conditional vector. The conditional vector is used to obtain conditional scaling parameters, conditional translation parameters, and conditional coupling parameters through learnable mapping. The first part scales the data according to the conditional scaling parameters and then adds conditional translation parameters to obtain the first coupled feature. The conditional vector and the first coupled feature together determine the conditional coupling parameters, which are then added to the second part to obtain the second coupled feature. The first and second coupled features calculate the category routing probability using a preset classification matrix and a Softmax function, respectively. Within the corresponding codebook, the codeword that minimizes the sum of the squared distance between the feature and the codeword, and the cross-entropy between the category routing probability and the candidate codeword index, is selected. The difference between the feature and the selected codeword is saved as the quantization residual. Finally, the various codewords, quantization residuals, and unified time anchors are combined to form a reversible semantic token sequence. The codewords in this sequence provide discrete semantic indices, the residuals preserve details that the codewords cannot express, and the unified time anchors maintain the temporal correspondence, enabling subsequent steps to recover key semantic features without accessing the original multimodal material.

[0066] Conditional affine coupling introduces a unified time anchor, narrative label, and quality confidence into feature transformation. Subspace features under different times, narrative segments, and quality conditions will receive different scaling, translation, and coupling strengths. The first and second coupled features generated by forward coupling have reversible recovery capabilities while preserving semantic information: using the same conditional vector, the second part is recovered by subtracting the conditional coupling parameter from the second coupled feature, and then the first part is recovered by subtracting the conditional translation parameter from the first coupled feature and scaling it according to the reciprocal of the conditional scaling parameter. This reversibility allows the reversible semantic token sequence to not only express the current semantics but also accurately recover subspace features when needed, providing a reliable prior for the reverse parsing reconstruction in step S50. Category routing quantization considers both the distance from features to codewords and the category prior cross-entropy when selecting the codebook, imposing stronger prior constraints on conflict-prone categories such as actions and shots, and avoiding aliasing caused by sharing codewords between different categories. The quantized residual and checksum are written into the token, enabling subsequent steps S40 to share the same token semantics when generating 2D AI short videos and 3D VR digital human scenes, reducing semantic drift between generation branches. In step S50, when reverse-analyzing from the 2D AI short video, features can be recovered based on the codewords, residuals, and unified time anchors in the token, and abnormal tokens can be identified through the checksum, thereby constraining semantic offsets and geometric inconsistencies during the reverse reconstruction process.

[0067] Traditional quantization methods typically use a single codebook for all modalities or categories of features, or directly quantize features into discrete indices, without preserving residuals and verification information. In the bidirectional conversion task between VR digital human scenes and AI short videos, different categories of features, such as character actions, shot timing, and scene space, have significant differences in their semantic space distribution. A unified codebook can easily lead to interference between action codewords and shot codewords, resulting in the loss of semantic details after quantization. When reverse-analyzing from 2D short videos, action, shot, and spatial labels shift during reconstruction, resulting in a 3D VR digital human scene that is inconsistent with the original semantics. This step sets up separate codebooks for five categories: character identity, character actions, shot timing, scene space, and narrative labels. The Softmax function is used to calculate category routing probabilities, assigning features to the corresponding codebooks to avoid aliasing caused by sharing codewords across different categories. Conditional affine coupling imbues the features before quantization with temporal anchors, narrative, and quality conditions, allowing for feature recovery from tokens when needed during reverse restoration. The reversible semantic token sequence contains codewords, residuals, and checksums, facilitating the generation of actions, shots, lip movements, and scenes by the multiple decoders in step S40 according to token type. It also facilitates reverse optimization in step S50 using the same token as a priori. Compared to traditional methods, this step preserves reversibility and residual information during quantization, improving semantic consistency and traceability in the bidirectional conversion process.

[0068] In a short VR digital human product demonstration video, text scripts, audio explanations, product keyframe images, and digital human motion videos were simultaneously captured on-site. In the unified semantic vector decoupled from the five categories of labels output in step S20, the character action subspace corresponds to the action semantics of "turning to point at the screen," the shot timing subspace corresponds to the timing semantics of "camera zooming in," the scene space subspace corresponds to the virtual showroom layout, and the narrative label corresponds to the product explanation segment. After receiving this unified semantic vector, this step sets up codebooks for character identity, character action, shot timing, scene space, and narrative label respectively. During processing, the action subspace features are split into a first part and a second part by channel, while simultaneously extracting the narrative label and combining it with the quality confidence of multi-source materials and a unified time anchor point to form a conditional vector. The conditional vector is used to obtain conditional scaling parameters, conditional translation parameters, and conditional coupling parameters through learnable mapping. The first part is scaled according to the conditional scaling parameters and then added to the conditional translation parameters to obtain the first coupling feature; the second part is added to the conditional coupling parameters to obtain the second coupling feature. Subsequently, the category routing probabilities of the first and second coupled features are calculated using a pre-set classification matrix and the Softmax function. The codeword that minimizes the sum of the squared distance between the feature and the codeword, and the cross-entropy between the category routing probability and the candidate codeword index, is selected from the action codebook, and the quantization residual is saved. Similarly quantization is performed on the shot temporal features and scene spatial features in their respective codebooks. Finally, all types of codewords, quantization residuals, and unified time anchors are combined to form a reversible semantic token sequence. In this sequence, action codewords correspond to the "turning to point at the screen" action mode, shot codewords correspond to the "camera zooming in" motion mode, spatial codewords correspond to the exhibition hall layout mode, residuals preserve details that codewords cannot express, and unified time anchors maintain the temporal correspondence between actions and shots. After receiving this reversible semantic token sequence in step S40, skeletal motion generation, camera trajectory generation, and scene graph generation can be driven respectively, without having to re-extract features from the original multimodal materials.

[0069] Step S40: Based on the reversible semantic token sequence, jointly drive multiple decoders to generate two-dimensional AI short videos and three-dimensional VR digital human scenes, and output a bimodal generated content set with consistent labels;

[0070] In this step, the bimodal content set refers to the collection of 2D AI short videos and 3D VR digital human scenes obtained by jointly driving multiple decoders with the same reversible semantic token sequence. This includes 2D video frame sequences, 3D scene graphs, skeletal motion sequences, camera trajectories, and lip-sync animation. After the reversible semantic token sequence enters this step, it is first split according to character identity, character actions, shot timing, scene space, and narrative tokens. The character action token drives the skeletal motion decoder; the shot timing token and character action token jointly drive the shot trajectory decoder; the scene space token and character identity token jointly drive the 3D scene graph decoder; and the narrative token and audio features participate in lip-sync generation. The skeletal motion decoder outputs the skeletal joint angle increments based on the character action tokens, audio beat and intonation features, and the latent state of the previous frame, setting upper limits on the angle increments and angular velocities to avoid jumps between adjacent frames. The shot trajectory decoder uses the shot timing tokens and character action tokens to calculate the weights of the spline basis function, generating camera extrinsic parameters to ensure that camera zooms, pans, or switches maintain temporal matching with character actions. The 3D scene graph decoder combines asset embedding, identity tokens, and spatial tokens into node features, and combines them with spatial relationship matrices and geometric relationships such as distance, orientation, and occlusion to form adjacency weights, determining character, prop, and lighting nodes and spatial relationship edges. Lip-sync generation obtains visual phonemes through audio time windows and text token classification, ensuring that lip movements correspond to speech accents. During generation, peak action times and camera transition times are obtained from 2D AI short videos and 3D VR digital human scenes, along with corresponding audio accent times. The absolute values ​​of the differences between peak action times and camera transition times, and between peak action times and audio accent times, are added to obtain the action-camera temporal closure error. The portion exceeding the preset allowable closure error is squared and multiplied by a preset closure penalty weight to constrain the decoder's generation temporal sequence. Subsequently, the temporally constrained 2D AI short videos and 3D VR digital human scenes are re-encoded into tokens. The readback consistency between the re-encoded tokens and the reversible semantic token sequence constrains the decoder, ensuring consistency in characters, actions, camera angles, and narrative tags between the 2D AI short videos and 3D VR digital human scenes, forming a tag-consistent bimodal generated content set.

[0071] This step uses a reversible semantic token sequence as a unified condition to simultaneously drive the action, camera, scene, and lip-sync decoders, enabling 2D AI short videos and 3D VR digital human scenes to share the same semantic representation during the generation phase, rather than aligning them after generation. Action tokens participate in both skeletal motion generation and camera trajectory weight calculation; camera tokens and action tokens jointly determine the spline trajectory; scene space tokens and identity tokens jointly construct the 3D scene graph; and lip-sync generation combines audio time windows and text tokens, thus establishing token-level associations between character actions, camera movements, scene layout, and lip movements. Peak action time, camera transition time, and audio accent time are incorporated into closure error constraints. A portion exceeding the preset allowable closure error is penalized with a squared penalty, prompting the decoder to proactively adjust the generation sequence during training or inference, reducing temporal misalignment between actions and cameras, and actions and audio. The readback consistency constraint between the re-encoded tokens and the reversible semantic token sequence further ensures that the generated content does not deviate from the original semantics in terms of characters, actions, cameras, and narrative tags. The resulting bimodal content set with consistent labels provides reliable input for step S50 to extract two-dimensional AI short videos and perform reverse parsing and reconstruction using reversible semantic token sequences as priors. This enables subsequent three-dimensional VR digital human interaction scenes to recover three-dimensional structures and interaction relationships consistent with the original semantics from two-dimensional videos.

[0072] Traditional forward generation typically generates actions, shots, lip movements, and scenes separately, then aligns them temporally in post-production. The action module is unaware of shot tokens, and the shot module is unaware of action tokens, easily leading to misalignment between character action peaks and shot transitions. Lip movement generation, relying solely on audio or text, is also prone to discrepancies between speech stress and text syllables. Scene generation lacks shared semantic constraints with actions and shots, resulting in inconsistencies in character identities, action types, or narrative labels between 2D videos and 3D scenes. This step uses a unified, reversible semantic token sequence to jointly drive multiple decoders. Action tokens participate in both action generation and shot weight calculation, shot tokens and action tokens jointly determine the spline trajectory, and scene space tokens and identity tokens jointly construct the 3D scene graph. Closure constraints are applied to the action, shot, and audio time windows using action-shot temporal closure errors. Compared to separate generation and subsequent stitching, this process generates 2D short videos and 3D VR scenes under the same token semantics, reducing action and shot label conflicts and outputting a bimodal generated content set with consistent labels.

[0073] In a short VR digital human product demonstration video, the live video simultaneously captured text scripts, audio explanations, keyframe images of the product, and digital human motion videos. In the reversible semantic token sequence output in step S30, the action codeword corresponds to the "turning to face the screen" motion mode, the camera codeword corresponds to the "camera zooming in" motion mode, and the spatial codeword corresponds to the exhibition hall layout mode. Residuals preserve details that the codewords cannot express, and a unified time anchor point maintains the temporal correspondence between the action and the camera. After receiving this reversible semantic token sequence, the action token drives the skeletal motion decoder to output the joint angle increment for "turning to face the screen." The angle increment and angular velocity are constrained by upper limits to prevent arm tremors. The camera timing token and the action token jointly calculate the spline basis function weights, adjusting the camera zooming speed according to the turning action. Lip shape generation combines the audio time window and the text token to synchronize lip movements with the explanation keywords. The 3D scene map places the digital human, exhibition stand, and lighting nodes based on the identity token and the spatial token. During the generation process, the peak action time, camera switching time, and audio accent time are extracted and the closure error is calculated. If the preset allowable closure error is exceeded, the token representation is adjusted in reverse. The final output includes a 2D AI short video and a 3D VR digital human scene. The two are consistent in terms of characters, actions, shots, and narrative tags, providing a foundation for step S50 to extract the 2D AI short video and reconstruct the 3D VR digital human interactive scene.

[0074] Step S50: Extract two-dimensional AI short videos from the bimodal content set with consistent tags, use the reversible semantic token sequence as a priori, and perform reverse parsing reconstruction task using a semantic-geometric joint optimization mechanism to output a three-dimensional VR digital human interaction scene.

[0075] In this step, the 3D VR digital human interaction scene refers to the 3D scene obtained through reverse parsing after extracting 2D AI short videos from a bimodal content set with consistent labels. This scene includes five types of information: character identity, character actions, shot sequence, scene space, and narrative tags. The input source for this scene is the 2D AI short video output from step S40. This short video shares the same reversible semantic token sequence with the 3D VR digital human scene; therefore, the characters, actions, shots, and narrative information in the 2D image have consistent labels with the 3D scene. During processing, the 2D AI short video is first frame-segmented, then shot segmentation, camera estimation, human pose estimation, and depth estimation are performed. Combined with speech recognition and optical character recognition, initial tokens for character identity, character actions, shot sequence, scene space, and narrative tags, as well as sparse 3D observations, are obtained. The character identity token is obtained by encoding face, clothing, and optical character recognition information; the character action token is obtained by encoding human pose; the shot sequence token is obtained by encoding camera trajectory; the scene space token is obtained by spatial encoding of depth map and sparse 3D points; and the narrative tag token is obtained by encoding speech recognition, optical character recognition, and shot rhythm through narrative encoding. These initial tokens serve as the initial values ​​for candidate tokens, using the reversible semantic token sequence generated in step S30 as a prior, and participate in the joint semantic-geometric optimization. The optimization objective is composed of the weighted sum of the square norms of the encoding differences between the 2D short video generated by the candidate tokens and the input 2D AI short video, the encoding differences between the 3D scene generated by the candidate tokens and the sparse 3D observations, and the deviation of the candidate tokens from the initial tokens, with each weight taking a preset value. Reversible loss and label consistency loss are also incorporated during the optimization process. The reversible loss maintains the reversible encoding characteristics of the tokens, while the label consistency loss calculates the cross-entropy for the five categories of labels: character, action, shot, space, and narrative. After optimization, optimized tokens are obtained. Shot sequence tokens and character action tokens are extracted from the optimized tokens to generate camera trajectory priors and skeletal joint angle priors, respectively. 3D points are obtained from sparse 3D observations, and corresponding 2D observations and skeletal joint 2D observations are obtained from the 2D AI short video. Camera and pose optimization are constrained by reprojection errors and deviations from the camera trajectory priors and skeletal joint angle priors, ultimately outputting a 3D VR digital human interaction scene consistent with the 2D AI short video in terms of character, action, shot, scene space, and narrative labels.

[0076] This step uses the reversible semantic token sequence generated in step S30 as a priori, and jointly optimizes the 2D short video reconstruction, 3D scene reconstruction, token prior deviation, reversibility loss, and label consistency loss. This ensures that the optimized tokens can reconstruct both the input 2D AI short video and the sparse 3D observations estimated from the 2D short video, without deviating from the reversible semantic token sequence. The 2D short video encoder and generator constrain the tokens to reconstruct the input 2D short video, while the 3D scene encoder and generator constrain the tokens to reconstruct the sparse 3D observations. The token prior term ensures that the optimized tokens do not deviate from the reversible semantic tokens generated in step S30. The reversibility loss and label consistency loss maintain the reversible encoding characteristics and the consistency of the five label classes, respectively. The resulting optimized tokens maintain stability in character identity, actions, camera angles, scene space, and narrative labels at the semantic level, and remain compatible with sparse 3D observations at the geometric level. Subsequently, camera timing tokens and character action tokens are extracted from the optimized tokens to generate camera trajectory priors and skeletal joint angle priors, respectively. Then, the camera and pose are optimized by constraining the reprojection error and the deviation from the relative camera trajectory priors and skeletal joint angle priors. The joint optimization of camera and pose further corrects the camera trajectory and human pose using projection error and token prior constraints, with preset values ​​for camera constraint weights and motion constraint weights. Occlusion robustness weights adjust the observation point weights based on the degree of occlusion and visibility; higher occlusion and lower visibility result in smaller geometric weights for observation points and larger token prior weights, making occluded areas rely more on token semantic inference. In this way, the 3D VR digital human interactive scene obtained from the reverse analysis of 2D AI short videos maintains consistency with the forward-generated content in terms of characters, actions, camera angles, scene space, and narrative tags, forming a two-way conversion loop. This scene can be directly used for VR interaction, allowing users to freely switch perspectives, trigger digital human actions, or view information on display boards without tag drift or geometric misalignment due to the 2D-to-3D conversion.

[0077] Traditional methods for reconstructing 3D scenes from 2D videos typically rely on multi-view geometry or monocular depth estimation. When characters are occluded, the camera moves rapidly, textures are missing, or lighting changes occur, geometric observation becomes unstable, easily leading to depth jumps, pose drift, and camera trajectory jitter. Without a unified semantic token prior, the reconstruction result can only maintain the geometric shape, making it difficult to maintain consistency between character identity, actions, and narrative labels. For example, when a digital human's arm occludes a display panel, pure geometric reconstruction might estimate the arm's depth as the panel's depth, resulting in confusion regarding the relationship between the panel and the arm. When the camera moves rapidly laterally, camera trajectory estimation may produce jitter, causing inconsistencies between the camera movement in the reconstructed scene and the original short video. When lighting changes cause texture loss, character identification may be incorrect, resulting in inconsistencies between the characters in the reconstructed scene and those in the original short video. This step uses a reversible semantic token sequence as a prior to jointly optimize 2D short video reconstruction, 3D scene reconstruction, token prior, reversible loss, and label consistency, and applies projection error and token prior constraints to the camera and pose, respectively. Occlusion robustness weights adjust the observation point weights based on the degree of occlusion and visibility, reducing the dependence of occluded areas on geometric errors and making them more constrained by token semantics. Thus, under the same conditions, this step can utilize the semantic information in the tokens to fill in unstable areas of geometric observation, maintaining consistency in character identity, actions, camera angles, scene space, and narrative tags, and reducing reverse reconstruction offset. Compared to pure geometric reconstruction, this step maintains tag consistency during 2D-to-3D reverse parsing, ensuring that the reconstructed scene not only has the correct geometric shape but also semantic content consistent with the original short video.

[0078] In a short VR digital human product demonstration video, the following were simultaneously captured on-site: a text script, audio explanation, keyframe images of the product, and video of the digital human's movements. In the 2D AI short video output in step S40, the digital human turns to point at a virtual display board, the camera slightly pans, and in some frames, the human's arm obscures the display board. This step extracts frames from this 2D AI short video, performs shot segmentation, camera estimation, human pose estimation, and depth estimation, and combines speech recognition and optical character recognition to obtain initial identity, action, shot, space, and narrative tokens. If only geometric reconstruction is used, the depth at the arm-obscured area may be incorrect, and the relationship between the display board and the arm may be confusing. After adopting this step, using the reversible semantic token sequence from step S30 as prior, the 2D reconstruction error and 3D observation error in the semantic-geometric joint optimization jointly correct the tokens. The label consistency loss constrains the action token to still correspond to "turning to point at the virtual display board," and the space token to still correspond to the display board position. The camera and pose joint optimization uses occlusion robust weights to reduce the geometric weight of occluded points and increase the prior weight of the tokens. The final output is a 3D VR digital human interactive scene, in which the character's identity, actions, camera trajectory, scene space, and narrative tags are consistent with the 2D short video, enabling bidirectional conversion. Users can freely rotate their viewpoint within this scene to observe the digital human turning and pointing at a display board. The content on the display board is consistent with the spoken script, and the camera movement is smooth, without any depth jumps or posture drift.

[0079] Example 2: Furthermore, the present invention provides a multimedia content-driven VR digital human scene and AI short video bidirectional conversion generation system, which employs the multimedia content-driven VR digital human scene and AI short video bidirectional conversion generation method and system described in the above embodiments, and can solve the technical problem of multimedia content-driven VR digital human scene and AI short video bidirectional conversion generation. The beneficial effects of the multimedia content-driven VR digital human scene and AI short video bidirectional conversion generation system provided by the present invention are the same as those of the multimedia content-driven VR digital human scene and AI short video bidirectional conversion generation method and system described in the above embodiments, and other technical features of the multimedia content-driven VR digital human scene and AI short video bidirectional conversion generation system are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0080] Example 3: This invention provides a multimedia content-driven VR digital human scene and AI short video bidirectional conversion generation device. The device includes at least one processor and a memory communicatively connected to the processor. The memory stores instructions executable by the processor, which are then executed to enable the processor to perform the multimedia content-driven VR digital human scene and AI short video bidirectional conversion generation method and system described in Example 1. The multimedia content-driven VR digital human scene and AI short video bidirectional conversion generation device in this invention can include, but is not limited to, mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. This multimedia content-driven VR digital human scene and AI short video bidirectional conversion generation device is merely an example and should not limit the functionality or scope of the invention. A multimedia content-driven VR digital human scene and AI short video bidirectional conversion generation device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to a program stored in a read-only memory or a program loaded from a storage device into a random access memory. The random access memory also stores various programs and data required for the operation of the multimedia content-driven VR digital human scene and AI short video bidirectional conversion generation device. The processing unit, read-only memory, and random access memory are interconnected via a bus. An I / O interface is also connected to the bus. Typically, the following systems can be connected to the I / O interface: input devices including touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices including liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices including magnetic tapes, hard disks, etc.; and communication devices. The communication device allows the multimedia content-driven VR digital human scene and AI short video bidirectional conversion generation device to wirelessly or wiredly communicate with other devices to exchange data. While a multimedia content-driven VR digital human scene and AI short video bidirectional conversion and generation device with various systems has been described, it should be understood that implementation or possession of all the described systems is not required. Alternatively, more or fewer systems may be implemented.

[0081] Example 4: This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the multimedia content-driven VR digital human scene and AI short video bidirectional conversion generation method and system described above. The computer program product provided by this invention can solve the technical problem of multimedia content-driven VR digital human scene and AI short video bidirectional conversion generation. Compared with the prior art, the beneficial effects of the computer program product provided by this invention are the same as those of the multimedia content-driven VR digital human scene and AI short video bidirectional conversion generation method and system provided in the above embodiments, and will not be repeated here.

[0082] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device, or installed from a read-only memory. When the computer program is executed by a processing device, it performs the functions defined in the methods of the embodiments disclosed in this invention.

[0083] It should be understood that the various parts disclosed in this invention can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics may be combined in any suitable manner in one or more embodiments or examples.

[0084] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the present invention and its equivalents, the present invention also intends to include these modifications and variations.

Claims

1. A method for bidirectional conversion and generation of VR digital human scenes and AI short videos driven by multimedia content, characterized in that, The method includes: Step S10: Obtain multi-source materials, and based on the multi-source materials, use the Huber robust weighted time anchoring mechanism to perform multimodal event time alignment tasks and output a unified time anchor point; Step S20: Based on the unified time anchor point, a shared basis plus modal low-rank incremental projection and synchronous hypergraph propagation mechanism is used to perform cross-modal semantic alignment and label decoupling tasks, and output a unified semantic vector of five types of decoupled labels; Step S30: Based on the unified semantic vector decoupled from the five types of tags, reversible semantic encoding preprocessing is performed using modal conditional reversible affine coupling and Softmax category routing quantization mechanism to output a reversible semantic token sequence; Step S40: Based on the reversible semantic token sequence, jointly drive multiple decoders to generate two-dimensional AI short videos and three-dimensional VR digital human scenes, and output a bimodal generated content set with consistent labels; Step S50: Extract two-dimensional AI short videos from the bimodal content set with consistent tags, use the reversible semantic token sequence as a priori, and perform reverse parsing reconstruction task using a semantic-geometric joint optimization mechanism to output a three-dimensional VR digital human interaction scene.

2. The multimedia content-driven VR digital human scene and AI short video bidirectional conversion generation method as described in claim 1, characterized in that, Step S10 specifically includes: Step S101: Extract event responses and quality scores from the multi-source materials, including text, audio, images, and video clips. Evaluate the semantic integrity of the text, the signal-to-noise ratio of the audio, the clarity of the image, and the visibility of the video pose to determine the quality score. Static images only provide the identity of the person and the attributes of the scene, and do not participate in the action time anchoring. Step S102: Map the event response to a unified time base to obtain an initial time base; divide the mass score of each mode participating in time anchoring by a preset temperature coefficient and take the exponent, then divide by the sum of the corresponding exponents of each mode to obtain the mode weight, wherein the temperature coefficient is between 0.3 and 0.8; Step S103: For each candidate time, calculate the multimodal average response from the event response, and calculate the Huber residual of each modal event response relative to the multimodal average response. Sum the results with the modal weights and add a penalty term for the squared time deviation relative to the initial time base. Take the candidate time with the smallest sum as the unified time anchor point. The coefficient of the penalty term is preset to be 0.2 to 0.

8.

3. The method for bidirectional conversion and generation of VR digital human scenes and AI short videos driven by multimedia content as described in claim 1, characterized in that, Step S20 specifically includes: Step S201: Select modal segments of the multi-source material according to the unified time anchor point, encode them respectively to obtain modal coding features, obtain the preset shared semantic basis and modal low-rank adaptation term, add the two to form a projection matrix, which is used to project the modal coding features to the unified semantic space; Step S202: Using the modal fragment as a node, construct a hyperedge by connecting the set of nodes under the same unified time anchor point or the same narrative event, establish a synchronous hypergraph, and propagate the projected semantic features between the nodes connected by the hyperedge; divide the unified semantic space into five subspaces: character identity, character action, shot sequence, scene space, and narrative tag; calculate the action shot conflict degree; and trigger reorthogonality when the action shot conflict degree is greater than a preset conflict threshold. Step S203: Train the unified semantic space through cross-modal comparison, hypergraph consistency, orthogonal decoupling of the five subspaces and encoding-decoding cyclic reconstruction constraints, and maintain the label categories with five-class label-assisted classification constraints, and output the unified semantic vector of the five-class label decoupling.

4. The multimedia content-driven VR digital human scene and AI short video bidirectional conversion generation method as described in claim 3, characterized in that, In step S202, the degree of conflict and re-orthogonality of the action shots are determined as follows: Dimensionless features of the same non-zero dimension are extracted from the character action and shot timing subspaces respectively. Combined with the unified time anchor point, action time anchor points, shot transition time anchor points, and corresponding time windows are determined from the modal segments. The action-shot conflict degree is calculated using the following formula: in, The degree of conflict in the aforementioned action shots; , These are features extracted from the character's actions and the shot sequence subspace, respectively. , These are the action time anchor points and the camera switching time anchor points, respectively, in seconds; The overlap of the corresponding time window, with a value ranging from 0 to 1; The cosine similarity function is used. For absolute values, It is a norm 2; the degree of conflict of the action shot is dimensionless, and the denominator is not less than 1; The conflict threshold is preset to 0.

35. When the conflict degree of the action shot exceeds the conflict threshold, the square of the excess is multiplied by a conflict penalty weight preset to 0.5 to 2.0 to obtain the reorthogonal loss. The unified semantic space is updated by backpropagation using the reorthogonal loss.

5. The multimedia content-driven VR digital human scene and AI short video bidirectional conversion generation method as described in claim 1, characterized in that, Step S30 specifically includes: Step S301: Extract subspace features of character identity, character action, shot sequence, scene space and narrative tag from the unified semantic vector decoupled from the five types of tags, and set up codebooks for each type of subspace feature; Step S302: The subspace features are split into a first part and a second part according to the channel. The narrative tags of the unified semantic vector decoupled from the five types of tags are extracted and the quality confidence of the multi-source material is obtained. The condition vector is formed with the unified time anchor point. The condition scaling parameter and the condition translation parameter are determined by the condition vector and the second part. The first part is multiplied element by element by the exponent of the condition scaling parameter and then the condition translation parameter is added to obtain the first coupling feature. The condition coupling parameter is determined by the condition vector and the first coupling feature and added to the second part to obtain the second coupling feature. Step S303: Calculate the category routing probabilities of the first and second coupled features using a preset classification matrix and Softmax function. Select the codeword that minimizes the sum of the squared distance between the feature and the codeword, and the cross-entropy of the category routing probability and the candidate codeword index, weighted by a preset weight. Save the difference between the feature and the selected codeword as the quantization residual. Combine the various codewords, the quantization residual, and the unified time anchor to form the reversible semantic token sequence. Use the same condition vector to recover the coupled features by adding the codeword and the residual. First, subtract the condition coupling parameter from the second coupled feature to recover the second part. Then, subtract the condition translation parameter from the first coupled feature and multiply each element by the negative exponent of the condition scaling parameter to recover the first part.

6. The multimedia content-driven VR digital human scene and AI short video bidirectional conversion generation method as described in claim 1, characterized in that, Step S40 specifically includes: Step S401: Extract character identity, character actions, shot sequence, scene space and narrative tokens from the reversible semantic token sequence, and jointly drive the action, shot and scene decoders to generate a two-dimensional AI short video and a three-dimensional VR digital human scene. Step S402: Obtain the peak action time and shot switching time from the generated 2D AI short video and 3D VR digital human scene, and obtain the corresponding audio accent time; add the absolute value of the difference between the peak action time and the shot switching time and the absolute value of the difference between the peak action time and the audio accent time to obtain the action shot timing closure error; for the part exceeding the preset allowable closure error of 0.03 seconds, square it and multiply it by the preset closure penalty weight of 2.0 to constrain the generation timing of the decoder; Step S403: Re-encode the temporally constrained 2D AI short video and 3D VR digital human scene into tokens, and constrain the decoder with the readback consistency between the re-encoded tokens and the reversible semantic token sequence, so that the 2D AI short video and 3D VR digital human scene are consistent in terms of characters, actions, shots and narrative tags, forming a bimodal generated content set with consistent tags.

7. The multimedia content-driven VR digital human scene and AI short video bidirectional conversion generation method as described in claim 1, characterized in that, Step S50 specifically includes: Step S501: Extract two-dimensional AI short videos from the bimodal content set with consistent tags, extract frames and perform shot segmentation, camera estimation, human pose estimation and depth estimation, and combine speech recognition and optical character recognition to obtain initial tokens and sparse three-dimensional observations of character identity, character actions, shot sequence, scene space and narrative tags. Step S502: Use the initial token as the initial value of the candidate token, and the reversible semantic token sequence as the prior; take the encoding difference between the two-dimensional short video generated by the candidate token and the two-dimensional AI short video, the encoding difference between the three-dimensional scene generated by the candidate token and the sparse three-dimensional observation, and the square norm of the deviation of the candidate token from the initial token as the objective, and perform semantic-geometric joint optimization to minimize the objective, so as to obtain the optimized token; Step S503: Extract the shot timing token and character action token from the optimized token, and generate camera trajectory prior and skeletal joint angle prior respectively; obtain 3D points from the sparse 3D observations, obtain corresponding 2D observations and skeletal joint 2D observations from the 2D AI short video, and optimize the camera and pose by constraining the reprojection error and the deviation relative to the camera trajectory prior and skeletal joint angle prior, and output a 3D VR digital human interaction scene that is consistent with the 2D AI short video in terms of character, action, shot, scene space and narrative tags.

8. A multimedia content-driven VR digital human scene and AI short video bidirectional conversion generation system, applied to the multimedia content-driven VR digital human scene and AI short video bidirectional conversion generation method according to any one of claims 1 to 7, characterized in that, The system includes: The multi-source material access and Huber robust time anchoring module is used to acquire multi-source materials, and based on the multi-source materials, the Huber robust weighted time anchoring mechanism is used to perform multimodal event time alignment tasks and output a unified time anchor point; The cross-modal unified semantic alignment module is used to perform cross-modal semantic alignment and label decoupling tasks based on the unified temporal anchor point using a shared basis plus modal low-rank incremental projection and synchronous hypergraph propagation mechanism, and outputs a unified semantic vector for decoupling five types of labels; The reversible semantic encoding module is used to perform reversible semantic encoding preprocessing based on the unified semantic vector decoupled from the five types of tags, using modal conditional reversible affine coupling and Softmax category routing quantization mechanism, and outputs a reversible semantic token sequence. The forward bidirectional conversion generation module is used to jointly drive multiple decoders to generate two-dimensional AI short videos and three-dimensional VR digital human scenes based on the reversible semantic token sequence, and output a bimodal generated content set with consistent labels. The reverse reconstruction and closed-loop correction module is used to extract two-dimensional AI short videos from the bimodal content set with consistent labels, and to perform reverse parsing and reconstruction tasks using the reversible semantic token sequence as a priori and a semantic-geometric joint optimization mechanism to output a three-dimensional VR digital human interaction scene.

9. A multimedia content-driven VR digital human scene and AI short video bidirectional conversion and generation device, characterized in that, The multimedia content-driven VR digital human scene and AI short video bidirectional conversion generation device includes: a memory, a processor, and a multimedia content-driven VR digital human scene and AI short video bidirectional conversion generation program stored on the memory and executable on the processor. When the multimedia content-driven VR digital human scene and AI short video bidirectional conversion generation program is executed by the processor, it implements the multimedia content-driven VR digital human scene and AI short video bidirectional conversion generation method according to any one of claims 1 to 7.

10. A computer program product, characterized in that, The computer program product includes a multimedia content-driven VR digital human scene and AI short video bidirectional conversion generation program. When the multimedia content-driven VR digital human scene and AI short video bidirectional conversion generation program is executed by the processor, it implements the multimedia content-driven VR digital human scene and AI short video bidirectional conversion generation method according to any one of claims 1 to 7.