Intelligent interaction method, device and equipment of cross-modal data and storage medium

Abstract

The application relates to the technical field of human-computer interaction and digital film and television cross, and provides a cross-modal data intelligent interaction method and device, equipment and a storage medium. According to the modal type of data, the received plot data set is subjected to feature extraction to obtain an initial feature set of each modal, and the initial feature set is projected to a preset common semantic space to generate a cross-modal plot feature sequence aligned in the semantic space; according to a preset semantic community set, the cross-modal plot feature sequence is subjected to clustering analysis and hierarchical compression coding to generate a plot semantic coding sequence; and according to a preset coding element priority, the plot semantic coding sequence is subjected to hierarchical decoding and time domain synchronous fusion rendering to generate an interactive plot data stream. Through modal alignment, semantic clustering, hierarchical coding, priority scheduling and time domain synchronous fusion, the application realizes low-delay and semantic-consistent interactive plot presentation.

Description

Technical Field

[0001] This application relates to the field of human-computer interaction and digital film and television technology, and in particular to an intelligent interaction method, device, equipment and storage medium for cross-modal data. Background Technology

[0002] Interactive short dramas are a form of audiovisual narrative characterized by short duration. They combine the plot compression of traditional short films with real-time interactive mechanisms, allowing viewers to participate in plot branches or trigger local changes through choices, input, or multimodal interactions (such as clicking, voice, facial expressions, and triggering events) as the story progresses, thereby obtaining multiple endings or personalized presentations. Its core lies in breaking down linear narratives into indexable and recombinable plot units, and determining the reading and rendering of subsequent segments based on rules or models at user interaction points to enhance immersion and participation.

[0003] Current interactive short dramas primarily rely on a combination of "branching videos" and UI selection: producers pre-create several linear segments and insert selection interfaces at key nodes, allowing users to switch to the corresponding segment on the client. Another approach uses "conditional flow control" to sequentially play multiple versions of material based on time or conditions. There's also a real-time rendering method based on game engines, using 3D scenes and scripts to drive characters and shots, achieving greater freedom and real-time responsiveness. Additionally, some works use chatbots or voice recognition to map natural language into plot decisions, or enhance interactivity through multi-screen interaction, AR overlay, and social voting. However, these methods mostly rely on discrete segment switching or predefined logic, and depend on network and client redundancy for multimodal consistency, bandwidth, and latency optimization.

[0004] The mainstream methods of existing interactive short dramas are based on fragmentation and version stacking. They cannot align visual, audio and text modalities at the semantic level and perform compact layered compression and priority transmission. This often leads to problems such as high bandwidth consumption, switching latency, modal desynchronization and loss of key semantic information or inconsistent rendering when users initiate interaction. This limits the immersive interactive experience with low latency, high robustness and semantic consistency. Summary of the Invention

[0005] In view of this, this application provides a method, apparatus, device, and storage medium for intelligent interaction of cross-modal data to solve the problem of insufficient collaborative capabilities between cross-modal data.

[0006] This application provides a method for intelligent interaction of cross-modal data, the method comprising: Based on the modality type of the data, feature extraction is performed on the received plot dataset to obtain the initial feature set of each modality, and the initial feature set is projected onto a preset common semantic space to generate a semantic space-aligned cross-modal plot feature sequence. Based on a preset semantic community set, cluster analysis and hierarchical compression encoding are performed on the cross-modal plot feature sequence to generate a semantic encoding sequence for the TV series. The semantic encoding sequence of the series is hierarchically decoded and temporally synchronized for fusion rendering based on the preset priority of encoding elements, generating an interactive plot data stream.

[0007] In an optional implementation, the step of extracting features from the received plot dataset according to the modality type of the data to obtain an initial feature set for each modality, and projecting the initial feature set onto a preset common semantic space to generate a semantic space-aligned cross-modal plot feature sequence includes: The received story dataset is classified according to its modality, and a subset of story data for each modality is generated. The plot data subset is subjected to feature extraction using a preset independent feature extraction strategy to obtain the initial feature set for each modality; The initial feature set is projected onto a preset public semantic space using a preset feature mapping function to generate a cross-modal semantically aligned feature set; Based on the timestamps between the data in the semantic alignment feature set, cross-modal plot synchronization analysis is performed on the semantic alignment feature set to construct a cross-modal plot feature sequence with semantic space alignment.

[0008] In an optional implementation, the step of performing cluster analysis and hierarchical compression encoding on the cross-modal plot feature sequence based on a preset semantic community set to generate a series semantic encoding sequence includes: Calculate the first feature similarity between feature subsequences of the same modality in the cross-modal plot feature sequence; Based on the first feature similarity and time order, cross-modal clustering analysis of candidate subplots is performed on the cross-modal plot feature sequence to obtain the initial subplot feature sequence of each subplot. The initial branch plot feature sequence is optimized, adjusted, and reorganized according to a preset semantic community set to generate a semantic representation feature sequence for each branch plot. According to a preset hierarchical encoding strategy, the semantic representation feature sequence is compressed and serialized to generate a semantic encoding vector sequence. According to the preset quantization codebook, the semantic encoding vector sequence is discretized and quantized, and a discrete index corresponding to each encoding element in the semantic encoding vector sequence is constructed to generate the semantic encoding sequence of the TV series.

[0009] In an optional implementation, after generating the episode semantic encoding sequence, the method further includes: Based on the preset priority of the encoding elements, the semantic encoding sequence of the TV series is subjected to layered encapsulation and unequal error protection encoding processing to generate encrypted data packets corresponding to each encoding element in the semantic encoding sequence of the TV series. A hierarchical scheduling control strategy for the encrypted data packets is constructed based on the priority of the encoded elements and the discrete index using a preset priority scheduling strategy.

[0010] In an optional implementation, the step of performing hierarchical decoding and temporally synchronized fusion rendering of the semantic encoding sequence of the TV series according to a preset encoding element priority to generate an interactive plot data stream includes: The acquired real-time resource data is allocated and processed according to the priority of the encoded elements to obtain the decoded resource data corresponding to the priority of each encoded element. Based on the received interactive data, a target discrete index is determined from the discrete index of the semantic coding sequence of the series, so as to obtain the target coding element and the corresponding target encrypted data packet from the semantic coding sequence of the series based on the target discrete index; The target encrypted data packets are transmitted and sorted according to the scheduling hierarchical control strategy to obtain the target plot encoding sequence; The target encrypted data packets in the target plot encoding sequence are parsed by a preset inverse quantization mapping strategy to restore the target encoding vector sequence corresponding to each target encrypted data packet; Based on the decoded resource data, the target encoded vector sequence is subjected to parallel hierarchical decoding of multimodal data to generate intermediate plot representation data streams for each modality; Global temporal synchronization calibration is performed on the intermediate plot representation data stream to obtain a temporally aligned intermediate plot representation data group sequence; According to the preset fusion rendering strategy, each data group in the intermediate plot representation data group sequence is rendered and synthesized to generate an interactive plot data stream.

[0011] In an optional implementation, after generating the cross-modal plot feature sequence, the method further includes: Calculate the second feature similarity between feature subsequences of different modalities in the cross-modal plot feature sequence according to a preset time window; When the similarity of the second feature is higher than the preset similarity threshold, the similarity of the second feature is converted into attention weights according to the preset weight mapping strategy; The feature subsequences in the cross-modal plot feature sequence are enhanced and fused according to the attention weights through a preset cross-modal attention mechanism, so as to update the feature subsequences in the cross-modal plot feature sequence.

[0012] A second aspect of this application provides an intelligent interaction device for cross-modal data, the device comprising: The feature extraction module is used to extract features from the received plot dataset according to the modality type of the data to obtain the initial feature set for each modality; The semantic alignment module is used to project the initial feature set onto a preset public semantic space to generate a semantic space-aligned cross-modal plot feature sequence. The hierarchical encoding module is used to perform cluster analysis and hierarchical compression encoding on the cross-modal plot feature sequence according to a preset semantic community set, so as to generate a semantic encoding sequence of the series. The decoding and rendering module is used to perform hierarchical decoding and temporally synchronized fusion rendering of the semantic encoding sequence of the series according to the preset encoding element priority, and generate an interactive plot data stream.

[0013] A third aspect of this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the intelligent interaction method for cross-modal data as described above.

[0014] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the intelligent interaction method for cross-modal data as described above.

[0015] In summary, this application includes at least the following beneficial technical effects: 1. Project the features of each modality onto a common semantic space and synchronize the timestamps to ensure that visual / audio / textual features are aligned in terms of semantics and temporal sequence, thereby making plot understanding and interactive decision-making more accurate.

[0016] 2. Through hierarchical compression coding, quantization codebook and unequal error protection, as well as priority encapsulation / scheduling, bandwidth-friendly, priority transmission of key elements and anti-packet loss capability are achieved.

[0017] 3. Enhanced attention weighting, hierarchical parallel decoding, and global temporal synchronous calibration support fast indexing, parallel decoding, and fusion rendering based on interactive triggers, ensuring a low-latency and smooth interactive experience. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart of an intelligent interaction method for cross-modal data provided in an embodiment of this application; Figure 2 This is a functional block diagram of an intelligent interactive device for cross-modal data provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application; Figure 4 This is a schematic diagram of the operating environment structure of the intelligent interaction method for cross-modal data provided in the embodiments of this application. Detailed Implementation

[0020] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] like Figure 4 The diagram shown illustrates the operational environment structure of the intelligent interaction method for cross-modal data provided in this application embodiment. The operational environment provided in this application embodiment includes the following structure.

[0022] The operating environment in this embodiment is based on a distributed collaborative computing architecture, which mainly consists of a central processing unit 4, a user interaction receiving terminal 8, and an interaction unit 6, forming a complete closed-loop system from data perception and semantic processing to immersive presentation. The interaction unit 6 includes a semantic storage network, a communication network, and a synchronization coordinator.

[0023] Among them, the central processing unit 4, as the core computing hub of the system, is usually deployed in the cloud or edge computing nodes. Its physical form can be a high-performance GPU server cluster or a dedicated artificial intelligence computing device. It is responsible for performing the most time-consuming computing tasks in the method, including but not limited to feature extraction and semantic alignment of multimodal plot data, clustering analysis and hierarchical compression encoding of cross-modal plot feature sequences, and generation of episode semantic encoding sequences. This unit has an optimized deep learning inference framework built in, which can efficiently schedule computing resources to perform complex neural network model operations.

[0024] User interaction receiving terminal 8 is the physical device that the user directly interacts with. It can take the form of augmented reality (AR) glasses, virtual reality (VR) headsets, high-performance smart TVs, or mobile terminals with powerful graphics processing capabilities. The core of this terminal includes a receiver module with data transmission and reception, local rendering, and display functions. On the one hand, this module receives the user's "interaction data" through a natural language interface, motion sensor, or touch screen and uploads it to the central processing unit. On the other hand, it receives the "series semantic encoding sequence" or partially generated streaming data from the network after compression and hierarchical transmission. Based on the terminal's own graphics, audio, and haptic rendering capabilities, it performs localized real-time decoding and rendering under the instructions of the synchronization coordinator. Finally, it seamlessly presents the generated images, sounds, and haptic feedback to the user, thus completing the interaction loop.

[0025] Furthermore, the system relies on a distributed, highly available interaction unit 6. Its semantic storage network consists of multiple storage nodes used to persistently store the encoded episode semantic sequences, preset quantization codebooks, semantic community sets, and various generative model parameters. The central processing unit 4 and the user interaction receiving terminal 8 can quickly load the required resources from the nearest storage node as needed, reducing latency and improving response speed. Simultaneously, the interaction unit 6 also includes a low-latency, high-bandwidth communication network for data transfer. This communication network not only transmits raw multimodal data, compressed semantic codes, and control commands, but also ensures that high-priority semantic data packets are transmitted to the receiving end reliably and preferentially through integrated priority scheduling and quality of service assurance mechanisms. Moreover, the interaction unit 6 also integrates a hardware-software combined logic module (i.e., a multimodal synchronization coordinator). The multimodal synchronization coordinator may be embedded in the central processing unit 4 or the user interaction receiving terminal 8. It uses a high-precision global clock as a reference and is responsible for scheduling and coordinating the execution timing of parallel tasks such as text generation, audio synthesis, image rendering and haptic feedback, ensuring that the output of all modalities is strictly aligned on the micro time scale, thereby providing users with a highly consistent immersive story experience.

[0026] In summary, the operating environment of the intelligent interaction method for cross-modal data provided in this application is a collaborative ecosystem that integrates cloud intelligence, edge computing, high-speed networks and heterogeneous terminals. Each module is interconnected through standardized interfaces and protocols, jointly supporting the complete technology chain from semantic understanding to immersive interaction.

[0027] like Figure 1 The diagram shown is a flowchart of an intelligent interaction method for cross-modal data provided in an embodiment of this application. The intelligent interaction method for cross-modal data provided in this application includes the following steps.

[0028] Step S1: Extract features from the received plot dataset according to the modality type of the data to obtain the initial feature set of each modality, and project the initial feature set to a preset common semantic space to generate a semantic space-aligned cross-modal plot feature sequence.

[0029] It should be understood that the plot dataset is a multimodal data collection specifically built for interactive short dramas. It is not a single file, but a heterogeneous data stream that is strictly aligned on the timeline and contains complete metadata. Its core components include script dialogue and scene descriptions in the text modality, character dialogue and environmental sound effects in the audio modality, scene images and character action sequences in the image modality, and preset force feedback and vibration mode parameters in the haptic modality. The construction process of this dataset is derived from standardized digital content production processes. For example, in the plot event of "the knight drawing his sword in the castle hall," the scriptwriter generates corresponding descriptive sentences, the recording studio simultaneously records the metallic friction sound of the sword being drawn and the echo of footsteps, the motion capture system and cameras capture visual materials from multiple angles, and the haptic designer defines the instantaneous impact and continuous vibration parameters of the handle based on physical simulation. All materials are aligned and encapsulated using a unified timecode in the post-production stage to form the plot dataset.

[0030] After the central processing unit loads the story dataset from the storage network, the heterogeneous data streams need to be classified to avoid interference between different formats during subsequent data operations, as the data of different modalities have drastically different data structures and information densities. For example, text is a discrete sequence of symbols while images are a continuous matrix of pixels. Based on the modality type of the data, the mixed data stream is separated into independent story data subsets. For example, all text descriptions are grouped into a text subset, and all audio waveforms are grouped into an audio subset. Next, this embodiment configures a pre-trained specialized feature extraction network for each modality, thereby enabling the application of independent feature extraction strategies for different data. For example, for the text subset, a language model based on the Transformer architecture is used to transform sentences into feature vectors containing contextual semantics; for the image subset, a deep convolutional network is used to extract contour, texture, and spatial relationship features from pixels; for the audio subset, it is first converted into a Mel spectrogram and then extracted using a convolutional network to extract spectral and temporal pattern features; for the tactile subset, a one-dimensional convolutional network is used to analyze the intensity and frequency profile of vibration signals. By performing specific feature extraction operations on each subset of plot data, the original high-dimensional perceptual data is reduced in dimensionality and transformed into a low-dimensional initial feature set that can represent its essential content, thereby reducing the difficulty of data processing.

[0031] Then, the initial feature sets obtained from the above operations are still in mutually isolated feature spaces. For example, text feature vectors and image feature vectors cannot be directly compared or correlated mathematically. Therefore, this application embodiment uses a feature mapping function to project each initial feature set to a preset public semantic space. The mapping function used in this application is typically implemented as a set of lightweight fully connected neural networks, each network dedicated to a specific modality. Its function is to linearly or non-linearly transform modality-specific feature vectors into a shared, uniform-dimensional vector space. This network optimizes the time-aligned metadata embedded in the dataset through a mechanism called contrastive learning: it forces feature pairs from different modalities describing the same plot moment (such as the text feature of "drawing a sword" and its corresponding sword sound audio feature) to be as close as possible in the public semantic space, while the distance between randomly sampled feature pairs from different moments is pushed further apart. After sufficient training, the feature mapping function can project semantic concepts such as "knight," "sword," and "metal collision," regardless of whether they come from text, sound, or image, to similar coordinate regions in the public semantic space, thereby generating cross-modal semantically aligned feature sets. The semantic alignment feature set essentially constructs a universal semantic language across senses. After obtaining the semantic alignment feature set, the system needs to restore and enhance its narrative structure in the temporal dimension. Specifically, based on the precise timestamp carried by each feature in the semantic alignment feature set, cross-modal plot synchronization analysis is performed. The semantic alignment features of all modalities within the same time point or a small time window are grouped and arranged strictly in chronological order, thereby constructing a semantically aligned cross-modal plot feature sequence. Each time point in this sequence corresponds to a feature snapshot containing multimodal semantic information, and the entire sequence constitutes the semantic skeleton of the plot development.

[0032] The above operations reorganize discrete multimodal data points into a coherent, machine-traceable temporal narrative flow, providing a structured data source for plot understanding and generation.

[0033] After generating the cross-modal plot feature sequence, an optional enhancement step is performed in this embodiment to further improve its robustness and information richness. First, based on a preset time window, the second feature similarity between feature subsequences of different modalities in the sequence is calculated. For example, the cosine similarity between visual and audio feature subsequences in the common semantic space within a certain time period is calculated. The time window setting is based on prior knowledge of plot coherence—related multimodal events always co-occur within a finite time range. When the calculated second feature similarity is higher than a preset similarity threshold, it indicates that the two cross-modal segments are highly semantically related; for example, a sword-wielding action in a scene is strongly correlated with the sound of wind breaking. Subsequently, according to a preset weight mapping strategy (e.g., through a sigmoid function), the values ​​of the second feature similarity exceeding the threshold are converted into an attention weight between 0 and 1, which quantifies the strength of this cross-modal association. Finally, using all the calculated attention weights, weighted fusion and information enhancement processing are performed on each feature subsequence in the original cross-modal plot feature sequence. The cross-modal attention mechanism used in this application embodiment is as follows: weighted summation of feature information from other modalities is used to update the feature representation of the current modality in order to adaptively strengthen the truly relevant cross-modal signals in the sequence (such as more closely integrating the visual features of the sword with the auditory features of the metallic sound), while suppressing noise interference that occurs simultaneously but is semantically irrelevant, thereby updating and outputting an enhanced cross-modal plot feature sequence with stronger noise resistance and clearer internal correlation.

[0034] For example, consider an interactive short drama containing the plot of "a knight charging from the end of the corridor to the door and suddenly drawing his sword." Its cross-modal plot feature sequence already includes time-stamp-aligned visual feature subsequences (such as the knight's figure approaching from afar in consecutive frames, and the action of reaching for the sword), audio feature subsequences (such as the sound of running footsteps increasing in intensity, and the sharp metallic sound of the sword blade rubbing against the scabbard), and tactile feature subsequences (such as low-frequency continuous signals simulating running vibrations, and high-frequency instantaneous signals simulating the impact of drawing the sword). The preset time window is set to 500ms, based on the physiological perception limit of human motion continuity and audio-visual synchronization, meaning the system only examines cross-modal feature pairs with time differences within this window. At time point T1, the system calculates the cosine similarity in the common semantic space between the visual feature subsequence representing "the knight running" and the audio feature subsequence representing "heavy footsteps" within a 500ms window before and after that moment. Since both visual movement and auditory rhythm semantically point to the common intention of "rapid approach," the similarity value calculated in this instance may be as high as 0.85. Simultaneously, the system also calculates the similarity between the same visual feature subsequence and the audio feature subsequence representing "background wind sound," but due to their weaker semantic association, the result may only be 0.2. A preset similarity threshold is set to 0.5 to filter out accidental associations. When the similarity value of "running visual - footsteps sound" (0.85) exceeds the threshold of 0.5, a preset weight mapping strategy is activated. This strategy maps the original similarity to attention weights using a sigmoid function, with the function parameters calibrated so that an input of 0.85 results in an output weight value close to 0.92. This weight of 0.92 signifies that, in subsequent fusion, the semantic information about "footsteps sound" in the audio modality will be injected into the visual feature update calculation at a high proportion. Conversely, for the "running visual - background wind sound" pair with a similarity of only 0.2, its mapped attention weight is close to zero, meaning that wind sound information will be effectively ignored. Subsequently, a pre-defined cross-modal attention mechanism operates using all these computationally generated attention weights. For the visual feature vector describing "the knight running," the mechanism aggregates semantic information from the audio features of "footsteps" with a weight of 0.92, while almost ignoring the features of "background wind sounds." Specifically, this is a process of weighted summation and linear transformation: the updated visual feature vector will not only contain the original visual content but also incorporate approximately 92% of the auditory semantics of "regularity" and "heaviness" extracted from the synchronized audio. Similarly, the audio features also receive semantic enhancements regarding "speed" and "direction" from the visual features.

[0035] Step S2: Perform cluster analysis and hierarchical compression encoding on the cross-modal plot feature sequence according to the preset semantic community set to generate the series semantic encoding sequence.

[0036] It should be understood that the aforementioned cross-modal plot feature sequence is still a lengthy, continuous temporal stream containing information from all modalities. To reduce the complexity of data transmission and processing, it needs to be converted into a highly structured, hierarchical, and easily transmitted discrete symbolic representation, namely, the episode semantic encoding sequence.

[0037] First, the central processing unit calculates the first feature similarity between feature subsequences of the same modality within the cross-modal plot feature sequence. Since feature changes within the same modality often more sensitively reflect the boundaries of plot units, for example, in a plot containing "reconnaissance," "dialogue," and "combat," the visual modality's features will continuously focus on characters' faces and lip movements within the "dialogue" segment, resulting in high internal similarity; however, when the plot transitions to "combat," the features rapidly change to large-scale actions and scene transitions, leading to a decrease in similarity. This application's embodiment uses a sliding window method to calculate the cosine similarity matrix (i.e., the first feature similarity) of feature fragments within each modality (text, audio, visual, etc.). The first feature similarity quantifies the coherence of each modality's changes over time, providing independent evidence from each sensory dimension for discovering potential plot turning points.

[0038] Subsequently, combining the inherent temporal order of the feature vectors and the first feature similarity, cross-modal clustering analysis of candidate subplots is performed on the complete cross-modal plot feature sequence. It should be understood that the clustering analysis operation in this embodiment does not rely solely on evidence from a single modality. Instead, it uses the first feature similarity calculated for each modality as a basic association metric, combined with the proximity of features along the time axis, and employs graph clustering or hierarchical clustering algorithms to perform a global analysis of the entire cross-modal sequence. Specifically, each moment in the entire sequence is considered a graph node, and the connection strength between nodes is determined by temporal proximity and weighted multimodal similarity. By applying a community detection algorithm based on modularity optimization to the graph, nodes can be divided into multiple communities. Nodes within each community exhibit high internal consistency across modal features and correspond to a temporally continuous and semantically complete candidate plot unit, such as automatically identified segments like "the knight infiltrates the corridor," "discovers the treasure," and "confronts the guard." All cross-modal features contained in each such community are extracted to form the initial subplot feature sequence for each subplot. The above operations automatically deconstructed a preliminary plot structure skeleton from the original time-series stream.

[0039] Next, the initial branch plot feature sequences are further optimized, adjusted, and reorganized using a pre-defined semantic community set. The semantic community set is a predefined or standardized plot unit template library learned from a large amount of drama knowledge, containing semantic descriptions of typical scenes such as "quiet exploration," "tense confrontation," "intense battle," and "emotional dialogue." The central processing unit matches and compares each initial branch plot feature sequence with the templates in the semantic community set. For example, an initial branch plot feature sequence might contain mixed features of "whispered dialogue" and "premonition of drawing a sword." By comparing it with the semantic community set, the system identifies that this segment better fits the typical feature distribution of the "tense confrontation" template. Based on this judgment, the system fine-tunes and reorganizes the features within this segment, potentially strengthening key semantic features related to "confrontation" while weakening secondary detail features, thereby generating a purer, more semantically clear, and narratively logical semantic representation feature sequence for each branch plot. Through the above operations, prior narrative knowledge is used to verify and refine the automated clustering results, improving the accuracy and rationality of the plot structure division.

[0040] After obtaining more refined semantic representation feature sequences for each subplot, the system compresses and serializes them according to a pre-defined hierarchical encoding strategy. This strategy is typically implemented as a two-stage process: In the first stage, a lightweight autoencoder is used to compress the semantic representation feature vector of each subplot, removing redundant information and retaining the core semantics, outputting a low-dimensional compressed encoding vector for the subplot. In the second stage, the compressed encoding vectors of all subplots are arranged in chronological order according to their plot occurrences, forming a high-level sequence. This sequence is then processed by a sequence encoder (such as a gated recurrent unit or a Transformer encoder) to capture the transitions and echoes between subplots, outputting a fixed-dimensional semantic encoding vector sequence that summarizes the overall structure of the entire series. Through these operations, a significant reduction in data volume is achieved while preserving the complete semantic context of the plot, resulting in substantial compression of the original massive feature data.

[0041] Finally, to adapt to network transmission and storage, the central processing unit discretizes and quantizes the continuous semantic encoded vector sequence according to a preset quantization codebook. The quantization codebook is a lookup table containing several prototype vectors. In this embodiment, the quantization operation searches for the prototype vector with the closest Euclidean distance in the codebook for each continuous value vector in the semantic encoded vector sequence, and replaces the original vector with the unique index (an integer between 1 and K) of that prototype vector. Thus, the continuous vector sequence is converted into a discrete integer index sequence, while simultaneously constructing and maintaining the correspondence between these indices and the original encoded elements (i.e., the prototype vectors in the codebook). Ultimately, the semantic encoded vector sequence is transformed into a sequence composed of discrete integer indices, i.e., the episode semantic encoded sequence. For example, a continuous vector describing a "battle" scene might be quantized to index "153," while a "dialogue" scene might be quantized to index "76." This discrete sequence has strong noise resistance and can be further compressed through entropy coding, becoming the plot's "semantic DNA" that can be transmitted efficiently and robustly over the network.

[0042] After generating the semantic encoded sequence of the series, to ensure its reliable transmission in complex network environments, in an optional implementation, the central processing unit performs the following transmission optimization steps. First, based on preset encoding element priorities, the sequence undergoes layered encapsulation and unequal error protection encoding. The encoding element priorities define the importance of different parts of the sequence; for example, indexes identifying key plot twists have high priority, while indexes describing environmental details have low priority. During layered encapsulation, high-priority indexes are encapsulated in data packets with stronger error correction capabilities (such as using forward error correction codes with higher redundancy), ensuring that the core semantics can still be accurately recovered in channels with packet loss or high error rates. Unequal error protection encoding applies stronger error correction coding (such as adding more redundant check bits) to high-priority data packets, making them more resistant to transmission errors; low-priority data packets use lighter checkpoints or no error correction at all. This generates encrypted data packets with different error resistance capabilities corresponding to the importance of each encoding element in the series. Subsequently, a hierarchical scheduling control strategy for encrypted data packets is constructed by combining a preset priority scheduling policy with the priority of encoded elements and the logical order of discrete indices in the storyline (implied by the indices). This hierarchical scheduling control strategy guides network devices to prioritize sending high-priority data packets during transmission, reserving bandwidth for them and selecting high-quality routes. This ensures that core storyline semantics can reach the user's interactive receiving terminal with low latency and high reliability, while secondary data is allowed to be delayed or dropped during congestion, thereby maximizing the user experience of critical data with limited network resources.

[0043] Step S3: Based on the preset priority of the encoding elements, perform hierarchical decoding and temporally synchronized fusion rendering of the semantic encoding sequence of the TV series to generate an interactive plot data stream.

[0044] It should be understood that, in the embodiments of this application, the execution of step S3 begins with the interactive intelligent restoration and presentation of the received episode semantic encoding sequence on the user interaction receiving terminal side, so as to convert the discrete and compressed semantic symbol stream into a multimodal plot data stream that the user can immerse themselves in in real time.

[0045] First, the user interaction receiving terminal allocates and processes the acquired real-time resource data. This real-time resource data refers to the available computing resources on the user interaction receiving terminal at the current moment, including GPU computing power, memory availability, network bandwidth, and battery life. Based on the encoding element priorities determined in step S2 (e.g., discrete indices identifying "key dialogue" or "core actions" have high priority, while "background details" indices have low priority), limited resources are dynamically allocated. More computing cores and faster processing threads are reserved or allocated to high-priority decoding tasks, while remaining resources are allocated to low-priority tasks or a delayed scheduling strategy is used, thereby obtaining the decoding resource data corresponding to the priority of each encoding element. This resource allocation ensures the smoothness of the interactive experience and the quality of the core content, ensuring that even when terminal resources fluctuate, the most important parts of the story perceived by the user always receive sufficient processing power for high-quality reproduction.

[0046] When a user inputs interactive data via natural language, gestures, or a controller, such as saying "I want to see how the knight opens the treasure chest," the user interaction receiving terminal immediately determines the target discrete index from the discrete index of the episode's semantic encoding sequence based on the received interactive data. The interactive data is first converted into a semantic query vector, and then searched and matched within the metadata index structure corresponding to the episode's semantic encoding sequence to quickly locate the discrete index range corresponding to the plot segment most relevant to the intention of "opening the treasure chest." Based on these target discrete indices, the user interaction receiving terminal requests the corresponding target encoded elements and corresponding target encrypted data packets from its local cache or via the network to the central processing unit. This enables non-linear access to the plot and interactive navigation, allowing users to directly jump to plot points of interest rather than passively watching linearly.

[0047] After acquiring the target encrypted data packet, the user-interactive receiving terminal sends a corresponding transmission request to the central processing unit (CPU) according to the scheduling hierarchy control strategy already established by the CPU. This allows the CPU to prioritize the target encrypted data packets for transmission. The transmission prioritization comprehensively considers the packet priority, the temporal dependencies between packets, and the current network conditions. For example, for the "open the treasure chest" storyline segment, the initial keyframe index data packet (high priority) will be placed in the highest priority transmission queue to ensure immediate delivery; while the detailed index data packet of the treasure chest texture (low priority) may be scheduled for later transmission. Through this prioritization process, the user-interactive receiving terminal obtains a target storyline encoding sequence that conforms to the storyline's logical progression and optimizes transmission efficiency. While this sequence may exhibit inconsistent packet arrival order at a microscopic level due to network jitter, at a macroscopic level, priority marking ensures the preservation of key content.

[0048] Next, the user interaction receiving terminal parses and restores the ordered target encrypted data packets. The inverse quantization mapping strategy adopted in this embodiment is as follows: First, each target encrypted data packet is decrypted and forward-corrected to recover the original discrete index values. Then, a reverse lookup table of the completely consistent quantization codebook obtained from the central processing unit is queried to precisely map each discrete index value back to its corresponding continuous value prototype vector, thereby restoring the target encoded vector sequence corresponding to each target encrypted data packet. The quantization codebook is stored locally on the user interaction receiving terminal after acquisition, or loaded in real-time from edge nodes. The above inverse quantization operation transforms the noise-resistant discrete symbols back into dense semantic features that can be understood by the generative model. Subsequently, the user interaction receiving terminal performs parallel hierarchical decoding of the restored target encoded vector sequence based on the previously allocated decoding resource data. Specifically, multiple parallel lightweight generative models are activated, including a text decoder, an audio decoder, an image decoder, and a haptic decoder. Each decoder takes a shared target encoded vector sequence as conditional input, but is allocated different computational precision and speed according to the priority of its generated content. For example, for the high-priority core action of "the knight opening the treasure chest," the image decoder uses a complete model to generate high-definition keyframes, while the audio decoder generates corresponding unlocking sound effects and ambient sounds. For the low-priority "treasure chest surface pattern," the image decoder may use a simplified model to quickly generate an approximate texture, or directly reuse a pre-stored texture template. All decoders work in parallel, outputting text streams, audio streams, image frame sequences, and haptic control signal streams, collectively forming the intermediate plot representation data streams for each modality. This hierarchical decoding operation maximizes the overall perceived quality of the experience within the limited computational constraints of the terminal, precisely allocating computational resources to the plot elements that users care about most.

[0049] Because the processing speeds of different modal decoders may vary, and jitter introduced by network transmission may cause slight deviations in packet arrival times, global temporal synchronization calibration of the generated intermediate plot representation data stream is necessary. The user interaction receiver relies on a high-precision global clock embedded in the user interaction receiver or issued from the central processing unit as a reference. Each intermediate data unit (such as a frame of image or a segment of audio) carries a precise timestamp recovered from the original semantic encoding. By comparing the actual arrival time of each data stream with the theoretical timestamp, the user interaction receiver uses techniques such as dynamically adjusting the audio playback buffer, image frame presentation queue, or inserting / discarding non-keyframes to align all modal data streams onto a unified, smoothly advancing timeline. This results in a temporally aligned sequence of intermediate plot representation data groups, eliminating flaws affecting immersion such as audio-visual asynchrony and haptic feedback delays, and ensuring that the "visual instant of the treasure chest opening," the "crisp sound of the lock popping open," and the "simulated vibration of the controller" occur strictly simultaneously.

[0050] Finally, the temporally aligned intermediate narrative representation data sequence is rendered and synthesized. The fusion rendering strategy employed in this embodiment coordinates the graphics rendering engine, audio mixing engine, and haptic driver interface on the user interaction receiving terminal. Specifically, the graphics rendering engine synthesizes image frame sequences with possible two-dimensional / three-dimensional graphic elements, applying color correction and lighting effects; the audio mixing engine mixes and spatializes dialogue, sound effects, and background music to create a stereo sound field; and the haptic driver interface converts haptic control signals into specific motor drive commands. The outputs of all engines are strictly synchronized and encapsulated into an interactive narrative data stream conforming to streaming media standards, directly driving the display screen to refresh images, the speakers to output sound, and the haptic actuators to vibrate, thereby presenting the user with a complete, coherent, immersive narrative experience that responds to their interactive intentions.

[0051] This application applies to the interdisciplinary fields of human-computer interaction and digital film and television technology. It extracts features from a received plot dataset based on the modality of the data to obtain initial feature sets for each modality. These initial feature sets are then projected onto a predefined common semantic space to generate a semantically aligned cross-modal plot feature sequence. Based on a predefined semantic community set, the cross-modal plot feature sequence is clustered and hierarchically compressed to generate a semantic encoding sequence for the series. Finally, based on predefined encoding element priorities, the semantic encoding sequence is hierarchically decoded and time-domain synchronized to generate an interactive plot data stream. This application achieves low-latency and semantically consistent interactive plot presentation through modality alignment, semantic clustering and hierarchical encoding, priority scheduling and time-domain synchronization fusion.

[0052] like Figure 2 The diagram shown is a functional block diagram of an intelligent interactive device for cross-modal data provided in an embodiment of this application.

[0053] In some embodiments, the intelligent interaction device 2 for cross-modal data may include multiple functional modules composed of computer program segments. The computer programs for each program segment in the intelligent interaction device 2 for cross-modal data may be stored in the server's memory and executed by at least one processor to perform (see details). Figure 1 (Description) Functionality of intelligent interaction methods for cross-modal data.

[0054] In this embodiment, the intelligent interaction device 2 for cross-modal data can be divided into multiple functional modules according to the functions it performs. These functional modules may include: a feature extraction module 21, a semantic alignment module 22, a hierarchical encoding module 23, a decoding and rendering module 24, a hierarchical scheduling module 25, and an enhancement and optimization module 26. The module referred to in this invention is a series of computer program segments that can be executed by at least one processor and perform a fixed function, stored in memory. In this embodiment, the functions of each module will be detailed in subsequent embodiments.

[0055] The feature extraction module 21 is used to extract features from the received plot dataset according to the modality type of the data to obtain the initial feature set of each modality; Semantic alignment module 22 is used to project the initial feature set onto a preset public semantic space to generate a semantic space-aligned cross-modal plot feature sequence; The hierarchical encoding module 23 is used to perform cluster analysis and hierarchical compression encoding on the cross-modal plot feature sequence according to a preset semantic community set to generate a semantic encoding sequence for the TV series. The decoding and rendering module 24 is used to perform hierarchical decoding and temporally synchronized fusion rendering of the semantic encoding sequence of the series according to the preset encoding element priority, and generate an interactive plot data stream.

[0056] In an optional implementation, the feature extraction module 21 and the semantic alignment module 22 are used for: The received story dataset is classified according to its modality, and a subset of story data for each modality is generated. The plot data subset is subjected to feature extraction using a preset independent feature extraction strategy to obtain the initial feature set for each modality; The initial feature set is projected onto a preset public semantic space using a preset feature mapping function to generate a cross-modal semantically aligned feature set; Based on the timestamps between the data in the semantic alignment feature set, cross-modal plot synchronization analysis is performed on the semantic alignment feature set to construct a cross-modal plot feature sequence with semantic space alignment.

[0057] In an optional implementation, the layered coding module 23 is used for: Calculate the first feature similarity between feature subsequences of the same modality in the cross-modal plot feature sequence; Based on the first feature similarity and time order, cross-modal clustering analysis of candidate subplots is performed on the cross-modal plot feature sequence to obtain the initial subplot feature sequence of each subplot. The initial branch plot feature sequence is optimized, adjusted, and reorganized according to a preset semantic community set to generate a semantic representation feature sequence for each branch plot. According to a preset hierarchical encoding strategy, the semantic representation feature sequence is compressed and serialized to generate a semantic encoding vector sequence. According to the preset quantization codebook, the semantic encoding vector sequence is discretized and quantized, and a discrete index corresponding to each encoding element in the semantic encoding vector sequence is constructed to generate the semantic encoding sequence of the TV series.

[0058] In an optional implementation, the intelligent interaction device 2 for cross-modal data further includes a hierarchical scheduling module 25, which is used for: Based on the preset priority of the encoding elements, the semantic encoding sequence of the TV series is subjected to layered encapsulation and unequal error protection encoding processing to generate encrypted data packets corresponding to each encoding element in the semantic encoding sequence of the TV series. A hierarchical scheduling control strategy for the encrypted data packets is constructed based on the priority of the encoded elements and the discrete index using a preset priority scheduling strategy.

[0059] In an optional implementation, the decoding and rendering module 24 is used for: The acquired real-time resource data is allocated and processed according to the priority of the encoded elements to obtain the decoded resource data corresponding to the priority of each encoded element. Based on the received interactive data, a target discrete index is determined from the discrete index of the semantic coding sequence of the series, so as to obtain the target coding element and the corresponding target encrypted data packet from the semantic coding sequence of the series based on the target discrete index; The target encrypted data packets are transmitted and sorted according to the scheduling hierarchical control strategy to obtain the target plot encoding sequence; The target encrypted data packets in the target plot encoding sequence are parsed by a preset inverse quantization mapping strategy to restore the target encoding vector sequence corresponding to each target encrypted data packet; Based on the decoded resource data, the target encoded vector sequence is subjected to parallel hierarchical decoding of multimodal data to generate intermediate plot representation data streams for each modality; Global temporal synchronization calibration is performed on the intermediate plot representation data stream to obtain a temporally aligned intermediate plot representation data group sequence; According to the preset fusion rendering strategy, each data group in the intermediate plot representation data group sequence is rendered and synthesized to generate an interactive plot data stream.

[0060] In an optional implementation, the intelligent interaction device 2 for cross-modal data further includes an enhancement and optimization module 26, which is used to: Calculate the second feature similarity between feature subsequences of different modalities in the cross-modal plot feature sequence according to a preset time window; When the similarity of the second feature is higher than the preset similarity threshold, the similarity of the second feature is converted into attention weights according to the preset weight mapping strategy; The feature subsequences in the cross-modal plot feature sequence are enhanced and fused according to the attention weights through a preset cross-modal attention mechanism, so as to update the feature subsequences in the cross-modal plot feature sequence.

[0061] It should be understood that the various variations and specific embodiments of the methods provided in the above embodiments are also applicable to the intelligent interaction device for cross-modal data in this embodiment. Through the foregoing detailed description of the intelligent interaction method for cross-modal data, those skilled in the art can clearly understand the implementation method of the intelligent interaction device for cross-modal data in this embodiment. For the sake of brevity, it will not be described in detail here.

[0062] like Figure 3 The diagram shown is a structural schematic of an electronic device provided in an embodiment of this application.

[0063] In a preferred embodiment of the present invention, the electronic device 3 may include, but is not limited to, a memory 31, at least one processor 32, and at least one communication bus 33.

[0064] Those skilled in the art should understand that Figure 3 The structure of the electronic device 3 shown does not constitute a limitation of the embodiments of the present invention. The electronic device 3 may also include more or fewer other hardware or software than shown, or different component arrangements.

[0065] In some embodiments, the electronic device 3 is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits, programmable gate arrays, digital processors, and embedded devices.

[0066] It should be noted that the electronic device 3 is merely an example. Other existing or future electronic products that are suitable for this application should also be included within the scope of protection of this application and are incorporated herein by reference.

[0067] In some embodiments, the memory 31 stores a computer program that, when executed by the at least one processor 32, implements all or part of the steps in the intelligent interaction method for cross-modal data as described. The memory 31 includes read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data. Further, the computer-readable storage medium may primarily include a program storage area and a data storage area, wherein the program storage area may store an operating system, at least one application program required for a function, etc.

[0068] In some embodiments, the at least one processor 32 is the control unit of the electronic device 3, connecting various components of the electronic device 3 via various interfaces and lines. It executes programs or modules stored in the memory 31 and calls data stored in the memory 31 to perform various functions and process data. For example, when the at least one processor 32 executes a computer program stored in the memory 31, it implements all or part of the steps of the intelligent interaction method for cross-modal data described in this application embodiment; or it implements all or part of the functions of the intelligent interaction device for cross-modal data. The at least one processor 32 may be composed of integrated circuits, such as a single-packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips.

[0069] In some embodiments, the at least one communication bus 33 is configured to enable communication between the memory 31 and the at least one processor 32, etc. Although not shown, the electronic device 3 may also include a power supply (e.g., a battery) to power the various components. Preferably, the power supply can be logically connected to the at least one processor 32 via a power management device, thereby enabling functions such as charging, discharging, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device 3 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0070] The integrated unit implemented as a software functional module described above can be stored in a computer-readable storage medium. This software functional module, stored in a storage medium, includes several instructions to cause an electronic device (which may be a personal computer, electronic device, or network device, etc.) or processor to execute portions of the methods described in the various embodiments of this application.

[0071] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0072] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0073] The above are all preferred embodiments of this application, and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the structure, shape and principle of this application should be covered within the scope of protection of this application.

Claims

1. A method for intelligent interaction of cross-modal data, characterized in that, The method includes: Based on the modality type of the data, feature extraction is performed on the received plot dataset to obtain the initial feature set of each modality, and the initial feature set is projected onto a preset common semantic space to generate a semantic space-aligned cross-modal plot feature sequence. Based on a preset semantic community set, cluster analysis and hierarchical compression encoding are performed on the cross-modal plot feature sequence to generate a semantic encoding sequence for the TV series. The semantic encoding sequence of the series is hierarchically decoded and temporally synchronized for fusion rendering based on the preset priority of encoding elements, generating an interactive plot data stream.

2. The intelligent interaction method for cross-modal data according to claim 1, characterized in that, The step of extracting features from the received plot dataset according to the modality type of the data to obtain an initial feature set for each modality, and projecting the initial feature set onto a preset common semantic space to generate a semantic space-aligned cross-modal plot feature sequence includes: The received story dataset is classified according to its modality, and a subset of story data for each modality is generated. The plot data subset is subjected to feature extraction using a preset independent feature extraction strategy to obtain the initial feature set for each modality; The initial feature set is projected onto a preset public semantic space using a preset feature mapping function to generate a cross-modal semantically aligned feature set; Based on the timestamps between the data in the semantic alignment feature set, cross-modal plot synchronization analysis is performed on the semantic alignment feature set to construct a cross-modal plot feature sequence with semantic space alignment.

3. The intelligent interaction method for cross-modal data according to claim 1, characterized in that, The step of performing cluster analysis and hierarchical compression encoding on the cross-modal plot feature sequence based on a preset semantic community set to generate a series semantic encoding sequence includes: Calculate the first feature similarity between feature subsequences of the same modality in the cross-modal plot feature sequence; Based on the first feature similarity and time order, cross-modal clustering analysis of candidate subplots is performed on the cross-modal plot feature sequence to obtain the initial subplot feature sequence of each subplot. The initial branch plot feature sequence is optimized, adjusted, and reorganized according to a preset semantic community set to generate a semantic representation feature sequence for each branch plot. According to a preset hierarchical encoding strategy, the semantic representation feature sequence is compressed and serialized to generate a semantic encoding vector sequence. According to the preset quantization codebook, the semantic encoding vector sequence is discretized and quantized, and a discrete index corresponding to each encoding element in the semantic encoding vector sequence is constructed to generate the semantic encoding sequence of the TV series.

4. The intelligent interaction method for cross-modal data according to claim 3, characterized in that, After generating the semantic encoding sequence of the series, the method further includes: Based on the preset priority of the encoding elements, the semantic encoding sequence of the TV series is subjected to layered encapsulation and unequal error protection encoding processing to generate encrypted data packets corresponding to each encoding element in the semantic encoding sequence of the TV series. A hierarchical scheduling control strategy for the encrypted data packets is constructed based on the priority of the encoded elements and the discrete index using a preset priority scheduling strategy.

5. The intelligent interaction method for cross-modal data according to claim 4, characterized in that, The step of performing hierarchical decoding and temporally synchronized fusion rendering of the semantic encoding sequence of the TV series according to the preset encoding element priority to generate an interactive plot data stream includes: The acquired real-time resource data is allocated and processed according to the priority of the encoded elements to obtain the decoded resource data corresponding to the priority of each encoded element. Based on the received interactive data, a target discrete index is determined from the discrete index of the semantic coding sequence of the series, so as to obtain the target coding element and the corresponding target encrypted data packet from the semantic coding sequence of the series based on the target discrete index; The target encrypted data packets are transmitted and sorted according to the scheduling hierarchical control strategy to obtain the target plot encoding sequence; The target encrypted data packets in the target plot encoding sequence are parsed by a preset inverse quantization mapping strategy to restore the target encoding vector sequence corresponding to each target encrypted data packet; Based on the decoded resource data, the target encoded vector sequence is subjected to parallel hierarchical decoding of multimodal data to generate intermediate plot representation data streams for each modality; Global temporal synchronization calibration is performed on the intermediate plot representation data stream to obtain a temporally aligned intermediate plot representation data group sequence; According to the preset fusion rendering strategy, each data group in the intermediate plot representation data group sequence is rendered and synthesized to generate an interactive plot data stream.

6. The intelligent interaction method for cross-modal data according to claim 1, characterized in that, After generating the cross-modal plot feature sequence, the method further includes: Calculate the second feature similarity between feature subsequences of different modalities in the cross-modal plot feature sequence according to a preset time window; When the similarity of the second feature is higher than the preset similarity threshold, the similarity of the second feature is converted into attention weights according to the preset weight mapping strategy; The feature subsequences in the cross-modal plot feature sequence are enhanced and fused according to the attention weights through a preset cross-modal attention mechanism, so as to update the feature subsequences in the cross-modal plot feature sequence.

7. A cross-modal data intelligent interaction device, applied to the cross-modal data intelligent interaction method of claim 1, characterized in that, The device includes: The feature extraction module is used to extract features from the received plot dataset according to the modality type of the data to obtain the initial feature set for each modality; The semantic alignment module is used to project the initial feature set onto a preset public semantic space to generate a semantic space-aligned cross-modal plot feature sequence. The hierarchical encoding module is used to perform cluster analysis and hierarchical compression encoding on the cross-modal plot feature sequence according to a preset semantic community set, so as to generate a semantic encoding sequence of the series. The decoding and rendering module is used to perform hierarchical decoding and temporally synchronized fusion rendering of the semantic encoding sequence of the series according to the preset encoding element priority, and generate an interactive plot data stream.

8. An electronic device, characterized in that, The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the intelligent interaction method for cross-modal data according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the intelligent interaction method for cross-modal data according to any one of claims 1 to 6.

Images