Omnimedia fusion method and system based on complementary fusion

Abstract

The present disclosure provides a full media fusion method and system based on complementary fusion, which relates to the field of converged communications. The method includes collecting different types of media data from multiple media sources, dividing them into several processable media units according to a preset segmentation strategy; performing feature extraction on different types of media data respectively to generate corresponding basic feature vectors; using a pseudo-query module to generate an associated pseudo-query vector; inputting the basic feature vector and the pseudo-query vector into an implicit interaction module, and outputting a fused media representation vector; storing the media representation vector and its corresponding media metadata in a vector database, and indexing the media representation vector in the vector database; in response to receiving a fusion request for a target media unit, obtaining media resources matching the target media unit by searching the vector database to perform cross-media display of the target media unit.

Description

Technical Field

[0001] The present disclosure relates to the field of converged communications, and in particular to a full media convergence method and system based on complementary fusion. Background Art

[0002] Multimodal data such as text, images, audio, video, and live streaming are increasingly widely used in various industries. Data of different modalities have significant differences in structure, semantics, and temporal distribution. Traditional single-modal or simple splicing processing methods often find it difficult to achieve deep integration and efficient matching of cross-modal content. Especially when it is necessary to combine and display data of multiple modalities (such as aligning video with text subtitles, mixing live streams with images / audio, etc.), if there is a lack of an effective complementary mechanism for multimodal features, it is easy to cause problems such as low retrieval and matching efficiency, large data redundancy, and inability to accurately align cross-modal information, making it difficult to meet the needs of multimedia integration in real-time and high-concurrency environments.

[0003] For example, the Chinese patent application with the announcement number CN113033647A discloses a multimodal feature fusion method, the main idea of ​​which is: first extract the features of each mode of the multimedia resource separately, combine the features of each mode in the modal dimension to form multi-channel features, and then generate fusion features through multi-channel convolution processing to achieve information complementarity between different modal features. This solution provides multiple convolution operations based on the convolution kernel in the dimension D direction, and supports technical means such as feature aggregation and stretching processing, hoping to improve the effect of multimodal feature fusion to a certain extent. This solution emphasizes that through convolution processing between channels, the influence of a single feature value can be effectively reduced, and the expression ability of the fusion feature can be improved. It is suitable for feature fusion scenarios of multimedia resources such as video image frame data, audio data and text data.

[0004] However, the above-mentioned solutions mainly focus on using convolution operations to perform relatively static “same-dimensional splicing” and channel convolution on multimodal features. They do not deeply consider how to reduce the computational overhead of online large-scale comparison in cross-media retrieval and real-time interaction scenarios, nor do they propose incremental fusion strategies for media types with strong temporal continuity, such as live streaming. Although it has a certain inter-channel complementary effect in the combination of multimodal features, when the system needs to dynamically supplement, synchronize in real time, or present on multiple terminals the potential needs and associations of different modalities in an online or highly concurrent environment, it lacks a more complete implicit interaction and offline perception mechanism, making it difficult to take into account both real-time and fusion accuracy. In addition, the processing method of only performing channel convolution on features of the same dimension is difficult to further reduce network load or improve cross-media retrieval efficiency when it is necessary to perform complementary queries on each media unit and realize online fusion scenarios. Therefore, there is still a need for a full-media fusion technology solution that is more adaptable to offline preprocessing and rapid online matching of multimodal data, and can perform low-latency, multi-terminal synchronous display of real-time data such as live streams, so as to overcome the limitations of existing solutions in cross-modal correlation accuracy and real-time mixed presentation. Summary of the Invention

[0005] In view of the deficiencies of the existing technology, the embodiments of the present disclosure provide a full media fusion method and system based on complementary fusion.

[0006] In a first aspect, an embodiment of the present disclosure provides an all-media fusion method based on complementary fusion, including:

[0007] Collect different types of media data from multiple media sources and perform preliminary formatting on the media data; wherein the media data includes: text, images, audio, video and live streaming data;

[0008] The formatted media data is divided into media units according to a preset segmentation strategy; wherein a media unit refers to an independent media segment within a time or space range, and each media unit includes at least one of the following: a piece of text, a frame or a video, an image or a group of images, and an audio segment;

[0009] Perform feature extraction on different types of media data to generate basic feature vectors corresponding to each media unit;

[0010] generating, using a pseudo query module, a pseudo query vector associated with each media unit based on the basic feature vector;

[0011] Inputting the basic feature vector and the pseudo query vector into an implicit interaction module, and outputting a fused media representation vector; storing the media representation vector and its corresponding media metadata in a vector database, and creating an index for the media representation vector in the vector database;

[0012] In response to receiving a fusion request for a target media unit, the media resources matching the target media unit are obtained by retrieving the vector database, and the target media unit and the matching media resources are synchronously synthesized according to the timestamp information of the matching media resources to perform cross-media display of the target media unit.

[0013] As an optional implementation manner, generating a pseudo query vector associated with each media unit includes:

[0014] Performing an attention mechanism operation on the basic feature vector to generate a potential demand representation;

[0015] A pseudo query vector is generated based on the potential demand representation, and the pseudo query vector is optimized using a reconstruction loss or a contrastive loss.

[0016] As an optional implementation manner, the outputting the fused media representation vector includes:

[0017] Concatenate the basic feature vector and the pseudo query vector to form an input sequence;

[0018] Processing the input sequence through a multi-head self-attention network to generate a fused media representation vector;

[0019] The fused media representation vector is output for indexing and retrieval.

[0020] As an optional implementation, storing the media representation vector and its corresponding media metadata in a vector database and indexing the media representation vector in the vector database includes:

[0021] Establishing a mapping relationship between the media representation vector and the media metadata, wherein the media metadata includes a media type, a timestamp, and a frame index;

[0022] Storing the media representation vector and its media metadata in a vector database;

[0023] The media representation vectors are indexed based on an approximate nearest neighbor search algorithm or a hash index algorithm.

[0024] As an optional implementation, the cross-media presentation includes:

[0025] receiving a fusion request for the target media unit;

[0026] Retrieving a candidate media unit from the vector database whose media representation vector similarity with the target media unit is higher than a preset threshold;

[0027] splicing and fusing the target media unit and the candidate media unit according to a set fusion strategy;

[0028] Output the fusion results to the front end for display.

[0029] As an optional implementation, the cross-media presentation further includes:

[0030] Slicing the live streaming data according to time periods, and generating the basic feature vector and pseudo query vector for each time period;

[0031] The fused media representation vectors of the live streaming data are incrementally written into the vector database, and the latest media representation vectors are used for matching during online retrieval.

[0032] As an optional implementation, the feature extraction includes:

[0033] For text, image, audio and video, a convolutional neural network, a visual transformer, a pre-trained language model or a speech recognition model is used to extract the basic feature vector from the original media data.

[0034] As an optional implementation, the cross-media presentation further includes: embedding the matching media resource into a rendering window corresponding to the target media unit.

[0035] In a second aspect, an embodiment of the present disclosure further provides an all-media fusion system based on complementary fusion, comprising: an acquisition module, a first processing module, a feature extraction module, a second processing module, a third processing module, and a display module;

[0036] The acquisition module is used to collect different types of media data from multiple media sources and perform preliminary formatting on the media data; wherein the media data includes: text, images, audio, video and live streaming data;

[0037] The first processing module is configured to divide the formatted media data into media units according to a preset segmentation strategy; wherein a media unit refers to a relative media segment within a time or space range, and each media unit includes at least one of the following: a piece of text, a frame or a video, an image or a group of images, and an audio segment;

[0038] The feature extraction module is used to extract features from different types of media data and generate basic feature vectors corresponding to each media unit;

[0039] The second processing module is configured to generate a pseudo query vector associated with each media unit based on the basic feature vector using a pseudo query module;

[0040] The third processing module is configured to input the basic feature vector and the pseudo query vector into an implicit interaction module and output a fused media representation vector; store the media representation vector and its corresponding media metadata in a vector database, and create an index for the media representation vector in the vector database;

[0041] The display module is used to, in response to receiving a fusion request for a target media unit, obtain media resources matching the target media unit by searching the vector database, and synchronously synthesize the target media unit with the matching media resources based on the timestamp information of the matching media resources to perform cross-media display of the target media unit.

[0042] Compared with the existing technology, the beneficial effect of the present invention is that different media (text, images, audio, video, live streaming) can use the same vector database to achieve cross-modal retrieval and complementarity, which is suitable for application in scenarios such as online teaching, audio-visual entertainment, conference live broadcast, and telemedicine. By adopting the approximate nearest neighbor search and timestamp synchronization of the vector database, only dot product calculation or similarity calculation is required in the online stage to complete cross-media matching and fusion, which greatly improves the fusion response speed. As the media data increases, the system only needs to extract features and implicitly interact with the newly added media units, write them into the vector database, and update the index, supporting large-scale distributed expansion. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A flowchart of an all-media fusion method based on complementary fusion provided in an embodiment of the present disclosure;

[0044] Figure 2 A flowchart of a method for establishing an index in a vector database provided in an embodiment of the present disclosure;

[0045] Figure 3 A schematic diagram of an all-media fusion system based on complementary fusion provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0046] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0047] The present invention provides a full media fusion method based on complementary fusion, see Figure 1 FIG. 1 is a flowchart of a full media fusion method based on complementary fusion provided by an embodiment of the present disclosure, wherein the method includes steps S101 to S106, wherein:

[0048] S101: Collecting different types of media data from multiple media sources and performing preliminary formatting on the media data; wherein the media data includes text, images, audio, video, and live streaming data;

[0049] S102: Divide the formatted media data into a plurality of processable media units according to a preset segmentation strategy; wherein a media unit refers to a relatively independent media segment within a temporal or spatial range, and each media unit includes at least one of the following: a text segment, a frame or a video segment, an image or a group of images, and an audio segment;

[0050] S103: Extract features for different types of media data and generate basic feature vectors corresponding to each media unit;

[0051] S104: using a pseudo query module to generate a pseudo query vector associated with each media unit based on the basic feature vector;

[0052] S105: Inputting the basic feature vector and the pseudo query vector into an implicit interaction module, outputting a fused media representation vector; storing the media representation vector and its corresponding media metadata into a vector database, and creating an index for the media representation vector in the vector database;

[0053] S106: In response to receiving a fusion request for a target media unit, media resources matching the target media unit are obtained by retrieving the vector database, and the target media unit and the matching media resources are synchronously synthesized according to the timestamp information of the matching media resources to perform cross-media display of the target media unit.

[0054] Regarding S101 above:

[0055] In the context of full media convergence, media data from different sources is diverse and formatted inconsistently. Direct subsequent analysis can easily lead to difficulties in identification or processing. By uniformly collecting media data from different sources and performing preliminary formatting, we can lay the foundation for subsequent feature extraction and cross-media matching.

[0056] In a specific implementation, different types of media data can be obtained from multiple media sources, including text databases, image storage services, audio recording and storage servers, video streaming platforms, and live broadcast servers.

[0057] For example, text data can be obtained from existing document libraries, news sources, or documents uploaded by users through API interfaces; image data can be read from image storage or materials uploaded by photographic equipment; audio and video data can be loaded from streaming platforms or local files; live streaming data can be captured through real-time acquisition ports.

[0058] In a specific implementation, the preliminary formatting of the collected original media data according to the respective media types may include:

[0059] For text, remove redundant spaces, unify the encoding format (such as UTF-8), and perform necessary cleaning;

[0060] For images, you can do proportional scaling in terms of length and width resolution or unify the image encoding format (such as JPEG / PNG);

[0061] For audio, unify the sampling rate, bit rate or audio encoding format (such as MP3 / WAV);

[0062] For videos, preliminary transcoding can be performed based on frame rate, resolution, or encoding format (such as H.264 / H.265);

[0063] For live streaming, the real-time stream is segmented into predetermined time periods (eg, 5 seconds or 10 seconds) and stored as temporary buffer files for subsequent processing.

[0064] In this way, through the above processing, a basic media file or data stream is generated for subsequent analysis and feature extraction.

[0065] Regarding S102 above:

[0066] After completing the unified formatting, the entire paragraph or article of media data still needs to be further divided into smaller "media units" so that the system can perform more refined feature extraction and matching of fragments of different types and different time / space ranges in subsequent steps.

[0067] In a specific implementation, the formatted media data is divided into a number of processable media units according to a preset segmentation strategy.

[0068] Among them, for text data, it can be divided based on paragraph or sentence-level granularity, and each paragraph of text corresponds to a media unit; for image data, each image or a group of images is regarded as a media unit; for audio or video data, a time segmentation strategy can be adopted, such as every 10 seconds or each key frame interval as a media unit; for live streaming data, the live stream can also be divided into multiple relatively independent media units according to the same time segmentation strategy.

[0069] It's important to note that in this disclosure, a media unit refers to a relatively independent piece of media within a temporal or spatial scope. This can include one or more of the following types: a piece of text, a frame or video clip, an image or group of images, and an audio clip. This categorization allows subsequent steps to process "media units" as the processing target, simplifying the management and analysis of multimodal data.

[0070] Regarding S103 above:

[0071] The key to multimodal fusion lies in converting different types of media into comparable and searchable vector representations. Without a unified feature vector representation, it would be difficult to achieve effective comparison and complementarity across media such as text, images, audio, and video.

[0072] In specific implementation, for each media unit obtained by division, it is necessary to use a feature extraction algorithm adapted to its media type to obtain a basic feature vector. For example:

[0073] For text feature extraction, you can use pre-trained language models (such as BERT, RoBERTa) or Transformer-based semantic encoders to perform vectorization operations on text units to generate semantic expression vectors.

[0074] For image feature extraction, CNN (such as ResNet, VGG) or Vision Transformer can be used to perform convolution / self-attention analysis on the image to extract the visual feature vector.

[0075] For audio feature extraction, Mel-Spectrogram or MFCC feature analysis can be performed on audio media units (including live stream segments), and the audio feature vector can be obtained by combining RNN or Transformer structures.

[0076] For video feature extraction, the spatiotemporal feature vectors can be extracted from video media units through key frame extraction or models such as 3D convolutional networks and video transformers.

[0077] It should be emphasized that if the video unit is long, average pooling can be performed on multiple frames to obtain an overall representation.

[0078] In this way, through the above multimodal feature extraction, the present disclosure finally generates a corresponding basic feature vector for each media unit and saves it in memory or temporary storage for subsequent processing.

[0079] Regarding S104 above:

[0080] Simple basic feature vectors can only reflect the content of a media unit itself, but fail to capture the complementary information that this media unit may need for other media resources. Performing complex cross-comparisons across all media units online would be computationally expensive and lack real-time performance.

[0081] In a specific implementation, the present disclosure inputs the basic feature vector of each media unit into a pseudo-query module (such as a lightweight Transformer decoder or a self-attention-based subnetwork) to generate a pseudo-query vector associated with each media unit. This pseudo-query vector can be understood as a potential expression of "the media unit's possible needs or connections with other media."

[0082] In a specific implementation, contrastive learning or reconstruction loss can be used in the training phase so that the pseudo query vector captures the core semantics and potential interaction information of the media unit.

[0083] In a specific implementation, the pseudo query vector may be generated in the following manner:

[0084] Input: basic feature vector;

[0085] Intermediate process: The pseudo query module performs multi-head self-attention calculation or decoding operation on the basic feature vector based on the trained parameters;

[0086] Output: Pseudo query vector associated with this media unit.

[0087] For example, if the media unit is a text describing a cooking process, the pseudo-query vector may include potential requirements for related ingredients, cooking temperature, and cooking time; while for an image of a specific scene, the pseudo-query vector may express associations with information such as location, theme, or linkable video.

[0088] As an optional implementation manner, generating a pseudo query vector associated with each media unit includes:

[0089] Performing an attention mechanism operation on the basic feature vector to generate a potential demand representation;

[0090] A pseudo query vector is generated based on the potential demand representation, and the pseudo query vector is optimized using reconstruction loss or contrast loss, so that the pseudo query vector can represent the association characteristics between the media unit and other media resources.

[0091] Among them, after the multimodal feature extraction is completed, each media unit has a basic feature vector, denoted as F base In order to further generate a pseudo query vector that can characterize the association features between the media unit and other media resources, the present disclosure introduces two steps: attention mechanism operation and reconstruction / contrast loss optimization.

[0092] In the specific implementation, F base The input is fed into a lightweight “attention generation submodule”, which can be implemented using a multi-head self-attention structure and includes at least the following key components:

[0093] 1. Linear mapping layer:

[0094] F base Mapped to three vector spaces: query (Q), key (K) and value (V). Specifically, a trainable weight matrix W can be set Q To generate the query vector Q = F base W Q , and generate K and V similarly.

[0095] Since the feature vectors of media units are often high in dimensionality, the linear mapping layer can reduce parameters while ensuring that information is not lost. This can be achieved by reducing dimensionality or maintaining the same dimensionality.

[0096] 2. Attention calculation layer:

[0097] The attention calculation layer performs dot product attention on Q, K, and V:

[0098]

[0099] Among them, Attention (Q, K, V) is used to assign learnable weights between input features to highlight the information most relevant to the current task; the softmax function is a normalization function that maps the input vector to the (0, 1) interval and makes its sum equal to 1, thereby obtaining the attention weight in the form of a probability distribution, d k is the dimension of the key vector.

[0100] It should be emphasized that for the case of only a single vector input (i.e., a single media unit F base ), this embodiment can be F base Consider it as batch processing or introduce virtual sequence length or fixed position encoding to ensure that the attention mechanism does not conflict in calculation. For example, if F base Corresponding to the video key frame sequence, multi-vector attention operations can also be performed at the frame level.

[0101] 3. Multi-head merging layer:

[0102] If multi-head attention is used, the outputs of each head are concatenated in the channel dimension and then linearly mapped to obtain the output vector A. This output vector A can be regarded as a "potential demand representation," representing an abstract expression of the media unit and external related elements after the attention mechanism.

[0103] Through the above attention mechanism operation, the potential demand representation A obtained can reflect the media unit's potential demand for cooperation or complementarity with other media resources, and preliminarily characterize the characteristics of "which external information the media unit may be associated with".

[0104] In addition, the study found that in actual system implementation, the simple self-attention operation cannot ensure that A accurately reflects the cross-media correlation features. Therefore, the present disclosure further optimizes A through reconstruction loss or contrast loss to obtain the final pseudo query vector (Q pseudo ).

[0105] For reconstruction loss:

[0106] In a specific implementation, if the system has additional annotations or contextual information about the media unit (such as key tags, summaries, corresponding subtitles, etc.), A can be input into a "small decoder" or classifier to form constraints by predicting or reconstructing the additional information.

[0107] For example, when the media unit is a piece of text and the text has manually annotated topic tags, the system can ask A to predict the topic tag; if the prediction is successful, it means that A has captured the core semantics of the text more accurately.

[0108] The loss function can be designed as cross entropy or mean square error, etc., and the parameters of the attention generation submodule and the merging layer are tuned through back propagation to make A have a higher semantic fit.

[0109] For contrast loss:

[0110] In specific implementations, if the system has positive and negative sample pairs (for example, "two video clips of the same scene" and "text descriptions of them with consistent timestamps", etc.), comparative learning can be used to make the vector distance between A and the matching samples closer and the vector distance between A and the mismatched samples farther.

[0111] For example, assuming that a media unit A and a media unit B are complementary resources, they can be marked as positive sample pairs, and the Q generated by the two pseudo The distance is closer; for random irrelevant media units C, it is a negative sample pair, making Q pseudo The distance is further. In this way, A will become more and more capable of distinguishing between related and unrelated things.

[0112] Commonly used formulas for contrast loss, such as InfoNCE or TripletLoss, can enable A to learn the ability to distinguish between cross-media retrieval or matching needs after multiple iterations.

[0113] Furthermore, in the specific implementation, the A produced by the above self-attention mechanism is continuously iterated and trained in the back propagation, and finally the pseudo query vector Q can be stably obtained. pseudo .

[0114] If reconstruction loss is used, the system can treat the output A as Q after training. pseudo If contrast loss is used, a linear transformation layer (MLP) or normalization operation can be added after A to obtain the final Q pseudo .

[0115] It should be noted that, at the feasibility level, the attention module and decoder / comparison network can be trained end-to-end or in stages during the training phase to ensure that each parameter update can reduce the prediction error or increase the discrimination, so that the pseudo-query vector not only retains the characteristic information of the original media unit, but also has the ability to "perceive needs" for other media.

[0116] It should be emphasized that in the implementation scheme of the present invention, the use of reconstruction loss (Reconstruction Loss) or contrastive loss (Contrastive Loss) is not just a conventional algorithm choice for model training, but is directly deeply coupled with the multimedia fusion application scenario, playing a key technical role in cross-modal data processing and online real-time display.

[0117] For example, relying solely on "basic feature vectors" to index or match media content will result in a problem in which it is difficult to fully express a media unit's possible complementary needs for other modal information. Therefore, large-scale cross-modal interactions or complex comparisons are often required in the online stage, resulting in a significant increase in latency. In the present invention, the concept of "pseudo-query vector" is proposed, so that each media unit can learn the "potential demand for external media content" during the offline or small-batch training stage. In order to enable this pseudo-query vector to more accurately and stably characterize cross-media associations or complementary needs, the present invention selects reconstruction loss or contrast loss in conjunction with the attention mechanism to overcome the limitation that pure perceptron or simple classification loss cannot fully capture the "demand-supply" semantic interaction between multiple modalities.

[0118] For example, in actual multimodal services, subtitles, keyword tags, meta-event descriptions, etc. are often configured for video clips or audio segments, or there is cross-annotation of the same content in different modalities (such as a video clip and its corresponding text description).

[0119] This invention utilizes a reconstruction loss, ensuring that the pseudo-query vector is "reconstructable" or "predictable" of these multimodal annotations. Once the model learns to reconstruct this auxiliary information through the pseudo-query vector during training, it indicates that the pseudo-query vector truly captures the intrinsic semantic connection between the media unit and the external complementary resources.

[0120] In addition, at the application level, the use of reconstruction loss greatly improves the accuracy of injecting "complementary demand perception" into each media unit in the offline stage, reduces the redundant inter-modal comparison overhead during online queries, and thus helps resource scheduling and integrated presentation in high-concurrency scenarios such as live broadcast and online on-demand.

[0121] Furthermore, in order to accelerate online retrieval and ensure fusion accuracy, the present invention can distinguish between cross-modal positive and negative sample pairs (such as audio and text describing it, or video frames and corresponding image illustrations, etc.) during the training phase.

[0122] As mentioned above, when it is determined that "media units A and B have a complementary relationship in a certain application scenario", they are marked as a positive sample pair, making the corresponding pseudo-query vector distance closer; conversely, if the two are not related, they are marked as a negative sample pair, increasing the vector distance.

[0123] In this way, contrastive loss is no longer just a general machine learning method; it is now integrated with the specific needs of multimedia convergence applications, enabling pre-learning of cross-modal correlation during offline training. When online, the system simply performs a vector search on the pseudo-query vector of the target media unit to quickly locate other media resources with the highest cross-media complementarity. Compared to similar solutions that do not use contrastive loss, this invention significantly improves both cross-modal matching and online query speed.

[0124] In this invention, by optimizing the "pseudo-query vector" through reconstruction loss or contrastive loss, the system can teach media units how to complement and match other media resources during offline training or small-batch training updates. In this case, the online phase requires only simple vector similarity calculations to determine the optimal fusion objects and presentation methods.

[0125] This technology directly serves the needs of online multimedia integration. For example, in live e-commerce scenarios, the product video clip currently being displayed by the host can be integrated with the most relevant product poster or accompanying instruction manual in real time through pop-up windows. In educational live broadcasts, courseware, exercises, or supplementary audio for the corresponding chapter can be recommended in a timely manner, significantly improving user acquisition efficiency and interactive experience.

[0126] Unlike simply optimizing neural networks using reconstruction loss or contrastive learning, this invention emphasizes integrating this training strategy into the overall process of multimodal feature extraction, implicit interaction, and vector database retrieval to reduce the burden of online computation and improve the efficiency of multimodal content complementation. In other words, this invention's use of reconstruction / contrastive loss goes beyond the simple deep learning algorithm level. Instead, it incorporates the system architecture requirements of offline-online multimedia data fusion to achieve the specific technical effects of cross-media complementarity: reducing latency, saving bandwidth, and improving retrieval accuracy and user satisfaction.

[0127] Furthermore, this embodiment can train the pseudo query generation subsystem in an offline environment (such as a GPU / TPU cluster), process existing media data on a large scale, and store the generated pseudo query vectors in a vector database.

[0128] When a new media unit is collected by the system and basic features are extracted, the attention module and comparison / reconstruction network are immediately called to generate a pseudo query vector, which is stored in the database for real-time retrieval.

[0129] In addition, if the system finds that some pseudo-query vectors do not match actual needs based on user feedback, the parameters of the attention mechanism can be regularly (or in real time) updated through small-batch training to enable the system to adapt to the environment and new media.

[0130] This enables the pseudo-query vector to more accurately characterize the associations between each media unit and other media resources, serving as the basis for subsequent cross-media complementary presentations. Compared to traditional single feature vector approaches, this implementation embeds the "perception of external media demand" attribute into each media unit during the offline phase, reducing the computational complexity of online retrieval and improving the system's robustness and responsiveness in multimodal fusion scenarios.

[0131] Regarding S105 above:

[0132] During the offline phase, by further fusing the "basic feature vector" and "pseudo-query vector," each media unit's final representation vector (the "fused media representation vector") incorporates its own characteristics while retaining a representation of the needs or relationships with other media. This eliminates the need for real-time interaction across all media during subsequent online searches, significantly reducing computational overhead.

[0133] In a specific implementation, the present disclosure inputs the "basic feature vector" and "pseudo query vector" of each media unit into the implicit interaction module. For example, a multi-layer Transformer or a multi-head self-attention network can be used to implement implicit interaction.

[0134] The implicit interaction operations include:

[0135] Through self-attention or cross-attention mechanisms, the "features of the media unit itself" and the "pseudo-query information expected or required by the media unit" are integrated to generate a fused media representation vector with a stronger "cross-media complementarity awareness";

[0136] The purpose of implicit interaction is to inject awareness of other media forms or content requirements into each media unit during the offline phase, thereby eliminating the need for large-scale cross-modal complex calculations during online retrieval.

[0137] Ultimately, the fused media representation vector output by the implicit interaction module can better characterize the semantic features of the media unit and its potential complementary relationship with other modalities, providing a basis for subsequent retrieval and synthesis.

[0138] Furthermore, the present disclosure stores the fused media representation vector together with the media metadata information of the media unit (such as media type, timestamp, frame index, text paragraph ID, live stream slice ID, etc.) in a vector database.

[0139] Exemplarily, the vector database may adopt a structure such as Faiss, Milvus, or HNSW to support large-scale vector similarity search.

[0140] Furthermore, after storage is complete, an approximate nearest neighbor (ANN) or hash index can be constructed based on the fused media representation vector to quickly perform subsequent retrieval and matching operations. At this point, each media unit has a unique identifier and index entry in the vector database, which can be searched in the online stage.

[0141] As an optional implementation manner, the outputting the fused media representation vector includes:

[0142] Concatenate the basic feature vector and the pseudo query vector to form an input sequence;

[0143] Processing the input sequence through a multi-head self-attention network to generate a fused media representation vector;

[0144] The fused media representation vector is output for indexing and retrieval.

[0145] In order to obtain the fused media representation vector and facilitate subsequent indexing and retrieval, in addition to the aforementioned generation and optimization of the pseudo query vector, the following operations can be further performed:

[0146] After completing the pseudo query vector Q pseudo and the basic eigenvector F base After generation, the system concatenates the two to form the input sequence.

[0147] For example, the following steps may be used:

[0148] The first is serialization processing, when F base With Q pseudo When both are single vectors, they can be directly spliced ​​according to the vector dimension; if both contain temporal information (such as video key frames or audio frame sequences), each frame vector can be first merged in time order or spatial order, and then the pseudo query vector can be appended to the beginning or end of the sequence to form the input sequence (S in ).

[0149] Doing so ensures that multimodal information and "demand / relevance" features are presented in the same input tensor, and subsequent processing layers no longer need to perform tedious cross-tensor operations.

[0150] Secondly, it is a multi-head self-attention network process. The system takes the input sequence (S in ) is input into a multi-head self-attention network (TransformerEncoder or similar structure). Each attention head calculates the attention weights between vectors in the sequence to capture temporal or semantic correlations.

[0151] If the base feature vector itself has a positional encoding, such as a video frame index or a text paragraph ID, this information can be retained when concatenating it with the pseudo query vector.

[0152] After the multi-head self-attention network is executed, a set of context enhancement vectors (H ctx In this embodiment, the vector that best represents the overall information of the sequence (such as the [CLS] position vector or the average pooling result of all vectors) can be selected as the fused media representation vector (F fused ).

[0153] Finally, the fused media representation vector (F fused ) is output to the storage or retrieval module for subsequent index retrieval.

[0154] In the specific implementation, in order to facilitate subsequent indexing, F fused Perform normalization (such as L2 regularization) or dimension reduction (such as PCA) to generate the representation vector (F final ). F final Compared with the original basic eigenvector F base and pseudo query vector Q pseudo , which can better reflect the integration characteristics of multimodal information and cross-media needs, and is suitable for similarity measurement or distance measurement in subsequent retrieval.

[0155] In this way, through the above-mentioned splicing, multi-head self-attention operation and vector output processing, the present invention can generate a fused media representation vector, providing a more complementary feature representation for the subsequent indexing and retrieval process of the vector database.

[0156] See also Figure 2 , which is a flowchart of a method for establishing an index in a vector database provided in an embodiment of the present disclosure. As an optional implementation, storing the media representation vector and its corresponding media metadata in the vector database and establishing an index for the media representation vector in the vector database includes steps S201 to S203, wherein:

[0157] S201: Establishing a mapping relationship between the media representation vector and the media metadata, where the media metadata includes a media type, a timestamp, and a frame index;

[0158] S202: Storing the media representation vector and its media metadata in a vector database;

[0159] S203: Create an index for the media representation vector based on an approximate nearest neighbor search algorithm or a hash index algorithm.

[0160] Regarding the above S201:

[0161] In a specific implementation, after generating the fused media representation vector F fused or F final When recording, the corresponding media metadata is recorded synchronously, including:

[0162] Media type (e.g., text, video, audio, image, or live stream segment);

[0163] Timestamp (in video or audio scenarios, identifies the start and end time of the segment in the original file; in live broadcast scenarios, identifies the actual playback or recording time);

[0164] Frame index (if it is a video key frame, it can record the frame number or continuous frame range).

[0165] By establishing a "mapping structure" (such as key-value or JSON format) containing the above media metadata for each fused media representation vector, refined matching can be achieved during future retrieval through vector similarity plus media metadata condition screening.

[0166] Regarding the above 202:

[0167] In specific implementation, a database or engine that supports large-scale vector search can be selected, such as Faiss, Milvus, HNSW, etc. fused The mapping structure is written to the database once, assigning a unique identifier (such as media_id) to each entry.

[0168] When a user query or system fusion request occurs, the stored vector table is retrieved using similarity calculation and the most similar entries are output.

[0169] Metadata fields (media type, timestamp, frame index) can be used to further perform secondary filtering or sorting in the search results.

[0170] Regarding the above S203:

[0171] In specific implementations, to improve retrieval efficiency, an index building process can be performed on all media representation vectors written into the database:

[0172] ANN (Approximate Nearest Neighbor) indexing: In high-dimensional space, all vectors are partitioned or a hierarchical graph structure (such as HNSW) is constructed to achieve O(logN) or sublinear approximate search during query time.

[0173] Hash index: Uses algorithms such as LSH (Locality-Sensitive Hashing) to bucket vectors in high-dimensional space. During queries, precise comparisons are performed only within the same or similar hash buckets, improving query speed.

[0174] The indexing process is usually performed offline in batches, but can also be updated incrementally as new media data is added.

[0175] For example, if the live broadcast data source continuously generates new media units, the system can regularly (or in real time) insert the newly generated fused media representation vectors into the database and update the ANN or hash index structure to ensure the timeliness of the retrieval.

[0176] By establishing, storing, and indexing these mapping relationships, the present invention enables fusion vector retrieval within large multimodal media repositories and combines this with media metadata for precise matching, facilitating cross-media fusion or alignment. Compared to traditional methods that store only the original document or file name, the present method significantly improves cross-modal retrieval speed and accuracy, and can be applied to a variety of scenarios, including online education, video retrieval, intelligent surveillance, and social media content recommendations.

[0177] Regarding S106 above:

[0178] After receiving a fusion request, cross-media recommendations or synthesis results need to be provided to users or downstream applications in a timely manner. By searching for similarities in the "fused media representation vectors" and combining them with media metadata (such as timestamps and frame indices), other media that can complement or enhance the content of the target media unit can be quickly located.

[0179] In a specific implementation, when the system receives a fusion request for a target media unit, it first searches the vector database for other media units whose fused media representation vectors are similar to the target media unit above a preset threshold. From the search results, it selects media resources that contain corresponding timestamp information and are complementary to the target media unit. Based on the timestamp or frame information of the retrieved media resources, the media resources are synchronized with the target media unit.

[0180] For example, if the target media unit is the Nth minute of a video, and the matched resource is an audio or subtitle text at the corresponding moment, the audio or subtitle text can be embedded into the video playback stream;

[0181] For another example, if the target media unit is a live stream segment, image / text information of the same time period or similar semantic points is found in the matching resources for real-time overlay display.

[0182] In a specific implementation, the present disclosure performs cross-modal combination or synchronous rendering of the target media unit and the matching media resources after synthesis.

[0183] For example, video and text can be displayed in split screen on the same playback page; audio can be superimposed on the video stream to form a new multimedia stream; corresponding images or text descriptions can pop up at key moments in live broadcast scenes, etc.

[0184] The final cross-media presentation result can be played, browsed or interacted in the client or front-end interface.

[0185] In this way, through the pseudo-query vector generation and implicit interaction module in the offline stage, the system integrates the potential complementary needs of different media units before storage, reducing the large-scale computing in the online stage.

[0186] As an optional implementation, the cross-media presentation includes:

[0187] receiving a fusion request for the target media unit;

[0188] Retrieving a candidate media unit from the vector database whose media representation vector similarity with the target media unit is higher than a preset threshold;

[0189] splicing and fusing the target media unit and the candidate media unit according to a set fusion strategy;

[0190] Output the fusion results to the front end for display.

[0191] In the specific implementation, during the online operation phase, the system waits for or listens for integration requests from the client, upper-layer business modules, or third-party applications. The request includes the following information:

[0192] Target media unit ID: for example, video_segment_id, audio_clip_id, text_paragraph_id, or other unique ID;

[0193] Fusion preferences: For example, users may specify the need to splice subtitles, insert images, or switch between multi-camera video clips;

[0194] Terminal / platform information: such as mobile, PC, or AR / VR device information, which can be used to differentiate the fusion strategy in the future.

[0195] This implementation sets a unified API interface on the server side, such as POST / media / fusion_request. When an external call is made, the system parses the request parameters and writes them into a queue or memory cache to trigger the next step of retrieval and fusion process.

[0196] In a specific implementation, after parsing the target media unit identifier, the following steps are performed in the vector database:

[0197] The corresponding “fused media representation vector” (e.g. F fused If the database supports fast ID-vector mapping, the vector can be directly obtained; if the system stores media representation vectors in separate libraries or shards, you must first locate the node that stores the target entry.

[0198] Based on the representation vector of the target media unit, an approximate nearest neighbor (ANN) search or a hash bucket search is performed to obtain a set of candidate media units {Candidate1, Candidate2, ...} whose similarity is higher than a preset threshold (eg, 0.8).

[0199] In addition, a coarse sorting (ANN) can be performed first followed by a fine sorting (exact dot product or cosine similarity) to ensure retrieval efficiency and accuracy.

[0200] The system sorts the retrieved candidate media units by similarity from high to low, and performs a secondary screening based on metadata information (such as timestamp, frame index, and media type). For example, in the case of video subtitles, only text or audio units with the same or similar timestamp as the current video period can be retained.

[0201] In a specific implementation, the present application maintains a configurable fusion strategy list in this embodiment, including:

[0202] Splicing methods: such as time-series splicing (merging by timestamp), image / video overlay, or split-screen;

[0203] Priority or weight: For example, in a video + text scenario, subtitles are displayed first; in an audio + video scenario, audio and video length alignment is prioritized.

[0204] Device characteristics: If the user end is a mobile device, you can choose to preload in segments; if it is a PC or high-performance AR / VR end, more complex three-dimensional overlay or multi-window rendering can be allowed.

[0205] For splicing processing:

[0206] If the candidate media unit contains timestamp information, the system can splice the target media unit with the candidate media unit in the same or similar time period. For example, in a live broadcast playback scenario, the video and subtitles at the same time can be merged.

[0207] For multiple images or videos retrieved simultaneously, overlay or picture-in-picture processing is performed according to the frame index to generate a new synthetic media stream.

[0208] If the candidate media unit is text, the text content can be displayed in real time below the video playback screen or in the sidebar in the form of scrolling subtitles, bubble prompts or barrages.

[0209] For audio + video fusion, an audio mixing engine (such as FFmpeg or self-developed mixing module) can be used to adjust the volume and sampling rate to keep it consistent with the target video clip.

[0210] If there are externally set special effects (such as AR filters, specific watermarks, etc.), they can be added according to preset rules during the synthesis stage.

[0211] This embodiment can use a variety of media processing tools (such as FFmpeg, GStreamer, etc.) or self-developed multimedia mixing engines to complete the splicing operation in a streaming or file manner and generate a final synthesized output stream or synthesized file.

[0212] Furthermore, the spliced ​​multimedia data is re-encoded or formatted. For example, the synthesized video stream is encoded and decoded in H.264 and packaged into MP4 or MPEG-TS format. The text information of the superimposed subtitles is rendered to generate a resource link that is convenient for web pages or apps to call.

[0213] If split-screen rendering is used, the video stream and text content can also be used as independent windows, and the front end can be laid out and played through HTML5 / JS or mobile SDK.

[0214] In a specific implementation, the synthesized multimedia file or stream address is returned to the requesting end (such as a client browser or app). The client plays the multimedia file adaptively based on the network conditions (such as bandwidth and latency) and device performance, or plays the multimedia file automatically based on system presets.

[0215] In live broadcast scenarios, fusion requests can be executed in the real-time processing pipeline, and the synthesized live stream can be pushed to the CDN or media server using protocols such as RTMP / HLS / WebRTC. The client can then watch the live stream synchronously with the host / viewer through the playback address.

[0216] In the on-demand scenario, the spliced ​​files can be stored in a media server or object storage, and the access URL is returned for the user to click to play in the front-end browser or APP interface.

[0217] Furthermore, this implementation can record user click-through rates, dwell time, and satisfaction ratings for the fused content on the front-end, transmitting this interaction information back to the server for dynamic optimization of subsequent fusion strategies. If it subsequently detects a user's preference for a particular fusion method, the system can prioritize that fusion strategy appropriately in the next splicing attempt.

[0218] In this way, the cross-media display process of the present invention for target media units not only efficiently retrieves candidate media units, but also implements multiple splicing methods through a predefined fusion strategy, ultimately presenting the processed results to the user terminal in a visual and diverse manner, thereby meeting the cross-media fusion needs in scenarios such as online education, live e-commerce, video conferencing, and entertainment content aggregation. Its overall process, from request to output, significantly improves the complementarity and visualization of content without sacrificing multimedia quality, and meets users' expectations for simultaneous browsing of multimodal information with a relatively fast response.

[0219] As an optional implementation, the cross-media presentation further includes:

[0220] Slicing the live streaming data according to time periods, and generating the basic feature vector and pseudo query vector for each time period;

[0221] The fused media representation vectors of the live streaming data are incrementally written into the vector database, and the latest media representation vectors are used for matching during online retrieval.

[0222] This application is adapted to large-scale, real-time data streams in live broadcast scenarios, and further performs time segmentation, incremental vector writing and online retrieval matching on live broadcast stream data.

[0223] In the specific implementation, after receiving the live stream data, the live content is continuously captured through the pre-deployed real-time acquisition module (such as FFmpeg push stream reception or WebRTC reception), and the live stream is divided into multiple discrete "live slices" according to the set time period (for example, every 5 seconds, every 10 seconds). After the slicing is completed, the system immediately performs basic feature vector (F base ) extraction.

[0224] For example, if the live stream is video content, the system performs key frame extraction or sparse sampling on the 5-second or 10-second video clip, and uses a video feature extraction model (such as 3D CNN or Video Transformer) to perform convolution / self-attention analysis on the key frames or sampled frames to generate a basic feature vector corresponding to the video slice.

[0225] If the live stream contains audio, the audio signal of this period can be further sampled at a unified rate and converted to Mel spectrum, and an audio coding network (such as CNN+RNN or Audio Transformer) can be used to extract audio features and merge them into the feature vector representation of the same live slice.

[0226] After obtaining the basic feature vector of the live slice, the pseudo query module is called to generate a pseudo query vector (Q pseudo Since live streaming data is often highly time-sensitive and lacks fixed file divisions, this implementation treats the time-sliced ​​video or audio segment as a "media unit," allowing subsequent pseudo-query vector generation and cross-media implicit interaction modules to utilize the same logic and model structure previously used for offline media.

[0227] Subsequently, the basic feature vector and the pseudo query vector are input into the implicit interaction module for fusion, and the output fused media representation vector (F fused ) represents the position of the live segment in the cross-media semantic space. Because live streaming is characterized by the continuous generation of new segments in real time, this implementation uses an incremental write mechanism to continuously insert the newly generated fused media representation vectors into the vector database.

[0228] In a specific implementation, a long connection with a vector database (such as Milvus, Faiss or HNSW) can be maintained in advance; after each slice is processed, the fused media representation vector (F fused ), the corresponding metadata (live broadcast room ID, start and end seconds of the time period, frame information) is written into the database to generate a new data entry.

[0229] If the database supports online index updates, the index construction module can be triggered after writing to perform ANN index update operations on the incremental vector samples; if the database uses micro-batch indexing, the index can be updated in batches within a short time interval to balance real-time performance and throughput.

[0230] When external requests require cross-media display or related retrieval of live streams, the vector database can be directly queried for the most recently inserted live slice representation vector. For example, in an educational live stream scenario, the fused media representation vector corresponding to a specific time period of the teacher's current live stream can be used to retrieve the most similar courseware images or presentation pages.

[0231] For example, the most recently inserted live segment vector (falling within the time period [T0-5s, T0]) can be searched based on the current time (such as T0), or conditional filtering can be performed based on the timestamp field to retrieve only the latest generated part to ensure that the result is synchronized with the live broadcast progress.

[0232] If other media resources with a similarity higher than a threshold are retrieved, the live broadcast slice can be spliced ​​or superimposed with the matched video, audio or text resources according to the fusion strategy, ultimately achieving a multimodal interactive experience of live broadcast and supplementary content on the front end.

[0233] Thus, through the aforementioned segmented slicing, incremental writing, and online retrieval process of live streaming data, this implementation enables real-time synchronization of cross-media convergence in live streaming scenarios. Each time a new slice is collected, segmented, and features extracted to generate a fused media representation vector, it is immediately written to the vector database and indexed and updated, allowing subsequent searches to retrieve the latest live broadcast status or semantic information. This significantly improves user experience in terms of latency and ensures that the full media convergence solution adapts to the ever-changing content needs of live streaming scenarios.

[0234] Based on the same inventive concept, the embodiments of the present disclosure also provide an all-media fusion system based on complementary fusion corresponding to the all-media fusion method based on complementary fusion. Since the principle of problem solving by the system in the embodiments of the present disclosure is similar to the above-mentioned all-media fusion method based on complementary fusion in the embodiments of the present disclosure, the implementation of the system can refer to the implementation of the method, and the repeated parts will not be repeated.

[0235] Reference Figure 3 , which is a schematic diagram of a full-media fusion system based on complementary fusion provided by an embodiment of the present disclosure, the system includes: an acquisition module 10, a first processing module 20, a feature extraction module 30, a second processing module 40, a third processing module 50, and a display module 60;

[0236] The acquisition module 10 is used to collect different types of media data from multiple media sources and perform preliminary formatting on the media data; wherein the media data includes: text, images, audio, video and live streaming data;

[0237] The first processing module 20 is configured to divide the formatted media data into a plurality of processable media units according to a preset segmentation strategy. A media unit refers to a relatively independent media segment within a temporal or spatial range, and each media unit includes at least one of the following: a text segment, a frame or a video segment, an image or a group of images, and an audio segment.

[0238] The feature extraction module 30 is used to extract features from different types of media data and generate basic feature vectors corresponding to each media unit;

[0239] The second processing module 40 is configured to generate a pseudo query vector associated with each media unit based on the basic feature vector using a pseudo query module;

[0240] The third processing module 50 is configured to input the basic feature vector and the pseudo query vector into an implicit interaction module and output a fused media representation vector; store the media representation vector and its corresponding media metadata in a vector database, and create an index for the media representation vector in the vector database;

[0241] The display module 60 is used to, in response to receiving a fusion request for a target media unit, obtain media resources matching the target media unit by searching the vector database, and synchronously synthesize the target media unit with the matching media resources based on the timestamp information of the matching media resources to perform cross-media display of the target media unit.

[0242] Those skilled in the art will understand that in the above-described methods of specific embodiments, the order in which the steps are presented does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible inherent logic. It should be understood that determining B based on A does not mean determining B based solely on A; B can also be determined based on A and / or other information.

[0243] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the present invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0244] Throughout this specification, references to terms such as "one embodiment," "example," or "specific example" indicate that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of the present invention. In this specification, schematic representations of these terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

Claims

1. The full media fusion method based on complementary fusion is characterized by: include: Collect different types of media data from multiple media sources and perform preliminary formatting on the media data; wherein the media data includes: text, images, audio, video and live streaming data; The formatted media data is divided into media units according to a preset segmentation strategy; wherein a media unit refers to an independent media segment within a time or space range, and each media unit includes at least one of the following: a piece of text, a frame or a video, an image or a group of images, and an audio segment; Perform feature extraction on different types of media data to generate basic feature vectors corresponding to each media unit; generating, using a pseudo query module, a pseudo query vector associated with each media unit based on the basic feature vector; Inputting the basic feature vector and the pseudo query vector into an implicit interaction module, and outputting a fused media representation vector; storing the media representation vector and its corresponding media metadata in a vector database, and creating an index for the media representation vector in the vector database; In response to receiving a fusion request for a target media unit, the media resources matching the target media unit are obtained by retrieving the vector database, and the target media unit and the matching media resources are synchronously synthesized according to the timestamp information of the matching media resources to perform cross-media display of the target media unit.

2. The full media fusion method based on complementary fusion according to claim 1, characterized in that: Generating a pseudo query vector associated with each media unit includes: Performing an attention mechanism operation on the basic feature vector to generate a potential demand representation; A pseudo query vector is generated based on the potential demand representation, and the pseudo query vector is optimized using a reconstruction loss or a contrastive loss.

3. The full media fusion method based on complementary fusion according to claim 2, characterized in that: The output fused media representation vector includes: Concatenate the basic feature vector and the pseudo query vector to form an input sequence; Processing the input sequence through a multi-head self-attention network to generate a fused media representation vector; The fused media representation vector is output for indexing and retrieval.

4. The full media fusion method based on complementary fusion according to claim 3 is characterized in that: Storing the media representation vector and its corresponding media metadata in a vector database and creating an index for the media representation vector in the vector database includes: Establishing a mapping relationship between the media representation vector and the media metadata, wherein the media metadata includes a media type, a timestamp, and a frame index; Storing the media representation vector and its media metadata in a vector database; The media representation vectors are indexed based on an approximate nearest neighbor search algorithm or a hash index algorithm.

5. The full media fusion method based on complementary fusion according to claim 4 is characterized in that: The cross-media presentation includes: receiving a fusion request for the target media unit; Retrieving a candidate media unit from the vector database whose media representation vector similarity with the target media unit is higher than a preset threshold; splicing and fusing the target media unit and the candidate media unit according to a set fusion strategy; Output the fusion results to the front end for display.

6. The full media fusion method based on complementary fusion according to claim 5, characterized in that: The cross-media presentation further includes: Slicing the live streaming data according to time periods, and generating the basic feature vector and pseudo query vector for each time period; The fused media representation vectors of the live streaming data are incrementally written into the vector database, and the latest media representation vectors are used for matching during online retrieval.

7. The full media fusion method based on complementary fusion according to claim 6, characterized in that: The feature extraction includes: For text, image, audio and video, convolutional neural network, visual transformer, pre-trained language model or speech recognition model are used to extract the basic feature vector from different types of media data.

8. The full media fusion method based on complementary fusion according to claim 7, characterized in that: The cross-media presentation further includes: embedding the matching media resource into a rendering window corresponding to the target media unit.

9. The full media fusion system based on complementary fusion is characterized by: include: An acquisition module, a first processing module, a feature extraction module, a second processing module, a third processing module, and a display module; The acquisition module is used to collect different types of media data from multiple media sources and perform preliminary formatting on the media data; wherein the media data includes: text, images, audio, video and live streaming data; The first processing module is configured to divide the formatted media data into media units according to a preset segmentation strategy; wherein a media unit refers to an independent media segment within a time or space range, and each media unit includes at least one of the following: a piece of text, a frame or a video, an image or a group of images, and an audio segment; The feature extraction module is used to extract features from different types of media data and generate basic feature vectors corresponding to each media unit; The second processing module is configured to generate a pseudo query vector associated with each media unit based on the basic feature vector using a pseudo query module; The third processing module is configured to input the basic feature vector and the pseudo query vector into an implicit interaction module and output a fused media representation vector; store the media representation vector and its corresponding media metadata in a vector database, and create an index for the media representation vector in the vector database; The display module is used to, in response to receiving a fusion request for a target media unit, obtain media resources matching the target media unit by searching the vector database, and synchronously synthesize the target media unit with the matching media resources based on the timestamp information of the matching media resources to perform cross-media display of the target media unit.

Images