Sparse data video retrieval method and system based on cross-modal feature hallucination completion

Abstract

This invention discloses a sparse data video retrieval method and system based on cross-modal feature illusion completion, belonging to the field of multimedia information processing technology. The method involves performing modal integrity detection on input video data, extracting source feature vectors of available modalities, and identifying missing modal types. A cross-modal mapping network is used to convert the source feature vectors into illusion feature vectors of missing modalities. The uncertainty confidence weights of these illusion feature vectors are calculated through multiple random forward propagations. A reverse mapping network is used to restore the illusion feature vectors to the reconstructed feature vectors of the corresponding source modalities, and the semantic distance between these reconstructed feature vectors and the source feature vectors is calculated. The uncertainty confidence weights are then corrected based on the consistency verification result of the semantic distance and a preset threshold. Based on the corrected uncertainty confidence weights, the source feature vectors and illusion feature vectors are dynamically weighted and fused to generate joint video features. Video retrieval is then performed based on the similarity between the joint video features and the query vector.

Description

Technical Field

[0001] This invention relates to the field of multimedia information processing technology, and in particular to a sparse data video retrieval method and system based on cross-modal feature illusion completion. Background Technology

[0002] With the rapid development of multimodal learning, content-based multimodal video retrieval technology has made significant progress. However, existing solutions typically assume that the input video data possesses complete and high-quality visual, audio, and text modalities. In practical applications, due to limitations in acquisition equipment, packet loss during transmission, or incomplete historical data formats, video data often exhibits modal sparsity or even missing modalities, such as silent surveillance footage or historical images without metadata. Existing technologies have the following significant drawbacks when dealing with such sparse data:

[0003] First, traditional multimodal fusion models are trained on complete modal data. When encountering modal gaps during testing and employing simple strategies like zero-padding, the input data distribution deviates significantly from the training distribution. This causes the fused feature vectors to fall into incorrect regions of the semantic space, resulting in a significant decrease in retrieval accuracy and precision. Second, some solutions attempt to complete information by generating original media data (such as synthesized audio waveforms or missing video frames). While this can restore modal integrity, its computational complexity is extremely high, making it difficult to meet the real-time requirements of large-scale video retrieval. Third, existing completion mechanisms generally lack the ability to evaluate the credibility of generated features. Cross-modal generation is inherently an ill-posed problem; the same visual content may correspond to multiple reasonable audio semantics. Existing models cannot effectively distinguish between reasonable hallucination features and logically flawed generation results, easily introducing erroneous semantics into the retrieval system and causing noise pollution. Finally, due to the above shortcomings, a large amount of legacy single-modal video data is considered low-value data in multimodal retrieval systems and cannot respond to queries containing missing modal semantics, resulting in a waste of data resources.

[0004] To address the aforementioned technical problems, this invention aims to provide a sparse data video retrieval method and system based on cross-modal feature illusion completion. Its objectives are: first, to avoid distribution shifts and ensure the stability of retrieval performance in scenarios with missing modalities by directly generating feature vectors of the missing modalities within the feature embedding space; second, to introduce an uncertainty confidence assessment and bidirectional consistency verification mechanism to verify the logical rationality of the illusion features in real time, preventing retrieval noise caused by erroneous completion; and third, to activate the retrieval value of existing sparse data, enabling low-quality videos such as those without sound or subtitles to be accurately retrieved through multimodal queries. Summary of the Invention

[0005] The present invention aims to at least partially solve one of the technical problems in the related art.

[0006] To address this, this invention proposes a sparse data video retrieval method based on cross-modal feature illusion completion. The method involves performing modal integrity detection on the input video data, extracting source feature vectors of available modalities, and identifying missing modal types. A cross-modal mapping network is used to convert the source feature vectors into illusion feature vectors of missing modalities, and the uncertainty confidence weights are calculated through multiple random forward propagations. A reverse mapping network is then used to restore the illusion feature vectors to reconstructed feature vectors, calculating their semantic distance from the source feature vectors, and correcting the uncertainty confidence weights based on consistency verification results. The source and illusion feature vectors are dynamically weighted and fused according to the corrected weights to generate joint video features. Video retrieval is then performed based on the similarity between the joint video features and the query vector, achieving accurate multimodal semantic retrieval in sparse data scenarios.

[0007] Another objective of this invention is to propose a sparse data video retrieval system based on cross-modal feature illusion completion.

[0008] To achieve the above objectives, this invention proposes a sparse data video retrieval method based on cross-modal feature illusion completion, comprising: Modal integrity detection is performed on the input video data, the source feature vectors of available modalities are extracted, and the types of missing modalities are identified; The source feature vector is converted into a hallucination feature vector with missing modality using a cross-modal mapping network, and the uncertainty confidence weight of the hallucination feature vector is calculated through multiple random forward propagations. The hallucination feature vector is restored to the corresponding source modality reconstructed feature vector through a reverse mapping network. The semantic distance between the reconstructed feature vector and the source feature vector is calculated. The uncertainty confidence weight is corrected based on the consistency verification result of the semantic distance and the preset threshold. The source feature vector and the hallucination feature vector are dynamically weighted and fused according to the corrected uncertainty confidence weight to generate joint video features, and video retrieval is performed based on the similarity between the joint video features and the query vector.

[0009] In one embodiment of the present invention, the step of performing modal integrity detection on the input video data, extracting the source feature vectors of available modalities, and identifying the type of missing modalities includes: The system receives input video data, performs visual detection, audio detection, and text detection, and constructs a modal mask vector based on the detection results. In the modal mask vector, a value of 1 indicates that the corresponding modality is available and of acceptable quality, while a value of 0 indicates that the corresponding modality is missing or unusable. For the available modalities marked as 1 in the modal mask vector, the pre-trained visual encoder, audio encoder, and text encoder are called respectively to extract the source feature vector with a uniform dimension. The modality type marked as 0 in the modality mask vector is identified as the type of the missing modality.

[0010] In one embodiment of the present invention, calculating the uncertainty confidence weight includes: Obtain the set of N illusion feature vectors generated after performing N forward propagations on the same source feature vector. As input data; Calculate the mean vector of the N hallucination feature vectors. The squared difference between each hallucination feature vector and the mean vector is calculated, summed, and divided by N to obtain the element-wise variance. The average of all element-wise variances is then calculated to obtain the overall uncertainty variance. ; The overall uncertainty variance Multiply by the preset adjustment hyperparameter Add 1 to form the denominator. Then, the constant 1 is divided by the denominator term to obtain the uncertainty confidence weight. The calculation formula is:

[0011] in, Output As dynamic weighting coefficients for the hallucination feature vectors in subsequent fusion steps.

[0012] In one embodiment of the present invention, the step of restoring the hallucination feature vector to the reconstructed feature vector of the corresponding source modality through a back-mapping network, calculating the semantic distance between the reconstructed feature vector and the source feature vector, and correcting the uncertainty confidence weight based on the consistency verification result of the semantic distance and a preset threshold includes: The pre-trained inverse mapping network is invoked, and the generated hallucination feature vector is input into the network to perform inverse mapping operation, thereby restoring the reconstructed feature vector of the corresponding source modality; The semantic distance between the reconstructed feature vector and the original true source feature vector is calculated using cosine distance as a metric. The semantic distance is compared with a preset consistency threshold: if the semantic distance is less than or equal to the consistency threshold, the currently calculated uncertainty confidence weight is retained; if the semantic distance is greater than the consistency threshold, the uncertainty confidence weight is forcibly corrected to zero.

[0013] In one embodiment of the present invention, calculating the semantic distance includes: Obtain the vector representations of the reconstructed feature vector and the source feature vector, and calculate their Euclidean norms in the multidimensional feature space as their respective moduli. The sum of the products of the corresponding dimensions of the two vectors is used as the dot product result. The dot product result is divided by the product of the two magnitudes to obtain the normalized cosine similarity value. Subtracting the normalized cosine similarity value from the constant 1 yields a semantic distance value ranging from 0 to 2. A smaller semantic distance value indicates greater semantic consistency, while a larger semantic distance value indicates a higher risk of logical errors.

[0014] In one embodiment of the present invention, the step of dynamically weighting and fusing the source feature vector and the hallucination feature vector according to the corrected uncertainty confidence weight to generate joint video features, and performing video retrieval based on the similarity between the joint video features and the query vector, includes: Obtain the preset visual modal weights, audio modal weights, and text modal weights as the basic fusion coefficients; The corrected uncertainty confidence weights are multiplied by the hallucination feature vector to obtain the weighted hallucination feature vector; The source feature vector is multiplied by its corresponding basic fusion coefficient, the weighted illusion feature vector is multiplied by the adjusted audio modality weight, the text source feature vector is multiplied by the text modality weight, and the product of the three is added together to generate the completed joint video feature. The user query text is encoded to obtain a query vector. The cosine similarity between the joint video features and the query vector is calculated as the retrieval score. The videos in the video library are sorted in descending order according to the retrieval score, and the top K retrieval results are returned.

[0015] In one embodiment of the present invention, generating joint video features includes: When a modality is detected to be missing, the missing modality type is automatically identified and the corresponding baseline weight strategy is adjusted: if an audio modality is missing, the baseline value of the audio modality weight is increased. When the corrected uncertainty confidence weight is not zero, the hallucination feature vector is multiplied by the weight and the adjusted audio modality weight in turn to achieve dynamic weighting under uncertainty perception. When the corrected uncertainty confidence weights are forcibly adjusted to zero, the fusion term corresponding to the illusion feature vector is automatically removed, causing the joint video features to degenerate into single-modal or multi-modal fusion features dominated only by the available source feature vectors.

[0016] In one embodiment of the present invention, a pre-completion operation performed during the index building phase is further included, the pre-completion operation comprising: Iterate through each video data in the video library and perform modal integrity detection to identify sparse data with missing modalities; For the identified sparse data, a cross-modal mapping network is invoked to pre-generate hallucination feature vectors for the missing modalities, and uncertainty confidence assessment and consistency verification are performed sequentially. The verified hallucination feature vectors are fused with the original source feature vectors to generate pre-complete joint video features, which are then stored in the video index library to support multimodal query requests that include missing modal semantics.

[0017] This invention also proposes a sparse data video retrieval system based on cross-modal feature illusion completion, comprising: The modality awareness module is used to perform modality integrity detection on input video data, extract the source feature vectors of available modalities, and identify the types of missing modalities; The hallucination confidence module is used to convert the source feature vector into a hallucination feature vector with missing modalities using a cross-modal mapping network, and to calculate the uncertainty confidence weight of the hallucination feature vector through multiple random forward propagations. The consistency verification module is used to restore the hallucination feature vector to the reconstructed feature vector of the corresponding source modality through the back mapping network, calculate the semantic distance between the reconstructed feature vector and the source feature vector, and correct the uncertainty confidence weight based on the consistency verification result of the semantic distance and the preset threshold. The fusion retrieval module is used to dynamically weight and fuse the source feature vector and the hallucination feature vector according to the corrected uncertainty confidence weight to generate joint video features, and to perform video retrieval based on the similarity between the joint video features and the query vector.

[0018] This invention presents a sparse data video retrieval method and system based on cross-modal feature illusion completion. By performing modal integrity detection and missing type identification on the input video data, it utilizes a cross-modal mapping network to generate illusion feature vectors for missing modalities and combines multiple random forward propagation calculations to determine uncertainty confidence weights. Furthermore, a reverse mapping network is used for bidirectional consistency verification to correct the weights. Finally, based on the corrected weights, the source features and illusion features are dynamically weighted and fused to generate joint video features, and similarity retrieval is performed. This effectively solves the problems of feature distribution shift, unreliable completion information, and retrieval performance degradation caused by modal missingness in existing technologies. It achieves integrated processing from modal perception and reliable completion to fusion retrieval, significantly improving the accuracy and robustness of multimodal video retrieval in sparse data scenarios, and enhancing the system's semantic retrieval capabilities and engineering applicability for low-quality videos such as silent or unsubtitled videos.

[0019] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0020] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 This is a flowchart of a sparse data video retrieval method based on cross-modal feature illusion completion according to an embodiment of the present invention; Figure 2 This is a system overall architecture diagram according to an embodiment of the present invention; Figure 3 This is a modal state sensing and encoding diagram of the method according to an embodiment of the present invention; Figure 4 This is a cross-modal feature illusion generation map generated by the method according to an embodiment of the present invention; Figure 5 This is an uncertainty confidence assessment graph of the method according to an embodiment of the present invention; Figure 6 This is a consistency check loop diagram of the method according to an embodiment of the present invention; Figure 7 This is a dynamic weighted fusion and retrieval graph based on the method of the present invention. Figure 8 This is a schematic diagram of the structure of a sparse data video retrieval system based on cross-modal feature illusion completion according to an embodiment of the present invention. Detailed Implementation

[0021] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0022] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0023] The following description, with reference to the accompanying drawings, describes a sparse data video retrieval method and system based on cross-modal feature illusion completion according to an embodiment of the present invention.

[0024] Figure 1 This is a flowchart of a sparse data video retrieval method based on cross-modal feature illusion completion, according to an embodiment of the present invention.

[0025] like Figure 1As shown, a sparse data video retrieval method based on cross-modal feature illusion completion includes the following steps: S1, perform modal integrity detection on the input video data, extract the source feature vectors of available modalities, and identify the types of missing modalities; S2, the source feature vector is converted into a hallucination feature vector with missing modality using a cross-modal mapping network, and the uncertainty confidence weight of the hallucination feature vector is calculated through multiple random forward propagations; S3, the hallucination feature vector is restored to the reconstructed feature vector of the corresponding source modality through the inverse mapping network, the semantic distance between the reconstructed feature vector and the source feature vector is calculated, and the uncertainty confidence weight is corrected based on the consistency verification result of the semantic distance and the preset threshold. S4. The source feature vector and the hallucination feature vector are dynamically weighted and fused according to the corrected uncertainty confidence weight to generate joint video features, and video retrieval is performed based on the similarity between the joint video features and the query vector.

[0026] Specifically, this invention provides a video retrieval method based on cross-modal feature illusion completion, which can be deployed on cloud servers or edge computing nodes, such as... Figure 2 As shown, the overall system architecture includes a data preprocessing module, a cross-modal feature illusion generation module, an uncertainty assessment module, a consistency verification module, and a dynamic fusion retrieval module.

[0027] Overall, this structure diagram illustrates the data flow from video input, modality detection, feature generation, verification to fusion retrieval. For detailed process information, please refer to the method steps.

[0028] The initial input of this invention is video data, which is assumed to contain three modalities: visual, audio, and text. Upon receiving the data, the invention senses the modal status. If all three modalities exist and are available, it directly extracts the corresponding features from each modal and fuses them using a weighted fusion method. These features are then used to calculate similarity with the retrieval vector, thereby enabling video library retrieval. If any of the three modalities are missing or exist but are unavailable, feature illusion completion is performed. The general process is as follows (taking the absence of an audio file as an example): Feature extraction is performed on existing visual and textual data. The extracted visual and textual features are then fed into the visual-to-audio conversion module and the text-to-audio conversion module, respectively, to generate audio features. Monte Carlo Dropout is introduced during the audio feature generation process to obtain confidence weights. The two generated audio features are then fed into the audio-to-video inverse conversion module and the audio-to-text inverse conversion module, respectively, to generate video features and audio features. The semantic distance between the original features and the inversely converted features (i.e., the semantic distance between the video features of the original audio file and the inversely converted video features; the semantic distance between the text features of the original audio file and the inversely converted text features) is calculated. If the semantic distance obtained in the previous step is greater than a predetermined threshold, the generated audio features and the original visual and text features are fused together, and the similarity is calculated with the query vector to achieve video library retrieval. If the semantic distances of both generated audio features are greater than the threshold, the larger one is taken as the final result for feature fusion. If the semantic distances of both generated audio features are less than the threshold, the confidence weights are reset to zero, and the corresponding audio features are automatically removed in the feature fusion stage.

[0029] Specifically, this invention mainly includes the following five core method steps (taking visual audio completion as an example): like Figure 3 As shown, this step mainly involves modal state perception and encoding, specifically including: the system receiving input video data. First, the modal integrity detection algorithm is executed: visual detection: checking if video frames exist, if the resolution is below a threshold, and if the keyframe occlusion rate is too high; audio detection: checking if audio tracks exist, if the signal-to-noise ratio is below a threshold, and if it is a silent track; text detection: checking if metadata and subtitle files exist; then, the modal mask vector is defined. ,in A value of 1 indicates that the modality is available and of acceptable quality, while 0 indicates that the modality is missing or unusable. For available modalities, source feature vectors are extracted using a pre-trained encoder: Visual features: It adopts VideoMAE or ViT architecture, with an output dimension of D; audio characteristics: Using AST or VGGish architecture, the output dimension is D; text features: It adopts the BERT architecture and has an output dimension of D.

[0030] like Figure 4 As shown, this step mainly involves cross-modal feature illusion generation. When a certain modality is missing (such as...), When [the target modality is reached], the pre-trained Cross-Modal Mapper network is invoked. This mapping network is a Transformer-based non-autoregressive generative model designed to learn the distribution mapping from source modality features to target modality features. .

[0031] Taking visual audio completion as an example (V2A Mapper): Input is visual features The output is the hallucination audio features. .

[0032] To increase the diversity of generated features and prevent pattern collapse, a noise vector is added to the input layer. , where I represents any integer and N represents a normal distribution.

[0033] The formula for generating the formula is as follows:

[0034] in, For mapping network functions, These are the network parameters. The network consists of a multi-head self-attention mechanism and a feedforward neural network (FFN), which can capture high-order correlations between visual events (such as object collisions and flames) and audio semantics (such as impact sounds and burning sounds).

[0035] like Figure 5 As shown, this step mainly involves uncertainty confidence assessment. Due to the uncertainty inherent in cross-modal generation, the system needs to evaluate the generated hallucinatory features. The feasibility of the experiment is assessed. This invention employs Monte Carlo Dropout (MC Dropout) technology for uncertainty estimation. The specific operation is as follows: During the inference phase, keep the Dropout layers in the mapping network enabled (typically, the Dropout rate during training is 0.1-0.3, which is also an empirical value derived from experiments); for the same input Perform N forward propagations (e.g., N = 10) to obtain N generated hallucination feature vectors. Calculate the mean of these N vectors. and variance :

[0036]

[0037] This variance is calculated as the mean of the element-wise variances, representing the overall uncertainty.

[0038] Define confidence weights :

[0039] in To adjust the hyperparameters. Variance The larger the weight, the more uncertain the model is about the generated result; the higher the weight. The lower the value, the less impact this feature will have in subsequent fusion processes.

[0040] like Figure 6 As shown, this step mainly involves a consistency check loop. To further prevent the illusion of logical errors (e.g., a detailed visual but generated explosion sound), this invention introduces a bidirectional consistency check mechanism (CycleConsistency Check), the specific process of which is as follows: Inverse mapping: Using a pre-trained audio-visual inverse mapper (A2V Mapper), the generated hallucinatory audio features are... Attempt to reduce to visual features :

[0041] Error calculation: Calculate the restored features With original real visual features The semantic distance between them. Cosine distance is used as the metric:

[0042] Threshold determination: Set a consistency threshold (e.g., 0.3), which is an empirical value obtained through data statistics based on actual conditions. If D ≤ Threshold, it means that the generated audio features are semantically consistent with the current video content, and the confidence weight is retained. If D > Threshold, it indicates a conflict between the generated audio features and the visual logic (e.g., the visual is "quiet reading," but the generated audio features are "noisy market"). In this case, it is forced to... Setting it to 0 means discarding the illusion feature and degenerating into a single-modal search.

[0043] like Figure 7 As shown, the true feature vector is fused with the hallucination feature vector after confidence weighting and verification to form the completed joint video features. The fusion formula is as follows:

[0044] in, Preset modal weights. In scenarios where modalities are missing, the system will automatically adjust the weighting strategy: if audio is missing, The baseline value was raised, but the final contribution depends on .like ,but The item is invalid.

[0045] Finally, calculation With query vector Similarity Score (obtained from the user's query text encoding):

[0046] The system sorts the video library according to the score and returns the Top-K search results.

[0047] Furthermore, in personal or enterprise cloud storage services, the video data uploaded by users daily comes from diverse sources and varies in quality. For example, audio-free surveillance footage uploaded from home cameras or historical data without metadata exported from older devices constitute typical "sparse data." Traditional cloud storage retrieval only supports metadata matching such as filenames, tags, or upload times, preventing users from accurately locating target content through semantic queries.

[0048] When a user enters the natural language query "video of someone clapping in a meeting room" into the cloud drive search box, the system first parses the query intent and generates a text feature vector T_{query}. Then, it traverses the cloud drive video library, performing modal state awareness for each video: if a video is detected to have no audio track (such as a silently recorded meeting video), the cross-modal feature illusion generation module is triggered. Based on visual features V_{feat} (such as clapping actions or meeting scene layout), it generates illusion audio features {\hat{A}}_{feat} (corresponding to the semantic embedding of "applause") in the embedding space. After uncertainty assessment and consistency verification, if the confidence level is met, the completed features are dynamically fused with the real visual features, and the similarity with T_{query} is calculated. Ultimately, even if the target video is completely missing audio, the system can still accurately recall and rank the meeting video at the top of the list through cross-modal reasoning of "visual → illusion audio".

[0049] In this example, the present invention eliminates the need for time-consuming audio synthesis or re-annotation of the original video, completing the task only in the feature space, thus ensuring the response time requirements for massive video retrieval on cloud drives. Simultaneously, the confidence verification mechanism effectively avoids retrieval noise caused by misinterpretations of images (such as mistaking a "book-turning action" for "applause"), significantly improving user experience and retrieval accuracy. This solution can be seamlessly integrated into existing cloud drive architectures, activating the semantic retrieval value of existing sparse videos.

[0050] The method of this invention effectively solves the problems of feature distribution shift, unreliable completion information, and retrieval performance degradation caused by modality loss in the prior art. It realizes integrated processing from modality perception, reliable completion to fusion retrieval, significantly improves the accuracy and robustness of multimodal video retrieval in sparse data scenarios, and enhances the system's semantic retrieval capability and engineering applicability for low-quality videos such as silent and unsubtitled videos.

[0051] To implement the method of the above embodiments, such as Figure 8 As shown, this invention proposes a sparse data video retrieval system 10 based on cross-modal feature illusion completion, comprising: The modality perception module 100 is used to perform modality integrity detection on the input video data, extract the source feature vectors of available modalities, and identify the types of missing modalities.

[0052] The hallucination confidence module 200 is used to convert the source feature vector into a hallucination feature vector with missing modalities using a cross-modal mapping network, and to calculate the uncertainty confidence weight of the hallucination feature vector through multiple random forward propagations.

[0053] The consistency verification module 300 is used to restore the hallucination feature vector to the reconstructed feature vector of the corresponding source modality through the inverse mapping network, calculate the semantic distance between the reconstructed feature vector and the source feature vector, and correct the uncertainty confidence weight based on the consistency verification result of the semantic distance and the preset threshold.

[0054] The fusion retrieval module 400 is used to dynamically weight and fuse the source feature vector and the hallucination feature vector according to the corrected uncertainty confidence weight to generate joint video features, and to perform video retrieval based on the similarity between the joint video features and the query vector.

[0055] The system of this invention effectively solves the problems of feature distribution shift and unreliable completion information caused by modality loss in the prior art. It realizes an integrated system architecture from modality perception, reliable completion to fusion retrieval, significantly improves the accuracy and robustness of multimodal video retrieval in sparse data scenarios, and enhances the system's semantic retrieval capability and engineering applicability for low-quality video data.

[0056] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0057] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

Claims

1. A sparse data video retrieval method based on cross-modal feature illusion completion, characterized in that, include: Modal integrity detection is performed on the input video data, the source feature vectors of available modalities are extracted, and the types of missing modalities are identified; The source feature vector is converted into a hallucination feature vector with missing modality using a cross-modal mapping network, and the uncertainty confidence weight of the hallucination feature vector is calculated through multiple random forward propagations. The hallucination feature vector is restored to the corresponding source modality reconstructed feature vector through a reverse mapping network. The semantic distance between the reconstructed feature vector and the source feature vector is calculated. The uncertainty confidence weight is corrected based on the consistency verification result of the semantic distance and the preset threshold. The source feature vector and the hallucination feature vector are dynamically weighted and fused according to the corrected uncertainty confidence weight to generate joint video features, and video retrieval is performed based on the similarity between the joint video features and the query vector.

2. The method according to claim 1, characterized in that, The modal integrity detection of the input video data, extracting the source feature vectors of available modalities, and identifying the types of missing modalities includes: The system receives input video data, performs visual detection, audio detection, and text detection, and constructs a modal mask vector based on the detection results. In the modal mask vector, a value of 1 indicates that the corresponding modality is available and of acceptable quality, while a value of 0 indicates that the corresponding modality is missing or unusable. For the available modalities marked as 1 in the modal mask vector, the pre-trained visual encoder, audio encoder, and text encoder are called respectively to extract the source feature vector with a uniform dimension. The modality type marked as 0 in the modality mask vector is identified as the type of the missing modality.

3. The method according to claim 1, characterized in that, The process of converting the source feature vector into a hallucination feature vector with missing modalities using a cross-modal mapping network, and calculating the uncertainty confidence weight of the hallucination feature vector through multiple random forward propagations, includes: A non-autoregressive generative model based on Transformer is invoked as a cross-modal mapping network to add a noise vector that follows a normal distribution to the source feature vector; The source feature vector with added noise vector is input into the cross-modal mapping network to generate the initial hallucination feature vector; During the inference phase, the Dropout layer in the cross-modal mapping network is kept on, and N forward propagations are performed on the same source feature vector to obtain N generated illusion feature vectors. The mean of the N vectors and the overall uncertainty variance are calculated. Based on the preset adjustment hyperparameters and the overall uncertainty variance, the uncertainty confidence weight is calculated using the reciprocal relationship formula, wherein the overall uncertainty variance is negatively correlated with the uncertainty confidence weight.

4. The method according to claim 3, characterized in that, Calculating the uncertainty confidence weights includes: Obtain the set of N illusion feature vectors generated after performing N forward propagations on the same source feature vector. As input data; Calculate the mean vector of the N hallucination feature vectors. The squared difference between each hallucination feature vector and the mean vector is calculated, summed, and divided by N to obtain the element-wise variance. The average of all element-wise variances is then calculated to obtain the overall uncertainty variance. ; The overall uncertainty variance Multiply by the preset adjustment hyperparameter Add 1 to form the denominator. Then, the constant 1 is divided by the denominator term to obtain the uncertainty confidence weight. The calculation formula is: in, Output As dynamic weighting coefficients for the hallucination feature vectors in subsequent fusion steps.

5. The method according to claim 1, characterized in that, The step of restoring the hallucination feature vector to the corresponding source modality using a back-mapping network, calculating the semantic distance between the reconstructed feature vector and the source feature vector, and correcting the uncertainty confidence weight based on the consistency verification result of the semantic distance and a preset threshold includes: The pre-trained inverse mapping network is invoked, and the generated hallucination feature vector is input into the network to perform inverse mapping operation, thereby restoring the reconstructed feature vector of the corresponding source modality; The semantic distance between the reconstructed feature vector and the original true source feature vector is calculated using cosine distance as a metric. The semantic distance is compared with a preset consistency threshold: if the semantic distance is less than or equal to the consistency threshold, the currently calculated uncertainty confidence weight is retained; if the semantic distance is greater than the consistency threshold, the uncertainty confidence weight is forcibly corrected to zero.

6. The method according to claim 5, characterized in that, Calculating semantic distance includes: Obtain the vector representations of the reconstructed feature vector and the source feature vector, and calculate their Euclidean norms in the multidimensional feature space as their respective moduli. The sum of the products of the corresponding dimensions of the two vectors is used as the dot product result. The dot product result is divided by the product of the two magnitudes to obtain the normalized cosine similarity value. Subtracting the normalized cosine similarity value from the constant 1 yields a semantic distance value ranging from 0 to 2. A smaller semantic distance value indicates greater semantic consistency, while a larger semantic distance value indicates a higher risk of logical errors.

7. The method according to claim 1, characterized in that, The step of dynamically weighting and fusing the source feature vector and the hallucination feature vector according to the corrected uncertainty confidence weight to generate joint video features, and performing video retrieval based on the similarity between the joint video features and the query vector, includes: Obtain the preset visual modal weights, audio modal weights, and text modal weights as the basic fusion coefficients; The corrected uncertainty confidence weights are multiplied by the hallucination feature vector to obtain the weighted hallucination feature vector; The source feature vector is multiplied by its corresponding basic fusion coefficient, the weighted illusion feature vector is multiplied by the adjusted audio modality weight, the text source feature vector is multiplied by the text modality weight, and the product of the three is added together to generate the completed joint video feature. The user query text is encoded to obtain a query vector. The cosine similarity between the joint video features and the query vector is calculated as the retrieval score. The videos in the video library are sorted in descending order according to the retrieval score, and the top K retrieval results are returned.

8. The method according to claim 7, characterized in that, Generate joint video features, including: When a modality is detected to be missing, the missing modality type is automatically identified and the corresponding baseline weight strategy is adjusted: if an audio modality is missing, the baseline value of the audio modality weight is increased. When the corrected uncertainty confidence weight is not zero, the hallucination feature vector is multiplied by the weight and the adjusted audio modality weight in turn to achieve dynamic weighting under uncertainty perception. When the corrected uncertainty confidence weights are forcibly adjusted to zero, the fusion term corresponding to the illusion feature vector is automatically removed, causing the joint video features to degenerate into single-modal or multi-modal fusion features dominated only by the available source feature vectors.

9. The method according to claim 1, characterized in that, It also includes a pre-completion operation performed during the index building phase, the pre-completion operation including: Iterate through each video data in the video library and perform modal integrity detection to identify sparse data with missing modalities; For the identified sparse data, a cross-modal mapping network is invoked to pre-generate hallucination feature vectors for the missing modalities, and uncertainty confidence assessment and consistency verification are performed sequentially. The verified hallucination feature vectors are fused with the original source feature vectors to generate pre-complete joint video features, which are then stored in the video index library to support multimodal query requests that include missing modal semantics.

10. A sparse data video retrieval system based on cross-modal feature illusion completion, characterized in that, include: The modality awareness module is used to perform modality integrity detection on input video data, extract the source feature vectors of available modalities, and identify the types of missing modalities; The hallucination confidence module is used to convert the source feature vector into a hallucination feature vector with missing modalities using a cross-modal mapping network, and to calculate the uncertainty confidence weight of the hallucination feature vector through multiple random forward propagations. The consistency verification module is used to restore the hallucination feature vector to the reconstructed feature vector of the corresponding source modality through the back mapping network, calculate the semantic distance between the reconstructed feature vector and the source feature vector, and correct the uncertainty confidence weight based on the consistency verification result of the semantic distance and the preset threshold. The fusion retrieval module is used to dynamically weight and fuse the source feature vector and the hallucination feature vector according to the corrected uncertainty confidence weight to generate joint video features, and to perform video retrieval based on the similarity between the joint video features and the query vector.

Images