An unsupervised semi-paired cross-modal retrieval method and system based on deep learning

Abstract

This invention discloses an unsupervised semi-paired cross-modal retrieval method and system based on deep learning, belonging to the field of artificial intelligence, to solve problems of labeled data dependency, asymmetric semantic association, and high-dimensional storage efficiency. This invention combines a dual-branch visual encoder and a dynamic cue text encoder, utilizing gated cross-attention to achieve dynamic weighting of visual-text features and suppress modal redundancy interference. A low-frequency semantic-guided generation enhancement strategy is used to improve long-tail keyword coverage; a two-stage quantized hierarchical index is constructed, employing coarse-grained clustering and fine-grained product quantization to compress feature storage, supporting real-time retrieval of millions of data points. A degradation-aware incremental maintenance mechanism monitors data distribution shifts through a KL divergence threshold, triggering index reconstruction to maintain long-term update accuracy. This method overcomes the limitations of traditional strong pairing models, achieving second-level localization of cross-modal sensitive content, effectively solving asymmetric semantic alignment, and improving retrieval efficiency.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and more specifically, to an unsupervised semi-paired cross-modal retrieval method and system based on deep learning. Background Technology

[0002] The rapid development of social media and mobile internet has spawned massive amounts of multimodal data, with users generating billions of pieces of text, images, and videos daily, becoming the core carriers of online information dissemination. In the analysis of breaking news and sensitive events, over 90% of trending topics rely on cross-modal interaction (such as images corroborating textual viewpoints and text interpreting visual metaphors). However, traditional single-modal retrieval technologies (such as keyword search and image feature matching) fail to meet the needs of complex scenarios due to their fragmented semantic connections between text and images. For example, users cannot retrieve in-depth report texts from vague on-site images, highlighting the urgent need for cross-modal semantic alignment technologies.

[0003] Current cross-modal retrieval methods mainly include: linear projection methods, which project different modal data into a shared subspace by learning a linear mapping function, but cannot effectively capture the nonlinear relationships of multimodal data (such as the association between abstract metaphors in text and visual symbols in images); and hash learning methods, which compress data dimensionality through binary encoding to improve retrieval speed, but suffer from low accuracy in complex scenarios due to semantic loss in low-bit representation. Furthermore, existing methods generally rely on strong pairing assumptions (i.e., strictly aligned image-text sample pairs), while in real-world scenarios, multimodal data often exhibits semi-pairing or dynamic incremental characteristics, making traditional static models difficult to adapt. This contradiction is particularly prominent in mobile scenarios where storage efficiency and real-time requirements are higher.

[0004] While cross-modal retrieval methods have achieved some success, they still face the following challenges: First, the reliance on expert annotation in specialized domains leads to a surge in model scaling costs; second, the asymmetric nature of data associations (such as image-text one-to-many / many-to-one mappings) causes semantic overfitting in traditional strong-pairing models; third, single-modal retrieval disrupts the complementarity of images and text, causing cross-modal semantic association to fail; and fourth, the storage requirements for high-dimensional features (such as 512-dimensional) indexes after grouping have increased dramatically, while hash encoding compression leads to a decrease in the recall rate of key features. These problems severely restrict the practical application of cross-modal technologies in high real-time scenarios such as network information analysis. Summary of the Invention

[0005] To address the limitations of existing cross-modal retrieval technologies in handling asymmetric associations, high annotation costs, and low storage efficiency, particularly their reliance on strong pairing data assumptions, semantic overfitting, and insufficient performance of high-dimensional feature indexing, this invention aims to provide a deep learning-based unsupervised semi-paired cross-modal retrieval technology. This technology integrates multi-scale visual encoding, learnable cue text alignment, gated cross-attention mechanisms, and two-stage quantization compression to achieve fine-grained cross-modal semantic association and real-time retrieval optimization, thereby improving the accuracy and efficiency of sensitive information detection.

[0006] To achieve the above technical objectives, this application provides an unsupervised semi-paired cross-modal retrieval method based on deep learning, comprising the following steps:

[0007] Based on multimodal data generated from social media and mobile internet, cross-modal fine-grained alignment is performed, followed by cross-modal feature fusion using a gated cross-attention mechanism;

[0008] Based on the results of cross-modal feature fusion, low-frequency semantic units are identified through word frequency statistics, supplementary text descriptions are generated using visual feature projection, and diverse cross-modal data are generated by combining annealing temperature sampling and KL divergence constraint bundle search strategy, while low-quality results are filtered out.

[0009] Based on the filtered and diverse cross-modal data, combined with dynamic hierarchical indexing algorithm and quantization compression technology, index reconstruction is triggered by monitoring data distribution offset through KL divergence, and cross-modal alignment is optimized by jointly comparing learning loss and generating loss to perform cross-modal retrieval.

[0010] Preferably, when performing cross-modal feature fusion, based on multimodal data, a dual-branch visual encoder and a learnable cue text encoder are constructed. After cross-modal fine-grained alignment, a gated cross-attention mechanism is introduced. Visual and text weights are dynamically allocated through the Sigmoid function to perform cross-modal feature fusion. Specifically, visual and text weights are dynamically allocated through the Sigmoid function, and cross-modal features are generated by element-weighted fusion to suppress interference from redundant modal information.

[0011] Preferably, when constructing the dual-branch visual encoder, the dual-branch visual encoder is constructed by combining CLIP pre-trained global features with an adaptive multi-scale pooling module. This is used to fuse multi-granularity local details through a multi-layer perceptron to achieve collaborative representation of global scene and local attributes.

[0012] Preferably, when constructing a dual-branch visual encoder, the adaptive multi-scale pooling module uses pooling operations with three grid sizes: 1×1, 2×2, and 4×4.

[0013] Preferably, when constructing a learnable cue text encoder, four dynamically adjustable cue vectors are inserted before the CLIP input sequence to adaptively capture task-related context.

[0014] Preferably, when constructing the learnable cue text encoder, four dynamically optimized cue vectors are inserted at the beginning of the CLIP input sequence, initialized as a normal distribution, and the CLS marker feature at the first position is aligned with the visual feature through linear projection.

[0015] Preferably, when combining dynamic hierarchical indexing algorithm and quantization compression technology, a two-stage quantization hierarchical index is adopted. Visual features are divided by coarse-grained clustering, and the features within the cluster are quantized and compressed by fine-grained product quantization. The probe range is dynamically selected and noise is filtered for matching by combining a dual threshold retrieval mechanism.

[0016] Preferably, when executing a two-stage quantization hierarchical index, the two-stage quantization hierarchical index includes:

[0017] Coarse-grained clustering encoding: The K-means++ algorithm is used to divide visual features into 1024 clusters, and an inverted index is constructed to accelerate retrieval;

[0018] Fine-grained product quantization: The 512-dimensional features within the cluster are divided into 8 subspaces, and each subspace is encoded with 8 bits.

[0019] Preferably, when performing cross-modal retrieval, the data distribution shift is monitored by using a KL divergence threshold Δ≥0.15 to trigger full index reconstruction; and the deviation is updated to control the retrieval accuracy decay to ≤2.1%.

[0020] In the coarse screening stage of cross-modal retrieval, Top-K candidates are quickly recalled based on the inverted index;

[0021] In the fine screening stage of cross-modal retrieval, the CLIP exact similarity between the candidate and the query is calculated, and the results are output by combining the semantic alignment threshold and the confidence threshold.

[0022] We employ a combination of symmetric cross-entropy contrastive loss (ITC) and generative cross-entropy loss (CE) for training.

[0023] This invention discloses a deep learning-based unsupervised semi-paired cross-modal retrieval system, which is used in the aforementioned deep learning-based unsupervised semi-paired cross-modal retrieval method. This system is suitable for medical multimodal analysis and cross-modal product recommendation scenarios, supporting fine-grained semantic association and second-level response under asymmetric data.

[0024] The system includes:

[0025] The cross-modal feature fusion module is used to perform cross-modal fine-grained alignment on multimodal data generated from social media and mobile internet, and then use a gated cross-attention mechanism to perform cross-modal feature fusion.

[0026] The data processing module is used to identify low-frequency semantic units through word frequency statistics based on the results of cross-modal feature fusion, generate supplementary text descriptions using visual feature projection, generate diverse cross-modal data by combining annealing temperature sampling and KL divergence constraint bundle search strategy, and filter low-quality results.

[0027] The cross-modal retrieval module is used to perform cross-modal retrieval based on filtered and diverse cross-modal data. It combines dynamic hierarchical indexing algorithm and quantization compression technology, uses KL divergence to monitor data distribution offset to trigger index reconstruction, and optimizes cross-modal alignment by combining contrastive learning loss and generation loss.

[0028] The present invention discloses the following technical effects:

[0029] The unsupervised semi-paired cross-modal retrieval algorithm (USCR) of this invention, combined with a dynamic hierarchical index (DHI) architecture, demonstrates significant advantages in efficient retrieval and semantic alignment of complex multimodal data. By integrating cross-modal fine-grained feature representation, dynamic generation enhancement, and high-compression indexing techniques, this method not only overcomes the semantic constraints of traditional strong pairing models but also solves the bottleneck problems of high-dimensional feature storage and incremental updates. It is particularly suitable for highly dynamic scenarios such as real-time detection of sensitive information, significantly improving the accuracy of cross-modal semantic association and the practicality of the system.

[0030] This invention proposes a multimodal feature fusion and dynamic alignment mechanism. Through collaborative optimization of a dual-branch visual encoder and a learnable cue text encoder, it overcomes the limitations of traditional single-modal feature extraction. An adaptive multi-scale pooling module achieves multi-granular feature fusion from pixel-level details to global structure, while dynamic cue vectors accurately capture task-related semantic context, solving the problem of modifier position deviation caused by traditional fixed templates. A gated cross-attention mechanism is combined to dynamically assign weights to visual and textual features, effectively suppressing redundant modal interference and significantly improving the flexibility of fine-grained semantic alignment.

[0031] This invention innovatively designs a cross-modal data augmentation strategy guided by low-frequency semantics. Based on visual feature projection, it generates diverse text descriptions that implicitly contain details such as material and lighting, solving the problem of missing low-frequency semantics caused by long-tail distribution. By dynamically identifying low-frequency words and increasing their generation probability, combined with annealing temperature sampling and diversity constraint search, it achieves cross-modal semantic expansion in non-paired scenarios, significantly reducing reliance on manually labeled data while avoiding the generation of duplicate or low-quality descriptions.

[0032] This invention constructs a two-stage quantized dynamic index architecture, achieving efficient compression and real-time retrieval of high-dimensional features through coarse-grained clustering and fine-grained product quantization techniques. A dynamic probe strategy, combined with semantic alignment and dual-threshold filtering based on confidence, significantly reduces noise matching interference. A degradation awareness mechanism, through data distribution offset monitoring, triggers adaptive index reconstruction, effectively addressing the long-term evolutionary characteristics of dynamically incremental data. This design ensures retrieval accuracy while supporting low storage overhead for millions of data points and meeting the high-concurrency, real-time response requirements of mobile devices.

[0033] Through the aforementioned breakthroughs in core technologies, this invention provides a systematic solution for semantic association and efficient retrieval of cross-modal data. It demonstrates stronger adaptability and reliability in practical scenarios such as screening for illegal content, and promotes the practical application of cross-modal technology in dynamic and open environments. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 This is a flowchart illustrating the method described in this invention;

[0036] Figure 2 This is a schematic diagram of the data augmentation model described in this invention;

[0037] Figure 3 This is a schematic diagram of the USCR model described in this invention. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0039] like Figures 1-3 As shown, this invention provides an unsupervised semi-paired cross-modal retrieval technology based on deep learning, specifically including the following:

[0040] S1. Multimodal Encoder Construction and Cross-Modal Dynamic Fusion Mechanism: A dual-branch visual encoder is proposed to overcome the limitations of traditional single-modal feature extraction: 1) Combining CLIP pre-trained global features with an adaptive multi-scale pooling module, multi-granularity local details (such as object parts and textures) are fused through a multilayer perceptron to achieve collaborative representation of global scenes and local attributes; 2) A learnable cue text encoder is designed, inserting four dynamically adjustable cue vectors before the CLIP input sequence to overcome the semantic constraints of traditional fixed templates (such as "Aphotoof{text}") and adaptively capture task-related context (such as modifier position preferences). Based on this, a gated cross-attention mechanism is introduced for the first time, dynamically allocating visual and text weights through Sigmoid to replace traditional concatenation / addition operations and solve the problems of multimodal redundancy and modal conflict.

[0041] S2. Data Preprocessing and Dynamic Augmentation Strategies: Input images are standardized and adapted to the resolution of the pre-trained model. Text is processed using sub-word segmentation to handle out-of-vocabulary words and complex semantic combinations. Low-frequency semantic units in the training set are statistically analyzed, and supplementary descriptions are generated through probability weighting and visual feature projection. Combined with annealing temperature sampling and diversity bundle search strategies, diverse cross-modal data is generated and low-quality results are filtered out to alleviate the scarcity of labeled data.

[0042] S3. Dynamic Hierarchical Index Architecture and Quantization Compression: A dual-stage quantization hierarchical index (DHI) is proposed to address the bottlenecks of high-dimensional feature storage and dynamic updates: 1) Coarse-grained clustering encoding: Visual features are divided into 1024 clusters using K-means++, constructing an inverted index to accelerate retrieval; 2) Fine-grained product quantization: Features within a cluster are divided into 8 subspaces, each encoded with 8 bits, achieving 87.5% feature dimensionality compression. A unique dual-threshold retrieval mechanism is implemented: The probe range is dynamically selected based on query similarity, and semantic alignment and confidence thresholds are combined to filter noise. A degradation-aware incremental maintenance mechanism is designed, using KL divergence to monitor data distribution drift, triggering full index reconstruction, and supporting incremental updates.

[0043] S4. Joint Optimization and Cross-Modal Retrieval Process: The visual encoder, text encoder, and decoder are jointly trained, and cross-modal alignment and text generation are collaboratively optimized through contrastive learning loss and generation loss. During the inference phase, real-time retrieval of multimodal queries is achieved based on a dynamic hierarchical index, extracting image and text features and calculating cross-modal similarity to output highly relevant retrieval results.

[0044] Fine-grained feature fusion and dynamic gating mechanism in multimodal encoders: This invention innovatively introduces an adaptive multi-scale pooling module into the visual encoder, employing pooling operations with three grid sizes: 1×1, 2×2, and 4×4, to capture pixel-level details, local region features, and global structural information respectively, solving the problem of insufficient fine-grained perception in traditional single-modal feature extraction. Through multi-scale feature independent projection and cross-layer fusion, the ability to represent local attributes such as object parts and textures is significantly enhanced. The text encoder designs dynamically learnable cue vectors, inserting four self-optimized semantic context vectors (initialized to a normal distribution) at the front of the CLIP input sequence, breaking the semantic constraints of traditional fixed templates and accurately capturing the positional preferences of task-related modifiers. The gating cross-attention mechanism achieves element-wise weighted fusion of visual and text features through dynamic weight allocation (visual weight 0.3-0.7), suppressing interference from redundant modal information. This design is significantly superior to traditional concatenation or addition operations.

[0045] Low-frequency semantic-guided generation control and dynamic data augmentation: Addressing the lack of low-frequency semantics due to long-tailed data distribution, a low-frequency word recognition mechanism based on word frequency statistics is proposed. During the generation stage, probability reweighting is used to force a 3-fold increase in the sampling probability of low-frequency words, covering long-tailed semantics such as "zebra stripes." Combined with visual feature-guided text augmentation, global image features are linearly projected into the text semantic space and fused with the original text at a fixed ratio (α = 0.6), driving the decoder to generate diverse descriptions of implicit material, lighting, and other visual details. Innovatively, annealing temperature sampling and KL divergence constraint search are introduced. The temperature parameter linearly decays from 1.5 to 0.5 to balance exploration and convergence, and the KL penalty term (λ = 0.5) forces candidate sequence differences, avoiding the generation of duplicate descriptions.

[0046] Dual-Stage Quantized Index Architecture and Degradation-Aware Incremental Maintenance: To address the challenges of high-dimensional feature storage and dynamic updates, a dual-stage quantized hierarchical index (DHI) is proposed. In the coarse-grained stage, K-means++ clustering is used to divide visual features into 1,024 clusters, constructing an inverted index to accelerate retrieval. In the fine-grained stage, product quantization (8 subspaces × 8 bits) is performed on features within each cluster, achieving 87.5% feature storage compression. A dual-threshold retrieval mechanism dynamically selects the probe range (10% or 5% of clusters) based on the maximum similarity between the query and the cluster center (threshold 0.7), and filters noise using a semantic alignment threshold (0.75) and a Top-1 confidence threshold (0.6). For incremental data updates, a degradation-aware mechanism is designed: KL divergence (Δ≥0.15) is used to monitor data distribution shifts, triggering full index reconstruction and ensuring monthly update deviation ≤5%, significantly outperforming traditional static indexes.

[0047] Cross-modal alignment and retrieval: This method achieves a dual-objective synergy of cross-modal feature alignment and text generation through joint optimization of symmetric cross-entropy contrastive loss (ITC) and generative loss (CE). ITC loss utilizes symmetric optimization of the intra-batch image-text similarity matrix to narrow the modal gap; CE loss drives the decoder to generate grammatically coherent text. The retrieval process employs a two-stage matching dynamic hierarchical index: the coarse screening stage quickly recalls candidates based on clustering results, while the fine screening stage calculates precise similarity and combines it with CLIP semantic ranking for output. This method achieves real-time retrieval (complexity O(logC)) on millions of data points and supports high-concurrency scenarios on mobile devices.

[0048] Example: Figure 2 As shown, this invention proposes a multimodal hybrid model based on encoder-decoder, which supports the following two collaborative operation modes: (1) Cross-modal feature alignment training mode: The visual and text encoders jointly optimize through contrastive loss to achieve fine-grained semantic alignment between images and text. The visual encoder integrates global semantics and multi-scale local features, while the text encoder adaptively captures task-related context through dynamic learnable cues and combines a gated cross-attention mechanism to suppress redundant information, ensuring consistency across the cross-modal feature space. (2) Visually guided diversity data augmentation mode: Based on the trained encoder, the model switches to the generative augmentation state, projects visual features into the text semantic space, and drives the decoder to generate diverse descriptions of implicit visual details. Combining low-frequency semantic guidance and diversity generation strategies, the coverage of sparse words is forcibly increased, and high-quality augmented text is retained through cross-modal semantic filtering. The two modes form a closed-loop optimization, realizing the collaborative iteration of cross-modal alignment and data augmentation, effectively solving the problems of annotation scarcity and long-tail distribution.

[0049] Step S1: Construction of multimodal encoder and cross-modal dynamic fusion

[0050] Step S1-1, the dual-branch visual encoder, includes a global feature extraction and adaptive multi-scale pooling (SR) module.

[0051] Input image I∈R in global feature extraction 3×224×224 Global features are extracted using the ViT-B / 32 model pre-trained with CLIP:

[0052] f clip = CLIP-Vision(I)∈R 512 ;

[0053] ViT-B / 32 divides the image into a 32×32 block sequence, which is then encoded by 12 layers of Transformer to generate a global semantic vector.

[0054] The adaptive multi-scale pooling module (SR) first performs K×K (K∈{1, 2, 4}) grid average pooling on the input image I to obtain feature maps of different granularities:

[0055] M k =AvgPool k (I)∈R C×H / k×W / k ;

[0056] Where C represents the number of channels in the input image, H and W are the height and width of the input image, respectively, k is the grid size for the pooling operation, and R... C×H / k×W / k Representing feature map M k It is a three-dimensional tensor in real number space.

[0057] Then, features at each scale in the feature projection and fusion process are processed by global average pooling into independent linear layers (Proj). k :R C →R 256 After mapping and concatenation, the data is input into a multilayer perceptron (MLP: dimensions 256→1024→512, ReLU activation) to generate locally enhanced features:

[0058]

[0059] Finally, feature reduction will reduce f clip with f sr After splicing, (f sr This represents the core output of the adaptive multi-scale pooling module (SR), which extracts local detail information (such as small object structures and texture features) from the image through multi-scale fusion of k×k grid pooling features (k∈{1,2,4}), complementing the global features of CLIP. This is then processed through a linear layer W. v ∈R 512×1024 Dimensionality reduction, outputting fused visual features:

[0060] f vis =W v ·[f clip ;f sr ]+b r ∈R 512 ;

[0061] Among them, W v ∈R 512×1024 It is the weight matrix of the linear projection layer, b r ∈R 512 It is a bias term.

[0062] Step S1-2, the learnable cue text encoder, includes cue vector embedding, text feature extraction, and feature alignment, with the following specific structure:

[0063] Cue vector embedding involves inserting four learnable cue vectors P∈R at the beginning of the CLIP text input sequence. 4×512 It is initialized as a normal distribution N(0, 0.02).

[0064] In text feature extraction, the input text is embedded into E by word segmentation. text ∈R L×512 After concatenating with the prompt vector, input it into the CLIP text encoder:

[0065]

[0066] Feature alignment is performed by taking the first position (CLS marker) and aligning the feature with the visual feature through linear projection:

[0067] f text =W t ·f text [0]+b t ∈R 512 ;

[0068] Steps S1-3 of the gated cross-attention mechanism include dynamic weight allocation and feature fusion, with the specific structure as follows:

[0069] Visual and textual features are fused using Sigmoid gating in dynamic weight allocation:

[0070] f gate =σ(W g [f vis ;f txt ]+b g );

[0071] Feature fusion generates fused features by weighting elements:

[0072] f fuse =f vis ⊙f gate +f txt ⊙(1-f gate );

[0073] Among them, f txt These are the text features output by the learnable cue text encoder; f gate It is a gating weight vector that dynamically allocates the fusion ratio between visual and textual elements; f fuse It is a cross-modal fusion feature, serving as a unified representation for downstream retrieval.

[0074] Step S2: Data Preprocessing and Dynamic Augmentation Strategies

[0075] Step S2-1, input standardization, includes image processing and text processing, with the following specific structure:

[0076] Image processing was performed with uniform scaling to 224×224 pixels and normalization parameters (mean 0.5, standard deviation 0.5). In text processing, the CLIP word segmenter processed the text, adding [SOS] and [EOS] markers, and truncating / padding to a length of 30 words.

[0077] Step S2-2, low-frequency semantic guidance enhancement, includes low-frequency word recognition, visual feature projection enhancement, and generation control. The specific structure is as follows:

[0078] Low-frequency word recognition statistically analyzes the word frequency of the training set and marks words with a frequency below 0.01% (e.g., "zebra stripes"). Global visual features are then enhanced through visual feature projection. vis Through linear layer W p ∈R 512×512 Mapped to the text space, it is fused with the original text features at α = 0.6:

[0079]

[0080] Among them, f txt These are the text features output by the learnable cue text encoder (defined in steps S1-2); b p It is the bias term of the visual feature projection layer; f aug It is the text space enhancement feature generated by visual feature projection; α is the fusion weight coefficient, which balances the ratio of the enhancement feature to the original text feature.

[0081] Annealing temperature sampling in generation control (τ) max =1.5→τ min =0.5) and KL divergence beam search (λ div =0.5) Suppress repeated generation, retain sim GLIP Results with a value ≥0.8.

[0082] Step S3: Dynamic Hierarchical Index Architecture and Quantization Compression

[0083] Step S3-1, the two-stage quantization encoding, includes coarse-grained clustering and fine-grained product quantization (PQ), with the specific structure as follows:

[0084] Coarse-grained clustering uses K-means++ to divide visual features into 1024 clusters and constructs an inverted index; fine-grained product quantization (PQ) divides the features within a cluster into 8 subspaces, each encoded with 8 bits.

[0085] Step S3-2, dual-threshold retrieval optimization, includes dynamic probe range and refined filtering, with the specific structure as follows:

[0086] The dynamic probe range dynamically selects the number of probe clusters based on query similarity:

[0087]

[0088] Refined filtration retention Candidates with a Top-1 confidence level ≥ 0.6.

[0089] Step S3-3, degradation-aware incremental maintenance, includes incremental updates and full reconstruction triggering, with the specific structure as follows:

[0090] New data D during incremental update new Keep only sim CLIP Samples with a value ≥0.8 are assigned to the nearest cluster and the PQ codebook is updated. Full reconstruction is triggered when the KL divergence Δ between the old and new data distributions is... KL When the value is ≥0.15, index rebuilding is triggered.

[0091] Step S4: Joint Optimization and Cross-Modal Retrieval Process

[0092] Step S4-1 The joint loss function includes contrastive loss and generative loss, with the following specific structure:

[0093] Contrast Loss (ITC): Symmetric cross-entropy loss optimizes the image-text similarity matrix S.

[0094]

[0095] Where S∈R N×N It is an image-text similarity matrix, S ij Let represent the cosine similarity between the i-th image and the j-th text segment; y is the transpose of matrix S, used to swap the retrieval directions of image and text (text → image); img ∈{0,1}N is the one-hot label vector of the image, representing the true image-text pairing relationship; y txt ∈{0,1} N It is the one-hot tag vector of the text, and y img symmetry.

[0096] Generation Loss (CE): Standard cross-entropy optimized text generation:

[0097]

[0098] Where T represents the maximum length of the text sequence to be generated; t represents the time step (position index) in the generation process, with a value ranging from 1 to T; and w represents any candidate word in the vocabulary V. t represents the ground truth word at time step t; I represents the input image to be retrieved.

[0099] Step S4-2, the two-stage retrieval process, includes a coarse screening stage and a fine screening stage, with the specific structure as follows:

[0100] The coarse screening stage retrieves Top-K candidates based on the DHI index (complexity O(logC)); the fine screening stage calculates the CLIP exact similarity between the candidates and the query, and outputs the results in conjunction with semantic ranking.

[0101] The main flow of the USCR algorithm is described as follows:

[0102]

[0103]

[0104]

[0105] Stability optimization for cross-modal encoders: Residual connections and batch normalization mechanisms are introduced into the multi-scale pooling module (SR) of a bi-branch visual encoder. The feature maps after pooling at each scale are linearly projected, added to the original features from the skip connections, and then fused using a multilayer perceptron (MLP). This design utilizes batch normalization to dynamically calibrate the feature distribution, mitigating the vanishing gradient problem, improving model training convergence speed, and enhancing the robustness of multi-scale features.

[0106] The incremental maintenance mechanism of the dynamic hierarchical index is designed as follows: For incremental data updates, a codebook dynamic expansion based on a greedy strategy is implemented: when the maximum similarity between new data and the existing cluster centers is lower than a threshold (0.65), a new cluster is automatically created; for samples assigned to old clusters, projection errors are compensated through residual quantization. Combined with KL divergence to monitor data distribution offset, this mechanism reduces the retrieval accuracy decay rate during updates and improves the index stability in long-term dynamic scenarios.

[0107] This invention can be applied to multiple aspects:

[0108] 1. Multimodal Intelligent Analysis and Early Warning System: This invention can build a real-time monitoring platform that accurately associates sensitive content (such as images and related texts of sudden events) in social media through unsupervised semi-paired cross-modal retrieval technology. Combined with dynamic hierarchical indexing, it can achieve second-level location of illegal content, providing enterprises with efficient early warning and decision support.

[0109] 2. Cross-modal product recommendation and intelligent shopping guide: Based on the dynamic alignment capability of visual and text features, the system can recommend semantically matched long-tail product descriptions based on the local product details (such as material texture) browsed by the user, solving the recommendation bias problem caused by the asymmetric association between images and text in traditional recommendation systems and improving the conversion rate of e-commerce platforms.

[0110] 3. Joint diagnosis of medical multimodal data: Through cross-modal fine-grained alignment technology, high-precision correlation between medical images (CT / MRI) and clinical text reports is achieved, which helps doctors quickly locate semantic contradictions between image features and medical record descriptions, thereby improving diagnostic efficiency and consistency.

[0111] 4. Open Environment Multimodal Data Analysis Engine: Supports efficient indexing and dynamic evolution analysis of incremental data, suitable for scenarios such as news event tracking and cross-platform content supervision. It continuously adapts to changes in data distribution through a degradation awareness mechanism to ensure long-term retrieval stability.

[0112] This invention provides an unsupervised semi-paired cross-modal retrieval algorithm (USCR) based on deep learning, combined with a dynamic hierarchical indexing (DHI) architecture, systematically solving the problems of high annotation costs, asymmetric association, and storage efficiency. It achieves fine-grained cross-modal alignment through a dual-branch visual encoder (fusing CLIP global features and multi-scale local details) and a text encoder with learnable prompts; a gated cross-attention mechanism is designed to dynamically fuse multimodal features, driving the generation of diverse descriptions. To address data sparsity, a multimodal feature-guided enhancement strategy is proposed: supplementary text is generated based on visual feature projection, and low-frequency semantics are probabilistically weighted to improve low-frequency word coverage; combined with the dynamic hierarchical indexing algorithm, a two-stage quantization encoding compression storage method achieves 87.5%, enabling real-time retrieval of millions of data points.

[0113] This invention optimizes the retrieval efficiency and semantic relevance of multimodal data by combining advanced cross-modal alignment technology and innovative dynamic indexing enhancement methods, and improves the accuracy and real-time performance of sensitive information identification, especially showing significant advantages in highly dynamic scenarios such as filtering illegal content.

[0114] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0115] In the description of this invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0116] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A deep learning-based unsupervised semi-paired cross-modal retrieval method, characterized in that, Includes the following steps: Based on multimodal data generated from social media and mobile internet, cross-modal fine-grained alignment is performed, followed by cross-modal feature fusion using a gated cross-attention mechanism; Based on the results of cross-modal feature fusion, low-frequency semantic units are identified through word frequency statistics, supplementary text descriptions are generated using visual feature projection, and diverse cross-modal data are generated by combining annealing temperature sampling and KL divergence constraint bundle search strategy, while low-quality results are filtered out. Based on the filtered and diverse cross-modal data, combined with dynamic hierarchical indexing algorithm and quantization compression technology, index reconstruction is triggered by monitoring data distribution offset through KL divergence, and cross-modal alignment is optimized by combining contrastive learning loss and generative loss to perform cross-modal retrieval; When performing cross-modal feature fusion, based on the multimodal data, a dual-branch visual encoder and a learnable cue text encoder are constructed. After cross-modal fine-grained alignment, a gated cross-attention mechanism is introduced. Visual and text weights are dynamically allocated through the Sigmoid function to perform cross-modal feature fusion. Specifically, visual and text weights are dynamically allocated through the Sigmoid function, and cross-modal features are generated by element-weighted fusion to suppress interference from redundant modal information. When constructing the dual-branch visual encoder, the dual-branch visual encoder is constructed by combining CLIP pre-trained global features with an adaptive multi-scale pooling module. This dual-branch visual encoder is used to fuse multi-granularity local details through a multilayer perceptron to achieve collaborative representation of global scene and local attributes. When constructing a dual-branch visual encoder, the adaptive multi-scale pooling module uses pooling operations with three grid sizes: 1×1, 2×2, and 4×4. When constructing a learnable cue text encoder, four dynamically adjustable cue vectors are inserted before the CLIP input sequence to adaptively capture task-related context. When constructing the learnable cue text encoder, four dynamically optimized cue vectors are inserted at the beginning of the CLIP input sequence, initialized as a normal distribution, and the CLS marker features at the first position are aligned with the visual features through linear projection.

2. The unsupervised semi-paired cross-modal retrieval method based on deep learning according to claim 1, characterized in that: When combining dynamic hierarchical indexing algorithm and quantization compression technology, a two-stage quantization hierarchical index is adopted. Visual features are divided by coarse-grained clustering, and the features within the cluster are compressed and stored by fine-grained product quantization encoding. The probe range is dynamically selected and noise is filtered for matching by combining a dual threshold retrieval mechanism.

3. The unsupervised semi-paired cross-modal retrieval method based on deep learning according to claim 2, characterized in that: When executing a two-stage quantization hierarchical index, the two-stage quantization hierarchical index includes: Coarse-grained clustering encoding: The K-means++ algorithm is used to divide visual features into 1024 clusters, and an inverted index is constructed to accelerate retrieval; Fine-grained product quantization: The 512-dimensional features within the cluster are divided into 8 subspaces, and each subspace is encoded with 8 bits.

4. The unsupervised semi-paired cross-modal retrieval method based on deep learning according to claim 3, characterized in that: When performing cross-modal retrieval, the data distribution shift is monitored by using a KL divergence threshold Δ≥0.15, triggering a full index reconstruction; and the deviation is updated to ensure that the retrieval accuracy decay is ≤2.1%. In the coarse screening stage of cross-modal retrieval, Top-K candidates are quickly recalled based on the inverted index; In the fine screening stage of cross-modal retrieval, the CLIP exact similarity between the candidate and the query is calculated, and the results are output by combining the semantic alignment threshold and the confidence threshold. We employ a combination of symmetric cross-entropy contrastive loss (ITC) and generative cross-entropy loss (CE) for training.

5. A deep learning-based unsupervised semi-paired cross-modal retrieval system, used to implement the deep learning-based unsupervised semi-paired cross-modal retrieval method as described in claim 1, characterized in that, This system is suitable for medical multimodal analysis and cross-modal product recommendation scenarios, supporting fine-grained semantic association and second-level response under asymmetric data. The system includes: The cross-modal feature fusion module is used to perform cross-modal fine-grained alignment on multimodal data generated from social media and mobile internet, and then use a gated cross-attention mechanism to perform cross-modal feature fusion. The data processing module is used to identify low-frequency semantic units through word frequency statistics based on the results of cross-modal feature fusion, generate supplementary text descriptions using visual feature projection, generate diverse cross-modal data by combining annealing temperature sampling and KL divergence constraint bundle search strategy, and filter low-quality results. The cross-modal retrieval module is used to perform cross-modal retrieval based on filtered and diverse cross-modal data. It combines dynamic hierarchical indexing algorithm and quantization compression technology, uses KL divergence to monitor data distribution offset to trigger index reconstruction, and optimizes cross-modal alignment by combining contrastive learning loss and generation loss.

Images