Feedback anchor point online cross-modal fast retrieval method and system based on semantic enhancement

Abstract

The application belongs to the technical field of multi-modal information processing, and provides a feedback anchor point online cross-modal fast retrieval method and system based on semantic enhancement. Under the condition of strictly limiting the source of evidence, a text enhancement description and an image enhancement description are respectively generated, and then a sample level confidence is obtained according to the consistency of the original representation and the enhanced representation. Then, a confidence weighted image composite feature, a text composite feature and a joint semantic vector are constructed. In the process of data flow, the binary hash code and the double modal hash function of the current data block are alternately updated by using the label correlation constraint label codebook, the semantic atom in the joint semantic space, the atomic residual compensation and the bounded history anchor memory. Finally, the projection score is threshold calibrated, and the hamming distance sorting result of the image to the text or the text to the image is output. The cross-block stability and retrieval sorting consistency of the hash code in the streaming update scene are improved.

Description

Technical Field

[0001] This invention belongs to the field of multimodal information processing technology, and particularly relates to a method and system for fast online cross-modal retrieval based on semantic enhancement feedback anchors. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] With the continuous generation of multimodal data such as images and text, cross-modal retrieval systems must eliminate heterogeneous differences between different modalities while controlling the storage and computational overhead of large-scale retrieval. Cross-modal hashing methods improve the efficiency of large-scale retrieval by mapping heterogeneous data to a common Hamming space, replacing high-dimensional real-valued vectors with binary codes, and using bitwise operations to accelerate retrieval. However, batch processing methods typically require re-accessing historical samples and updating the entire model when new data arrives, making them unsuitable for continuous data stream scenarios; updating the model only with the current data block can easily disrupt the semantic consistency between new and old samples.

[0004] Existing online hashing schemes such as DOCH achieve incremental learning by preserving the Hamming space similarity between new and old data and discrete optimization; SEOCH constructs a semantic similarity matrix by using label semantic embedding to maintain semantic relationships; manifold ranking methods use anchor points to approximate the correlation of streaming data; class semantic guidance schemes mine category associations through word embedding; semantic enhancement strategies use two-step online hashing to reduce retraining overhead; and fine-grained semantic methods use large models to generate category descriptions to enhance representations.

[0005] However, the above solutions still face common problems: generative augmentation often relies on real labels or paired modalities, which can easily introduce non-generalizable semantic leakage; the augmentation quality of different samples varies significantly, and fixed fusion weights cannot suppress misdescriptions; in online scenarios, limited historical storage makes it difficult to take into account the semantic relationship, label relationship and binary code stability of new and old data; multi-label category codes may be highly correlated or repetitive, and category conflicts are prone to occur under short code conditions; updating based on the current block is prone to forgetting the historical structure, while full revisiting leads to a continuous increase in storage and computing overhead. Summary of the Invention

[0006] To address at least one of the technical problems mentioned above, this invention provides a method and system for fast online cross-modal retrieval of feedback anchors based on semantic enhancement.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: The first aspect of the present invention provides a fast online cross-modal retrieval method based on semantically enhanced feedback anchors, comprising the following steps: Read multimodal data containing paired images, text, and optional multi-labels, and generate a unique identifier for each sample; divide the training data stream according to the preset arrival order and data block size, and divide the query set and retrieval library; Semantic enhancement with modality isolation is performed on text and images respectively. Image confidence and text confidence are calculated based on the consistency between the original and enhanced representations of the two modalities. Image composite features, text composite features and joint semantic vectors are constructed by combining the original and enhanced representations of the two modalities and the image confidence and text confidence. Using only training data, a label codebook and semantic atoms are constructed. During the arrival of the data stream, the binary hash code and bimodal hash function of the current data block are alternately updated using the label codebook constrained by label relevance, semantic atoms in the joint semantic space, atomic residual compensation, and bounded historical anchor memory, until the relative change of the overall objective function of the data block is less than a preset threshold or the maximum number of iterations is reached, and the image and text projection scores are output. Threshold calibration is performed on the image and text projection scores. Binary hash codes for the query set and retrieval database are generated using the final image projection matrix and text projection matrix, respectively. The Hamming distance between the cross-modal hash codes is calculated and sorted in ascending order. The image-to-text retrieval results and text-to-image retrieval results are then output.

[0008] Furthermore, the semantic enhancement of modality isolation for text and images respectively includes: For text branches, the original title, description, user tags or other text fields of the current sample are used as the only visible evidence input to the generation model. The generated results use structured fields to record summaries, entities, attributes, actions, relationships, scenarios, search keywords, uncertainties, confidence statements and short descriptions for use by the visual-language encoder. For the image branch, the image pixels of the current sample are used as the only visible evidence to input into the visual generation model. The generated result describes the visible objects, attributes, actions, spatial relationships and scenes with structured fields corresponding to the text branch. Establish cache keys for each of the two generation branches, determined by the sample identifier, enhancement stage, input content hash, prompt template hash, and model version identifier.

[0009] Furthermore, the image composite features Textual composite features Represented as: , The joint semantic vector is represented as: ,in, , , and These represent the original image representation, the image-enhanced description representation, the original text representation, and the text-enhanced description, respectively. and Representing the confidence level of image semantic consistency and the confidence level of text semantic consistency. This indicates L2 normalization.

[0010] Furthermore, a label codebook is constructed based on the label matrix in the training partition. When the hash code length is sufficient and the number of categories is small, mutually orthogonal codewords are selected from the Hadamard matrix. When the number of categories is greater than the code length or the label correlation is complex, codewords are selected from the balanced candidate code pool, and the correlation penalty of category pairs is adjusted by the label co-occurrence similarity. Semantic atoms are basic semantic representation units determined by the label information, sample distribution, or semantic clustering results of training data, used to characterize representative category semantics or local semantic structures in the dataset.

[0011] Furthermore, the alternating update of the binary hash code and bimodal hash function of the current data block using the tag codebook constrained by tag relevance, semantic atoms in the joint semantic space, atomic residual compensation, and bounded historical anchor memory includes: Read the t-th training data block and calculate the semantic atom soft assignment matrix based on the similarity between the current joint semantic vector and each semantic atom. ; Using the feedback matrix from the previous time step With semantic atomic soft allocation matrix Predict the structured quantization residual of the current block Generate the current tag code target based on the tag codebook. It also integrates the label sharing relationship and semantic similarity relationship between the current sample and the historical anchor point to form a current-historical similarity matrix. and effective historical constraint weights ; Based on structured quantitative residuals Construct a residual feedback term based on the current-historical similarity matrix. and effective historical constraint weights Construct a current-history similarity preservation term, and build the current block objective function with residual feedback term, image projection fitting term, text projection fitting term, projection matrix regularization term, tag code constraint term, current-history similarity preservation term, two-modal projection score consistency term, bit balance term, and bit decorrelation term; While maintaining binary and orthogonal constraints, the image projection matrix, text projection matrix, orthogonal rotation matrix, continuous latent representation, and current block binary hash code are alternately updated using a block coordinate descent method until the relative change of the objective function is less than a preset threshold or the maximum number of iterations is reached.

[0012] Furthermore, the construction of bounded historical anchor memory includes: in a global memory approach, performing reservoir sampling on arriving samples: adding directly when the memory is not full; and when the memory is full, sampling the first sample... One known sample The probability of replacing a historical position. In the hierarchical memory approach, the sample's atom is first determined by the largest soft-allocation atom, and then reservoir sampling is performed independently within the capacity of each atom to maintain coverage of different semantic regions.

[0013] Furthermore, the overall objective function for the current data block is: , , , , , in, This represents the quantized residual feedback term, used to... As a historical semantic residual offset, it is injected into the current quantization; Represents the image projection fitting term, the text projection fitting term, and the projection matrix regularization term; Indicates the tag code constraint item; Indicates the current-historical similarity preservation term; This represents the consistency term of the two-modal projection fractions; Represents bit balancing terms and bit decorrelation terms; Indicated by the tag codebook With the current block label matrix The target of the combined sample label code; This represents the binary hash code matrix of the current data block. , Control the quantization term and the atomic compensation intensity separately. and Controlling two-mode fitting, The regularization weights are the projection matrix weights. To control the target of the tag code, In order to maintain historical relations, To control the consistency of the two-modal projection scores, bit balancing and bit decorrelation terms are used to suppress the correlation between constant bits and different bits, respectively. and For image and text hash projection matrices, The length of the hash code. For continuous hidden variables in the current block, It is an orthogonal rotation matrix. and These are the composite feature matrices of the current block image and text, respectively.

[0014] A second aspect of the present invention provides an online cross-modal fast retrieval system based on semantically enhanced feedback anchors, comprising: The query set and retrieval library construction module is used to read multimodal data containing paired images, text, and optional multi-labels, generate a unique identifier for each sample, divide the training data stream according to the preset arrival order and data block size, and divide the query set and retrieval library. The isolated semantic enhancement module is used to perform modal isolation semantic enhancement on text and images respectively. It calculates image confidence and text confidence based on the consistency of the extracted original and enhanced representations of the two modalities. It constructs image composite features, text composite features and joint semantic vectors by combining the original and enhanced representations of the two modalities and the image confidence and text confidence respectively. The hash learning module is used to construct a label codebook and semantic atoms using only training data. During the arrival of the data stream, it uses the label codebook constrained by label relevance, semantic atoms in the joint semantic space, atomic residual compensation, and bounded history anchor memory to alternately update the binary hash code and bimodal hash function of the current data block until the relative change of the overall objective function of the data block is less than a preset threshold or the maximum number of iterations is reached, and outputs the image and text projection scores. The retrieval module is used to perform threshold calibration on the projection scores of images and text, generate binary hash codes for the query set and the retrieval library using the final image projection matrix and text projection matrix respectively, calculate the Hamming distance between the cross-modal hash codes and sort them in ascending order, and output the image-to-text retrieval results and the text-to-image retrieval results.

[0015] A third aspect of the present invention provides a computer-readable storage medium.

[0016] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the semantically enhanced feedback anchor-based online cross-modal fast retrieval method described above.

[0017] A fourth aspect of the present invention provides a computer device.

[0018] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the semantically enhanced feedback anchor-based online cross-modal fast retrieval method described above.

[0019] Compared with the prior art, the beneficial effects of the present invention are: 1. This invention, through the combined action of semantic atomic soft allocation, historical feedback matrix, and structured quantization residuals, predicts reusable quantization deviations in historical data blocks as residual compensations for the current block and injects them into the quantization target. This solves the problem that existing online cross-modal hashing typically discards quantization errors as local errors of the current block and makes it difficult to reuse similar semantic residuals across blocks. It achieves coordinated correction of the current binary code and historical semantic residual directions, improving the cross-block stability of hash codes and the consistency of retrieval and ranking in streaming update scenarios.

[0020] 2. This invention uses a fixed-budget historical anchor point memory to store column-aligned historical binary codes, joint semantic vectors, and label vectors. It also uses the fused current-history similarity matrix to constrain the inner product relationship between the current continuous latent representation and the historical binary anchor point. This solves the problem of easily forgetting the semantic relationship between new and old samples when not all historical samples can be revisited. It achieves the maintenance of historical binary structure compatibility with limited storage and reduces the storage and computational overhead caused by full retraining or full history storage.

[0021] 3. This invention addresses the problems of supervision leakage, cross-modal shortcuts, and misrepresentations that generative enhancement may introduce by means of modal isolation semantic enhancement and sample-level confidence fusion, so that the text enhancement branch does not access the paired image and the image enhancement branch does not access the paired text. It also uses the consistency between the original representation and the enhanced representation to regulate the energy of the enhanced information, thereby achieving the suppression of noisy enhancement results and the utilization of reliable enhanced semantics.

[0022] 4. This invention addresses the problems of multi-label category code conflict and weakened semantics of low-frequency labels by constraining the label codebook and sample label code targets through label co-occurrence relationship constraints, under the conditions of short hash codes and multi-label category correlation, and constraining codeword correlation, maximum correlation, bit balance and repeated codewords, thereby realizing a label-level binary semantic reference that is more suitable for cross-modal supervised hash learning.

[0023] 5. This invention solves the problems of high cost of revisiting all historical features and the extra quantization loss caused by continuous relaxation and requantization in online updates by using block coordinate descent under binary constraints and orthogonal constraints and cumulative update of projection statistics. It realizes direct alternating update of the current block hash code, image projection and text projection, and outputs a graphic-text binary code that can be directly used for Hamming distance calculation.

[0024] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0025] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0026] Figure 1 This is a flowchart of the online cross-modal fast retrieval method based on semantic enhancement feedback anchors provided in the embodiments of the present invention. Detailed Implementation

[0027] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0028] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0029] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0030] Example 1 like Figure 1 As shown, this embodiment provides a fast online cross-modal retrieval method based on semantically enhanced feedback anchors, including the following steps: S1: Read multimodal data containing paired images, text, and optional multi-labels, and generate a unique identifier for each sample; divide the training data stream according to the preset arrival order and data block size, and divide the query set and retrieval library to ensure that query samples do not participate in model training, semantic atom construction, and historical anchor sampling.

[0031] In this embodiment, let the set of continuously arriving cross-modal samples be . , No. training samples It must contain at least paired images. and text In supervised or semi-supervised implementations, multiple label vectors may also be included. ,in The training samples are divided into T data blocks in the order of arrival. The t-th data block contains... Each sample. The query set does not participate in semantic atom selection, tag codebook statistics, threshold fitting, or historical anchor sampling.

[0032] S2: Perform modality-isolated semantic enhancement on both text and image. Calculate image confidence and text confidence based on the consistency between the extracted original and enhanced representations of the two modalities. Construct composite image features by combining the original and enhanced representations of the two modalities, as well as the image and text confidence scores. Text composite features and joint semantic vector ; Specifically, the steps include the following: S201: Perform semantic enhancement with modal isolation on text and images respectively; In this embodiment, modal isolation semantic enhancement is performed on text and image respectively, so that the text enhancement branch only accesses text evidence and the image enhancement branch only accesses image evidence; Specifically, for text branches, the original title, description, user label, or other text fields of the current sample are used as the only visible evidence input to the generation model. For example, Qwen / Qwen3-VL-8B-Instruct can be used. The generation model must not access the paired image, ground truth label, category vocabulary, file name, dataset name, training / query split, or neighboring samples of the sample.

[0033] The generated results use structured fields to record summaries, entities, attributes, actions, relationships, scenes, search keywords, uncertainties, confidence statements, and short descriptions for use by the visual-language encoder. For structured text containing location or time fields, location and time are used only as context and are not used to infer objects, identities, weather, or events in the image that are not supported by the title or description.

[0034] For the image branch, the image pixels of the current sample are used as the only visible evidence input to the visual generation model, such as using Qwen / Qwen3-VL-8B-Instruct. The generation model must not access paired text, ground truth labels, category vocabularies, filenames, dataset names, training / query splits, or neighboring samples. The generated results describe visible objects, attributes, actions, spatial relationships, and scenes using structured fields corresponding to the text branch. Uncertain content is written as an uncertain item without making a definitive statement.

[0035] A cache key is established for each of the two generation branches, determined by the sample identifier, enhancement stage, input content hash, prompt template hash, and model version identifier. Successfully generated samples are directly read from the cache during repeated runs. If the model call fails or the returned content cannot be parsed into the specified structure, the text branch reverts to the original text, and the image branch reverts to a preset uncertain image description, with the revert status and error reason recorded. The two branches can be processed concurrently, but their input evidence remains isolated.

[0036] S202: Calculate image confidence and text confidence based on the consistency of the extracted original and enhanced representations of the two modalities; In this embodiment, a frozen visual-language encoder is used to extract the original image representation, the original text representation, the image-enhanced descriptive representation, and the text-enhanced descriptive representation. Let the output dimension of the frozen visual-language encoder be... For the i-th sample, the original image representation, image augmentation description representation, original text representation, and text augmentation description representation are denoted as follows: , , and The original image, original text, image augmentation description, and text augmentation description are fed into the frozen visual-language encoder to obtain... , , and L2 normalization was performed on each sample. For samples containing multiple original texts, each text was encoded separately, the mean was calculated, and then normalization was performed to reduce the scale bias caused by the difference in the number of texts.

[0037] Calculate the cosine consistency between the original representation and the same-modal enhanced representation, and linearly map them to the [0,1] interval to obtain the image semantic consistency confidence score. and text semantic consistency confidence : , Since all input representations have been L2 normalized, the inner product equals the cosine similarity. When the enhanced description matches the original content, the confidence level is high; when the enhanced description deviates from the original content, the confidence level decreases. If an enhancement branch is not enabled or fails to generate, its confidence level can be set to zero and the corresponding enhancement term omitted.

[0038] S203: Construct composite image features by combining the original and enhanced representations of the two modalities, as well as image confidence and text confidence. Text composite features and joint semantic vector ; In this embodiment, the original image representation is concatenated with the image enhancement representation weighted by the square root of the image confidence level to obtain the image composite feature; the text composite feature is obtained in the same way. The two original representations are summed with the two enhancement representations weighted by the corresponding confidence levels and normalized to obtain the joint semantic vector.

[0039] Image composite features and text composite features are represented as follows: , in, This represents vector concatenation, where the square root of the confidence level is applied to the augmented components so that the square energy of the augmented components after concatenation is equal to the sum of the squared values ​​of the augmented components. or It is proportional to the original components while preserving complete information. The composite features are used for subsequent modal hashing projection learning.

[0040] The joint semantic vector is represented as: , in, The L2 normalization is represented by summing the four representations in the same dimension of the joint semantic vector. The enhancement components are linearly weighted by confidence and used for semantic atom construction, semantic similarity calculation between the current sample and historical anchors, and atomic residual compensation. Thus, the composite modality features retain the original / enhanced components of each modality, while the joint semantic vector provides cross-modality shared semantic coordinates.

[0041] S3: Construct a tag codebook and semantic atoms using only training data; In this embodiment, the tag codebook is selected from Hadamard code, supervised greedy code, or correlation-optimized code; Construct a tag codebook based on the tag matrix from the training partition. When the hash code length is sufficient and the number of categories is small, mutually orthogonal codewords can be selected from the Hadamard matrix; when the number of categories is greater than the code length or the label correlation is complex, codewords are selected from the balanced candidate code pool, and the correlation penalty of category pairs is adjusted by the label co-occurrence similarity.

[0042] The co-occurrence similarity between tag a and tag b is denoted as The target for tag codebook optimization can be written as: , in, Let r be the correlation coefficient of the inner product of the two tag codewords after normalization. A larger value indicates that the codewords of the two tags are more similar. For tag codebook, or Indicates the first Each tag corresponds to Bit codeword, or Indicates the first Each tag corresponds to Bit codeword, The length of the hash code. This represents the mean of the k-th bit across all categories of codewords. This represents the average value of each bit in the a-th category codeword. , , , Non-negative weights. Weights are reduced through candidate code selection or bit-by-bit flipping. It also rejects repetitive typing, allowing highly relevant tags to be retained while keeping less relevant tags as separate as possible.

[0043] As a further embodiment, for multi-label samples, the corresponding label codewords are combined according to their label sets, and then normalized by square root or mean based on the number of valid labels to form the sample label code target: , in, Optional label frequency weights; This represents element-wise product. For low-frequency labels, their weights can be increased based on the label frequency in the training set, but not using query set label statistics.

[0044] Construct K semantic atoms on the joint semantic vector set of the training stream. A semantic atom is a basic semantic representation unit determined by the label information, sample distribution, or semantic clustering results of the training data. It is used to characterize the representative category semantics or local semantic structure in the dataset. In this embodiment, each semantic atom corresponds to a relatively stable semantic center or semantic anchor point, which can be used to organize the semantic association between the current sample and historical samples, and to provide a reusable semantic reference for online cross-modal hashing learning.

[0045] Semantic atoms are obtained using farthest point initialization and spherical clustering. Specifically, atoms are first initialized using the farthest point strategy, and then iterated using spherical clustering: each joint semantic vector is assigned to the atom with the largest inner product, and the mean of samples in the same cluster is calculated and normalized; if a cluster is empty, the atom of the empty cluster is replaced by the sample with the lowest maximum similarity to the existing atom.

[0046] In a simplified implementation, sample vectors can also be selected as atoms from the training stream at equal intervals or randomly.

[0047] S4: Online hash learning based on semantic atomic residuals and bounded anchor memory; In this embodiment, during the arrival of the data stream, the binary hash code and bimodal hash function of the current data block are alternately updated using the tag codebook constrained by tag correlation, semantic atoms in the joint semantic space, atomic residual compensation, and bounded historical anchor memory. Specifically, the steps include the following: S401: Read the t-th training data block and calculate the semantic atom soft assignment matrix based on the similarity between the current joint semantic vector and each semantic atom. ; Specifically, when the t-th data block arrives, its joint semantic matrix is... Computation relative to semantic atoms Temperature soft assignment matrix The elements in the temperature soft-assignment matrix are represented as follows: , in, >0 represents the atomic temperature. and Indicates the first The semantic atom and the first One semantic atom; set up Given the atomic residual compensation matrix, the atomic compensation amount for the current block is: Soft allocation allows the use of the feedback matrix from the previous time step. With soft allocation matrix This allows a sample to be represented by multiple similar atoms, reducing abrupt changes near hard cluster boundaries.

[0048] S402: Utilize the feedback matrix from the previous time step With soft allocation matrix Predict the structured quantization residual of the current block Simultaneously, generate the current tag code target based on the tag codebook. It also integrates the label sharing relationship and semantic similarity relationship between the current sample and the historical anchor point to form a current-historical similarity matrix. and effective historical constraint weights ; In this embodiment, the feedback matrix from the previous time step is used. With soft allocation matrix Predict the structured quantization residual of the current block Specifically: Let Given the atomic residual compensation matrix, the atomic compensation amount for the current block is: Soft allocation allows the use of the feedback matrix from the previous time step. With soft allocation matrix This allows a sample to be represented by multiple similar atoms, reducing abrupt changes near hard cluster boundaries.

[0049] In this embodiment, a bounded historical anchor memory is established. ,in Save the historical anchor point binary code. Save the joint semantic vector aligned with its columns. Save the label vector. In the global memory mode, perform reservoir sampling on arriving samples: add directly when the memory is not full; when the memory is full, perform sampling on the first sample. One known sample The probability of replacing a historical position. For memory budgeting. In the hierarchical memory approach, the atom to which the sample belongs is first determined by the largest soft-allocation atom, and then reservoir sampling is performed independently within the capacity of each atom to maintain coverage of different semantic regions.

[0050] Construct a fusion similarity matrix between the current sample and historical anchor points. .set up This is a tag-sharing relationship. For a joint semantic cosine relation, then: , in, ∈[0,1]. When the label is unavailable, you can let =0, using only joint semantic similarity; improves when the labels are reliable. This similarity matrix aligns historical binary codes, historical semantics, and historical labels to the same set of anchor columns.

[0051] In this embodiment, to avoid a sudden increase in the weight of historical items due to changes in the scale of historical memory, the effective historical constraint weight is kept constant. According to the current memory column number Normalization: , Or use in another implementation The above. That is, in S403 The historical constraint weights in the overall objective function are used to ensure that the historical relationship preservation terms at different training stages have a comparable scale, thus avoiding issues when the number of historical anchor points changes. The impact suddenly becomes larger. S403: The objective function of the current block is composed of structured quantization residual feedback term, image projection fitting term, text projection fitting term, label code constraint term, current-historical similarity preservation term, and projection regularization term; set up The current block contains consecutive hidden variables. Let X be an orthogonal rotation matrix. I X T These are the composite feature matrices of the current image and text block, respectively. The overall objective function for the current data block is: , , , , , in, This represents the quantized residual feedback term, used to... As a historical semantic residual offset, it is injected into the current quantization; Represents the image projection fitting term, the text projection fitting term, and the projection matrix regularization term; Indicates the tag code constraint item; Indicates the current-historical similarity preservation term; This represents the consistency term of the two-modal projection fractions; Represents bit balancing terms and bit decorrelation terms; Indicated by the tag codebook With the current block label matrix The target of the combined sample label code; This represents the binary hash code matrix of the current data block. , Control the quantization term and the atomic compensation intensity separately. and Controlling two-mode fitting, The regularization weights are the projection matrix weights. To control the target of the tag code, In order to maintain historical relations, To control the consistency of the two-modal projection scores, bit balancing and bit decorrelation terms are used to suppress the correlation between constant bits and different bits, respectively, reducing the impact of repeated bits on the distinguishability of the hash code. When there are no historical anchors, It is zero. The sign is obtained directly using the following formula, and the corresponding bit from the previous round is retained when the score is exactly zero: ， in, To suppress imbalances based on the mean of each bit in the current block, To suppress correlations between different bits based on the bit correlation matrix after removing the diagonal, the above update maintains... The discrete constraint {-1,+1} does not require finding a continuous relaxation solution before quantization.

[0052] S404: While maintaining binary and orthogonal constraints, the image projection matrix, text projection matrix, orthogonal rotation matrix, continuous latent representation, and current block binary hash code are updated alternately using a block coordinate descent method until the relative change of the objective function is less than a preset threshold or the maximum number of iterations is reached.

[0053] Iterative updates using block coordinate descent , , , and When other variables are fixed, and Updated using closed linear equations with ridge regularization; when When the value is greater than 0, the two projection matrices alternately use the projection results of the other mode to form coupled right-hand terms. Through the Performing singular value decomposition yields orthogonal Procrustes solutions. By including Solve for an r×r linear system.

[0054] After the current block iteration is completed, its quantization residual is utilized. Update the atomic residual compensation matrix: , in, >0 indicates a stable term. ∈[0,1] represents the historical smoothing coefficient. The current block sample is then written into the bounded historical anchor memory according to S402. For bimodal hash projection, only the accumulated values ​​are... and Sufficient statistics are available, so there is no need to reload all historical features when subsequent data blocks arrive.

[0055] S405: After the training of the current data block is completed, its image features, text features and binary code contributions are written into the cumulative sufficient statistics; the current quantization residual is calculated, the mapping from semantic atoms to quantization residuals is estimated by ridge regression, and the feedback matrix is ​​updated by exponential sliding; the binary code, joint semantic vector and label vector of the current block are written into the bounded historical anchor memory, and when the capacity limit is reached, global reservoir sampling or reservoir sampling layered by semantic atoms is used to replace the historical anchor.

[0056] S406: Determine if there are still unprocessed data blocks in the training data stream; if so, let t=t+1, and return to step four using the updated feedback matrix, cumulative sufficient statistics, and historical anchor memory; if not, proceed to the next step.

[0057] S5: Use only the existing training data to calibrate the bit-by-bit thresholds and bit polarities of image projection and text projection respectively; use the final image projection matrix and text projection matrix to generate binary hash codes for the query set and retrieval library respectively, calculate the Hamming distance between the cross-modal hash codes and sort them from smallest to largest, and output the image-to-text retrieval results and text-to-image retrieval results.

[0058] In this embodiment, after completing the learning of the current block, the image and text projection scores are calculated using only the training samples that have appeared up to the current block.

[0059] As one example, zero can be used as the threshold for each bit; the adaptive implementation calculates the median or preset quantile of the seen training scores for each bit, and multiplies it by the image threshold ratio and the text threshold ratio respectively to obtain the results. , Another implementation can fix the image threshold to zero, select the text threshold and bit polarity, so that the text projection binary result has a high consistency rate with the image projection binary result of the same sample.

[0060] Query any image and any text query Calculate separately: , in, , ∈{-1,+1} r For bit polarity vectors, This is an element-wise indicator function. The same method is used to generate binary codes for the search library.

[0061] Calculate the Hamming distance between the query code and the heteromodal retrieval library code: , in, This indicates that the query sample is obtained after corresponding modal projection, threshold calibration, and bit polarity calibration. Bit-2 hash code, Indicates the first modality in the heterogeneous retrieval database candidate samples Both are binary hash codes, and belong to... .

[0062] Samples are sorted in ascending order based on Hamming distance. For samples with the same distance, the sample label order can be maintained or a secondary sorting can be performed using projected similarity scores. Image queries are compared with the text database to obtain image-to-text retrieval results, and text queries are compared with the image database to obtain text-to-image retrieval results.

[0063] As a further embodiment, semantic generation submits cached transactions in batches; each feature branch of the frozen encoder records a completion flag per sample row; composite features, similarity matrices, and anchor matrices record completion flags per data block. When configuration, prompt templates, model versions, input content, or sample order changes, the invalid cache is isolated from the current results by the corresponding hash identifier.

[0064] Before exporting the data file for use in the online model, verify the uniqueness of sample identifiers, the mutual exclusion of the training set and query set, that anchor points do not come from the query set, that the beginning and end of block pointers are valid, that all feature values ​​are finite numbers, and that the average L2 norm of the features to be normalized is verified. The final export is stopped if any critical verification fails. During export, a temporary file is first written, and after passing the audit, the final data file is formed using atomic replacement.

[0065] The output file should include at least the original and augmented representations, semantic consistency confidence, image and text composite features, joint semantic vectors, multi-label matrix, training / query / retrieval partition, data flow order, block pointers, and audit information. The online model reads the above fields in the same sample order and outputs the updated image hash projection, text hash projection, label codebook, atomic residual compensation matrix, and historical anchor memory state block by block.

[0066] To verify the technical effectiveness of this invention compared to existing online cross-modal hashing methods, it was compared with FDDH, OSCMFH, DOCMH, ROHLSE, SHOH, SSOCH, and PEOCH under the same training / query / retrieval partitioning, data stream order, hash code length, and Hamming distance ranking evaluation method. The evaluation metric used was the mean precision (mAP) on a fixed retrieval database. The test datasets included IAPR TC-12, NUS-WIDE, and MIRFLICKR-25K, with hash code lengths of 16 bits, 32 bits, 48 ​​bits, and 64 bits, respectively, as shown in Table 1. Table 1. Comparison of average mAP between existing methods and the present invention on a fixed search database.

[0067] As shown in Table 1, this invention achieves optimal results in 11 out of 12 dataset-code length combinations, only slightly lower than ROHLSE in the 16-bit setting of IAPRT-12. When averaging across three datasets, four code lengths, and bidirectional retrieval directions, the average mAP of this invention is 0.7536, stronger than ROHLSE's 0.7331, the best performing method among the comparisons, representing an absolute improvement of 0.0205. On MIRFLICKR-25K, the average mAP of this invention is improved from 0.7949 of the best existing method to 0.8163. These results demonstrate that this invention, through the collaborative constraints of structured semantic state, semantic atomic residual feedback, and historical anchor relationship maintenance, can improve the semantic stability and retrieval ranking quality of cross-modal hash codes during streaming online updates.

[0068] Example 2 This embodiment provides an online cross-modal fast retrieval system based on semantically enhanced feedback anchors, including: The query set and retrieval library construction module is used to read multimodal data containing paired images, text, and optional multi-labels, generate a unique identifier for each sample, divide the training data stream according to the preset arrival order and data block size, and divide the query set and retrieval library. The isolated semantic enhancement module is used to perform modal isolation semantic enhancement on text and images respectively. It calculates image confidence and text confidence based on the consistency of the extracted original and enhanced representations of the two modalities. It constructs image composite features, text composite features and joint semantic vectors by combining the original and enhanced representations of the two modalities and the image confidence and text confidence respectively. The hash learning module is used to construct a label codebook and semantic atoms using only training data. During the arrival of the data stream, it uses the label codebook constrained by label relevance, semantic atoms in the joint semantic space, atomic residual compensation, and bounded history anchor memory to alternately update the binary hash code and bimodal hash function of the current data block until the relative change of the overall objective function of the data block is less than a preset threshold or the maximum number of iterations is reached, and outputs the image and text projection scores. The retrieval module is used to perform threshold calibration on the projection scores of images and text, generate binary hash codes for the query set and the retrieval library using the final image projection matrix and text projection matrix respectively, calculate the Hamming distance between the cross-modal hash codes and sort them in ascending order, and output the image-to-text retrieval results and the text-to-image retrieval results.

[0069] It should be noted that the specific implementation of the online cross-modal fast retrieval system based on semantic enhancement feedback anchors in this embodiment of the invention is similar to the specific implementation of the online cross-modal fast retrieval method based on semantic enhancement feedback anchors in this embodiment of the invention. For details, please refer to the description in the method section. To reduce redundancy, it will not be repeated here.

[0070] Example 3 This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in the semantically enhanced feedback anchor online cross-modal fast retrieval method described above.

[0071] Example 4 This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the semantically enhanced feedback anchor online cross-modal fast retrieval method described above.

[0072] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0073] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0074] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0075] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0076] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0077] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A fast online cross-modal retrieval method based on semantically enhanced feedback anchors, characterized in that, include: Read multimodal data containing paired images, text, and optional multi-labels, and generate a unique identifier for each sample; The training data stream is divided according to the preset arrival order and data block size, and the query set and retrieval library are also divided. Semantic enhancement with modality isolation is performed on text and images respectively. Image confidence and text confidence are calculated based on the consistency between the original and enhanced representations of the two modalities. Image composite features, text composite features and joint semantic vectors are constructed by combining the original and enhanced representations of the two modalities and the image confidence and text confidence. Using only training data, a label codebook and semantic atoms are constructed. During the arrival of the data stream, the binary hash code and bimodal hash function of the current data block are alternately updated using the label codebook constrained by label relevance, semantic atoms in the joint semantic space, atomic residual compensation, and bounded historical anchor memory, until the relative change of the overall objective function of the data block is less than a preset threshold or the maximum number of iterations is reached, and the image and text projection scores are output. Threshold calibration is performed on the image and text projection scores. Binary hash codes for the query set and retrieval database are generated using the final image projection matrix and text projection matrix, respectively. The Hamming distance between the cross-modal hash codes is calculated and sorted in ascending order. The image-to-text retrieval results and text-to-image retrieval results are then output.

2. The online cross-modal fast retrieval method based on semantic enhancement feedback anchors as described in claim 1, characterized in that, The semantic enhancement that performs modal isolation on text and images respectively includes: For text branches, the original title, description, user tags or other text fields of the current sample are used as the only visible evidence input to the generation model. The generated results use structured fields to record summaries, entities, attributes, actions, relationships, scenarios, search keywords, uncertainties, confidence statements and short descriptions for use by the visual-language encoder. For the image branch, the image pixels of the current sample are used as the only visible evidence to input into the visual generation model. The generated result describes the visible objects, attributes, actions, spatial relationships and scenes with structured fields corresponding to the text branch. Establish cache keys for each of the two generation branches, determined by the sample identifier, enhancement stage, input content hash, prompt template hash, and model version identifier.

3. The online cross-modal fast retrieval method based on semantic enhancement feedback anchors as described in claim 1, characterized in that, The image composite features Textual composite features Represented as: ， The joint semantic vector is represented as: ,in, , , and These represent the original image representation, the image-enhanced description representation, the original text representation, and the text-enhanced description, respectively. and Representing the confidence level of image semantic consistency and the confidence level of text semantic consistency. This indicates L2 normalization.

4. The online cross-modal fast retrieval method based on semantic enhancement feedback anchors as described in claim 1, characterized in that, A label codebook is constructed based on the label matrix in the training partition. When the hash code length is sufficient and the number of categories is small, mutually orthogonal codewords are selected from the Hadamard matrix. When the number of categories is greater than the code length or the label correlation is complex, codewords are selected from the balanced candidate code pool, and the correlation penalty of category pairs is adjusted by the label co-occurrence similarity. Semantic atoms are basic semantic representation units determined by the label information, sample distribution, or semantic clustering results of training data, used to characterize representative category semantics or local semantic structures in the dataset.

5. The online cross-modal fast retrieval method based on semantic enhancement feedback anchors as described in claim 1, characterized in that, The method of alternately updating the binary hash code and bimodal hash function of the current data block using a tag codebook constrained by tag correlation, semantic atoms in the joint semantic space, atomic residual compensation, and bounded historical anchor memory includes: Read the t-th training data block and calculate the semantic atom soft assignment matrix based on the similarity between the current joint semantic vector and each semantic atom. ; Using the feedback matrix from the previous time step With semantic atomic soft allocation matrix Predict the structured quantization residual of the current block Generate the current tag code target based on the tag codebook. It also integrates the label sharing relationship and semantic similarity relationship between the current sample and the historical anchor point to form a current-historical similarity matrix. and effective historical constraint weights ; Based on structured quantitative residuals Construct a residual feedback term based on the current-historical similarity matrix. and effective historical constraint weights Construct a current-history similarity preservation term, and build the current block objective function with residual feedback term, image projection fitting term, text projection fitting term, projection matrix regularization term, tag code constraint term, current-history similarity preservation term, two-modal projection score consistency term, bit balance term, and bit decorrelation term; While maintaining binary and orthogonal constraints, the image projection matrix, text projection matrix, orthogonal rotation matrix, continuous latent representation, and current block binary hash code are alternately updated using a block coordinate descent method until the relative change of the objective function is less than a preset threshold or the maximum number of iterations is reached.

6. The online cross-modal fast retrieval method based on semantic enhancement feedback anchors as described in claim 1, characterized in that, The construction of bounded historical anchor memory includes: in a global memory approach, performing reservoir sampling on arriving samples: adding samples directly when the memory is not full; and sampling the first sample when the memory is full. One known sample The probability of replacing a historical position. In the hierarchical memory approach, the sample's atom is first determined by the largest soft-allocation atom, and then reservoir sampling is performed independently within the capacity of each atom to maintain coverage of different semantic regions.

7. The online cross-modal fast retrieval method based on semantic enhancement feedback anchors as described in claim 1, characterized in that, The overall objective function for the current data block is: ， ， ， ， ， in, This represents the quantized residual feedback term, used to... As a historical semantic residual offset, it is injected into the current quantization; Represents the image projection fitting term, the text projection fitting term, and the projection matrix regularization term; Indicates the tag code constraint item; Indicates the current-historical similarity preservation term; This represents the consistency term of the two-modal projection fractions; Represents bit balancing terms and bit decorrelation terms; Indicated by the tag codebook With the current block label matrix The target of the combined sample label code; This represents the binary hash code matrix of the current data block. , Control the quantization term and the atomic compensation intensity separately. and Controlling two-mode fitting, The regularization weights are the projection matrix weights. To control the target of the tag code, In order to maintain historical relations, To control the consistency of the two-modal projection scores, bit balancing and bit decorrelation terms are used to suppress the correlation between constant bits and different bits, respectively. and For image and text hash projection matrices, The length of the hash code. For continuous hidden variables in the current block, It is an orthogonal rotation matrix. and These are the composite feature matrices of the current block image and text, respectively.

8. A rapid online cross-modal retrieval system based on semantically enhanced feedback anchors, characterized in that, include: The query set and retrieval library building module is used to read multimodal data containing paired images, text, and optional multi-labels, and generate a unique identifier for each sample; The training data stream is divided according to the preset arrival order and data block size, and the query set and retrieval library are also divided. The isolated semantic enhancement module is used to perform modal isolation semantic enhancement on text and images respectively. It calculates image confidence and text confidence based on the consistency of the extracted original and enhanced representations of the two modalities. It constructs image composite features, text composite features and joint semantic vectors by combining the original and enhanced representations of the two modalities and the image confidence and text confidence respectively. The hash learning module is used to construct a label codebook and semantic atoms using only training data. During the arrival of the data stream, it uses the label codebook constrained by label relevance, semantic atoms in the joint semantic space, atomic residual compensation, and bounded history anchor memory to alternately update the binary hash code and bimodal hash function of the current data block until the relative change of the overall objective function of the data block is less than a preset threshold or the maximum number of iterations is reached, and outputs the image and text projection scores. The retrieval module is used to perform threshold calibration on the projection scores of images and text, generate binary hash codes for the query set and the retrieval library using the final image projection matrix and text projection matrix respectively, calculate the Hamming distance between the cross-modal hash codes and sort them in ascending order, and output the image-to-text retrieval results and the text-to-image retrieval results.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements the steps in the online cross-modal fast retrieval method based on semantically enhanced feedback anchors as described in any one of claims 1-7.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the online cross-modal fast retrieval method based on semantic enhancement feedback anchors as described in any one of claims 1-7.

Images