A robust cross-modal retrieval method based on consistent correction and mining

Abstract

The application provides a robust cross-modal retrieval method based on consistency correction and mining, relates to the technical field of cross-modal retrieval, and comprises the following steps: S1, obtaining a data set to be retrieved and performing data preprocessing on the data set; S2, training two separate neural network models A and B by using a cross-modal information noise contrast estimation loss function; S3, feeding the preprocessed data set into the model A and the model B respectively to perform collaborative division; S4, performing consistency correction and consistency mining on the divided data set; S5, repeatedly performing the steps S3-S4 to perform robust cross-modal retrieval training until the network A and the network B converge to obtain the trained network A and the trained network B; and S6, performing cross-modal retrieval by using the trained network A and the trained network B to obtain a retrieval result.

Description

Technical Field

[0001] This invention relates to the field of cross-modal retrieval technology, and more specifically, to a robust cross-modal retrieval method based on consistency correction and mining. Background Technology

[0002] Cross-modal retrieval aims to search for semantically related samples from different modalities. The key is to reduce the gap between features of different modalities with the same semantics and increase the similarity between features of different modalities with the same semantics. To this end, existing cross-modal retrieval methods mainly focus on in-depth research into the cross-modal consistency hidden in paired data (positive sample pairs), so that features of different modalities with the same semantics can be brought closer to each other in the feature space.

[0003] Existing cross-modal retrieval methods can be broadly categorized into two types: 1) Coarse-grained cross-modal retrieval methods. These methods typically employ different neural networks to identify features of specific modalities. For example, convolutional neural networks are used to acquire features of the image modality, while gated recurrent units are used to acquire features of the text modality. The distance between features of paired modalities is then reduced, while features of unpaired modalities (negative sample pairs) are pushed away. 2) Fine-grained cross-modal retrieval methods. These methods design different fine-grained approaches based on the characteristics of different modalities to focus on the details of different modalities and align these features from a global perspective. These methods employ various fine-grained consistency metrics, such as multi-level attention mechanisms and similarity graph mechanisms.

[0004] While the two types of methods mentioned above have achieved good performance, most existing methods experience performance degradation when encountering incorrectly associated data pairs. For example, text descriptions may not match image content, or audio may not match video.

[0005] Based on this, this application proposes a robust cross-modal retrieval method based on consistency correction and mining to solve the above problems. Summary of the Invention

[0006] The purpose of this invention is to provide a robust cross-modal retrieval method based on consistency correction and mining, which can achieve robust cross-modal retrieval of noise associations.

[0007] The technical solution of this invention is as follows:

[0008] In a first aspect, this application provides a robust cross-modal retrieval method based on consistency correction and mining, which includes the following steps:

[0009] S1. Obtain the dataset to be retrieved and perform data preprocessing on it;

[0010] S2. Two separate neural network models A and B are trained using the cross-modal information noise contrast estimation loss function;

[0011] S3. Feed the preprocessed dataset into Model A and Model B respectively for collaborative partitioning;

[0012] S4. Perform consistency correction and consistency mining on the partitioned dataset;

[0013] S5. Repeat steps S3 to S4 to perform robust cross-modal retrieval training until network A and network B converge to obtain the trained network A and network B.

[0014] S6. Use the trained networks A and B to perform cross-modal retrieval to obtain retrieval results.

[0015] Furthermore, in step S2, the calculation formulas used in training two separate neural network models A and B using the cross-modal information noise contrast estimation loss function include:

[0016]

[0017] Among them, l i Let exp(·) represent the loss formed by the i-th data pair in the network, τ represent the exponential function with the natural constant e as the base, and s represent the temperature coefficient. i，i s i，j and s j，i Both are the inner product of the feature vectors of the data points, and N represents the number of data pairs in a batch.

[0018] Furthermore, step S3 specifically includes:

[0019] The preprocessed datasets are fed into Model A and Model B respectively. The loss function values ​​calculated by Model A and Model B are modeled by a Gaussian mixture model composed of two Gaussian distributions to obtain the Gaussian distribution components.

[0020] Samples are selected from the dataset, and the dataset is divided into clean subsets, fuzzy subsets, and noisy subsets according to the posterior probability of the Gaussian distribution component with the smaller mean in each sample.

[0021] Furthermore, the aforementioned consistency correction includes:

[0022] The correspondence between data in the partitioned dataset is corrected based on the predictions of the correspondence between data in the dataset by Model A and Model B, and the posterior probabilities. Specifically, this includes:

[0023] For data pairs in the clean subset, the correspondence of the i-th data sample used to train network A is corrected as follows: The correspondence between the i-th pair of data samples used to train network B is corrected as follows: The specific calculation method is as follows:

[0024]

[0025]

[0026] in, This represents the correspondence between the i-th pair of data samples used to train network A after correction within the clean subset. Let y represent the correspondence between the i-th pair of data samples used to train network B after correction within the clean subset. i This represents the correspondence between the i-th pair of data samples used to train network A and network B in the dataset. These are all intermediate parameters. Let represent the posterior probability of model A for the i-th pair of samples. This represents the posterior probability derived by model B for the i-th pair of samples;

[0027] For data pairs in the fuzzy subset, the correspondence of the i-th data sample used to train network A is corrected as follows: The correspondence between the i-th pair of data samples used to train network B is corrected as follows: The specific calculation method is as follows:

[0028]

[0029]

[0030] in, This represents the correspondence between the i-th pair of data samples used to train network A after correction within the fuzzy subset. This represents the correspondence between the i-th pair of data samples used to train network B after correction within the fuzzy subset;

[0031] For data pairs in the noisy subset, the correspondence of the i-th data sample used to train network A is corrected as follows: The correspondence between the i-th pair of data samples used to train network B is corrected as follows: The specific calculation method is as follows:

[0032]

[0033]

[0034]

[0035]

[0036] in, This represents the correspondence between the i-th pair of data samples used to train network A after correction within the noise subset. Let l represent the correspondence between the i-th pair of data samples used to train network B after correction within the noise subset. i Let exp(·) represent the loss formed by the i-th data pair in the network, τ represent the exponential function with the natural constant e as the base, and s represent the temperature coefficient. A i，i s A i，j s A j，i s B i，i s B i，j and s B j，i All are inner products between the feature vectors of the data points, and N represents the number of data pairs in a batch.

[0037] Furthermore, the aforementioned consistency mining includes performing consistency mining on negative samples in the consistency-corrected dataset, and its calculation process includes:

[0038]

[0039]

[0040]

[0041] in, and Let y' and y' represent the consistency of all negative samples in all negative sample pairs in the current batch. i A This represents the correspondence between the i-th pair of data samples after correction in the previous step, used to train network A, y′. j A This represents the correspondence between the j-th data sample and the data sample used to train network A after being corrected in the previous step, s. A i， j 、 s A i，k and s A k，j All are inner products between the feature vectors of the data points, where N represents the number of data pairs in a batch, and β represents the threshold. 干净 N 模糊 and N 噪声 These represent the number of sample pairs in the clean subset, fuzzy subset, and noisy subset, respectively. and These represent the mean values ​​of the correspondence between the corrected data pairs.

[0042] Furthermore, in step S5, the aforementioned cross-modal retrieval training process includes:

[0043]

[0044] Where L is the training loss function, l i Let y′ represent the loss formed by the i-th pair of data in the network. i This represents the correspondence between the corrected data pairs. and This represents the consistency after threshold filtering, exp(·) represents the exponential function with the natural constant e as the base, τ represents the temperature coefficient, and s i，j s i，k and s k，j All are inner products between the feature vectors of the data points, and N represents the number of data pairs in a batch.

[0045] Furthermore, in step S6, the cross-modal retrieval method described above includes:

[0046] Similarity calculations were performed on the test data using trained networks A and B, respectively.

[0047] The outputs of the two networks are averaged and sorted to obtain the retrieval results.

[0048] Secondly, this application provides an electronic device, characterized in that it includes:

[0049] Memory, used to store one or more programs;

[0050] processor;

[0051] When one or more of the above programs are executed by the above processor, a robust cross-modal retrieval method based on consistency correction and mining, as described in any of the first aspects above, is implemented.

[0052] Thirdly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a robust cross-modal retrieval method based on consistency correction and mining as described in any of the first aspects above.

[0053] Compared with the prior art, the present invention has at least the following advantages or beneficial effects:

[0054] (1) The present invention provides a robust cross-modal retrieval method based on consistency correction and mining, which adopts a collaborative learning approach to detect and correct the correspondence of erroneous positive sample pairs (i.e. consistency correction), laying the foundation for exploring potential consistency.

[0055] (2) This invention explores and utilizes different potential consistency (i.e. consistency mining) to achieve robust cross-modal retrieval, which can find matching data for data pairs that should be discarded, making full use of the internal information of the data, thereby improving the performance of cross-modal retrieval. Attached Figure Description

[0056] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0057] Figure 1 This is a flowchart illustrating the steps of a robust cross-modal retrieval method based on consistency correction and mining according to the present invention.

[0058] Figure 2 This is a schematic block diagram of an electronic device.

[0059] Icons: 101, memory; 102, processor; 103, communication interface. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0061] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0062] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0063] It should be noted that, in this document, the term "comprising" or any other variation thereof is intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0064] In the description of this application, it should also be noted that, unless otherwise explicitly specified and limited, the terms "set" and "connection" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0065] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the various embodiments and features described below can be combined with each other.

[0066] Example 1

[0067] Please see Figure 1 , Figure 1 The diagram shows the steps of a robust cross-modal retrieval method based on consistency correction and mining provided in an embodiment of this application.

[0068] In a first aspect, this application provides a robust cross-modal retrieval method based on consistency correction and mining, which includes the following steps:

[0069] S1. Obtain the dataset to be retrieved and perform data preprocessing on it;

[0070] S2. Two separate neural network models A and B are trained using the cross-modal information noise contrast estimation loss function;

[0071] S3. Feed the preprocessed dataset into Model A and Model B respectively for collaborative partitioning;

[0072] S4. Perform consistency correction and consistency mining on the partitioned dataset;

[0073] S5. Repeat steps S3 to S4 to perform robust cross-modal retrieval training until network A and network B converge to obtain the trained network A and network B.

[0074] S6. Use the trained networks A and B to perform cross-modal retrieval to obtain retrieval results.

[0075] In a preferred embodiment, the calculation formula used in step S2 during the training of two separate neural network models A and B using the cross-modal information noise contrastive estimation loss function includes:

[0076]

[0077] Among them, l i Let exp(·) represent the loss formed by the i-th data pair in the network, τ represent the exponential function with the natural constant e as the base, and s represent the temperature coefficient. i，i s i，j and s j，i Both are the inner product of the feature vectors of the data points, and N represents the number of data pairs in a batch.

[0078] As a preferred embodiment, step S3 specifically includes:

[0079] The preprocessed datasets are fed into Model A and Model B respectively. The loss function values ​​calculated by Model A and Model B are modeled by a Gaussian mixture model composed of two Gaussian distributions to obtain the Gaussian distribution components.

[0080] Samples are selected from the dataset, and the dataset is divided into clean subsets, fuzzy subsets, and noisy subsets according to the posterior probability of the Gaussian distribution component with the smaller mean in each sample.

[0081] Specifically, let the posterior probability of model A for the i-th pair of samples be... The posterior probability derived by model B for the i-th pair of samples is when and When, this data pair belongs to a clean subset; when and At this point, the data pair belongs to the noisy subset; the remaining data pairs belong to the fuzzy subset. After the dataset is collaboratively partitioned, network A and network B will perform optimization alternately.

[0082] As a preferred implementation method, the consistency correction includes:

[0083] The correspondence between data in the partitioned dataset is corrected based on the predictions of the correspondence between data in the dataset by Model A and Model B, and the posterior probabilities. Specifically, this includes:

[0084] For data pairs in the clean subset, the correspondence of the i-th data sample used to train network A is corrected as follows: The correspondence between the i-th pair of data samples used to train network B is corrected as follows: The specific calculation method is as follows:

[0085]

[0086]

[0087] in, This represents the correspondence between the i-th pair of data samples used to train network A after correction within the clean subset. Let y represent the correspondence between the i-th pair of data samples used to train network B after correction within the clean subset. i This represents the correspondence between the i-th pair of data samples used to train network A and network B in the dataset. These are all intermediate parameters. Let represent the posterior probability of model A for the i-th pair of samples. This represents the posterior probability derived by model B for the i-th pair of samples;

[0088] For data pairs in the fuzzy subset, the correspondence of the i-th data sample used to train network A is corrected as follows: The correspondence between the i-th pair of data samples used to train network B is corrected as follows: The specific calculation method is as follows:

[0089]

[0090]

[0091] in, This represents the correspondence between the i-th pair of data samples used to train network A after correction within the fuzzy subset. This represents the correspondence between the i-th pair of data samples used to train network B after correction within the fuzzy subset;

[0092] For data pairs in the noisy subset, the correspondence of the i-th data sample used to train network A is corrected as follows: The correspondence between the i-th pair of data samples used to train network B is corrected as follows: The specific calculation method is as follows:

[0093]

[0094]

[0095]

[0096]

[0097] in, This represents the correspondence between the i-th pair of data samples used to train network A after correction within the noise subset. Let l represent the correspondence between the i-th pair of data samples used to train network B after correction within the noise subset. i Let exp(·) represent the loss formed by the i-th data pair in the network, τ represent the exponential function with the natural constant e as the base, and s represent the temperature coefficient. A i，i s A i，j s A j，i s B i，i s B i，j and s B j，i All are inner products between the feature vectors of the data points, and N represents the number of data pairs in a batch.

[0098] As a preferred implementation, consistency mining includes performing consistency mining on negative samples in the consistency-corrected dataset, and the calculation process includes:

[0099]

[0100]

[0101]

[0102] in, and Let y' and y' represent the consistency of all negative samples in all negative sample pairs in the current batch. i A This represents the correspondence between the i-th pair of data samples after correction in the previous step, used to train network A, y′. j A This represents the correspondence between the j-th data sample and the data sample used to train network A after being corrected in the previous step, s. A i，j s A i，k and s A k，j All are inner products between the feature vectors of the data points, where N represents the number of data pairs in a batch, and β represents the threshold. 干净 N 模糊 and N 噪声 These represent the number of sample pairs in the clean subset, fuzzy subset, and noisy subset, respectively. and These represent the mean values ​​of the correspondence between the corrected data pairs.

[0103] In a preferred implementation, step S5, the cross-modal retrieval training process includes:

[0104]

[0105] Where L is the training loss function, l i Let y′ represent the loss formed by the i-th pair of data in the network. i This represents the correspondence between the corrected data pairs. and This represents the consistency after threshold filtering, exp(·) represents the exponential function with the natural constant e as the base, τ represents the temperature coefficient, and s i，j s i，k and s k，j All are inner products between the feature vectors of the data points, and N represents the number of data pairs in a batch.

[0106] In a preferred embodiment, step S6 includes the following method for cross-modal retrieval:

[0107] Similarity calculations were performed on the test data using trained networks A and B, respectively.

[0108] The outputs of the two networks are averaged and sorted to obtain the retrieval results.

[0109] Specifically, the cross-modal retrieval process is as follows:

[0110] Taking cross-modal image-text retrieval as an example, two networks, Network A and Network B, are trained. First, the two networks are used to extract features from the input images (N images) and text (M sentences, where M and N are not necessarily equal), respectively, to obtain the corresponding image feature vectors I1, ..., I2. N and text feature vectors T1, ..., T M Then, the similarity between image and text features is calculated pairwise using the vector inner product, resulting in two different similarity matrices S. A and S B Then, add the two matrices together and take the average to obtain... Then, based on the retrieval needs, the bow (which stores all similarity values) is processed. For example, if a text-to-image retrieval is needed, the top 1 (R@1), top 5 (R@5), and top 10 (R@10) images most similar to each text are found. Then, we check if the image we want is among them. If it is, the count is incremented by 1; if not, it is not counted. Then, by dividing by the total number of tests, we can get the values ​​of R@1, R@5, and R@10.

[0111] Example 2

[0112] To further illustrate the present invention, this embodiment 2 is described in the following four parts, and experiments are conducted using three image and text datasets to verify the effectiveness of the method:

[0113] We acquired the dataset and performed data preprocessing: For all given image-text pairs, we used a pre-trained Faster-RCNN neural network to extract region features from 36 images. Each region was encoded as a 2048-dimensional feature vector, with a 36*2048 matrix representing the features of one image. For the input text, we obtained the encoding of each word as a feature description of the text using word embedding, with a maximum word count (dictionary size) of 300.

[0114] Network structure: For two neural network models A and B, it is necessary to receive image features and text features, project them into the same common space, and calculate the similarity between the input image features and text features.

[0115] Experimental Procedure: 1) Prepare the dataset to be retrieved; 2) Preprocess the data; 3) Feed the processed data into Model A and Model B for individual network warm-up; 4) Model A and Model B collaboratively partition the dataset; 5) Correct for consistency in the input data; 6) Mine for consistency in the input data; 7) Robust cross-modal retrieval training, and repeat steps 4) to 7) until Network A and Network B converge; 8) Calculate the similarity of the test data using Network A and Network B respectively, average the results of the two networks, and then sort the results using the similarity to obtain the retrieval results.

[0116] Experimental metrics: R@1 and R@10. R@K is defined as recall, which is the percentage of correctly retrieved samples among the top K samples in the retrieved data. The higher the value, the better the performance.

[0117] Experiment 1

[0118] Conceptual Caption 3M: This is a large-scale network dataset consisting of approximately 3.3 million image-text pairs. In our experiments, we used a randomly selected subset, Conceptual Caption 152K, for evaluation. Conceptual Caption 152K contains 150,000 pairs for training, 1,000 pairs for debugging, and another 1,000 pairs for testing. This dataset itself contains 3% to 20% noisy data. The experimental results are as follows:

[0119] Conceptual Caption 152K Experiment Results

[0120]

[0121] As can be seen from the table, for data pairs corresponding to noise in real-world environments, the present invention is almost on par with or significantly improves upon other methods. This indicates that in practical applications, the present invention can effectively alleviate the performance degradation caused by data pairs corresponding to noise, and compared to two other methods with the same purpose, the present invention demonstrates certain superiority in performance.

[0122] [Experiment 2]

[0123] Flickr30K: This dataset contains 31,000 images collected from the Flickr website, each with five text descriptions, all manually annotated, resulting in a total of 155,000 data pairs. We selected 5,000 data pairs for debugging, 5,000 for testing, and the remainder (145,000 data pairs) for training. Since this dataset itself does not contain noisy data pairs, to test the robustness of this invention to noisy data pairs, a portion of the training data was randomly shuffled at proportions of 20%, 40%, 60%, and 80%. The experimental results are as follows:

[0124] Flickr 30K

[0125]

[0126] As can be seen from the experimental results table, this method significantly improves upon other methods across all metrics for different proportions of noise data. This indicates that in practical applications, even with unknown noise rates, this invention can adapt well to training data and improve the robustness of the retrieval model. Compared to two other methods with the same objective, this method demonstrates certain performance advantages.

[0127] Experiment 3

[0128] Microsoft COCO: This dataset contains 123,287 images, each with 5 manually annotated text descriptions, totaling 616,435 image-text pairs. In this experiment, 566,435 pairs were used for training, 25,000 pairs for performance testing, and the remaining data for debugging. Since this dataset itself does not contain noisy data, to test the robustness of this invention to noisy data, a portion of the training data was randomly shuffled at proportions of 20%, 40%, 60%, and 80%. The experimental results are as follows:

[0129] MicroSoftCOCO

[0130]

[0131]

[0132] As can be seen from the table, for larger datasets and under different noise levels, this method shows significant improvements over other methods across almost all metrics. This demonstrates that, in practical applications, this invention can adapt to big data environments and environments with unknown noise levels, verifying its robustness. Compared to two other methods with the same objective, this method exhibits superior performance, proving that it can better utilize the internal information of the data and is more accurate in correcting and mining consistency.

[0133] Example 3

[0134] Please see Figure 2 , Figure 2 This is a schematic structural block diagram of an electronic device provided in Embodiment 3 of this application.

[0135] An electronic device includes a memory 101, a processor 102, and a communication interface 103. The memory 101, processor 102, and communication interface 103 are electrically connected directly or indirectly to enable data transmission or interaction. For example, these components can be electrically connected to each other via one or more communication buses or signal lines. The memory 101 can be used to store software programs and modules. The processor 102 executes the software programs and modules stored in the memory 101 to perform various functional applications and data processing. The communication interface 103 can be used for signaling or data communication with other node devices.

[0136] The memory 101 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0137] The processor 102 can be an integrated circuit chip with signal processing capabilities. The processor 102 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0138] It is understood that the structure shown in the figure is for illustrative purposes only. A robust cross-modal retrieval method based on consistency correction and mining may include more or fewer components than those shown in the figure, or have a different configuration. The components shown in the figure can be implemented in hardware, software, or a combination thereof.

[0139] In the embodiments provided in this application, it should be understood that the disclosed methods can also be implemented in other ways. The embodiments described above are merely illustrative. For example, the flowcharts or block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of methods and computer program products according to various embodiments of this application. In this regard, each block in the flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0140] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0141] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0142] In summary, the robust cross-modal retrieval method based on consistency correction and mining provided in this application employs collaborative learning to detect and correct the correspondence of erroneous positive sample pairs, laying the foundation for exploring potential consistency. At the same time, by utilizing different potential consistency (i.e., consistency mining), robust cross-modal retrieval is achieved, which can find paired data for data pairs that should be discarded, making full use of the internal information of the data, thereby improving the performance of cross-modal retrieval.

[0143] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

[0144] It will be apparent to those skilled in the art that this application is not limited to the details of the exemplary embodiments described above, and that this application can be implemented in other specific forms without departing from the spirit or essential characteristics of this application. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of this application is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within this application. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A robust cross-modal retrieval method based on consistency correction and mining, characterized in that, Includes the following steps: S1. Obtain the dataset to be retrieved and perform data preprocessing on it; S2. Two separate neural network models A and B are trained using the cross-modal information noise contrast estimation loss function; S3. Feed the preprocessed dataset into Model A and Model B respectively for collaborative partitioning; S4. Perform consistency correction and consistency mining on the partitioned dataset; S5. Repeat steps S3 to S4 to perform robust cross-modal retrieval training until model A and model B converge to obtain the trained model A and model B. S6. Use the trained model A and model B to perform cross-modal retrieval to obtain retrieval results; In step S2, the calculation formula used in training two separate neural network models A and B using the cross-modal information noise contrast estimation loss function is... include: ， in, Indicates the first Regarding the loss of data to the network, Represented by natural constant An exponential function with base 1. , , and Both are the inner product of the feature vectors of the data points. This represents the number of data logarithms in a batch. In step S5, the cross-modal retrieval training process includes: ， in, To train the loss function, Indicates the first Regarding the loss of data to the network, This represents the correspondence between the corrected data pairs. and This indicates consistency after threshold filtering. Represented by natural constant An exponential function with base 1. Indicates the temperature coefficient. , and All are the inner products between the feature vectors of the data points. This represents the number of data logs in a batch.

2. The robust cross-modal retrieval method based on consistency correction and mining as described in claim 1, characterized in that, Step S3 specifically includes: The preprocessed datasets are fed into Model A and Model B respectively. The loss function values ​​calculated by Model A and Model B are modeled by a Gaussian mixture model composed of two Gaussian distributions to obtain the Gaussian distribution components. Samples are selected from the dataset, and the dataset is divided into clean subsets, fuzzy subsets, and noisy subsets according to the posterior probability of the Gaussian distribution component with the smaller mean in each sample.

3. The robust cross-modal retrieval method based on consistency correction and mining as described in claim 2, characterized in that, The consistency correction includes: The correspondence between data in the partitioned dataset is corrected based on the predictions of the correspondence between data in the dataset by Model A and Model B, and the posterior probabilities. Specifically, this includes: For the data pairs in the clean subset, the first one used to train model A The correspondence between the data samples was corrected to: The first one used to train model B The correspondence between the data samples was corrected to: The specific calculation method is as follows: in, The clean subset represents the corrected version used for training model A. The correspondence between data samples, This represents the corrected version of the first subset used for training model B within the clean subset. The correspondence between data samples, Represents the dataset used to train model A and model B. The correspondence between data samples, These are all intermediate parameters. Representing model A for the first The posterior probability derived from the sample. Model B represents the first... The posterior probability derived from the sample; For the data pairs in the fuzzy subset, the first one used to train model A The correspondence between the data samples was corrected to: The first one used to train model B The correspondence between the data samples was corrected to: The specific calculation method is as follows: ， ， in, The fuzzy subset represents the modified version used for training model A. The correspondence between data samples, The fuzzy subset represents the corrected version used for training model B. The correspondence between data samples; For the data pairs in the noise subset, the first one used to train model A The correspondence between the data samples was corrected to: The first one used to train model B The correspondence between the data samples was corrected to: The specific calculation method is as follows: ， ， ， ， in, The corrected version of the noise subset used to train model A represents the first... The correspondence between data samples, The corrected version of the noise subset used to train model B represents the first... The correspondence between data samples, Represented by natural constant An exponential function with base 1. Indicates the temperature coefficient. , , , , and All are the inner products between the feature vectors of the data points. This represents the number of data logs in a batch.

4. The robust cross-modal retrieval method based on consistency correction and mining as described in claim 3, characterized in that, The consistency mining includes performing consistency mining on negative samples in the consistency-corrected dataset, and the calculation process includes: ， in, and These represent the consistency of all negative samples in the row and column negative sample pairs of the current batch, respectively. In the previous step, the first The correspondence between the corrected data samples and those used to train model A. In the previous step, the first The correspondence between the corrected data samples and those used to train model A. , and All are the inner products between the feature vectors of the data points. This represents the number of data logarithms in a batch. Indicates the threshold. , and These represent the number of sample pairs in the clean subset, fuzzy subset, and noisy subset, respectively. , and These represent the mean values ​​of the correspondence between the corrected data pairs.

5. A robust cross-modal retrieval method based on consistency correction and mining as described in claim 1, characterized in that, In step S6, the cross-modal retrieval method includes: The similarity between the trained model A and model B and the test data is calculated. The outputs of the two networks are averaged and sorted to obtain the retrieval results.

6. An electronic device, characterized in that, include: Memory, used to store one or more programs; processor; When the processor executes the one or more programs, it implements a robust cross-modal retrieval method based on consistency correction and mining as described in any one of claims 1-5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements a robust cross-modal retrieval method based on consistency correction and mining as described in any one of claims 1-5.

Images