A target level cross-modal image-text retrieval method and a storage medium

Abstract

The application discloses a target-level cross-modal image-text retrieval method and a storage medium. The method constructs image-target information paired data, integrates the position information of the target into the text title features in the feature extraction process, effectively learns the correlation between the target and the text in the image, and improves the accuracy and interpretability of the image-text retrieval. The cross-modal gap distribution is added to the image features, and the similarity of the image features and the target features is maximized by combining the gap cosine loss function, so that the modal gap problem of the image-text data is solved. The secondary training module is used for further training of the error retrieval data, so that a target-level cross-modal image-text retrieval model with complete training is obtained, and the performance of the model is further improved.

Description

Technical Field

[0001] This invention relates to the field of computer application technology, specifically to a target-level cross-modal image and text retrieval method and storage medium. Background Technology

[0002] Cross-modal image-text retrieval is the process of automatically retrieving data of different modalities from image and text data using computers. It includes image-to-text (I2T) and text-to-image (T2I) tasks. Due to the differences in the underlying feature spaces of images and text, measuring the content similarity between different modalities is a highly challenging task. Therefore, reducing the heterogeneity gap between image and text data while preserving effective contrastive features is a key issue in solving cross-modal retrieval problems.

[0003] Current related technologies mainly involve processing images and text at the feature level to align image and text features. Patent CN117520590B discloses a cross-modal image-text retrieval method for oceanographic data: First, a global similarity measurement module performs similarity clustering on global image features and global text features respectively, extracting reconstructed global image features and reconstructed global text features. Then, a multi-layer guidance module fuses and reconstructs local image features, local text features, reconstructed global image features, and reconstructed global text features to obtain aligned image features and aligned text features with multi-modal information, thus aligning information between the image and text modalities. Patent CN117648459B discloses a cross-modal image-text retrieval method for highly similar oceanographic remote sensing data: First, image and text features are extracted; then, a cross-modal retrieval model is trained, including a fine-grained sparse representation module, a cross-modal information bias calculation module, and a semantic perception loss module; finally, the image or text to be retrieved is input into the trained cross-modal retrieval model to obtain the corresponding retrieval results.

[0004] However, the above-mentioned existing technologies still have the following shortcomings: (1) They do not consider the specific location information of the target in the image, and cannot accurately explain the relationship between the image and the text, which seriously affects the accuracy and interpretability of image and text retrieval; (2) Simply calculating the semantic similarity of paired samples by loss function cannot effectively solve the modal gap problem of image and text data; (3) They do not consider the improvement of image and text retrieval model performance at the data level. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a target-level cross-modal image and text retrieval method, specifically embodied in:

[0006] (1) Obtain a training sample set of N sets of image-target information. Where ν iIt is an image sample. It is a set of targets, each set of targets including s groups of text titles t j - Set of two-dimensional point coordinates Each set of two-dimensional point coordinates contains m two-dimensional points, x k Represents the x-coordinate, y k Represents the vertical axis.

[0007] (2) Construct a target-level cross-modal image and text retrieval model and train the model using the training sample set.

[0008] The construction of the target-level cross-modal image and text retrieval model includes the following steps:

[0009] S1: Use a two-dimensional point position mapping function to map the position information of each target to α(ψ). j The text title is encoded using a text encoder to obtain the feature η(t). j ).

[0010] S2: Concatenate the location information mapping and text title features, and input them into the target feature extractor τ(·,·) for feature learning to obtain the target features. Target features The formula is as follows:

[0011] S3: Use a visual encoder to learn image features and obtain image features.

[0012] S4: Add the cross-modal gap distribution p to the image features The final image feature representation is v i ′,

[0013] Final image feature representation v i The formula is as follows:

[0014] S5: Maximize the final image feature v using the gap cosine loss function. i ′ and target features Based on the similarity, a target-level cross-modal image and text retrieval model is trained and obtained.

[0015] The gap cosine loss function is as follows:

[0016] Preferably, the performance of the model can be evaluated using a validation sample set, and a secondary training module can be constructed to further improve the model's performance. The construction of the secondary training module includes the following steps:

[0017] The performance of the model on two retrieval tasks was evaluated using a validation sample set, including the image retrieval text title task (I2T) and the text title retrieval image task (T2I).

[0018] The erroneously retrieved images from the I2T task are divided into a first image set and a second image set. Multiple target sets matching the first image set are obtained and re-added to the model for training.

[0019] The erroneously retrieved text titles of the T2I task are divided into a first text title set and a second text title set. Target location information is added to each text title in the first text title set to form a first target set. Multiple image sets matching the first target set are obtained and re-added to the model for training.

[0020] The model was validated using a second image set and a second text title set, resulting in a fully trained target-level cross-modal image and text retrieval model.

[0021] (3) Input the image or text title to be retrieved into the fully trained target-level cross-modal image and text retrieval model, and the model returns the cross-modal retrieval results.

[0022] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described cross-modal image and text retrieval method.

[0023] Technical features and beneficial effects of the present invention:

[0024] A data set of image-target information pairs was constructed, and the target's location information was integrated into the text title features during feature extraction. This effectively learned the association between the target and text in the image, improving the accuracy and interpretability of image-text retrieval. The modal gap problem of image-text data was solved by adding the cross-modal gap distribution to image features and combining it with the gap cosine loss function to maximize the similarity between image features and target features. Furthermore, a secondary training module was used to further train the model on erroneous retrieval data, resulting in a fully trained target-level cross-modal image-text retrieval model, further improving the model's performance. Attached Figure Description

[0025] Figure 1 This is a flowchart of the present invention;

[0026] Figure 2 This is a flowchart of the model training, model validation, and model testing process in the embodiment. Detailed Implementation

[0027] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments, so as to facilitate a better understanding of the technical solution of the present invention.

[0028] like Figure 1 As shown, the present invention mainly includes the following steps:

[0029] (1) Obtain a training sample set of N sets of image-target information. Where ν i It is an image sample. It is a set of targets, each set of targets including s groups of text titles t j - Set of two-dimensional point coordinates Each set of two-dimensional point coordinates contains m two-dimensional points, x k Represents the x-coordinate, y k Represents the vertical axis.

[0030] (2) A target-level cross-modal image and text retrieval model is obtained by training the training sample set.

[0031] (3) The performance of the model is evaluated using the validation sample set, and a secondary training module is constructed to further improve the performance of the model.

[0032] (4) Input the image or text title to be retrieved into a fully trained target-level cross-modal image and text retrieval model, and the model returns the cross-modal retrieval results.

[0033] In this embodiment, the model training process of the target-level cross-modal image and text retrieval method is as follows:

[0034] Obtain the training sample set of N sets of images-target set. Where ν i It is an image sample. It is a set of targets, each set of targets including s groups of text titles t j - Set of two-dimensional point coordinates Each set of two-dimensional point coordinates contains m two-dimensional points, x k Represents the x-coordinate, y k Represents the vertical axis.

[0035] The position information of each target is mapped to α(ψ) using a two-dimensional point position mapping function. j The text title is encoded using a text encoder to obtain the feature η(t). j ).

[0036] The location information and text title features are concatenated and input into the target feature extractor τ(·,·) for feature learning to obtain the target features. Target features The formula is as follows:

[0037] Image features are obtained by using a visual encoder to learn image features.

[0038] Add the cross-modal gap distribution p to image features Obtain the final image feature representation v i ′;

[0039] Final image feature representation v i The formula is as follows:

[0040] Maximize the final image feature v using the chasm cosine loss function. i ′ and target features Based on the similarity, a target-level cross-modal image and text retrieval model is trained and obtained.

[0041] The gap cosine loss function is as follows:

[0042] In this embodiment, the model validation process for the target-level cross-modal image and text retrieval method is as follows:

[0043] The performance of the model on two retrieval tasks was evaluated using a validation sample set, including the image retrieval text title task (I2T) and the text title retrieval image task (T2I).

[0044] The erroneously retrieved images from the I2T task are divided into a first image set and a second image set. Multiple target sets matching the first image set are obtained and re-added to the model for training.

[0045] The erroneously retrieved text titles of the T2I task are divided into a first text title set and a second text title set. Target location information is added to each text title in the first text title set to form a first target set. Multiple image sets matching the first target set are obtained and re-added to the model for training.

[0046] The model was validated using a second image set and a second text title set, resulting in a fully trained target-level cross-modal image and text retrieval model.

[0047] In this embodiment, the model testing process for a target-level cross-modal image and text retrieval method is as follows:

[0048] The image or text title to be retrieved is input into a well-trained target-level cross-modal image and text retrieval model, and the model returns cross-modal retrieval results.

Claims

1. A target level cross-modal image-text retrieval method, characterized in that, The method comprises: A training sample set of N groups of image-target information is acquired wherein v i is an image sample, o i is a target information set, Each target information set comprises s groups of text titles t j - a set of two-dimensional point coordinate positions ψ j , Each set of two-dimensional point coordinate positions ψ j contains m two-dimensional points, x k denotes the abscissa, y k denotes the ordinate; constructing a target-level cross-modal image-text retrieval model, training the target-level cross-modal image-text retrieval model using a training sample set; inputting an image or text title to be retrieved into the trained target-level cross-modal image-text retrieval model, and returning a cross-modal retrieval result by the model; the construction of the target-level cross-modal image-text retrieval model comprises the following steps: (1) using a two-dimensional point position mapping function to map the position information of each target to α(ψ j ), using a text encoder to encode the text title to obtain a feature η(t j ); (2) The position information mapping and the text title features are connected and input into a target feature extractor τ(·,·) for feature learning to obtain target features (3) using a visual encoder to learn features of the image to obtain image features ; (4) adding the cross-modal gap distribution p to the image features The final image feature representation is (5) adopting a gap cosine loss function maximizing similarity of final image features v i and target features to train and obtain a target-level cross-modal image-text retrieval model.

2. The method of claim 1, wherein, The method further comprises evaluating the performance of the target-level cross-modal image-text retrieval model using a validation sample set, and constructing a secondary training module to further improve the performance of the model; the construction of the secondary training module comprises the following steps: (1) evaluating the performance of the model on two retrieval tasks, including an image retrieval text title task I2T and a text title retrieval image task T2I, using a validation sample set; (2) dividing the incorrect retrieval images of the I2T task into a first image set and a second image set, obtaining a plurality of target sets matched with the first image set, and rejoining the model for training; (3) dividing the incorrect retrieval text titles of the T2I task into a first text title set and a second text title set, adding target position information to each text title of the first text title set to form a first target set, obtaining a plurality of image sets matched with the first target set, and rejoining the model for training; (4) using the second image set and the second text title set to verify the model, and obtaining a trained target-level cross-modal image-text retrieval model.

3. A non-transitory computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the steps of the target-level cross-modal image-text retrieval method of claim 1 or 2.

Images