Kazakh image-text retrieval method based on transfer learning

Abstract

The invention belongs to the technical field of cross-modal retrieval, and discloses a Kazakh image-text retrieval method based on transfer learning, which comprises the following steps: acquiring and processing image-text training data, and pre-training an English source language encoder and an image encoder; constructing a Kazakh encoder based on the model parameters of the English source language encoder, and embedding a noise filtering module; based on image-text alignment knowledge and image-text training data between the English source language encoder and the image encoder, performing knowledge migration and training on the target language encoder to obtain a trained Kazakh encoder; and constructing a Kazakh image-text retrieval model based on the image encoder and the trained Kazakh encoder to realize a Kazakh image-text retrieval task. The model realizes alignment of a Kazakh text and an image in a cross-modal space based on a proposed triangular alignment knowledge migration strategy, the image-text retrieval capability of the model on Kazakh is enhanced, and meanwhile, the robustness of the model on translation noise is enhanced by adopting a noise filtering module.

Description

Technical Field

[0001] The present invention belongs to the technical field of cross-modal retrieval, and in particular relates to a Kazakh image-text retrieval method based on transfer learning. Background Art

[0002] As an interdisciplinary field of information science, computer science and linguistics, Kazakh image and text retrieval research faces many challenges. First of all, Kazakh belongs to the Altaic language family, and its grammatical structure, vocabulary composition and expression habits are unique, which makes it difficult to build accurate image understanding and text retrieval models. In addition, compared with resource-rich languages, Kazakh information resources are relatively scarce, and the lack of resources such as corpora and dictionaries limits the performance of Kazakh image and text retrieval systems. These challenges show that it is challenging to build a key image and text retrieval technology that can deeply understand Kazakh image and text data.

[0003] In recent years, multimodal representation learning of joint vision and language has promoted the development of pre-trained models such as Efficientclip, Clip-adapter, and Blip-2. In particular, the CLIP model, which aligns image and text features through comparative learning, has state-of-the-art performance on zero-shot vision-language tasks. Although these models have good generalization capabilities, training models such as CLIP requires a lot of data and computing resources. Large model training usually requires hundreds of millions of image-text pairs for training. Currently, image-text datasets only involve high-resource languages ​​such as English, Chinese, and Spanish, and lack image-text databases for low-resource languages ​​such as foreign Kazakh.

[0004] With the popularity of machine translation (MT), generating pseudo-visual and target language pairs through MT and establishing the correspondence between them has become a solution. Specifically, a large amount of visual data and translated target language image-text pairs (V+T) are used to pre-train the model, but this method ignores the impact of translation noise.

[0005] Although some progress has been made in the field of multilingual cross-modal retrieval, there are still some shortcomings in the image-text retrieval work for Kazakh: on the one hand, although the multilingual visual language model has achieved visual language alignment for multiple languages, due to the imbalance of different language data in the pre-training stage, the pre-training model tends to align visual text for resource-rich languages, and the alignment performance for low-resource languages ​​such as Kazakh is poor. On the other hand, existing research obtains non-English data through machine translation to meet the needs of cross-modal retrieval tasks. However, this type of method ignores the noise problem in machine translation. This translation bias will result in the inability to accurately align the text content with the corresponding image during the image-text alignment process, thereby reducing the final image-text retrieval effect. Summary of the invention

[0006] The purpose of the present invention is to provide a Kazakh image-text retrieval method based on transfer learning to solve the problems existing in the above-mentioned prior art.

[0007] To achieve the above object, the present invention provides a Kazakh image-text retrieval method based on transfer learning, comprising:

[0008] Acquire image-text training data, wherein the image-text training data includes text descriptions and corresponding images, wherein the text descriptions include source language titles and corresponding target language titles, wherein the source language titles are English titles, and the target language titles are Kazakh titles;

[0009] Building a target language encoder based on the model parameters of the English source language encoder, and embedding a noise filtering model in the target language encoder;

[0010] Based on the image-text alignment knowledge between the English source language encoder and the image encoder and the image-text training data, knowledge transfer and training are performed on the target language encoder to obtain a trained target language encoder;

[0011] A Kazakh image-text retrieval model is constructed based on the image encoder and the trained target language encoder, and Kazakh image-text retrieval is performed based on the Kazakh image-text retrieval model.

[0012] Optionally, the process of obtaining the target language title specifically includes:

[0013] The source language title is translated based on a machine translation method to obtain the target language title.

[0014] Optionally, the English source language encoder is constructed based on an XLM-RobertaBase model, and the image encoder is constructed based on an OpenCLIP ViT-B model.

[0015] Optionally, knowledge transfer and training of the target language encoder is performed based on the image-text alignment knowledge between the English source language encoder and the image encoder and the image-text training data, specifically including:

[0016] Inputting a source language into an English source language encoder to extract text features and obtain source language text features, inputting a corresponding image into the image encoder to extract image features and obtain image features, inputting a Kazakh text into the target language encoder, filtering the input Kazakh text through a noise filtering model to obtain filtered data, and performing text feature extraction on the filtered data to obtain target language text features;

[0017] Aligning the target language features output by the target language encoder with the image features output by the image encoder based on a contrastive learning strategy;

[0018] Aligning the target language features output by the target language encoder with the source language features output by the English source language encoder;

[0019] Based on the bidirectional KL divergence loss, the image-text alignment knowledge between the English source language encoder and the image encoder is transferred to the target language encoder to obtain a trained target language encoder.

[0020] Optionally, the processing process of the noise filtering model specifically includes:

[0021] Extracting correct semantic features of the source language in the English source language encoder;

[0022] The correct semantic features in Kazakh are screened based on the attention mechanism and the correct semantic features of the source language to obtain the filtered data.

[0023] Optionally, the training process of the Kazakh image-text retrieval model specifically includes:

[0024] The source language text is input into the English source language encoder for feature extraction to obtain source language text features, the Kazakh text is input into the target language encoder for feature filtering and extraction to obtain Kazakh text features, the image is input into the image encoder for image feature extraction to obtain corresponding image features, the cross-modal loss between the target language features and the image features is calculated, the cross-language loss between the source language features and the target language features is calculated, the bidirectional KL divergence loss between the three features is calculated, and finally the weighted sum of the three losses is taken as the final loss, and training is performed with the goal of minimizing the final loss to obtain a trained Kazakh image-text retrieval model.

[0025] The technical effects of the present invention are:

[0026] The present invention constructs a foreign Kazakh graphic and text dataset based on manual and machine translation, providing a data basis for the research of AI intelligent analysis related work in foreign Kazakh languages.

[0027] The present invention adopts a knowledge transfer method to extract the rich language-independent alignment knowledge of the pre-trained model in English, and transfers it to the Kazakh encoder to enhance its alignment capability with the visual encoder and improve the retrieval accuracy of the current image-text retrieval task in foreign Kazakh.

[0028] In the face of the translation noise problem, the present invention adopts a noise filtering module (NFM) based on the idea of ​​attention mechanism to filter Kazakh text with translation noise. Then, through cross-language alignment, the features of Kazakh text and English text are strengthened, and the robustness of the model to translation noise is enhanced. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0030] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0031] Figure 1 Schematic diagram of a Kazakh image-text retrieval method based on transfer learning in an embodiment of the present invention; wherein, Figure 1 (a) is a model structure diagram in an embodiment of the present invention, Figure 1 (b) is the alignment relationship in the triangular alignment knowledge transfer method in the embodiment of the present invention;

[0032] Figure 2 An example of translation noise (left) and a schematic diagram of the structure of a noise filtering module NFM in an embodiment of the present invention. DETAILED DESCRIPTION

[0033] Various exemplary embodiments of the present invention will now be described in detail. This detailed description should not be considered as limiting the present invention, but should be understood as a more detailed description of certain aspects, features, and embodiments of the present invention.

[0034] It should be understood that the terms described in the present invention are only for describing special embodiments and are not intended to limit the present invention. In addition, for the numerical range in the present invention, it should be understood that each intermediate value between the upper and lower limits of the scope is also specifically disclosed. Each smaller range between the intermediate value in any stated value or stated range and any other stated value or intermediate value in the described range is also included in the present invention. The upper and lower limits of these smaller ranges can be independently included or excluded in the scope.

[0035] Unless otherwise indicated, all technical and scientific terms used herein have the same meanings as those of ordinary skill in the art generally understood by the present invention. Although the present invention describes only preferred methods, any method similar or equivalent to that described herein may also be used in the implementation or testing of the present invention. All documents mentioned in this specification are incorporated by reference to disclose and describe the methods related to the documents. In the event of a conflict with any incorporated document, the content of this specification shall prevail.

[0036] It will be apparent to those skilled in the art that various modifications and variations may be made to the specific embodiments of the present invention description without departing from the scope or spirit of the present invention. Other embodiments derived from the present invention description will be apparent to those skilled in the art. The present application description and examples are exemplary only.

[0037] The words “include,” “including,” “have,” “contain,” etc. used in this article are open-ended terms, meaning including but not limited to.

[0038] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0039] Embodiment 1

[0040] like Figure 1 - Figure 2 As shown, in this embodiment, a Kazakh image-text retrieval method based on transfer learning is provided, including: obtaining image-text training data, the image-text training data including text descriptions and corresponding images, the text descriptions including source language titles and corresponding target language titles, the source language titles are English titles, and the target language titles are Kazakh titles; constructing a target language encoder based on the model parameters of the English source language encoder, and embedding a noise filtering model in the target language encoder; performing knowledge transfer and training on the target language encoder based on the image-text alignment knowledge between the English source language encoder and the image encoder and the image-text training data to obtain a trained target language encoder; constructing a Kazakh image-text retrieval model based on the image encoder and the trained target language encoder, and performing Kazakh image-text retrieval based on the Kazakh image-text retrieval model.

[0041] In this embodiment, the English source language encoder parameters are copied as the target language encoder parameters, and the noise filtering module is embedded into the target language encoder to obtain the target language encoder that needs to be trained. An initial image-text retrieval model is constructed based on the English source language encoder, the image encoder and the target language encoder, and the initial image-text retrieval model is trained based on the image-text training data.

[0042] A trained image-text retrieval model is obtained; and image-text retrieval in a target language is performed based on the trained image-text retrieval model.

[0043] In order to realize the image and text retrieval task for foreign Kazakh languages, this embodiment uses machine translation to construct a foreign Kazakh image and text dataset to solve the problem of insufficient data and provide data support for research.

[0044] In order to solve the problem that the current model does not work well on Kazakh due to data imbalance, a "triangle alignment knowledge transfer" method is proposed to enhance the retrieval effect of the current model on foreign Kazakh languages. The model's ability to understand Kazakh and images is realized through cross-modal alignment knowledge transfer. At the same time, language-independent alignment knowledge transfer is used to provide additional alignment knowledge for the alignment training process, thereby enhancing the cross-modal retrieval performance of the model. Considering the translation noise of the dataset, a noise filtering NFM module is designed and embedded in the Kazakh text encoder. The cross-language alignment knowledge transfer training module NFM is used to filter the translation noise and enhance the robustness of the model alignment translation noise.

[0045] The triangular alignment knowledge transfer method mainly includes three parts: Cross Modal Align Knowledge Transfer CMA-KT, Cross Language Align Knowledge Transfer CLA-KT and Language Independent Alignment Knowledge Transfer LIA-KT. The alignment knowledge in the visual language pre-training model OpenCLIP ViT-B / 32XLM-RobertaBase is transferred to the target language encoder to improve the alignment ability between the target language encoder and the image. The relationship between them is as follows: Figure 1 (b) At the same time, considering that the dataset based on machine translation has the problem of translation noise, before performing "triangle alignment knowledge transfer", a noise filtering module (Noise Filter Model-NFM) is designed in the target language encoder to filter translation noise and enhance the model's robustness to translation noise. The following are the details of the method:

[0046] set up It is a dataset containing annotated paired images and English, with a data size of N. During the training process, there are no manually annotated image and Kazakh sample pairs, so we can only rely on some external tools, such as machine translation or parallel corpus, to use machine translation to expand the training data D to Where S iThe source language is English, T i is the target Kazakh language, V i is the corresponding image. English will be referred to as the source language and Kazakh as the target language.

[0047] This example uses the pre-trained multilingual multimodal model OpenCLIP ViT-B / 32XLM-Roberta Base as the base. The model has been trained 12.8 billion times on the LAION-5B dataset with a batch size of 90k and has excellent performance in aligning English text and images. At the same time, a two-stream architecture is adopted, using XLM-RoBERTaBase and ViTBase as text and image encoders respectively, and the image encoder resolution is 32×32. They are used as the English source language encoder F S and image encoder F V , and initialize a text encoder with the same parameters as the target language encoder F T ,like Figure 1 (a). Where F S and F V Freeze, retain the original alignment knowledge, and train the target language encoder F T , making it better to align images on foreign Kazakh languages.

[0048] Figure 1 (a) is the model structure diagram of the Kazakh image-text retrieval algorithm, which mainly Figure 1 (b) The triangle alignment knowledge transfer method is trained. The structure diagram contains the English source language encoder, image encoder and target language encoder. Figure 1 (b) Shows the alignment relationship in the “triangular alignment knowledge transfer” method.

[0049] Figure 1 This is a schematic diagram of the Kazakh image-text retrieval method based on transfer learning. Figure (a) is a model structure diagram, including the English source language encoder F S 、Image encoder F V and the target language encoder F T , where snowflakes represent parameter freezing and do not participate in training, and flames represent parameters for training. And the corresponding source language features f generated by the encoder S , image features f V and the target language features f T , where K scm and K tcmThey are the source language cross-modal alignment knowledge and the target language cross-modal alignment knowledge. They are mainly trained by the triangular alignment knowledge transfer method shown in Figure (b). NFM is a noise filtering module, which is mainly embedded after the last Transformer layer of the target language encoder and before the Proj projection layer to filter the noise of the target language features. Figure (b) shows the alignment relationship in the triangular alignment knowledge transfer method, which mainly targets the source language features f output by the model. S , image features f V and the target language features f T Conduct alignment knowledge transfer training.

[0050] Figure 2 Translation noise example (left) and noise filtering module NFM (right). The left side is a translation noise example. The correct translation is indicated in blue, and the incorrect translation is indicated in orange. First, the semantics of the source language "poking at him" (poking him) changes to "οны ” (hitting the guitar). Secondly, the object of the action changes from “him” to “οны” (guitar), and the action changes from “poking” to “ ” (knock). Combined with the image, it can be seen that there is a description gap between the translated text and the image, so it is called pseudo alignment. The right side is the proposed noise filtering module. This method mainly uses the attention idea to filter the target language features based on the correct semantic features of the source language. It is mainly divided into two stages. Stage 1: Filter weight calculation, including similarity calculation (CosSim) and weight normalization (softmax). Stage 2: Use weights to perform weighted calculation. Filter translation noise.

[0051] Noise filtering module (NFM): Consider a dataset constructed using machine translation, where the alignment between the target language and the image is a pseudo alignment and there is translation noise. Inspired by the SGRAF work that uses the relationship between local and global alignments to suppress the interference of unimportant alignments, a noise filtering module is embedded in the target language encoder. Based on the idea of ​​the attention mechanism, the correct semantic features of the source language are used to filter the correct semantic features in the target language text, thereby reducing the impact of translation noise on the overall semantics in the alignment. The specific structure is as follows: Figure 2 shown.

[0052] Specifically, first, extract F T The last hidden state of the last layer Transformer module

[0053] As the target language context feature, we also extract F S The features in the last hidden state of the corresponding layer As the correct global semantic feature of this layer. Then calculate h T and The similarity weight w between them is calculated as follows:

[0054]

[0055] Where L represents the length of the target text.

[0056] Then, through h T and The similarity weight w between filters h T Relevant semantic features in translation to suppress translation noise:

[0057]

[0058] Get the filtered global target language features

[0059] Finally, the NFM-filtered Then pass through the target encoder F T The last layer Proj obtains the final feature f of the target language T , Proj is the last fully connected layer through which the feature passes, which mainly controls the final f T The feature dimension and f S The dimensions are the same.

[0060]

[0061] Through this module, the translation noise is filtered with the help of the correct semantic features of the source language, reducing the information bias caused by translation noise and enhancing the noise robustness of the model training data.

[0062] Triangular Alignment Knowledge Transfer: This method mainly consists of three parts: the first is Cross-modal Alignment Knowledge Transfer (CMA-KT), which trains the target language features and visual image features to align in the semantic space through contrastive learning (loss). The second is Cross-language Alignment Knowledge Transfer (CLA-KT), which aligns the target language features with the source language features to enhance the model's robustness in dealing with target language translation noise.

[0063] The third is image-text knowledge transfer (LIA-KT), which improves the model's cross-modal alignment ability for the target language Kazakh by extracting the alignment knowledge of the pre-trained model in English.

[0064] Cross-modal Alignment Knowledge Transfer (CMA-KT): Contrastive learning has been shown to be effective in unimodal and cross-modal representation learning. T and F VIn the alignment of modal shared semantic space, this embodiment adopts contrastive learning as the basic strategy. By shortening the feature distance of related sample pairs and expanding the feature distance of unrelated sample pairs, the model can have a clearer distinction between different samples, thereby improving the model's cross-modal alignment ability for Kazakh language and images.

[0065] In contrastive learning, model parameters are trained by making the features of paired samples close or separate. Specifically, consider a training batch containing N samples, where is the target language feature and image feature of the i-th sample pair. In addition, consider the negative samples within the batch, that is, for is a positive sample, all other f j (where j≠i) are all negative samples. Note: L ct Defined as image-to-text loss and text-to-image loss The contrast loss is expressed as follows:

[0066]

[0067] in

[0068]

[0069] exp(.) is an exponential function with the natural constant e as the base, and τ is the temperature parameter, which is used to control the smoothness of the distribution in the similarity calculation and adjust the sensitivity of the model.

[0070] Cross-language Alignment Knowledge Transfer (CLA-KT): Considering that there is translation noise in the target language text after translation, there is a deviation between the target language semantics and the source language semantics in the modal shared space. Sentence-level cross-lingual alignment is proposed to align the semantics of the target language and source language in a cross-modal shared space, thereby enhancing the robustness of the target language encoder to noise.

[0071] Directly use encoder F for source language text S Encode to obtain f S The target language text is obtained through the noise filtering module NFM Then through F T The last layer Proj is obtained, as shown in formula (4), and the final target language text feature is f T .

[0072] Computing cross-lingual sentence similarity scores using sentence-level representations

[0073]

[0074] Finally, for different languages, they are regarded as different modalities in the shared semantic space, and the symmetric InfoNCE loss function is used to optimize the target language encoder and learn discriminative cross-lingual features. It can be expressed as:

[0075]

[0076] Language Independent Alignment Knowledge Transfer (LIA-KT): Inspired by relational knowledge distillation (RKD), RKD transfers the relationship between data instances through distillation losses based on distance and angle to improve the performance of the student model. Different from RKD, this embodiment extracts the similarity between different instances in a batch as relevant knowledge, which is called Language Independent Alignment Knowledge (LIAK). The English source language encoder F is transferred using a bidirectional KL divergence loss. S The LIAK in the target language encoder F is transferred to the target language encoder F T It makes up for the shortcomings of the above-mentioned cross-modal alignment based on contrastive learning, thereby enhancing the model's ability to align the target language with the image.

[0077] Among them, the source language cross-modal alignment knowledge is represented as K scm , the target language cross-modal alignment knowledge is represented as K tcm , the calculation formula is as follows:

[0078]

[0079] sim(,) is the cosine similarity calculation formula.

[0080] In order to transfer the alignment knowledge of the pre-trained model to F T In this paper, a bidirectional KL divergence loss is designed to reduce the K tcm To K scm The distribution gap of F T Learn the language-independent alignment knowledge of the pre-trained model. The knowledge transfer loss is expressed as:

[0081]

[0082] Where KL(||) represents the Kullback-Leiber divergence, τ is the temperature parameter, which adjusts the smoothness of knowledge expression. Represents the transpose of the knowledge matrix. scm Knowledge transfer to K tcm , which can improve the alignment quality between visual and target language representations, thereby improving retrieval performance.

[0083] Training and reasoning (mainly explaining the total training loss and the main reasoning process): This embodiment trains the model by minimizing the combination of the above losses. The total loss function is defined as follows:

[0084] L=λ 1 L ct +λ 2 L cl +λ 3 L KL (14)

[0085] Among them, 1 ,λ 2 ,λ 3 It is a hyperparameter used to balance the importance of each loss, and the sum is 1.

[0086] After the model is trained, the model only retains the target language encoder parameters. In the inference phase, given a target language sentence as a query, the query and the image to be queried are subjected to feature extraction using the target language encoder and image encoder, and finally the cross-modal cosine similarity between it and the image to be queried is calculated, and then the candidate images are sorted in descending order as the final retrieval result. Image retrieval text tasks are also feasible.

[0087] This embodiment constructs a foreign Kazakh text and image dataset based on manual and machine translation, providing a data basis for research on AI intelligent analysis of foreign Kazakh languages.

[0088] This embodiment adopts a knowledge transfer method to extract the rich language-independent alignment knowledge of the pre-trained model in English, and transfers it to the Kazakh encoder to enhance its alignment capability with the visual encoder and improve the retrieval accuracy of the current image-text retrieval task in foreign Kazakh.

[0089] In this embodiment, in order to solve the translation noise problem, a noise filtering module (NFM) based on the attention mechanism is used to filter Kazakh text with translation noise. Then, through cross-language alignment, the features of Kazakh and English texts are strengthened to enhance the robustness of the model to translation noise.

[0090] The above is only a preferred specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A Kazakh image-text retrieval method based on transfer learning, characterized in that: include: Acquire image-text training data, wherein the image-text training data includes text descriptions and corresponding images, wherein the text descriptions include source language titles and corresponding target language titles, wherein the source language titles are English titles, and the target language titles are Kazakh titles; Building a target language encoder based on the model parameters of the English source language encoder, and embedding a noise filtering model in the target language encoder; Based on the image-text alignment knowledge between the English source language encoder and the image encoder and the image-text training data, knowledge transfer and training are performed on the target language encoder to obtain a trained target language encoder; A Kazakh image-text retrieval model is constructed based on the image encoder and the trained target language encoder, and Kazakh image-text retrieval is performed based on the Kazakh image-text retrieval model.

2. According to the Kazakh image-text retrieval method based on transfer learning according to claim 1, it is characterized in that: The process of obtaining the target language title specifically includes: The source language title is translated based on a machine translation method to obtain the target language title.

3. The Kazakh image-text retrieval method based on transfer learning according to claim 1, characterized in that: The English source language encoder is constructed based on the XLM-Roberta Base model, and the image encoder is constructed based on the OpenCLIP ViT-B model.

4. The Kazakh image-text retrieval method based on transfer learning according to claim 1, characterized in that: Based on the image-text alignment knowledge between the English source language encoder and the image encoder and the image-text training data, knowledge transfer and training are performed on the target language encoder, specifically including: Inputting a source language into an English source language encoder to extract text features and obtain source language text features, inputting a corresponding image into the image encoder to extract image features and obtain image features, inputting a Kazakh text into the target language encoder, filtering the input Kazakh text through a noise filtering model to obtain filtered data, and performing text feature extraction on the filtered data to obtain target language text features; Aligning the target language features output by the target language encoder with the image features output by the image encoder based on a contrastive learning strategy; Aligning the target language features output by the target language encoder with the source language features output by the English source language encoder; Based on the bidirectional KL divergence loss, the image-text alignment knowledge between the English source language encoder and the image encoder is transferred to the target language encoder to obtain a trained target language encoder.

5. The Kazakh image-text retrieval method based on transfer learning according to claim 4 is characterized in that: The processing process of the noise filtering model specifically includes: Extracting correct semantic features of the source language in the English source language encoder; The correct semantic features in Kazakh are screened based on the attention mechanism and the correct semantic features of the source language to obtain the filtered data.

6. The Kazakh image-text retrieval method based on transfer learning according to claim 1, characterized in that: The training process of the Kazakh image-text retrieval model specifically includes: The source language text is input into the English source language encoder for feature extraction to obtain source language text features, the Kazakh text is input into the target language encoder for feature filtering and extraction to obtain Kazakh text features, the image is input into the image encoder for image feature extraction to obtain corresponding image features, the cross-modal loss between the target language features and the image features is calculated, the cross-language loss between the source language features and the target language features is calculated, the bidirectional KL divergence loss between the three features is calculated, and finally the weighted sum of the three losses is taken as the final loss, and training is performed with the goal of minimizing the final loss to obtain a trained Kazakh image-text retrieval model.

Images