A picture-text cross-modal retrieval network training method, application method and electronic equipment

Abstract

The application relates to a kind of text-image cross-modal retrieval network training method, application method and electronic equipment, comprising: obtaining text-image training dataset;Text-image cross-modal retrieval network model consisting of image embedder, text embedder and text-image combination module is constructed;Text-image training dataset is used as the input of the text-image cross-modal retrieval network, sequentially according to the image invariant while text mask modeling operation, text invariant while image mask modeling operation, image text matching operation, semantic alignment processing operation and iterative training text-image cross-modal retrieval network, and the text-image cross-modal retrieval network that is trained is obtained.This application represents semantic information as dense low latitude real value vector based on image embedder, text embedder and text-image combination module, extracts fine-grained semantic information in data;And through image text matching operation and semantic alignment processing operation, the semantic connection of entity and relationship is calculated, and the semantic alignment problem on the entity and relationship level between modal data is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of information retrieval, in particular to a kind of picture-text cross-modal retrieval network training method, application method and electronic equipment. BACKGROUND

[0002] Now, with the rapid development of Internet technology, the various information of multimedia network has appeared explosive growth, the information received by people has not only limited to text, image, sound or video alone one form of information, but multiple types of data to describe the same object or the same event, and each different type of data can be regarded as a kind of modality.For the common image-text combined information, when carrying out cross-modal data retrieval, due to the distribution characteristics of different modal data and inconsistent feature representation, the existing cross-modal retrieval method mainly learns uniform space for different modal data, i.e. different modal feature representation is mapped from original single-modal space to cross-modal uniform space, to obtain uniform representation that can directly measure cross-modal similarity.But the existing method mainly focuses on global feature learning of image, text and other data, ignores the fine-grained semantic information in these data, and cannot realize semantic alignment between modal data at entity and relationship level. SUMMARY

[0003] Therefore, it is necessary to provide a kind of picture-text cross-modal retrieval network training method, application method and electronic equipment, for solving the problem that the fine-grained semantic information contained in the data is ignored in the prior art, and the semantic alignment between modal data at entity and relationship level cannot be realized.

[0004] In order to solve the above problems, the present application provides a kind of picture-text cross-modal retrieval network training method, comprising:

[0005] obtain picture-text training data set;

[0006] constructing picture-text cross-modal retrieval network model composed of image embedder, text embedder and picture-text combination module;

[0007] picture-text training data set is used as the input of the picture-text cross-modal retrieval network, and the picture-text cross-modal retrieval network is trained in turn according to image invariant and text mask modeling operation, text invariant and image mask modeling operation, image-text matching operation, semantic alignment processing operation and iteration, to obtain trained picture-text cross-modal retrieval network.

[0008] Further, the image embedder is used for slicing operation to obtain slice block for the input image of the picture-text training data set, encoding each slice block and obtaining visual feature vector and position information of each slice block, to obtain image input;

[0009] The text embedder is configured to perform a tokenization operation on input texts of the image-text training dataset, encode text tokens, and obtain word embeddings and position information of each text token to obtain text inputs;

[0010] The image-text combination module is configured to combine the image inputs and the text inputs to obtain joint inputs, wherein the joint inputs include image-text pairs formed by pairing any image input with all text inputs, and wherein consistent image-text pairs are positive samples and inconsistent image-text pairs are negative samples, and any image input or text input has only one positive sample.

[0011] Further, the image-invariant and text-masked modeling operation includes:

[0012] The image inputs of the image-text training set are kept unchanged, and part of the text inputs are randomly masked, and the masked part of the text inputs is processed by marking, replacing, or keeping unchanged;

[0013] Based on the interactive observation of the text inputs that are not masked and all image inputs, and by taking a minimization of a negative log-likelihood function as a loss function, the image-text cross-modal retrieval network model is trained to predict the masked text inputs.

[0014] Further, the text-invariant and image-masked modeling operation includes:

[0015] The text inputs of the image-text training set are kept unchanged, and part of the image inputs are randomly masked, and the masked image inputs are replaced with zero vectors for visual feature vectors;

[0016] Based on the interactive observation of the image inputs that are not masked and all text inputs, and by taking a minimization of a cross-entropy loss function as a loss function, the image-text cross-modal retrieval network model is trained to reconstruct the masked image inputs.

[0017] Further, the image-text matching operation includes:

[0018] Special token inputs are created to the image-text cross-modal retrieval network, and outputs of the special tokens through the image-text cross-modal retrieval network are taken as positive samples of the joint inputs;

[0019] The image inputs or the text inputs in the positive samples are randomly replaced to obtain negative sample pairs, a minimization of a cross-entropy loss function is taken as a loss function, the image-text cross-modal retrieval network model is trained, and a matching score of the input image-text pairs is obtained through a full connection layer and an activation function, and the instance-level alignment of the image inputs and the text inputs is maintained by taking the matching score of the positive sample pairs to be higher than the matching score of the negative sample pairs.

[0020] Further, the creation of the special tokens includes:

[0021] Randomly initialize an initialization vector with the same dimension as the image input or text input, and input the initialization vector after being spliced with the image-text pair into the image-text cross-modal retrieval network, and obtain the special token after training and optimization of the initialization vector through self-attention analysis.

[0022] Further, the semantic alignment processing operation includes:

[0023] The positive sample is sampled for training, the corresponding relationship between the text input and the image input is analyzed through the self-attention weight, the image input is calibrated using the text input self-attention, the attention distribution between the two modalities is aligned and calibrated through the text input and the image input, the semantic alignment of the entity and the corresponding relationship is maintained, and the KL divergence function is used as a loss function.

[0024] Further, the iteration training of the image-text cross-modal retrieval network to obtain the training complete image-text cross-modal retrieval network includes:

[0025] The special token is input into the image-text cross-modal retrieval network, the output of the special token through the image-text cross-modal retrieval network is used as a positive sample of joint input, and the matching score of the input image-text pair is obtained through the full connection layer and the activation function.

[0026] All sample pairs are iteratively trained and the matching score of the sample pair is learned, and a minimum binary cross-entropy function is used as a loss function until the loss is no longer reduced, and a complete image-text cross-modal retrieval network is trained.

[0027] The application also provides an image-text cross-modal retrieval network application method, which includes:

[0028] Obtaining an image or text to be retrieved;

[0029] Inputting the image or text to be retrieved into the training complete image-text cross-modal retrieval network to retrieve similar images and texts, wherein the training complete image-text cross-modal retrieval network is determined according to the image-text cross-modal retrieval network training method according to any one of the above.

[0030] The image-text cross-modal retrieval network outputs to obtain similar images and texts.

[0031] The application also provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the program, the image-text cross-modal retrieval network training method according to any one of the above and / or the image-text cross-modal retrieval network application method according to the above is implemented.

[0032] Compared with the prior art, the beneficial effects of the above-mentioned embodiment are that the image embedding device and the text embedding device and the image-text combination module are used to represent the semantic information of the research object as a dense low-dimensional real value vector, and the semantic connection of the entity and the relationship is efficiently calculated in the low-dimensional space through the image-text matching operation and the semantic alignment processing operation. In this way, the problem of ignoring the fine-grained semantic information contained in the data and being unable to realize the semantic alignment of the entity and the relationship between the modal data at the level in the cross-modal retrieval process is solved. BRIEF DESCRIPTION OF DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0034] Figure 1 The flowchart of one embodiment of the image-text cross-modal retrieval network training method provided by the present application is shown in the figure.

[0035] Figure 2 The structure diagram of one embodiment of the image-text cross-modal retrieval network provided by the present application is shown in the figure.

[0036] Figure 3 The flowchart of one embodiment of the image-text cross-modal retrieval network application method provided by the present application is shown in the figure.

[0037] Figure 4 The structure diagram of one embodiment of the electronic device provided by the present application is shown in the figure. DETAILED DESCRIPTION

[0038] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, any other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0039] It should be understood that the drawings of the schematic diagram are not drawn to scale. The flowchart used in the present application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can not be implemented in sequence, and the steps without logical context relationship can be reversed in sequence or implemented simultaneously. In addition, one or more other operations can be added to the flowchart or removed from the flowchart by those skilled in the art under the guidance of the content of the present application.

[0040] Some of the block diagrams shown in the drawings are functional entities, which do not necessarily have to correspond to physically or logically independent entities. These functional entities can be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different network and / or processor systems and / or microcontroller systems.

[0041] Reference herein to an "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment can be included in at least one embodiment of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are they necessarily mutually exclusive or alternative embodiments. It is explicitly and implicitly understood that the embodiments described herein can be combined with each other.

[0042] Figure 1 An embodiment flowchart of the image-text cross-modal retrieval network training method provided by the application is shown in FIG. 1, which shows that the image-text cross-modal retrieval network training method comprises: Figure 1

[0043] S101, acquiring an image-text training dataset;

[0044] S102, constructing an image-text cross-modal retrieval network model composed of an image embedder, a text embedder and an image-text combination module;

[0045] S103, taking the image-text training dataset as input of the image-text cross-modal retrieval network, sequentially performing image-invariant and text-mask modeling operation, text-invariant and image-mask modeling operation, image-text matching operation, semantic alignment processing operation and iterative training of the image-text cross-modal retrieval network, to obtain a trained image-text cross-modal retrieval network.

[0046] In the image-text cross-modal retrieval network provided by the application, the semantic information of the research object is represented as a dense low-dimensional real value vector by constructing an image embedder, a text embedder and an image-text combination module, fine-grained semantic information in the data is extracted, and the semantic connection of entities and relationships is calculated through image-text matching operation and semantic alignment processing operation, to realize semantic alignment of entities and relationships at the level of modal data.

[0047] In one specific embodiment of the application, the image embedder is configured to perform slicing operation on the input image of the image-text training dataset to obtain sliced blocks, encode each sliced block and obtain a visual feature vector and position information of each sliced block, to obtain image input;

[0048] The text embedder is configured to perform wordpiece division operation on the input text of the image-text training dataset, encode the text wordpiece and obtain word embedding and position information of each wordpiece, to obtain text input. ​

[0049] The image-text combination module is used for combining the image input and the text input to obtain a joint input, the joint input includes pairing any image input with all text inputs to form an image-text pair, it is specified that the image-text pair in which the image and the text are consistent is a positive sample, and the image-text pair in which the image and the text are inconsistent is a negative sample, and any image input or text input has only one positive sample.

[0050] Specifically, the data set sample used in the embodiment of the present application selects image data and text data as initial data through preprocessing, and after excluding repeated data in the data set through a URL matching algorithm, the image in the data set is normalized to an image in a 224*224*3 format according to the input requirements of the retrieval model of the embodiment to obtain an image-text training data set.

[0051] As shown in Figure 2 The image-text cross-modal retrieval network 200 is composed of an image embedding module 201, a text embedding module 202 and an image-text combination module 203. The image embedding module 201 divides the input image in a unified shape into 16*16 size slice blocks, the number of slice blocks is fixed at 196, and the visual features of each slice block are extracted by using a linear projection method, and then the initial position of each slice block is encoded by using a 7-dimensional vector L = [x1, y1, x2, y2, w, h, w*h] (standardized upper / lower / left / right coordinates, width, height and area). Then the visual features and position encodings are respectively passed through a fully connected layer, the outputs of the two fully connected layers are added and normalized to obtain the image input V = [v1, v2, …, v 196 ].

[0052] The text embedding module 202 constructs a text vocabulary, and maps word labels to digital indexes starting from 0. The input text sentence is labeled as a single word by the vocabulary, and then the word is respectively passed through an embedding layer to obtain the embedding of the word and its position feature, and then the embeddings are added and normalized to obtain the text input W = [w1, w2, …, wn], where n is the number of words. n ].

[0053] The image-text combination module 203 combines the image input and the text input as a joint input, and pairs any image with each text to form an image-text pair, and specifies that the image-text pair in which the image and the text are consistent is a positive sample, and the image-text pair in which the image and the text are inconsistent is a negative sample. Then a standard Transformer encoder is used to process the joint input corresponding to each image-text pair, the number of encoders is 12, the vector dimension is 768, the number of self-attention heads is 12, and an additional segment is added in the joint input to distinguish the image input and the text input. In the continuous multiple encoder blocks, the joint context representation of each image slice block and each word is learned through the multi-head self-attention mechanism.

[0054] In specific embodiments of the present application, the image-invariant and text-masked modeling operation comprises:

[0055] The image input of the image-text training set is kept unchanged, and part of the text input is randomly masked, and the masked part of the text is processed by marking, replacing or keeping unchanged;

[0056] Based on the interaction observation of the text input which is not masked and all image inputs, and taking the minimization of the negative log likelihood function as the loss function, the image-text cross-modal retrieval network model is trained to predict the masked text input.

[0057] Specifically, the input image slice block is kept complete, part of the word units is randomly selected from the input text with a probability of 15%, the masked word units are replaced with other word units with a probability of 10%, kept unchanged with a probability of 10%, and represented by a special mark [MASK] with a probability of 80%. The image-text cross-modal retrieval network model is trained to predict the masked word units by using the remaining unmasked text input and all image inputs, and in the training process, the following minimization of the negative log likelihood function is used as the loss function:

[0058] L=-E (W,V)～D log P(W m |W \m ,V)

[0059] Wherein, W represents the text, V represents the image, m represents the index set of the masked word units in the text vocabulary, W m represents the masked word units, W \ represents the remaining unmasked word units, and (W, V) represents the image-text pair sampled from the entire image-text training data set.

[0060] In specific embodiments of the present application, the image-invariant and text-masked modeling operation comprises:

[0061] The text input of the image-text training set is kept unchanged, and part of the image input is randomly masked, and the masked image input is replaced with a zero vector for visual feature vectors;

[0062] Based on the interaction observation of the image input which is not masked and all text inputs, and taking the minimization of the cross-entropy loss function as the loss function, the image-text cross-modal retrieval network model is trained to reconstruct the masked image input.

[0063] Specifically, the input text is kept intact, and image slice blocks are randomly masked with a probability of 15% from the input, and the masked image slice blocks are replaced with zero vectors to replace their visual feature vectors. The image-text cross-modal retrieval network model is trained to reconstruct the masked image input by using the remaining unmasked image input and all text input, and the following minimum cross-entropy loss function is used as the loss function in the training process:

[0064]

[0065] wherein m represents the number of masked image slice blocks, represents the model output of the i-th masked part, represents the label of the i-th masked image slice block, and represents the object class contained in the masked image slice block when the image slice block is not covered; represents the probability that the i-th masked image slice block corresponds to the output belonging to the object contained in each image slice, and the normalized distribution is obtained through the full connection layer and the softmax function; and CE represents the cross-entropy loss function.

[0066] In specific embodiments of the present application, the image-text matching operation comprises:

[0067] creating a special token input into the image-text cross-modal retrieval network, and using the output of the special token through the image-text cross-modal retrieval network as a positive sample of the joint input;

[0068] randomly replacing the image input or the text input in the positive sample to obtain a negative sample pair, using the minimum cross-entropy loss function as the loss function, training the image-text cross-modal retrieval network model, and obtaining the matching score of the input image-text pair through the full connection layer and the activation function, and keeping the instance-level alignment of the image input and the text input by making the matching score of the positive sample pair higher than the matching score of the negative sample pair.

[0069] In specific embodiments of the present application, creating a special token comprises:

[0070] randomly initializing an initialization vector with the same dimension as the image input or the text input, and inputting the initialization vector into the image-text cross-modal retrieval network after being spliced with the image-text pair, and obtaining the special token by training and optimizing the initialization vector through self-attention analysis.

[0071] Specifically, a single initialization vector with the same dimension as the image input or the text input is randomly initialized, and the initialization vector is spliced with the image-text pair and input into the image-text cross-modal retrieval network for training, and the special token [CLS] is obtained by continuously training and optimizing the initialization vector through the self-attention mechanism of the Transformer encoder.

[0072] The special word element [CLS] is extracted by the output of the image-text cross-modal retrieval network as the joint representation of the input image-text pair, and a sample is randomly selected from other image-text pairs to replace the image or text in the positive sample pair to create a negative sample pair; During training, both positive and negative image-text pairs are sampled, and the output is passed through a fully connected layer and a sigmoid function to obtain the matching score of the image-text pair, denoted as s(w, v); Learn their matching scores so that the positive sample pair score is as high as possible and the negative sample pair score is as low as possible to achieve instance-level alignment of the entire input image and text, and the following binary cross-entropy function is used as the loss function during the training process:

[0073] L = -E (,)～D [ylogs(W,V)+(1-)log(1-(W,V))]

[0074] In specific embodiments of the present application, the semantic alignment processing operation includes:

[0075] The positive sample is sampled for training, the corresponding relationship between the text input and the image input is analyzed through the self-attention weight, and the image input is calibrated using the text input self-attention, and the attention distribution between the two modalities is aligned with the text input and the image input, the semantic alignment of the entity and the corresponding relationship is maintained, and the KL divergence function is used as the loss function.

[0076] Specifically, the image-text pair of the sampled positive sample is used as the training sample, the corresponding relationship between each word element of the text input and each slice block of the image input is analyzed by generating self-attention weights through the image-text cross-modal retrieval model, and the image slice block corresponding to each word element is found. In the training process, due to the existence of noise in the redundant image information, the picture input is calibrated using the text input self-attention, and the attention distribution of the two modalities is aligned with the text input and the image input, so as to realize the semantic alignment of the entity and the corresponding relationship. In this training process, the following KL divergence function is used as the loss function:

[0077]

[0078] Where n represents the number of word elements of the text input, represents the text self-attention weight of the i-th word element, represents the visual representation of the text self-attention weight of the i-th word element, D KL represents the KL divergence.

[0079] In specific embodiments of the present application, the image-text cross-modal retrieval network is iteratively trained to obtain a fully trained image-text cross-modal retrieval network, which includes:

[0080] input the special word into the image-text cross-modal retrieval network, obtain the output of the special word through the image-text cross-modal retrieval network as a positive sample of joint input, pass through the full connection layer and the activation function, and obtain a matching score of the input image-text pair;

[0081] Iterative training of all sample pairs and learning of the matching scores of the sample pairs is performed until the loss is no longer reduced, and a complete image-text cross-modal retrieval network is trained.

[0082] Specifically, the output of the special word [CLS] through the image-text cross-modal retrieval network is extracted as a joint representation of the input image-text pair, and the joint representation is passed through a full connection layer and a sigmoid function to obtain a matching score of the input image-text pair, denoted as s(w, v). All image-text pairs in the image-text training data set and the corresponding data set of the actual retrieval task are obtained, and the matching scores of all sample pairs in the image-text training data set are learned by minimizing the binary cross-entropy function as a loss function, and the matching scores of the sample pairs are continuously trained and optimized until the loss is no longer reduced, and a complete image-text cross-modal retrieval network is obtained.

[0083] In summary, the image-text cross-modal retrieval network training method provided by the application represents semantic information as a dense low-dimensional real value vector based on an image embedder, a text embedder and an image-text combination module, extracts fine-grained semantic information in data, and calculates semantic connections of entities and relationships through image-text matching operations and semantic alignment processing operations to realize semantic alignment of entities and relationships at the level of modal data.

[0084] The embodiment of the image-text cross-modal retrieval network application method provided by the application further provides an image-text cross-modal retrieval network application method. Figure 3 As can be seen, Figure 3 The flowchart of one embodiment of the image-text cross-modal retrieval network application method provided by the application comprises steps S301 to S303.

[0085] S301, obtaining an image or text to be retrieved;

[0086] S302, inputting the image or text to be retrieved into the trained complete image-text cross-modal retrieval network to retrieve similar images and texts, wherein the trained complete image-text cross-modal retrieval network is determined according to the image-text cross-modal retrieval network training method described above.

[0087] S303, the image-text cross-modal retrieval network outputs to obtain similar images and texts.

[0088] In the embodiment of the present application, first, the image or text to be retrieved is effectively acquired; then the image or text to be retrieved is effectively identified by using the above-mentioned trained complete image-text cross-modal retrieval network, cross-modal retrieval is performed on it, and corresponding similar images and texts can be output.

[0089] The technical method of the present application will be described more clearly in combination with a specific application scenario, and the effectiveness of the present application will be evaluated, and the specific process is as follows:

[0090] I. Preparation of data set:

[0091] A large-scale data set COCO, Visual Genome (VG) and Conceptual (CC) is acquired, and image-text pairs in the data set are selected as an initial data set; the original training set and the validation set of each data set are merged, and the URL matching algorithm is used to exclude repeated data in each data set; and according to the input requirements of the image-text cross-modal retrieval network, the image shape in the initial data set is normalized to 244*244*3, and an image-text training data set is obtained.

[0092] II. Application process:

[0093] According to the specific embodiment of the above-mentioned image-text cross-modal retrieval network training method, the initial model is a standard Transformer encoder architecture, the number of encoders is 12, the vector dimension is 768, the number of self-attention heads is 12, and an additional segment embedding is added to distinguish image input and text input; the image-text pairs extracted from the image-text training data set are used to train the image-text cross-modal retrieval network until the loss is no longer reduced, and the trained image-text cross-modal retrieval network is obtained.

[0094] The image or text to be retrieved is input into the image-text cross-modal retrieval network, and similar images and texts are output.

[0095] The present application also provides an electronic device, such as Figure 4 as shown, Figure 4 The structure schematic diagram of an embodiment of the electronic device provided by the present application, the electronic device 400 includes a memory 401, a processor 402, and a computer program stored on the memory 401 and executable on the processor 402, when the processor 402 executes the program, the image-text cross-modal retrieval network training method and / or the medical image retrieval network application method as described above are implemented.

[0096] For example, the computer program can be divided into one or more modules / units, one or more modules / units are stored in the memory 401 and executed by the processor 402 to complete the present application. One or more modules / units can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program in the electronic device 400.

[0097] The electronic device 400 can be a desktop computer, a notebook computer, a palm computer, a smart phone or the like.

[0098] The memory 401 can be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a secure digital (SD) card, a flash card and the like. The memory 401 is used to store programs, and the processor 402 executes the programs after receiving an execution instruction. The method defined by the flow disclosed in any of the foregoing embodiments of the present application can be applied to the processor 402 or implemented by the processor 402.

[0099] The processor 402 can be an integrated circuit chip having a signal processing capability. The processor 402 can be a general purpose processor, including a central processing unit (CPU), a network processor (NP) and the like; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC) or the like. It can implement or execute the disclosed methods, steps and logic block diagrams in the embodiments of the present application. The general purpose processor can also be a microprocessor or any conventional processor or the like.

[0100] It can be understood that, Figure 4 The structure shown is only a structural schematic diagram of the electronic device 400, and the electronic device 400 can further include more or less components than those shown. Figure 4 The components shown in the foregoing embodiments can be realized by hardware, software or a combination thereof. Figure 4

[0101] The above is only a preferred embodiment of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application.​

Claims

1. A method for training a cross-modal retrieval network for image-text, characterized in that, The method comprises the following steps: obtain a caption training dataset; build a caption cross-modal retrieval network model composed of an image embedding module, a text embedding module, and a caption combination module, the image embedding module is used to slice the input image of the caption training dataset to obtain slice blocks, encode each slice block, and obtain visual feature vectors and position information of each slice block to obtain image input, the text embedding module is used to divide the input text of the caption training dataset into word units, encode the text word units, and obtain word embedding and position information of each word unit to obtain text input, the caption combination module is used to combine the image input and the text input to obtain joint input, the joint input includes pairing any image input with all text inputs to form image-text pairs, and the image-text pairs that match are defined as positive samples, and the image-text pairs that do not match are defined as negative samples, and any image input or text input has only one positive sample; input the caption training dataset into the caption cross-modal retrieval network, and iteratively train the caption cross-modal retrieval network in the order of image-invariant and text-mask modeling operation, text-invariant and image-mask modeling operation, image-text matching operation, and semantic alignment processing operation to obtain a trained caption cross-modal retrieval network; the image-invariant and text-mask modeling operation comprises: keeping the image input of the caption training dataset unchanged, randomly masking part of the text input, and processing the masked part of the text input by marking, replacing, or keeping unchanged; based on the interactive observation of the text input that is not masked and all image inputs, and taking the minimization of the negative log-likelihood function as the loss function, training the caption cross-modal retrieval network model to predict the masked text input; the text-invariant and image-mask modeling operation comprises: keeping the text input of the caption training dataset unchanged, randomly masking part of the image input, and replacing the visual feature vector of the masked image input with a zero vector; based on the interactive observation of the image input that is not masked and all text inputs, and taking the minimization of the cross-entropy loss function as the loss function, training the caption cross-modal retrieval network model to reconstruct the masked image input.

2. The text-image cross-modal retrieval network training method according to claim 1, characterized in that, the image-text matching operation comprises: creating a special word unit input into the caption cross-modal retrieval network, and taking the output of the special word unit through the caption cross-modal retrieval network as the positive sample of the joint input; randomly replacing the image input or the text input in the positive sample to obtain a negative sample pair, training the caption cross-modal retrieval network model by taking the minimization of the cross-entropy loss function as the loss function, and obtaining the matching score of the input image-text pair through the full connection layer and the activation function, and keeping the instance-level alignment of the image input and the text input by making the matching score of the positive sample pair higher than the matching score of the negative sample pair.

3. The text-image cross-modal retrieval network training method according to claim 2, characterized in that, the creation of the special word unit comprises: randomly initializing an initialization vector with the same dimension as the image input or the text input, and inputting the initialization vector into the caption cross-modal retrieval network after being spliced with the image-text pair, and obtaining the special word unit by training and optimizing the initialization vector through self-attention analysis.

4. The text-image cross-modal retrieval network training method according to claim 3, characterized in that, the semantic alignment processing operation comprises: The positive sample is sampled for training, the corresponding relationship between the text input and the image input is analyzed through the self-attention weight, and the image input is calibrated using the text input self-attention, the attention distribution between the two modalities is aligned and calibrated using the text input and the image input, the semantic alignment of the entity and the corresponding relationship is maintained, and a KL divergence function is used as a loss function.

5. The text-image cross-modal retrieval network training method according to claim 4, characterized in that, The iterative training of the image-text cross-modal retrieval network includes: The special token is input into the image-text cross-modal retrieval network, and the output of the special token through the image-text cross-modal retrieval network is used as a positive sample of the joint input, and a matching score of the input image-text pair is obtained through the full connection layer and the activation function; All sample pairs are iteratively trained and the matching scores of the sample pairs are learned, a binary cross-entropy function is used as a loss function, and the training is completed until the loss is no longer reduced, and a complete image-text cross-modal retrieval network is obtained.

6. A method for applying a text-image cross-modal retrieval network, characterized in that, It includes: Obtaining an image or text to be retrieved; The image or text to be retrieved is input into the trained image-text cross-modal retrieval network to retrieve similar images and texts, wherein the trained image-text cross-modal retrieval network is determined according to the image-text cross-modal retrieval network training method in any one of claims 1 to 5; The image-text cross-modal retrieval network outputs similar images and texts.

7. An electronic device, comprising: It includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and when the processor executes the program, the image-text cross-modal retrieval network training method in any one of claims 1 to 5 and / or the image-text cross-modal retrieval network application method in claim 6 is implemented.

Images