Image-text matching method and device and storage medium

Abstract

The present disclosure relates to a method and device for image-text matching and a storage medium. The method for image-text matching comprises: obtaining an image to be matched; inputting the image into a pre-trained image-text encoding model to encode the image and obtain an image vector; determining a text vector similar to the image vector from pre-stored text vectors; wherein the image-text encoding model comprises a text encoding sub-network for encoding text during training, and the pre-stored text vectors are determined by encoding pre-set texts using the text encoding sub-network; and determining the texts corresponding to a preset number of text vectors similar to the image vector as the texts matched with the image. The present disclosure can improve the efficiency of image-text matching of a server and reduce the system delay of the server.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to image-text matching methods, apparatus and storage media. Background Technology

[0002] Multimodal retrieval is a novel retrieval method that enables data retrieval across different modalities. For example, in image-text retrieval, a user can input an image to retrieve descriptive text matching that image, or a user can input text to retrieve an image described by that text.

[0003] Taking image-based text retrieval as an example, the server needs to roughly search the text library for candidate texts that are highly relevant to the input image based on the image input by the user. After obtaining the candidate texts, the server uses a text encoding model to encode the candidate texts to obtain their text vectors. Finally, the matching degree between the candidate texts and the user's input image is determined based on the text vectors of the candidate texts.

[0004] When performing image-text matching using the above method, the server needs to go through several steps, including preliminary retrieval, encoding, and calculation of matching degree, before it can achieve image-text matching, resulting in low server processing efficiency. Summary of the Invention

[0005] To overcome the problems existing in related technologies, this disclosure provides a text-image matching method, apparatus and storage medium.

[0006] According to a first aspect of the present disclosure, a text-image matching method is provided. The text-image matching method includes: acquiring an image to be matched; inputting the image into a pre-trained text-image encoding model to encode the image to obtain an image vector; determining text vectors similar to the image vector from pre-stored text vectors; wherein the text-image encoding model further includes a text encoding sub-network for encoding text during training, and the pre-stored text vectors are determined by encoding preset text through the text encoding sub-network; and determining the text corresponding to a preset number of text vectors similar to the image vector as the text matching the image.

[0007] In one example, determining a text vector similar to the image vector from pre-stored text vectors includes:

[0008] Determine the cosine distance between each text vector in the pre-stored text vector and the image vector;

[0009] Based on the calculated cosine distance, text vectors whose cosine distance to the image vector is greater than a set distance threshold are obtained, and

[0010] Text vectors whose cosine distance is greater than a set distance threshold are identified as text vectors similar to the image vector.

[0011] In one example, the pre-stored text vector is determined by encoding the preset text through the text encoding sub-network: the image encoding model is invoked; the preset text is input into the image encoding model, and the input preset text is encoded through the text encoding sub-network included in the image encoding model to obtain the pre-stored text vector.

[0012] In one example, the image encoding model is trained based on a first training sample pair and a second training sample pair. The first training sample pair includes the preset text and image samples that match the preset text, and the second training sample pair includes the preset text and image samples that do not match the preset text.

[0013] According to a second aspect of the present disclosure, a method for training a graph-text coding model is provided. The method includes: determining a preset text sample; determining an image that matches the preset text sample to obtain a first training sample pair, and determining an image that does not match the preset text sample to obtain a second training sample pair; and training a graph-text coding model based on the first training sample pair and the second training sample pair.

[0014] In one example, the image-to-text encoding model includes an image encoding subnetwork and a text encoding subnetwork. The step of training the image-to-text encoding model based on the first training sample pair and the second training sample pair includes: extracting image vectors of each image sample in the first training sample pair and the second training sample pair using the image encoding subnetwork, and extracting text vectors of each text sample in the first training sample pair and the second training sample pair using the text encoding subnetwork; determining a first cosine distance and a second cosine distance based on the image vectors of the image samples and the text vectors of the text samples, wherein the first cosine distance is the cosine distance between the text vector and the image vector in the first training sample pair, and the second cosine distance is the cosine distance between the text vector and the image vector in the second training sample pair; adjusting the training parameters of the image encoding subnetwork and the text encoding subnetwork according to the first cosine distance, the second cosine distance, and a loss function to obtain the image-to-text encoding model that satisfies the loss value; wherein the image-to-text encoding model satisfies the condition that the first cosine distance is greater than a preset first distance threshold, and the second cosine distance is less than a preset second distance threshold.

[0015] According to a third aspect of the present disclosure, an image-text matching apparatus is provided, comprising: an acquisition unit configured to acquire an image to be matched; a processing unit configured to input the image into a pre-trained image-text coding model and encode the image to obtain an image vector; and a determination unit configured to determine text vectors similar to the image vector from pre-stored text vectors; wherein the image-text coding model further includes a text coding sub-network for encoding text during training, and the pre-stored text vectors are determined by encoding preset text through the text coding sub-network; the determination unit is further configured to determine text corresponding to a preset number of text vectors similar to the image vector as text matching the image.

[0016] In one example, the determining unit determines text vectors similar to the image vector from pre-stored text vectors in the following manner: determining the cosine distance between each text vector in the pre-stored text vectors and the image vector; obtaining text vectors whose cosine distance to the image vector is greater than a set distance threshold based on the calculated cosine distance, and determining the text vectors whose cosine distance is greater than the set distance threshold as text vectors similar to the image vector.

[0017] In one example, the determining unit determines the pre-stored text vector by encoding the preset text through the text encoding sub-network in the following manner: calling the image-text encoding model; inputting the preset text into the image-text encoding model, and encoding the input preset text through the text encoding sub-network included in the image-text encoding model to obtain the pre-stored text vector.

[0018] In one example, the image encoding model is trained based on a first training sample pair and a second training sample pair. The first training sample pair includes the preset text and image samples that match the preset text, and the second training sample pair includes the preset text and image samples that do not match the preset text.

[0019] According to a fourth aspect of the present disclosure, a graphic encoding model training apparatus is provided. The graphic encoding model training apparatus includes: a determining unit configured to determine a preset text sample, and to determine an image that matches the preset text sample to obtain a first training sample pair, and to determine an image that does not match the preset text sample to obtain a second training sample pair; and a training unit configured to train a graphic encoding model based on the first training sample pair and the second training sample pair.

[0020] In one example, the image-to-text encoding model includes an image encoding subnetwork and a text encoding subnetwork. The training unit trains the image-to-text encoding model based on the first training sample pair and the second training sample pair in the following manner: The image encoding subnetwork extracts the image vectors of each image sample in the first training sample pair and the second training sample pair, respectively; the text encoding subnetwork extracts the text vectors of each text sample in the first training sample pair and the second training sample pair, respectively; based on the image vectors of the image samples and the text vectors of the text samples, a first cosine distance and a second cosine distance are determined, wherein the first cosine distance is the cosine distance between the text vector and the image vector in the first training sample pair, and the second cosine distance is the cosine distance between the text vector and the image vector in the second training sample pair; according to the first cosine distance, the second cosine distance, and the loss function, the training parameters of the image encoding subnetwork and the text encoding subnetwork are adjusted to obtain the image-to-text encoding model that satisfies the loss value; wherein the image-to-text encoding model satisfies the condition that the first cosine distance is greater than a preset first distance threshold, and the second cosine distance is less than a preset second distance threshold.

[0021] According to a fifth aspect of this disclosure, a text-image matching apparatus is provided, comprising: a memory configured to store instructions; and a processor configured to invoke the instructions to execute the text-image matching method of the first aspect or any example thereof.

[0022] According to a sixth aspect of this disclosure, a non-transitory computer-readable storage medium is provided, which stores computer-executable instructions that, when executed by a processor, perform the image-text matching method of the first aspect or any example of the first aspect.

[0023] According to the seventh aspect of this disclosure, a graph-text coding model training apparatus is provided, comprising: a memory configured to store instructions; and a processor configured to invoke the instructions to execute the graph-text coding model training method of the second aspect or any example thereof.

[0024] According to the eighth aspect of this disclosure, a non-transitory computer-readable storage medium is provided, which stores computer-executable instructions that, when executed by a processor, perform the graph-text coding model training method of the second aspect or any example of the second aspect.

[0025] The technical solutions provided by the embodiments of this disclosure can include the following beneficial effects: A server for image-text matching acquires an image to be matched and inputs it into a pre-trained image-text coding model. The model encodes the image to be matched to obtain an image vector. Then, it determines text vectors similar to the image vector of the image to be matched from pre-stored text vectors. The text corresponding to a predetermined number of text vectors similar to the image vector of the image to be matched is determined as the text matching the image. Therefore, the server for image-text matching does not need to first search for multiple candidate texts matching the user-input image in a preset text list, nor does it need to encode the candidate texts in real time to obtain text vectors, thus improving the image-text matching efficiency of the server and reducing the system latency of the image-text matching server.

[0026] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0027] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0028] Figure 1 This is a flowchart illustrating a text-image matching method according to an exemplary embodiment.

[0029] Figure 2 This is a flowchart illustrating a text-image matching method according to an exemplary embodiment.

[0030] Figure 3 This is a flowchart illustrating the training of a text encoding model and an image encoding model according to an exemplary embodiment.

[0031] Figure 4 This is a block diagram illustrating a text-image matching device according to an exemplary embodiment.

[0032] Figure 5 This is a block diagram of a graphic encoding model training device according to an exemplary embodiment.

[0033] Figure 6 This is a block diagram illustrating an apparatus according to an exemplary embodiment. Detailed Implementation

[0034] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0035] The technical solutions of the exemplary embodiments disclosed herein can be applied to application scenarios in image-text matching systems, whereby the system returns text matching the image input by the user. In this scenario, the image-text matching system may include, for example, a terminal for inputting the image to be matched and a server for performing text matching on the input image. The user terminal includes, but is not limited to, fixed or mobile electronic devices such as smartphones, tablets, laptops, desktop computers, and e-book readers. The server for image-text matching can be an independent application service device or a service cluster composed of multiple servers. In practical applications, it can be a cloud server, cloud host, virtual center, etc. This disclosure does not limit the structure or implementation of the server.

[0036] In related technologies, multimodal retrieval is used. For example, when retrieving descriptive text that matches an input image, the user inputs an image with the text to be matched on the terminal. After receiving the image input by the user, the server for image-text matching needs to search for multiple candidate texts that match the user's input image in the text library. After obtaining the candidate texts, the candidate texts are encoded based on a network model to obtain text vectors of the candidate texts. Then, based on the text vectors of the candidate texts, the text that matches the user's input image is determined and fed back to the terminal for the user to select.

[0037] Because the server used for image-text matching needs to go through the above-mentioned multiple steps in the process of performing image-text matching, the server used for image-text matching is inefficient and suffers from severe system latency when performing the operation.

[0038] This disclosure provides an image-text matching method. In this method, a server for image-text matching acquires an image to be matched and inputs it into a pre-trained image-text encoding model. The model encodes the image to obtain an image vector, and then determines text vectors similar to the image vector from a pre-stored set of text vectors. The text corresponding to a predetermined number of text vectors similar to the image vector is then identified as the text matching the image. Therefore, the server for image-text matching does not need to first search for multiple candidate texts matching the user-input image in a preset set of texts, nor does it need to encode the candidate texts in real-time to obtain text vectors, thus improving the image-text matching efficiency and reducing system latency.

[0039] Figure 1 This is a flowchart illustrating a text-image matching method according to an exemplary embodiment, such as... Figure 1 As shown, the image-text matching method includes the following steps.

[0040] In step S11, the image to be matched with the text is obtained.

[0041] In this disclosure, the image to be matched with the text and image can be one or more images input by the user, or it can be image frames extracted from the video after the user inputs the video.

[0042] In step S12, the image is input into a pre-trained image encoding model to encode the image and obtain an image vector.

[0043] The image-text encoding model disclosed herein can encode images and / or text based on input images and / or text, and output image vectors and / or text vectors.

[0044] Furthermore, in this disclosure, the image to be matched is input into the image coding model, and the image coding sub-network encodes the image to be matched to obtain the image vector of the image to be matched.

[0045] In step S13, a text vector similar to the image vector is determined from the pre-stored text vectors.

[0046] In practical applications, since images and text come from two heterogeneous spaces, in order to directly measure the similarity between images and text, images and text can be mapped into one space, and the image vector of the image and the text vector of the text can be used to measure the similarity between the images and text.

[0047] Based on the principle of vector similarity, in one embodiment, this disclosure can retrieve text vectors that are similar to image vectors from pre-stored text vectors, and determine the text corresponding to a preset number of text vectors that are similar to image vectors as the text that matches the image.

[0048] The pre-stored text vector is obtained by encoding the preset text based on the text encoding sub-network included in the graph-text encoding model.

[0049] In one implementation, the text encoding subnetwork may be, for example, a Bidirectional Encoder Representations from Transformers (BERT) neural network.

[0050] In step S14, the text corresponding to a preset number of text vectors that are similar to the image vector is determined as the text that matches the image.

[0051] In this disclosure, after obtaining the image to be matched with text, a text vector similar to the image vector of the image to be matched can be retrieved from the pre-stored text vectors based on the image vector of the image to be matched. The text corresponding to a preset number of text vectors similar to the image vector of the image to be matched is then returned to the user for selection.

[0052] For example, a user inputs a landscape image. After encoding the image using a text-image encoding model to obtain an image vector, the system retrieves text vectors similar to the input landscape image vector from a pre-stored text vector database. It then returns a predetermined number of text vectors with similarity to the landscape image vector, containing text such as classical poems or elegant prose, for the user to choose from. This significantly reduces the time required for the user to add text to the landscape photo. Furthermore, the user can use the predetermined number of text vectors with similarity to the landscape image vector for further image processing and other operations.

[0053] In an exemplary embodiment of this disclosure, a server for image-text matching acquires an image to be matched and inputs it into a pre-trained image-text encoding model. The model encodes the image to be matched to obtain an image vector. Then, it determines text vectors similar to the image vector of the image to be matched from pre-stored text vectors. The text corresponding to a predetermined number of text vectors similar to the image vector of the image to be matched is determined as the text that matches the image. Therefore, the server for image-text matching does not need to first search for multiple candidate texts matching the user-input image in a preset text library, nor does it need to encode the candidate texts in real time to obtain text vectors, thus improving the image-text matching efficiency of the server and reducing the system latency of the image-text matching server.

[0054] Figure 2 This is a flowchart illustrating a text-image matching method according to an exemplary embodiment, such as... Figure 2 As shown, the image-text matching method includes the following steps.

[0055] In step S21, the image to be matched with the text is obtained.

[0056] In step S22, the image is input into a pre-trained image encoding model to encode the image and obtain an image vector.

[0057] In step S23, the cosine distance between each text vector in the pre-stored text vector and the image vector is determined. Based on the calculated cosine distance, text vectors whose cosine distance to the image vector is greater than a set distance threshold are obtained, and text vectors whose cosine distance is greater than the set distance threshold are determined as text vectors similar to the image vector.

[0058] Based on the principle of vector similarity, in one embodiment, this disclosure measures the similarity between two vectors by measuring the cosine distance between them, i.e., the cosine of the angle between the two vectors. When the cosine distance between two vectors approaches 1, the two vectors are considered similar. When the cosine distance between two vectors approaches 0, the two vectors are considered dissimilar.

[0059] Based on the characteristics of cosine distance, a distance threshold for cosine distance similarity to image vectors can be set. For example, the distance threshold for cosine distance similarity to image vectors can be set to 0.98. That is, when the cosine distance between a text vector in the pre-stored text vectors and the image vector is greater than 0.98, the text vectors in the pre-stored text vectors with a cosine distance greater than 0.98 to the image vectors are determined as text vectors similar to the image vectors. Furthermore, the text corresponding to a predetermined number of text vectors similar to the image vectors can be determined as text matching the image.

[0060] For example, a user-input image of a mountain and flowing water scene is obtained. After encoding the image using a text-to-image (TTO) model to obtain its image vector, the cosine distance between each pre-stored text vector and the image vector is determined. Based on the calculated cosine distance, text vectors with a cosine distance greater than 0.98 are selected and considered similar to the image vector. The text corresponding to a predetermined number of similar text vectors—such as ancient poems about mountains and flowing water or other beautiful texts related to the scene—is returned to the user for selection. Returning matching text based on the user's input significantly reduces the time required to add text to the mountain and flowing water image. Furthermore, the user can use the predetermined number of similar text vectors for further image processing.

[0061] For example, when acquiring a video input from a user, since the content differences between adjacent video frames are minimal, the input video can be sampled every preset number of video frames to obtain individual image frames. After encoding each image frame using a graph-text coding model to obtain its image vector, the cosine distance between each pre-stored text vector and the image vector of each image frame is determined. Based on the calculated cosine distance, text vectors with a cosine distance greater than 0.98 are selected and considered similar to the image vectors of each image frame. The text corresponding to the preset number of text vectors similar to the image vectors of each image frame—for example, including elegant text related to each image frame—is returned to the user for selection.

[0062] In step S24, the text corresponding to a preset number of text vectors that are similar to the image vector is determined as the text that matches the image.

[0063] In an exemplary embodiment of this disclosure, a server for image-text matching acquires an image to be matched and encodes it using a preset image-text encoding model to obtain an image vector. Then, it determines the cosine distance between each text vector in a pre-stored set of text vectors and the image vector. Based on the calculated cosine distance, text vectors with a cosine distance greater than a set distance threshold are identified as text vectors similar to the image vector. Therefore, the server for image-text matching does not need to first search a text library for multiple candidate texts matching the user's input image, nor does it need to encode the candidate texts in real time to obtain text vectors, thus improving the image-text matching efficiency and reducing processing latency.

[0064] To improve the accuracy of image-text matching and reduce the processing latency for image-text matching, in a real-time manner, this disclosure may employ an end-to-end training method to train the image-text coding model.

[0065] Figure 3 This is a flowchart illustrating the training of a text coding subnetwork and an image coding model according to an exemplary embodiment, such as... Figure 3 As shown, it includes the following steps.

[0066] In step S31, the first training sample pair and the second training sample pair are obtained.

[0067] In one embodiment, this disclosure allows for training a text-image matching model based on a first training sample pair and a second training sample pair. The first training sample pair includes preset text and image samples that match the preset text, while the second training sample pair includes preset text and image samples that do not match the preset text. For example, 70% of the sample pairs from the first and second training sample pairs can be used as the training dataset to train the text-image matching model, and 30% of the sample pairs from the first and second training sample pairs can be used as the test dataset to validate the text-image matching model.

[0068] For ease of description, this disclosure refers to a training sample pair consisting of a preset text sample and an image sample that matches the preset text as a first training sample pair. A training sample pair consisting of a preset text and an image sample that does not match the preset text is referred to as a second training sample pair.

[0069] In one embodiment, the first training sample pair and the second training sample pair in this disclosure are determined, for example, in the following manner:

[0070] Retrieve preset text, which can be a single statement or a combination of statements with complete semantics. The statement referred to here can be a statement in any natural language, such as a large amount of text including Tang poems, Song lyrics, song lyrics, famous quotes, classic movie lines, etc.

[0071] Construct an image that matches the text in the preset text to obtain the first training sample pair, and construct an image that does not match the preset text to obtain the second training sample pair.

[0072] In step S32, the image-text matching model is trained and optimized.

[0073] In this disclosure, the image-text matching model can be a model comprising an image encoding subnetwork and a text encoding subnetwork, wherein the image encoding subnetwork can be a ResNet or a VGG network. The text encoding subnetwork can be, for example, a Bidirectional Encoder Representations from Transformers (BERT) neural network.

[0074] To avoid omissions and incomplete retrieval when searching multiple candidate texts that match the user's input image from a text retrieval database, and to avoid processing delays caused by retrieving candidate texts, in one embodiment, this disclosure can train the image-text matching model using an end-to-end training method. That is, by jointly training the image encoding subnetwork and the text encoding subnetwork, image-text matching can be performed on the image of the text to be matched from a global perspective, thereby improving the accuracy of image-text matching, improving the processing efficiency of image-text matching, and reducing the processing delay of the image-text matching server.

[0075] Since the first training sample pair consists of text from the text retrieval database and image samples that match the text in the text retrieval database, the cosine distance between the text vector and the image vector in the first training sample pair should approach 1. The second training sample pair consists of text from the text retrieval database and image samples that do not match the text in the text retrieval database, so the cosine distance between the text vector and the image vector in the second training sample pair should approach 0.

[0076] Therefore, when training the image-text matching model, which includes an image encoding subnetwork and a text encoding subnetwork, in an end-to-end manner, the first training sample pair and the second training sample pair are input into the image-text matching model. The image encoding subnetwork in the image-text matching model encodes each image sample in the first and second training sample pairs, and the image-text encoding model outputs the image vector of the image sample. Similarly, the text encoding subnetwork in the image-text matching model encodes each text sample in the first and second training sample pairs, and the image-text encoding model outputs the text vector of the text sample.

[0077] Subsequently, based on the image vectors of the image samples and the text vectors of the text samples, the cosine distance between the text vectors and image vectors in the first training sample pair is determined, as well as the cosine distance between the text vectors and image vectors in the second training sample is determined.

[0078] For ease of description, this disclosure refers to the cosine distance between the text vector and the image vector in the first training sample pair as the first cosine distance, and the cosine distance between the text vector and the image vector in the second training sample pair as the second cosine distance.

[0079] The parameters of the image coding model are adjusted based on the first cosine distance, the second cosine distance, and the loss function. Specifically, the training parameters of the image coding sub-network and the text coding sub-network included in the image coding model are adjusted respectively, so that the image coding model satisfies the condition that the first cosine distance is greater than a preset first distance threshold and the second cosine distance is less than a preset second distance threshold.

[0080] For example, the preset first distance threshold is 0.97, and the preset second distance threshold is 0.02. The training parameters of the image coding sub-network and the text coding sub-network are adjusted using the cross-entropy loss function, so that the image vectors and text vectors output from the image-text coding model satisfy the following conditions: the first cosine distance between the text vector and the image vector in the first training sample pair is greater than 0.97, and the second cosine distance between the text vector and the image vector in the second training sample pair is less than 0.02. Thus, the trained image-text coding model is obtained.

[0081] In step S33, the test dataset is input into the trained image-text matching model for verification, and a verified image-text matching model is obtained.

[0082] In this disclosure, a test dataset can be input into a trained image-text matching model for verification, resulting in a verified image-text matching model.

[0083] Therefore, after training the image-text coding model, the text coding sub-network included in the trained image-text coding model is used to encode all the text in the preset text. Then, when the user needs to perform image-text matching, the trained image-text coding model encodes the image to be matched and obtains the image vector. Based on the cosine distance between the image vector and the text vector, the text vectors similar to the image vector to be matched can be directly retrieved from the pre-stored text vectors. The text corresponding to the preset number of text vectors similar to the image vector to be matched is determined as the text that matches the image and is output for the user to select.

[0084] Furthermore, to quickly retrieve text vectors similar to the image vectors to be matched, the text encoding sub-network included in the image-text encoding model can encode all text in the preset text into text vectors. These encoded text vectors can then be used to construct a vector retrieval library through similarity search, such as Faiss. Since Faiss supports searches at the billion-level, a vector retrieval library built on Faiss can significantly improve retrieval efficiency when searching for text vectors similar to image vectors, while maintaining search accuracy, thereby enhancing the efficiency of image-text matching.

[0085] In an exemplary embodiment of this disclosure, an image-text matching model, including an image encoding subnetwork and a text encoding subnetwork, is trained end-to-end. This ensures that the trained image-text encoding model satisfies the following conditions: the cosine distance between the text vector and the image vector in the first training sample pair is greater than a preset first distance threshold, and the second cosine distance is less than a preset second distance threshold. Therefore, when performing image-text matching on the acquired image, the image is simply input into the image-text encoding model. The model encodes the acquired image to obtain an image vector. Based on all text vectors determined by the preset text encoding subnetwork included in the image-text matching model, text vectors similar to the image vector can be determined. This eliminates the need to first select candidate text in a text retrieval database and encode the candidate text in real time. Further matching with the image vector based on the encoded candidate text vector improves the image-text matching efficiency of the server and reduces the processing latency of image-text matching.

[0086] Based on the same concept, this disclosure also provides a text-image matching device.

[0087] It is understood that the image-text matching device provided in this disclosure includes hardware structures and / or software modules corresponding to each function in order to achieve the above-mentioned functions. In conjunction with the units and algorithm steps of the various examples disclosed in this disclosure, this disclosure can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the technical solutions of this disclosure.

[0088] Figure 4 This is a block diagram illustrating a text-image matching device according to an exemplary embodiment. (Refer to...) Figure 4 The image and text matching device 400 includes an acquisition unit 401, a processing unit 402, and a determination unit 403.

[0089] The acquisition unit 401 is configured to acquire an image to be matched with text; the processing unit 402 is configured to input the image into a pre-trained text encoding model to encode the image and obtain an image vector; the determination unit 403 is configured to determine text vectors similar to the image vector from pre-stored text vectors; wherein, the text encoding model further includes a text encoding sub-network for encoding text during training, and the pre-stored text vectors are determined by encoding preset text through the text encoding sub-network; the determination unit 403 is further configured to determine the text corresponding to a preset number of text vectors similar to the image vector as the text matching the image.

[0090] In one example, the determining unit 403 determines text vectors similar to the image vector from pre-stored text vectors in the following manner: determining the cosine distance between each text vector in the pre-stored text vectors and the image vector; obtaining text vectors whose cosine distance to the image vector is greater than a set distance threshold based on the calculated cosine distance, and determining the text vectors whose cosine distance is greater than the set distance threshold as text vectors similar to the image vector.

[0091] In one example, the determining unit 403 determines the pre-stored text vector by encoding the preset text through the text encoding sub-network in the following manner: calling the image-text encoding model; inputting the preset text into the image-text encoding model, and encoding the input preset text through the text encoding sub-network included in the image-text encoding model to obtain the pre-stored text vector.

[0092] In one example, the image encoding model is trained based on a first training sample pair and a second training sample pair. The first training sample pair includes the preset text and image samples that match the preset text, and the second training sample pair includes the preset text and image samples that do not match the preset text.

[0093] Figure 5 This is a block diagram illustrating a graph encoding model training device according to an exemplary embodiment. (Refer to...) Figure 5 The image encoding model training device 500 includes a determination unit 501 and a training unit 502.

[0094] The determining unit 501 is configured to determine a preset text sample and an image that matches the preset text sample to obtain a first training sample pair, and to determine an image that does not match the preset text sample to obtain a second training sample pair; the training unit 502 is configured to train a graph-text coding model based on the first training sample pair and the second training sample pair.

[0095] In one example, the image-text coding model includes an image coding sub-network and a text coding sub-network. The training unit 502 trains the image-text coding model based on the first training sample pair and the second training sample pair in the following manner: The image coding sub-network extracts the image vectors of each image sample in the first training sample pair and the second training sample pair, respectively; the text coding sub-network extracts the text vectors of each text sample in the first training sample pair and the second training sample pair, respectively; based on the image vectors of the image samples and the text vectors of the text samples, a first cosine distance and a second cosine distance are determined, wherein the first cosine distance is the cosine distance between the text vector and the image vector in the first training sample pair, and the second cosine distance is the cosine distance between the text vector and the image vector in the second training sample pair; according to the first cosine distance, the second cosine distance, and the loss function, the training parameters of the image coding sub-network and the text coding sub-network are adjusted to obtain the image-text coding model that satisfies the loss value; wherein the image-text coding model satisfies the condition that the first cosine distance is greater than a preset first distance threshold, and the second cosine distance is less than a preset second distance threshold.

[0096] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0097] Figure 6 This is a block diagram illustrating an apparatus 600 for image-text matching according to an exemplary embodiment. For example, apparatus 600 may be provided as a server. See also... Figure 6 The device 600 includes a processing component 622, which further includes one or more processors, and memory resources represented by memory 632 for storing instructions, such as application programs, that can be executed by the processing component 622. The application programs stored in memory 632 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 622 is configured to execute instructions to perform the aforementioned text-image matching method.

[0098] Device 600 may also include a power supply component 626 configured to perform power management of device 600, a wired or wireless network interface 660 configured to connect device 600 to a network, and an input / output (I / O) interface 668. Device 600 may operate on an operating system stored in memory 632, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or similar.

[0099] It is understood that in this disclosure, "multiple" refers to two or more, and other quantifiers are similar. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. The singular forms "a," "the," and "the" are also intended to include the plural forms unless the context clearly indicates otherwise.

[0100] It is further understood that the terms "first," "second," etc., are used to describe various types of information, but this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another, and do not indicate a specific order or degree of importance. In fact, the expressions "first," "second," etc., are completely interchangeable. For example, without departing from the scope of this disclosure, first information can also be referred to as second information, and similarly, second information can also be referred to as first information.

[0101] It can be further understood that, unless otherwise specified, "connection" includes both direct connections where no other components exist between the two parties and indirect connections where other components exist between them.

[0102] It is further understood that although operations are described in a specific order in the accompanying drawings in the embodiments of this disclosure, this should not be construed as requiring these operations to be performed in the specific order or serial order shown, or requiring all of the shown operations to be performed to obtain the desired result. In certain environments, multitasking and parallel processing may be advantageous.

[0103] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0104] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method for matching images and text, characterized in that, The method includes: Obtain the image to be matched with text and image; The image is input into a pre-trained image encoding model to encode the image and obtain an image vector; Determine text vectors similar to the image vectors from pre-stored text vectors stored in a vector retrieval library; Based on the pre-stored text vectors in the vector retrieval library, retrieve text vectors similar to the image vector, and determine the text corresponding to a preset number of text vectors similar to the image vector as the text that matches the image; The step of determining a text vector similar to the image vector from pre-stored text vectors includes: Determine the cosine distance between each text vector in the pre-stored text vector and the image vector; Based on the calculated cosine distance, text vectors whose cosine distance to the image vector is greater than a set distance threshold are obtained, and Text vectors whose cosine distance is greater than a set distance threshold are identified as text vectors similar to the image vector; The image-text encoding model further includes an image encoding subnetwork and a text encoding subnetwork during training. The text encoding subnetwork is used to encode text. The pre-stored text vector is determined by encoding the preset text through the text encoding subnetwork. After determining the pre-stored text vector, the vector retrieval library is constructed by similarity search on the pre-stored text vector. The image-text encoding model is jointly trained end-to-end based on the image encoding subnetwork and the text encoding subnetwork.

2. The image-text matching method according to claim 1, characterized in that, The pre-stored text vector is determined by encoding the preset text through the text encoding sub-network, including: The preset text is input into the image-text encoding model, and the input preset text is encoded by the text encoding sub-network included in the image-text encoding model to obtain the pre-stored text vector.

3. The image-text matching method according to claim 1, characterized in that, The image-text encoding model is trained based on a first training sample pair and a second training sample pair. The first training sample pair includes the preset text and image samples that match the preset text. The second training sample pair includes the preset text and image samples that do not match the preset text.

4. A text-image matching device, characterized in that, The device includes: The acquisition unit is configured to acquire the image to be matched with the image; The processing unit is configured to input the image into a pre-trained image coding model, encode the image, and obtain an image vector; The determining unit is configured to determine a text vector similar to the image vector from a pre-stored text vector, the pre-stored text vector being stored in a vector retrieval library; The determining unit is further configured to retrieve text vectors similar to the image vector based on pre-stored text vectors in the vector retrieval library, and determine the text corresponding to a preset number of text vectors similar to the image vector as the text matching the image. The determining unit determines text vectors similar to the image vectors from pre-stored text vectors in the following manner: Determine the cosine distance between each text vector in the pre-stored text vector and the image vector; Based on the calculated cosine distance, text vectors whose cosine distance to the image vector is greater than a set distance threshold are obtained, and Text vectors whose cosine distance is greater than a set distance threshold are identified as text vectors similar to the image vector; The image-text encoding model further includes an image encoding subnetwork and a text encoding subnetwork during training. The text encoding subnetwork is used to encode text. The pre-stored text vector is determined by encoding the preset text through the text encoding subnetwork. After determining the pre-stored text vector, the vector retrieval library is constructed by similarity search on the pre-stored text vector. The image-text encoding model is jointly trained end-to-end based on the image encoding subnetwork and the text encoding subnetwork.

5. The image-text matching device according to claim 4, characterized in that, The determining unit determines the pre-stored text vector by encoding the preset text through the text encoding sub-network in the following manner: Invoke the aforementioned image encoding model; The preset text is input into the image-text encoding model, and the input preset text is encoded by the text encoding sub-network included in the image-text encoding model to obtain the pre-stored text vector.

6. The image-text matching device according to claim 4, characterized in that, The image-text encoding model is trained based on a first training sample pair and a second training sample pair. The first training sample pair includes the preset text and image samples that match the preset text. The second training sample pair includes the preset text and image samples that do not match the preset text.

7. A text-image matching device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to execute the image-text matching method according to any one of claims 1-3.

8. A non-transitory computer-readable storage medium, wherein when instructions in the storage medium are executed by a processor of a mobile terminal, the mobile terminal is able to perform the image-text matching method according to any one of claims 1-3.

Images