Text image matching method and device, equipment and storage medium

Abstract

This application relates to a text-image matching method, apparatus, device, and storage medium, belonging to the field of image processing technology. The method includes: acquiring first text and a first image; acquiring text features at least two scales corresponding to the first text; fusing the text features at least two scales to obtain a first multi-scale fusion feature corresponding to the first text; acquiring image features at least two scales corresponding to the first image; fusing the image features at least two scales to obtain a second multi-scale fusion feature corresponding to the first image; obtaining a feature similarity between the first text and the first image based on the first and second multi-scale fusion features; and determining a matching relationship between the first text and the first image based on the feature similarity. This solution, through natural language processing and computer vision technology, considers the feature similarity between text and images at different feature scales, thereby improving the matching accuracy between text and images.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a text-image matching method, apparatus, device, and storage medium. Background Technology

[0002] Image processing is an important research direction in the field of computer vision. Building image classification models based on machine learning and then using these models for image recognition is a widely used image recognition method.

[0003] In related technologies, when it is necessary to find a matching image for text information input by a user, an image feature extraction model and a text feature extraction model can be constructed based on machine learning. The text feature extraction model is used to extract the text feature vector corresponding to the text input by the user, and the image feature extraction model is used to obtain the image feature vector corresponding to each candidate image. Based on the similarity between the vectors, the matching degree between the image and the text is selected, and then the image that is closest to the input text is selected.

[0004] In the above technical solutions, the accuracy of text-image matching between the developed image feature extraction model and the text feature extraction model is relatively low. Summary of the Invention

[0005] This application provides a text-image matching method, apparatus, device, and storage medium. It improves the accuracy of text-image matching by fusing text features from multiple scales and image features from multiple scales, and by determining the matching relationship between text and image based on the feature similarity between the fused text features and image features. The technical solution is as follows:

[0006] On the one hand, a text-image matching method is provided, the method comprising:

[0007] Get the first text and the first image;

[0008] Obtain text features at least two scales corresponding to the first text;

[0009] By fusing the text features at least two scales, the first multi-scale fusion feature corresponding to the first text is obtained;

[0010] Obtain image features at least two scales corresponding to the first image;

[0011] By fusing the image features at least two scales, a second multi-scale fusion feature corresponding to the first image is obtained;

[0012] Based on the first multi-scale fusion feature corresponding to the first text and the second multi-scale fusion feature corresponding to the first image, the feature similarity between the first text and the first image is obtained.

[0013] Based on the feature similarity, the matching relationship between the first text and the first image is determined.

[0014] On another front, a method for training a text-image matching model is provided, the method comprising:

[0015] Obtain a training sample set, which contains sample text and sample images that match the sample text;

[0016] The sample text is input into the text feature extraction branch of the text image matching model to obtain text features at least two scales corresponding to the sample text;

[0017] The sample image is input into the image feature extraction branch of the text image matching model to obtain image features at least two scales corresponding to the sample image;

[0018] By fusing text features at least two scales corresponding to the sample text, a first multi-scale fusion feature corresponding to the sample text is obtained.

[0019] The image features at least two scales corresponding to the sample image are fused to obtain the second multi-scale fusion feature corresponding to the sample image.

[0020] Input the first multi-scale fusion feature corresponding to the sample text and the second multi-scale fusion feature corresponding to the sample image into the loss function to obtain the loss function value corresponding to the sample text;

[0021] The text-image matching model is updated based on the loss function value corresponding to the sample text.

[0022] In another aspect, a text-image matching model training device is provided, the device comprising:

[0023] The text-image acquisition module is used to acquire the first text and the first image;

[0024] The text feature acquisition module is used to acquire text features at least two scales corresponding to the first text;

[0025] The first feature fusion module is used to fuse text features of at least two scales corresponding to the first text to obtain the first multi-scale fusion feature corresponding to the first text.

[0026] An image feature acquisition module is used to acquire image features at least two scales corresponding to the first image;

[0027] The second feature fusion module is used to fuse the image features of the at least two scales to obtain the second multi-scale fusion feature corresponding to the first image.

[0028] The similarity acquisition module is used to acquire the feature similarity between the first text and the first image based on the first multi-scale fusion feature corresponding to the first text and the second multi-scale fusion feature corresponding to the first image.

[0029] The matching relationship acquisition module is used to determine the matching relationship between the first text and the first image based on the feature similarity.

[0030] In one possible implementation, the first feature fusion module is used to input the text features of at least two scales into the first feature fusion branch of the text image matching model to obtain the first multi-scale fusion feature corresponding to the first text.

[0031] The second feature fusion module is used to input the image features of at least two scales into the second feature fusion branch of the text image matching model to obtain the second multi-scale fusion feature corresponding to the first image.

[0032] In one possible implementation, the first feature fusion branch is a fully connected layer or a deep learning model;

[0033] Alternatively, the second feature fusion branch may be the fully connected part or a deep learning model.

[0034] In one possible implementation, the image feature acquisition module is used to,

[0035] The first image is input into the image feature extraction branch of the text image matching model to obtain image features at least two scales corresponding to the first image.

[0036] In one possible implementation, the image features at least two scales include global image features and local image features at at least one scale; the image feature extraction branch includes a global feature extraction layer and at least one local feature extraction layer; the global feature extraction layer includes at least two feature extraction layers; the local feature extraction layer includes at least one feature extraction layer; the feature extraction layer is used to extract image features;

[0037] The image feature acquisition module includes:

[0038] A global image feature acquisition unit is used to acquire global image features corresponding to the first image based on the first image and the global feature extraction layer in the image feature extraction branch.

[0039] The local image feature acquisition unit is used to acquire local image features of at least one scale corresponding to the first image based on the first image and at least one local feature extraction layer in the image feature extraction branch.

[0040] In one possible implementation, the local image feature acquisition unit includes:

[0041] The first intermediate feature acquisition subunit is used to extract features from the first image based on a first specified number of feature extraction layers in the global feature extraction layer, and to obtain the first layer intermediate image features corresponding to the first image.

[0042] The intermediate local feature acquisition subunit is used to segment the intermediate image features of the first layer and obtain at least two intermediate local features corresponding to the intermediate image features of the first layer.

[0043] The local feature stitching subunit is used to stitch together at least two intermediate local features corresponding to the intermediate image features of the first layer to obtain the intermediate fusion feature of the first layer; the intermediate fusion feature of the first layer is an image feature that is different from the intermediate image features of the first layer; the size of the intermediate fusion feature of the first layer is the same as that of the intermediate image features of the first layer.

[0044] The local feature acquisition subunit is used to acquire local image features of at least one scale corresponding to the first image based on the intermediate fusion features of the first layer and at least one local feature extraction layer in the image feature extraction branch.

[0045] In one possible implementation, the image feature extraction branch includes N local feature extraction layers; the local feature acquisition subunit is used for,

[0046] Based on the intermediate image features of the first layer and the first local feature extraction layer in the image feature extraction branch, the local image features of the first scale corresponding to the first image are obtained.

[0047] Based on the second specified number of feature extraction layers in the (i-1)th local feature extraction layer, feature extraction is performed on the intermediate fusion features of the (i-1)th layer to obtain the intermediate image features of the i-th layer.

[0048] The intermediate image features of the i-th layer are segmented to obtain at least two intermediate local features corresponding to the intermediate image features of the i-th layer;

[0049] At least two intermediate local features corresponding to the intermediate image features of the i-th layer are concatenated to obtain the intermediate fusion feature of the i-th layer; the intermediate fusion feature of the i-th layer is an image feature that is different from the intermediate image features of the i-th layer; the intermediate fusion feature of the i-th layer has the same size as the intermediate image features of the i-th layer.

[0050] Based on the intermediate fusion features of the i-th layer and the i-th local feature extraction layer in the image feature extraction branch, the local image features of the i-th scale corresponding to the first image are obtained; where 2≤i≤N, and i and N are integers.

[0051] In one possible implementation, the text feature acquisition module is used to,

[0052] The first text is input into the text feature extraction branch of the text image matching model to obtain at least two scales of text features corresponding to the first text; the text feature extraction branch is a neural network model used to extract text features.

[0053] In one possible implementation, the text feature acquisition module is used to,

[0054] Obtain at least two scales of sub-text from the first text; the at least two scales of sub-text include global text and at least one scale of local text; the scale of the global text is larger than the scale of the local text;

[0055] Input at least two scales of subtext of the first text into the text feature extraction branch of the text image model to obtain at least two scales of text features corresponding to the first text.

[0056] In one possible implementation, the device further includes:

[0057] The training sample set acquisition module is used to acquire a training sample set, which includes sample text and sample images that match the sample text.

[0058] The sample text feature acquisition module is used to input the sample text into the text feature extraction branch of the text image matching model to obtain text features at least two scales corresponding to the sample text;

[0059] The sample image feature acquisition module is used to input the first sample image into the image feature extraction branch of the text image matching model to acquire image features at least two scales corresponding to the first sample image.

[0060] The sample text fusion module is used to fuse text features of at least two scales corresponding to the sample text to obtain the first multi-scale fusion feature corresponding to the sample text.

[0061] The sample image fusion module is used to fuse image features of at least two scales corresponding to the first sample image to obtain the second multi-scale fusion feature corresponding to the sample image.

[0062] The loss function value acquisition module is used to input the first multi-scale fusion feature corresponding to the sample text and the second multi-scale fusion feature corresponding to the first sample image into the loss function to obtain the loss function value corresponding to the sample text.

[0063] The matching model update module is used to update the text image matching model based on the loss function value corresponding to the sample text.

[0064] In another aspect, a text-image matching model training device is provided, the device comprising:

[0065] The training sample set acquisition module is used to acquire a training sample set, which includes sample text and sample images that match the sample text.

[0066] The sample text feature acquisition module is used to input the sample text into the text feature extraction branch of the text image matching model to obtain text features at least two scales corresponding to the sample text;

[0067] The sample image feature acquisition module is used to input the first sample image into the image feature extraction branch of the text image matching model to acquire image features at least two scales corresponding to the first sample image.

[0068] The sample text fusion module is used to fuse text features of at least two scales corresponding to the sample text to obtain the first multi-scale fusion feature corresponding to the sample text.

[0069] The sample image fusion module is used to fuse image features of at least two scales corresponding to the first sample image to obtain a second fused feature corresponding to the sample image.

[0070] The loss function value acquisition module is used to input the second multi-scale fusion feature corresponding to the sample text and the second fusion feature corresponding to the first sample image into the loss function to obtain the loss function value corresponding to the sample text.

[0071] The matching model update module is used to update the text image matching model based on the loss function value corresponding to the sample text.

[0072] In another aspect, a computer device is provided, the computer device comprising a processor and a memory, the memory storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the above-described text image matching method; or, the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the above-described text image matching model training method.

[0073] In another aspect, a computer-readable storage medium is provided, wherein at least one instruction, at least one program, code set, or instruction set is stored in the storage medium, wherein the at least one instruction, the at least one program, the code set, or instruction set is loaded and executed by a processor to implement the above-described text image matching method; or, the at least one instruction, the at least one program, the code set, or instruction set is loaded and executed by the processor to implement the above-described text image matching model training method.

[0074] In another aspect, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the aforementioned text-image matching method; or, the processor of the computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the aforementioned text-image matching model training method.

[0075] The technical solution provided in this application may include the following beneficial effects:

[0076] The above scheme obtains features at least two scales corresponding to the first text and the first image, fuses the text features at least two scales into a first multi-scale fusion feature, fuses the image features at least two scales into a second multi-scale fusion feature, and determines the matching relationship between the first text and the first image based on the similarity between the first multi-scale fusion feature and the second multi-scale fusion feature. The above scheme determines the matching relationship between text and image by the similarity between the fusion features of text and image at multiple scales, and at the same time considers the feature similarity between text and image at different feature scales, thereby improving the matching accuracy between text and image. Attached Figure Description

[0077] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0078] Figure 1 This is a model training and text-image matching framework diagram illustrated according to an exemplary embodiment.

[0079] Figure 2 This is a flowchart illustrating a text-image matching method according to an exemplary embodiment.

[0080] Figure 3 It shows Figure 2 The illustrated embodiment is a schematic diagram of a text-image matching model.

[0081] Figure 4 This is a flowchart illustrating a text-image matching model training method according to an exemplary embodiment.

[0082] Figure 5 This is a flowchart illustrating a text image matching model training and text image matching method according to an exemplary embodiment.

[0083] Figure 6 The diagram illustrates a text feature extraction branch according to an embodiment of this application.

[0084] Figure 7 A schematic diagram of a local feature segmentation combination according to an embodiment of this application is shown.

[0085] Figure 8 A schematic diagram of an image recognition residual network according to an embodiment of this application is shown.

[0086] Figure 9 A schematic diagram of a text-image matching model according to an embodiment of this application is shown.

[0087] Figure 10 A schematic diagram of a text-image matching application according to an embodiment of this application is shown.

[0088] Figure 11 This is a schematic diagram illustrating a model training and text-image matching framework according to an exemplary embodiment.

[0089] Figure 12 This is a structural block diagram of a text image matching model training device according to an exemplary embodiment.

[0090] Figure 13 This is a structural block diagram of a text-image matching device according to an exemplary embodiment.

[0091] Figure 14 This is a schematic diagram of the structure of a computer device according to an exemplary embodiment. Detailed Implementation

[0092] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0093] Before describing the various embodiments shown in this application, several concepts involved in this application will be introduced first:

[0094] 1) Artificial Intelligence (AI)

[0095] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0096] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0097] 2) Computer Vision (CV)

[0098] Computer vision is the science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing and measuring targets, and then performs image processing to create images more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D (3D) technology, virtual reality, augmented reality, map building, and other technologies.

[0099] 3) Machine Learning (ML)

[0100] Machine learning is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory, among others. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learn-by-doing.

[0101] 4) Natural Language Processing (NLP)

[0102] Natural Language Processing (NLP) is an important field within computer science and artificial intelligence. It studies the theories and methods for enabling effective communication between humans and computers using natural language. NLP is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language—the language people use in daily life—and thus it has a close relationship with linguistic research. NLP technologies typically include text processing, semantic understanding, machine translation, question answering, and knowledge graphs.

[0103] The solution in this application includes a model training phase and a text-image matching phase. Figure 1 This is a model training and text-image matching framework diagram illustrated according to an exemplary embodiment. For example... Figure 1As shown, in the model training phase, the model training device 110 trains a text image matching model with high accuracy using a pre-set training sample set. In the text image matching phase, the text image matching device 120, based on the trained text image matching model and the input target text, searches for the image with the highest similarity to the input target text in the candidate image set as the candidate image for matching the target text.

[0104] The aforementioned model training device 110 and text image matching device 120 can be computer devices with machine learning capabilities, such as terminals or servers.

[0105] Optionally, the model training device 110 and the text-image matching device 120 can be the same device, or they can be different devices. Furthermore, when the model training device 110 and the text-image matching device 120 are different devices, they can be of the same type, such as both being servers; or they can be of different types. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. Terminals and servers can be connected directly or indirectly via wired or wireless communication, and this application does not impose any restrictions on this.

[0106] Figure 2 This is a flowchart illustrating a text-image matching method according to an exemplary embodiment. The method can be derived from the above... Figure 1 The text-image matching device in the illustrated embodiment performs this operation. For example... Figure 2 As shown, the text-image matching method may include the following steps:

[0107] Step 201: Obtain the first text and the first image.

[0108] In one possible implementation, the first text is text containing descriptive information about an object, and the first image is one of the images in a candidate image set corresponding to the first text. The candidate image set for the first text may contain at least two candidate images, and the first image is any one of the at least two candidate images.

[0109] In one possible implementation, the first image may be an image pre-stored in the text image matching device.

[0110] In one possible implementation, the first image may be a first image acquired in response to the acquisition of the first text.

[0111] That is, the first image can be an image pre-stored in a text-image matching device, or it can be an image that the user inputs at the same time as the first text.

[0112] Step 202: Obtain text features at least two scales corresponding to the first text.

[0113] The text features at different scales include text features extracted from text features at different information levels. These information levels can be any one of the following: article, paragraph, sentence, or word. In other words, text features at different scales can be text features corresponding to an article, a paragraph, a sentence, or a word.

[0114] In one possible implementation, the text feature can be a text feature vector extracted from the first text using a deep neural network.

[0115] At this point, the scale of the text features can represent the size of the text when the deep neural network extracts feature vectors from the first text. For example, when using an article as input for feature extraction via a deep neural network, the extracted text features are article-scale text features. These article-scale text features consider the individual features of paragraphs, sentences, and words within the entire article, as well as the interactions between paragraphs, sentences, and words. Therefore, the article scale is the largest text feature scale. Conversely, when using words as input for feature extraction via a deep neural network, the extracted text features are word-scale text features. These word-scale features only consider the text features of the word itself during feature extraction. Therefore, word features are the smallest scale text features.

[0116] Step 203: Fuse the text features of at least two scales to obtain the first multi-scale fused feature corresponding to the first text.

[0117] Since the first multi-scale fusion feature is obtained by fusing text features of at least two scales of the first text, the first multi-scale fusion feature simultaneously contains text features of at least two scales of the first text.

[0118] In one possible implementation, each scale of text features may contain one or more text features of the same scale but different features.

[0119] Step 204: Obtain image features at least two scales corresponding to the first image.

[0120] In one possible implementation, when the first image is an image pre-stored in a text image matching device, the second multi-scale fusion feature corresponding to the first image is an image pre-stored in the text image matching device. That is, the text image matching device has pre-performed image feature extraction on the first image before the first text input, and extracted image features of the first image at least two scales.

[0121] In another possible implementation, when the first image is a first image acquired in response to the acquisition of the first text, the first image is an image that is simultaneously input into the text image matching device along with the first text. In this case, the text image matching device performs image feature extraction on the first image to obtain image features at least two scales corresponding to the first image.

[0122] Step 205: Fuse the image features of at least two scales to obtain the second multi-scale fused feature corresponding to the first image.

[0123] In one possible implementation, image features at different scales are features extracted from images of different sizes, where image size can refer to the image resolution. Images of different resolutions contain varying amounts of information; generally, higher resolution images contain more information, and therefore, the scale of image features extracted from higher resolution images is also larger.

[0124] Step 206: Based on the first multi-scale fusion feature corresponding to the first text and the second multi-scale fusion feature corresponding to the first image, obtain the feature similarity between the first text and the first image.

[0125] The feature similarity between the first text and the first image is used to indicate the degree of similarity between the information described by the first text and the information described by the first image.

[0126] Step 207: Based on the feature similarity, determine the matching relationship between the first text and the first image.

[0127] In one possible implementation, the matching relationship between the first text and the first image is determined based on the relationship between the feature similarity and the similarity threshold.

[0128] Once the feature similarity between the first text and the first image is determined, the feature similarity can be compared with a similarity threshold. If the feature similarity is higher than the similarity threshold, the first text and the first image are considered to match; if the feature similarity is not higher than the similarity threshold, the first text and the first image are considered not to match.

[0129] In one possible implementation, the matching relationship between the first text and the first image is the matching confidence level between the first text and the first image, which indicates the likelihood that the first text and the first image are mutually matched.

[0130] In one possible implementation, the text features of the first text at at least two scales include global text features and at least one local text feature. The global text features and local text features of the first text are fused together to obtain the text fusion feature of the first text (i.e., the first multi-scale fusion feature), which simultaneously possesses both global and local features of the first text. The image features of the first image at at least two scales also include global image features and at least one local image feature. The global image features and local image features of the first image are fused together to obtain the image fusion feature of each first image (the second multi-scale fusion feature), which also simultaneously possesses both global and local features of the first image. At this time, the matching relationship between the first text and the first image is determined based on the text fusion feature of the first text and the image fusion feature of the first image. This can simultaneously consider both the local and global features of the image and the text. When the first image matches both a partial description of the text and a global description of the text, the first image is used as the target image for matching the first text, thereby improving the matching accuracy of text and image features.

[0131] In one possible implementation, the first image is one of the candidate images in the candidate image set. The candidate images can be sorted according to the feature similarity between the text fusion features of the first text and the image fusion features of the candidate images, and the one with the highest feature similarity can be obtained as the target image; or, the candidate images with the highest corresponding feature similarity among the candidate images can be obtained as the target image.

[0132] In one possible implementation, candidate images with feature similarity greater than a similarity threshold are obtained as target images.

[0133] In other words, candidate images with feature similarity greater than a threshold have a chance of being the target image corresponding to the first text. At this point, all candidate images with feature similarity greater than the similarity threshold are taken as the target images corresponding to the first text, which improves the accuracy of obtaining the image that matches the first text.

[0134] The solution shown in the embodiments of this application can be applied in at least the following scenarios:

[0135] 1) Obtain the image that matches the text.

[0136] When selecting an image that matches the text from multiple images using the scheme shown in the embodiments of this application, the multiple images can be images pre-stored in the text image matching device. At this time, the text image matching device has already extracted image features of multiple scales corresponding to each image and fused them into second multi-scale fusion features corresponding to each image. At this time, the text image matching device only needs to extract text features of multiple scales from the text and fuse the text features of multiple scales. Then, the feature similarity is compared between the fused first multi-scale fusion features and the second multi-scale fusion features corresponding to each image to determine the image that matches the text in each image.

[0137] 2) Obtain the text that matches the image.

[0138] When selecting text that matches an image from multiple texts using the scheme shown in the embodiments of this application, the multiple texts can be texts pre-stored in the text image matching device. At this time, the text image matching device has already extracted text features of multiple scales corresponding to each text and fused them into a first multi-scale fusion feature corresponding to each text. At this time, the text image matching device only needs to extract multi-scale image features from the image and fuse the image features of multiple scales. Then, the feature similarity is compared between the fused second multi-scale fusion feature and the first multi-scale fusion feature corresponding to each text to determine the text that matches the image in each text.

[0139] 3) Determine the matching degree between the text and the image.

[0140] When determining the matching relationship between an image and text using the scheme shown in the embodiments of this application, the text and the image can be input into the text-image matching device simultaneously. At this time, the text-image matching device extracts text features at multiple scales from the text and fuses them into a first multi-scale fusion feature; the text-image matching device extracts image features at multiple scales from the image and fuses them into a second multi-scale fusion feature; then the text matching device compares the feature similarity between the first multi-scale fusion feature and the second multi-scale fusion feature to determine the matching degree between the text and the image.

[0141] In summary, the solution presented in this application obtains features at least two scales corresponding to the first text and the first image, fuses the text features at least two scales into a first multi-scale fusion feature, fuses the image features at least two scales into a second multi-scale fusion feature, and determines the matching relationship between the first text and the first image based on the similarity between the first multi-scale fusion feature and the second multi-scale fusion feature. This solution determines the matching relationship between text and image by using the similarity between the fusion features of text and image at multiple scales, while also considering the feature similarity between text and image at different feature scales, thus improving the matching accuracy between text and image.

[0142] Figure 3 It shows Figure 2 The illustrated embodiment relates to a text-image matching flowchart. For example... Figure 3 As shown, the first text 301 and each candidate image in the candidate image set 302 corresponding to the first text are used as the first image input to the text image matching model 310. The text image matching model 310 performs text image matching on the input first text and the first image. When the feature similarity between the first text and the first image satisfies the matching relationship, that is, when the first image and the first text are mutually matched images and text, the candidate image that matches the first text 301 in the candidate image set 302 is determined as the target image 320 and output.

[0143] In the text image matching model 310, each candidate image in the candidate image set 302 is used as the first image input image feature extraction branch 311. The image feature extraction branch 311 performs image feature extraction on the first image to obtain the image feature 312 corresponding to the first image. The image feature 312 corresponding to the first image contains the global image feature of the first image corresponding to each candidate image and at least one local image feature.

[0144] The first text 301 is input into the text image matching model 310, and the text feature extraction branch 313 performs text feature extraction on the first text to obtain the first text feature 314 corresponding to the first text. The first text feature 314 includes the text global feature corresponding to the first text and at least one text local feature.

[0145] The text fusion feature is obtained by fusing the first text feature 314 corresponding to the first text 301 and the image fusion feature is obtained by fusing the first image feature 312 corresponding to the first image in the candidate image set 302. The feature similarity 315 between the text fusion feature and the image fusion feature corresponding to the first image is calculated. The target image 320 is selected from each candidate image corresponding to the first image and output according to the size of the feature similarity.

[0146] Figure 4 This is a flowchart illustrating a text-image matching model training method according to an exemplary embodiment. The method can be derived from the above... Figure 1 The text-image matching device in the illustrated embodiment performs, as follows: Figure 4 As shown, the text-image matching model training method may include the following steps:

[0147] Step 401: Obtain a training sample set, which contains sample text and a first sample image that matches the sample text.

[0148] In one possible implementation, the sample text is text containing descriptive information about a certain object, and the sample image is an image that matches the sample text. Here, "matching the sample text" means that the sample image contains all the content described by the sample text; that is, all the content described by the sample text can find corresponding image features in the sample image.

[0149] Step 402: Input the sample text into the text feature extraction branch of the text image matching model to obtain at least two scales of text features corresponding to the sample text.

[0150] The text features at different scales corresponding to the sample text include text features extracted from text features at different information levels. The aforementioned information levels can be any one of article, paragraph, sentence, or word; that is, text features at different scales can be text features at the article scale, text features at the paragraph scale, text features at the sentence scale, or text features at the word scale.

[0151] In one possible implementation, the text feature extraction branch can be a deep neural network model. In this case, the scale of the text features can represent the text size corresponding to the feature vector extraction performed by the deep neural network on the sample text. For example, when using a deep neural network to extract features from an article-level sample text as input, the extracted text features are article-scale text features. When extracting article-scale text features from this sample text, the features of paragraphs, sentences, and words within the entire article, as well as the interactions between paragraphs, sentences, and words, are considered. Therefore, the maximum text feature scale of this sample text is the article scale. Conversely, when using a deep neural network to extract features from words in the sample text as input, the extracted text features are word-scale text features. Furthermore, when extracting features from words in the sample text, only the text features of the word itself are considered. Therefore, word features are the smallest scale text features of this sample text.

[0152] Step 403: Input the first sample image into the image feature extraction branch of the text image matching model to obtain image features at least two scales corresponding to the first sample image.

[0153] Specifically, at least two scales of image features are extracted from first sample images at different scales, where different scales refer to different resolutions of the first sample images. For first sample images of different resolutions, the amount of information contained in the images also varies. Generally, images with higher resolutions contain more information. Feature extraction from images with higher resolutions takes into account more image information, and therefore the scale of the obtained image features is also larger.

[0154] Step 404: Fuse at least two scales of text features corresponding to the sample text to obtain the first multi-scale fusion feature corresponding to the sample text.

[0155] Since the first multi-scale fusion feature is obtained by fusing text features of at least two scales of the sample text, the first multi-scale fusion feature simultaneously contains text features of at least two scales of the sample text.

[0156] In one possible implementation, each scale of text features may contain one or more text features of the same scale but different features.

[0157] Step 405: Fuse image features of at least two scales corresponding to the first sample image to obtain the second multi-scale fused feature corresponding to the first sample image.

[0158] Since the second multi-scale fusion feature is obtained by fusing image features of at least two scales of the first sample image, the second multi-scale fusion feature simultaneously contains the features of image features of at least two scales of the first sample image.

[0159] Step 406: Input the first multi-scale fusion feature corresponding to the sample text and the second multi-scale fusion feature corresponding to the first sample image into the loss function to obtain the loss function value corresponding to the sample text.

[0160] In one possible implementation, the loss function is constructed based on the feature similarity between the first multi-scale fused feature and the second multi-scale fused feature. The loss function obtains its value based on the feature similarity between the first and second multi-scale fused features.

[0161] Step 407: Update the text-image matching model based on the loss function value corresponding to the sample text.

[0162] The text-image matching model is updated based on the loss function value using a backpropagation algorithm.

[0163] In one possible implementation, based on the loss function value, the text feature extraction branch and the image feature extraction branch of the text-image matching model are updated simultaneously using the backpropagation algorithm.

[0164] In summary, the above scheme obtains features at least two scales corresponding to the first text and the first image respectively, fuses the text features at least two scales into a first multi-scale fusion feature, fuses the image features at least two scales into a second multi-scale fusion feature, and determines the matching relationship between the first text and the first image based on the similarity between the first multi-scale fusion feature and the second multi-scale fusion feature. This scheme determines the matching relationship between text and image by using the similarity between the fusion features of text and image at multiple scales, while also considering the feature similarity between text and image at different feature scales, thus improving the matching accuracy between text and image.

[0165] Figure 5 This is a flowchart illustrating a text-image matching model training and text-image matching method according to an exemplary embodiment. The method can be executed by a model training device and a text-image matching device, which can be implemented as a single computer device or belong to different computer devices. Taking the model training device as the server and the text-image matching device as the terminal as an example, as follows... Figure 5 As shown, the method may include the following steps:

[0166] Step 501: Obtain the training sample set.

[0167] The training sample set contains sample text and sample images that match the sample text.

[0168] In one possible implementation, the features of the first sample image match the features of the sample text, meaning that the content displayed in the first sample image should correspond to the content displayed in the sample text.

[0169] In one possible implementation, the features of the first sample image include those present in the sample text.

[0170] Since images and text belong to different modalities of information display, their display features, information content, and information intensity are different. It is difficult to guarantee that the features displayed by the image and text are completely consistent. Therefore, the features displayed in the sample image may include other features besides those corresponding to the sample text.

[0171] For example, for the sample text "a man in black clothes", the image content of the sample image can be "a man in black clothes walking on the street". The features "black clothes" and "man" in the sample text are presented as images in the first sample image. In addition, the image usually displays features that are not related to the sample text (such as environmental features).

[0172] In one possible implementation, the training sample set also includes a second sample image; the second sample image is an image that does not match the sample text.

[0173] The second sample image can be a pre-defined image that does not match the sample text. That is, the data used in one training session of the training sample set includes the sample text, the first sample image, and the second sample image. In one training process of training the text image matching model, the sample text, the first sample image, and the second sample image need to be used simultaneously.

[0174] In one possible implementation, the second sample image may be an image randomly obtained from the training sample set, other than the first sample image.

[0175] Step 502: Input the sample text into the text feature extraction branch of the text image matching model to obtain at least two scales of text features corresponding to the sample text.

[0176] In one possible implementation, the subtext at least two scales includes a first sample subtext and a second sample subtext of the sample text.

[0177] In one possible implementation, the first sample subtext is at least one sentence text segmented from the sample text; the second sample subtext is at least one word text in the sample text. For example, the first sample subtext may be a clause corresponding to a sample text segmented from the sample text; while the second sample subtext may be at least one word in the sample text; the sample subtext simultaneously contains both the clause corresponding to the sample text and the word corresponding to the sample text, thus simultaneously containing text features of the sample text at two different scales.

[0178] For example, when the sample text is "a man wearing black clothes, about 1.8 meters tall, and about 30 years old", the first sample text can be at least one of "a man wearing black clothes", "about 1.8 meters tall", and "about 30 years old"; the second sample text can be at least one of the words "man", "black", "1.8 meters", etc.

[0179] In one possible implementation, the text feature extraction branch could be a BERT (Bidirectional Encoder Representations from Transformers) model.

[0180] The BERT model is a pre-trained model obtained by running a self-supervised learning method on a massive corpus. The BERT model can be fine-tuned by using a small number of sample texts to convert the sample text into a feature vector containing the semantic features of the sample text relatively accurately.

[0181] The sample text is input into the BERT model, and the output is the feature vector of the semantic features corresponding to the sample text. Since the feature vector (i.e. the global text feature) is the feature vector obtained by inputting the entire text of the sample text into the BERT model, the feature vector contains the global features corresponding to the sample text, which is the vector information obtained based on the overall semantics of the sample text.

[0182] In one possible implementation, at least one sample subtext of the sample text is input into the text feature extraction branch to obtain at least one local feature of the sample text.

[0183] The sample subtext contains a portion of the text content of the sample text, that is, the at least one local feature is a feature extracted from the portion of the text content of the at least one sample subtext.

[0184] In one possible implementation, when the sample subtext contains the first sample subtext and the second sample subtext, the first sample subtext and the second sample subtext are input into the text feature extraction branch to obtain the local features corresponding to the first sample subtext and the local features corresponding to the second sample subtext.

[0185] Here, the first sample subtext is the short sentence corresponding to the sample text. The local features obtained when the first sample subtext is input into the text feature extraction branch are the local features extracted from the text of the short sentence through the text feature extraction branch. In other words, the local features corresponding to the first sample subtext are sentence-level text features.

[0186] The second sample subtext is the word corresponding to the sample text. When the second sample subtext is input into the text feature extraction branch, the local features obtained are the local features extracted from the text of the word part through the text feature extraction branch. In other words, the local features corresponding to the second sample subtext are word-level text features.

[0187] At this point, the local features of the sample text can include both sentence-level and word-level text features. For example, if the sample text is "a man wearing black clothes, about 1.8 meters tall, and approximately 30 years old", the local features can include local features corresponding to at least one of the sentences "a man wearing black clothes", "about 1.8 meters tall", and "approximately 30 years old", as well as local features corresponding to at least one of the words "man", "black", "1.8 meters", etc.

[0188] In one possible implementation, at least two scales of sub-text of the sample text are obtained; the at least two scales of sub-text include global text and local text of at least one scale; the scale of the global text is larger than the scale of the local text; the at least two scales of sub-text of the sample text are input into the text feature extraction branch of the text image model to obtain the text features of at least two scales corresponding to the sample text.

[0189] The global text contains all the text content of the sample text, while the local text contains less text content than the global text. The global text and at least one scale of local text are input into the text feature extraction branch to obtain text features at at least two scales, namely, the at least two scales of text features include global text features and at least one local text feature.

[0190] Please refer to Figure 6 This illustrates a schematic diagram of a text feature extraction branch according to an embodiment of this application. Figure 6 As shown, sample text 610 is identified by CLS (Classification) and the entire sample text is classified into global sample text and clause sample text. Then, the global sample text, clause sample text, and word sample text are input into BERT model 600 to obtain sample text feature 620 corresponding to sample text 610.

[0191] First, add the CLS identifier 611 before the entire sentence description. The CLS identifier is used to represent the content of all text in the sample text 610. The global text feature 621 corresponding to the input of the CLS identifier 611 into BERT represents the features of the entire sample text description.

[0192] Then divide the entire description into two clauses, such as Figure 6As shown, CLS identifiers 612 and 613 are set after the CLS identifier 611 corresponding to the entire sentence of sample text 610. CLS identifiers 612 and 613 divide the sample text into two clauses. CLS identifiers 612 and 613 are used by the BERT model 600 to output the corresponding first clause text features 622 and second clause text features 623, representing the features corresponding to the sample text portions contained in the two clauses, respectively. For example, when the sample text is "A man wearing black clothes, about 1.8 meters tall, approximately 30 years old", CLS identifier 611 represents all the text content in sample text 610, that is, all the text content corresponding to "A man wearing black clothes, about 1.8 meters tall, approximately 30 years old"; CLS identifier 612 indicates part of the content in the sample text, "A man wearing black clothes"; and CLS identifier 613 indicates another part of the content in the sample text, "about 1.8 meters tall, approximately 30 years old".

[0193] Figure 6 The symbols “XX” and “XXX” are used to represent the words segmented from the sample text, such as “height”, “black”, “clothes”, “man”, etc. For each word in the sample text, it is input into the BERT model 600, which outputs the corresponding word text features. The sample text features 620 are composed of the word text features, the first clause text features 622, the second clause text features 623, and the global text features 621, serving as text features at different scales obtained by extracting the sample text using the BERT model.

[0194] Step 503: Input the first sample image into the image feature extraction branch of the text image matching model to obtain image features at least two scales corresponding to the first sample image.

[0195] In one possible implementation, the image features corresponding to the first sample image at least two scales include global image features and local image features at at least one scale; the image feature extraction branch includes a global feature extraction layer and at least one local feature extraction layer; the global feature extraction layer includes at least two feature extraction layers; the local feature extraction layer includes at least one feature extraction layer; the feature extraction layer is used to extract image features; based on the first sample image and each feature extraction layer of the global feature extraction layer, the global features of the first sample image are obtained.

[0196] In one possible implementation, the global feature extraction layer acquires the first sample image, extracts the image features of the first sample image through the first feature extraction layer of the global feature extraction layer, and uses it as the feature map of the first sample image. The feature map is then passed to the next feature extraction layer so that the next feature extraction layer can extract features from the feature map. The above process is repeated until the last feature extraction layer of the global feature extraction layer extracts features from the feature map extracted by the previous feature extraction layer to obtain the global features corresponding to the sample image.

[0197] In one possible implementation, the image feature extraction branch can be a ResNet (Deep Residual Network) for image recognition, where each feature extraction layer is a Resblock (residual module) of the ResNet. The Resblock is used to progressively extract image features through residual connections. This ResNet network model comes from the paper "Deep Residual Learning for Image Recognition," in which engineers such as Kaiming He designed a residual learning structure (i.e., resblock) to improve the learning rate of deep networks while addressing gradient vanishing or gradient explosion.

[0198] In one possible implementation, when the image feature extraction branch is ResNet, the number of resblocks in the global feature extraction layer of the image branch model, as well as the number of convolutional kernels, kernel size, stride, and other hyperparameters of each layer can be pre-set according to the sample images in the sample dataset to improve the recognition ability of the sample images in the sample dataset.

[0199] For example, when the resolution of the sample image is large, the stride can be increased and the number of convolution kernels can be increased to reduce the resolution of the sample text while keeping the number of Resblocks in the global feature extraction layer unchanged. At the same time, the number of parameters obtained by convolution kernels can be increased to ensure that enough image features are extracted to improve the recognition ability of the sample image.

[0200] In one possible implementation, at least one local feature of the first sample image is obtained based on the first sample image and at least one local feature extraction layer of the image feature extraction branch.

[0201] The local feature extraction layer contains at least one feature extraction layer; the number of feature extraction layers in the local feature extraction layer is less than that in the global feature extraction layer.

[0202] In one possible implementation, the first sample image is input into the global feature extraction layer to obtain the first layer intermediate image features of the first sample image; the first layer intermediate image features are the intermediate image features corresponding to the first local feature extraction layer in the image feature extraction branch; the intermediate image features have more feature parameters than the global image features; based on the first layer intermediate image features and at least one local feature extraction layer of the image feature extraction branch, at least one local feature of the first sample image is obtained.

[0203] The first intermediate image features are obtained by extracting image features from the first sample image through a partial feature extraction layer of the global feature extraction layer.

[0204] In one possible implementation, feature extraction is performed on the first sample image based on a first specified number of feature extraction layers in the global feature extraction layer to obtain the first layer of intermediate image features of the first sample image.

[0205] The first specified number can be preset, that is, when the first sample image is subjected to feature extraction through a preset number of feature extraction layers in the global feature extraction layer, the first intermediate image features of the first sample image can be obtained and passed to the local feature extraction layer; the first specified number is less than the number of feature extraction layers in the global feature extraction layer.

[0206] After the first image sample is processed by a first specified number of feature extraction layers, the features are then passed to the local feature extraction layer. This allows the local features obtained by the local feature extraction layer to have a certain similarity to the global features, and the local features and global features can more realistically reflect the image features of the first image sample.

[0207] In one possible implementation, the first intermediate image features are segmented to obtain at least two intermediate local features corresponding to the first intermediate image features; based on the at least two intermediate local features corresponding to the first intermediate image features, intermediate fusion features corresponding to the first intermediate image features are obtained; based on the intermediate fusion features corresponding to the first intermediate image features and the first local feature extraction layer of the image feature extraction branch, the first local features of the first sample image are obtained.

[0208] In one possible implementation, the first intermediate image features are averaged based on spatial height information to obtain a first intermediate local feature and a second intermediate local feature corresponding to the first intermediate image features, wherein the spatial height of the first intermediate local feature is greater than that of the second intermediate local feature.

[0209] In one possible implementation, the first intermediate local feature and the second intermediate local feature are randomly concatenated to obtain the intermediate fusion feature corresponding to the first layer intermediate image feature.

[0210] After the first layer of intermediate image features are averaged according to height, the first intermediate local feature and the second intermediate local feature are obtained. The first intermediate local feature and the second intermediate local feature are then combined in a random order according to height (random combination means that the initial order after segmentation is shuffled and then combined, and the spatial order of each feature after shuffling is different from that before segmentation). The intermediate fusion feature corresponding to the first layer of intermediate image features is obtained, wherein the feature size of the intermediate fusion feature is the same as the feature size of the first layer of intermediate image features.

[0211] Figure 7 A schematic diagram illustrating a local feature segmentation combination according to an embodiment of this application is shown. Figure 7 As shown, intermediate image feature 701 is an image feature in the form of a feature map obtained by extracting features from the image based on a specified number of feature extraction layers. Figure 7 The example uses numbers to represent the image feature values ​​at various locations on the feature map.

[0212] The intermediate image feature 701 can be divided equally along the horizontal direction to obtain a first intermediate local feature 702 and a second intermediate local feature 703. Then, the first intermediate local feature 702 and the second intermediate local feature 703 are combined in a random order to generate an intermediate fusion feature 704. The intermediate fusion feature 704 is a different feature from the intermediate image feature 701, and the feature size of the intermediate fusion feature 704 is the same as that of the intermediate image feature 701. That is, the first intermediate local feature 702 and the second intermediate local feature 703 are combined in a spatial order different from that before the segmentation.

[0213] By segmenting and rearranging the intermediate local features in spatial orientation as described above, the influence of the spatial order between the upper and lower modules on the image features is reduced when the feature extraction layer extracts the image features. The feature extraction layer is more likely to notice local features. This achieves the goal of reducing the influence of spatial orientation features on the extraction of local features by shuffling the spatial order of the feature maps while keeping the feature map size unchanged, thereby improving the accuracy of local feature extraction.

[0214] The first layer of intermediate image features can also be segmented using other segmentation methods (such as vertical segmentation, region segmentation, etc.) and divided into more intermediate local features. These intermediate local features can be recombined by random combination, exchange of order, etc.

[0215] In one possible implementation, the first local feature includes a first local sub-feature and a second local sub-feature.

[0216] In one possible implementation, based on the intermediate fusion features corresponding to the first layer intermediate image features and the first local feature extraction layer of the image feature extraction branch, the first local features to be segmented of the first sample image are obtained; based on the first local features to be segmented of the first sample image, average segmentation is performed according to spatial height information to obtain the first local sub-features and the second local sub-features of the first sample image.

[0217] The first local feature to be segmented in the first sample image can be obtained by extracting intermediate fusion features through the feature extraction layer in the first local feature extraction layer. The intermediate fusion features are obtained by randomly splicing the first intermediate local features and the second intermediate local features. They still have the features extracted from each pixel in the sample image. Therefore, the first local feature to be segmented is still the feature corresponding to all pixels in the sample image.

[0218] The first local feature (first local sub-feature and second local sub-feature) obtained by averaging the first local feature to be segmented based on spatial height information has local features of the sample image.

[0219] In one possible implementation, based on the second specified number of feature extraction layers in the (i-1)th local feature extraction layer, feature extraction is performed on the intermediate fusion features corresponding to the (i-1)th local feature extraction layer to obtain the intermediate image features of the i-th layer; the intermediate image features of the i-th layer are segmented to obtain at least two intermediate local features corresponding to the intermediate image features of the i-th layer; based on the at least two intermediate local features corresponding to the intermediate image features of the i-th layer, the intermediate fusion features corresponding to the intermediate image features of the i-th layer are obtained; based on the intermediate fusion features corresponding to the intermediate image features of the i-th layer, and each feature extraction layer of the i-th local feature extraction layer, the local image features of the first sample image at the i-th scale are obtained; where 2≤i≤N, and i and N are integers.

[0220] When the image feature extraction branch contains at least two local feature extraction layers, the lower local feature extraction layer (i.e., the local feature extraction layer other than the first local feature extraction layer) can input the intermediate image features corresponding to the local feature extraction layer into the local feature extraction layer to obtain the local features to be segmented corresponding to the local feature extraction layer. By segmenting the local features to be segmented corresponding to the local feature extraction layer, the local features corresponding to the local feature extraction layer can be obtained.

[0221] The intermediate image features corresponding to this local feature extraction layer are obtained by extracting features from the intermediate fusion features corresponding to the previous local feature extraction layer using a specified number of feature extraction layers. Specifically, the intermediate fusion features corresponding to the local feature extraction layer can be obtained by segmenting the intermediate image features of this layer to obtain at least two intermediate local features, and then fusing these intermediate local features.

[0222] In one possible implementation, the second specified quantity is determined based on the number of layers in the local feature extraction layer.

[0223] That is, the second specified number of local feature extraction layers in each layer can be different.

[0224] In one possible implementation, the number of feature extraction layers in the local feature extraction layer is inversely related to the number of layers in the local feature extraction layer.

[0225] The larger the number of local feature extraction layers, the more times the corresponding intermediate image features are extracted by the feature extraction layers, and therefore the smaller the image resolution. At this time, the fewer feature extraction layers are needed to extract local features from the intermediate image features.

[0226] Please refer to Figure 8 This illustrates a schematic diagram of an image recognition residual network according to an embodiment of this application. Figure 8 As shown, taking the residual network with a global feature extraction layer and two local feature extraction layers as an example, the sample image 801 is input into the image recognition residual network 800, and the features of the sample image 801 are extracted through the global feature extraction layer, that is, the feature extraction layer composed of the first Resblock layer (the feature size is the image-level feature map) and global average pooling is performed to obtain global image features 811 (i.e., the image features at the first scale). The overall image feature dimension of the global image features 811 is 768.

[0227] Sample image 801 is then processed by a specified number of elements in the global feature extraction layer. Figure 8 The global feature extraction layer extracts features from only a portion of the sample image using two Resblocks. The generated intermediate image features are then transmitted to the first local feature extraction layer, where they are averaged and then randomly combined to obtain the intermediate fused feature 802 corresponding to the first local feature extraction layer. The specific steps of segmentation and combination are as follows: Figure 7 As shown, it will not be elaborated further here.

[0228] The intermediate fusion feature 802 corresponding to the first local feature extraction layer continues to extract features through the resblock of the first local feature extraction layer to obtain the local segmentation feature of the first local feature extraction layer. The local segmentation feature is an image-level feature map. The local sub-features after the local segmentation feature is averaged are then subjected to global average pooling to obtain the first local feature 812 (second-scale image feature) corresponding to the first local feature extraction layer. The feature dimension of each local sub-feature in the first local feature 812 is 768.

[0229] The intermediate fusion feature 802 corresponding to the first local feature extraction layer is extracted by a specified number of Resblocks in the first local feature extraction layer, and then divided into three parts by average segmentation. The three parts of the features are then combined in random order to generate the intermediate fusion feature 803 corresponding to the second local feature extraction layer. The intermediate fusion feature 803 is further extracted by the Resblocks of the second local feature extraction layer to obtain the local to-be-segmented feature of the second local feature extraction layer. The local to-be-segmented feature is an image-level feature map. The local to-be-segmented feature is then divided into three parts of local sub-features by average segmentation and global average pooling is performed to obtain the second local feature 813 (the third-scale image feature) corresponding to the second local feature extraction layer. The feature dimension of each local sub-feature in the second local feature 813 is 768.

[0230] exist Figure 8 In this context, global image features 811, first local features 812, and second local features 813 constitute the sample image features corresponding to the sample image.

[0231] Step 504: Fuse the text features of at least two scales corresponding to the sample text to obtain the first multi-scale fusion feature corresponding to the sample text.

[0232] In one possible implementation, at least two scales of text features corresponding to the sample text are input into the first feature fusion branch of the text image matching model to obtain the first multi-scale fusion feature corresponding to the sample text.

[0233] In one possible implementation, the first feature fusion branch is a fully connected layer or a deep learning model.

[0234] In one possible implementation, the first multi-scale fusion feature of the sample text is obtained through a fully connected layer based on the text features of at least two scales of the sample text.

[0235] In convolutional neural networks, fully connected layers can be used as classifiers to map distributed feature representations to sample label spaces. This involves directly connecting global and local features, and then using fully connected layers to perform weighted summation of the corresponding dimensional information of global and local features to obtain fused features corresponding to global and local features.

[0236] For example, the global feature dimension of the sample text is 768, and there are 6 local features of the sample text, all with a feature dimension of 768. In this case, the fully connected layer directly connects the global feature and the local feature to form a connected feature with a dimension of 7*768. The fully connected layer then performs a weighted summation of the 7 feature values ​​corresponding to the 768 dimensions of the global and local features to obtain a text fusion feature with a feature dimension of 768.

[0237] In one possible implementation of this application embodiment, the feature dimension can also be a value preset by the developer.

[0238] When fusing image or text features at different scales using fully connected layers, features at multiple different scales are weighted by adding features of the same type (i.e., features extracted by the same convolutional kernel) from features at different scales. This reduces the number of features in the fused feature set. Furthermore, by adding features of the same type extracted by the same convolutional kernel, the data characteristics corresponding to features at different scales are preserved to some extent, so that the fused feature set is influenced by features at multiple different scales simultaneously.

[0239] In another possible implementation, the text features of the sample text at at least two scales are input into the first feature fusion branch built based on a deep learning model to obtain the first multi-scale fusion feature of the sample text.

[0240] The first feature fusion branch can be a pre-trained, directly usable deep learning model; or the first feature fusion branch needs to be trained using the sample text and the first sample image.

[0241] Step 505: Fuse image features of at least two scales corresponding to the first sample image to obtain the second multi-scale fused feature corresponding to the first sample image.

[0242] In one possible implementation, at least two scales of text features corresponding to the first sample image are input into the second feature fusion branch of the text image matching model to obtain the first multi-scale fusion feature corresponding to the first sample image.

[0243] In one possible implementation, the second feature fusion branch is a fully connected layer or a deep learning model.

[0244] In one possible implementation, the first feature fusion branch and the second feature fusion branch can both be feature fusion branches with the same structure, or they can be feature fusion branches with different structures. For example, the first feature fusion branch is a fully connected layer and the second feature fusion branch is a deep learning model; or the first feature fusion branch is a deep learning model and the second feature fusion branch is a fully connected layer; or both of the first feature fusion branches are fully connected layers or deep learning models.

[0245] Step 506: Input the first multi-scale fusion feature corresponding to the sample text and the second multi-scale fusion feature corresponding to the first sample image into the loss function to obtain the loss function value corresponding to the sample text.

[0246] In one possible implementation, based on the first multi-scale fusion feature of the sample text and the second multi-scale fusion feature of the first sample image, the feature similarity between the features of the sample text and the features of the first sample image is obtained; and based on the feature similarity between the sample text and the first sample image, the loss function value corresponding to the sample text is obtained.

[0247] In one possible implementation, the vector distance between the first multi-scale fusion feature of the sample text and the second multi-scale fusion feature of the first sample image is obtained as the feature similarity between the first multi-scale fusion feature of the sample text and the second multi-scale fusion feature of the first sample image.

[0248] The vector distance can be the Euclidean distance between the first multi-scale fusion feature of the sample text and the second multi-scale fusion feature of the first sample image, or it can be the cosine distance between the first multi-scale fusion feature of the sample text and the second multi-scale fusion feature of the first sample image.

[0249] After obtaining the global and local features of the sample text and the global and local features of the first sample image, it is necessary to fuse the global and local features of the sample text to obtain the first multi-scale fusion feature, and fuse the global and local features of the first sample image to obtain the second multi-scale fusion feature. In order to determine the degree of matching between the sample text and the first sample image by comparing the similarity between the first multi-scale fusion feature and the second multi-scale fusion feature of the first sample image.

[0250] Among them, the first multi-scale fusion feature and the second multi-scale fusion feature are features obtained by fusing global features and local features. Therefore, the first multi-scale fusion feature and the second multi-scale fusion feature have feature information at both global and local scales. By matching the first multi-scale fusion feature with the second multi-scale fusion feature of the first sample image, the local feature similarity and the overall feature similarity between the text and the image can be considered at the same time, thereby improving the accuracy of text-image matching.

[0251] Step 507: Update the text-image matching model based on the loss function value corresponding to the sample text.

[0252] In one possible implementation, the loss function could be the cross-entropy loss function.

[0253] In one possible implementation, the text feature extraction branch and the image feature extraction branch can be updated using the backpropagation algorithm based on the loss function value.

[0254] The text feature extraction branch and the image feature extraction branch are updated simultaneously using the loss function value. This loss function value is obtained based on the feature similarity between the first multi-scale fusion feature of the sample text and the second multi-scale fusion feature of the first sample image. In other words, the loss function value has features of multiple scales of the sample text and the first sample image. That is, when the text feature extraction branch and the image feature extraction branch are updated according to the loss function, they simultaneously consider the features of multiple scales of the sample text and the first sample image, as well as the feature similarity between the sample text and the first sample image. The updated image feature extraction branch in the text image matching model can pay more attention to the features of different scales of the text, and the text feature extraction branch in the text image matching model can also pay more attention to the features of different scales of the image. Therefore, the text image matching model after training has a high accuracy in matching text and images.

[0255] In one possible implementation, the second sample image is input into the image feature extraction branch of the text-image matching model to obtain image features at least two scales corresponding to the second sample image; the image features at least two scales corresponding to the second sample image are fused to obtain a second multi-scale fusion feature corresponding to the second sample image; the first multi-scale fusion feature corresponding to the sample text, the second multi-scale fusion feature corresponding to the first sample image, and the second multi-scale fusion feature corresponding to the second sample image are input into a loss function to obtain a loss function value corresponding to the sample text; based on the loss function value, the text feature extraction branch and the image feature extraction branch are updated.

[0256] The loss function for this text-image matching model can also be triplet loss. The triplet loss function is shown below:

[0257]

[0258] The triplet loss function takes a triplet (a, p, n) as input, where a is the anchor (origin), i.e., the target sample; p is the positive sample, i.e., the sample of the same class as a; n is the negative sample, i.e., the sample of a different class than a; and margin is the boundary value set by the text-image matching model.

[0259] In the scheme shown in the embodiments of this application, the origin 'a' represents the sample text feature (first multi-scale fusion feature) corresponding to the sample text, the positive sample 'p' represents the first sample image feature (second multi-scale fusion feature corresponding to the first sample image) matching the sample text, and the negative sample 'n' represents the second sample image feature (second multi-scale fusion feature corresponding to the second sample image) matching the second sample text. Triplet loss is typically applied to fine-grained recognition at the individual level. Traditional classification is for broad categories like flowers, birds, and dogs, but some requirements demand precision down to the individual level, such as facial recognition of a specific person. In text-image matching, triplet loss can improve the matching accuracy between images and text by selecting the image closest to the text from similar candidate images as the target image for matching.

[0260] In one possible implementation, when the sample text is the text description information corresponding to a face, the loss function can also be arcface (Additive Angular Margin Loss).

[0261] In one possible implementation, the loss function could also be CMPM (Cross-Modal Projection Matching) or CMPC (Cross-Modal Projection Classification).

[0262] In this process, CMPM constructs a probability distribution for the matching of image-text pairs in each batch, and uses the KL (Kullback-Leibler) divergence of the network's predicted and ground values ​​to constrain the network, hoping that the projection of the network from one modality to another will be as close as possible to the true distribution.

[0263] CMPM constructs a normalized softmax classification objective function by mimicking Norm-softmax (a normalized flexible maximum transfer function), which performs projections from one modality to another. The weights of the fully connected layers connected to the softmax function are normalized, ensuring that the optimization objective is to achieve good classification between different projections.

[0264] Step 508: Obtain the first text and the first image.

[0265] In one possible implementation, the first text is of the same type as the sample text.

[0266] That is, the first text and the sample text are in the same language, and the content described by the first text and the sample text can be of the same type. For example, the appearance and clothing features of the person described in the sample text are more likely to be considered by the text-image matching model trained based on the sample text and the corresponding first sample image. Therefore, when the first text is also a description of a person's appearance, its matching accuracy is higher.

[0267] Step 509: Input the first text into the text feature extraction branch of the text image matching model to obtain at least two scales of text features corresponding to the first text.

[0268] The text feature extraction branch is a neural network model used to extract text features.

[0269] In one possible implementation, at least two scales of sub-text of the first text are obtained; the at least two scales of sub-text include global text and local text of at least one scale; the scale of the global text is larger than the scale of the local text; the at least two scales of sub-text of the first text are input into the text feature extraction branch of the text image model to obtain the text features of at least two scales corresponding to the first text.

[0270] The global text contains all the text content of the first text, while the local text contains less text content than the global text. The global text and at least one scale of local text are input into the text feature extraction branch to obtain text features at at least two scales, namely, the text features at at least two scales include global text features and at least one local text feature.

[0271] In one possible implementation, the text feature extraction method for the first text can also be achieved through... Figure 6 As shown, the BERT model is used to extract features at multiple scales from the first text simultaneously.

[0272] Step 510: Input the text features of at least two scales into the first feature fusion branch of the text image matching model to obtain the first multi-scale fusion feature corresponding to the first text.

[0273] In one possible implementation, the first feature fusion branch is a fully connected layer or a deep learning model.

[0274] In one possible implementation, the first multi-scale fusion feature of the first text is obtained through a fully connected layer based on the text features of the first text at at least two scales.

[0275] In convolutional neural networks, fully connected layers can be used as classifiers to map distributed feature representations to sample label spaces. This involves directly connecting global and local features, and then using fully connected layers to perform weighted summation of the corresponding dimensional information of global and local features to obtain fused features corresponding to global and local features.

[0276] For example, the global feature dimension of the first text is 768, and the local feature of the first text has 6 features, all with a feature dimension of 768. In this case, the fully connected layer directly connects the global feature and the local feature to form a connected feature with a dimension of 7*768. The fully connected layer then performs a weighted summation of the 7 feature values ​​corresponding to the 768 dimensions of the global and local features to obtain the text fusion feature with a feature dimension of 768.

[0277] In one possible implementation of this application embodiment, the feature dimension can also be a value preset by the developer.

[0278] When fusing image or text features at different scales using fully connected layers, features at multiple different scales are weighted by adding features of the same type (i.e., features extracted by the same convolutional kernel) from features at different scales. This reduces the number of features in the fused feature set. Furthermore, by adding features of the same type extracted by the same convolutional kernel, the data characteristics corresponding to features at different scales are preserved to some extent, so that the fused feature set is influenced by features at multiple different scales simultaneously.

[0279] In another possible implementation, at least two scales of text features of the first text are input into a first feature fusion branch constructed based on a deep learning model to obtain the first multi-scale fused features of the first text. The first feature fusion branch can be a pre-trained, directly usable deep learning model; or the first feature fusion branch can be trained using the first text and the first sample image.

[0280] Step 511: Input the first image into the image feature extraction branch of the text image matching model to obtain image features at least two scales corresponding to the first image.

[0281] In one possible implementation, the image features at least two scales include global image features and local image features at at least one scale; the image feature extraction branch includes a global feature extraction layer and at least one local feature extraction layer; the global feature extraction layer includes at least two feature extraction layers; the local feature extraction layer includes at least one feature extraction layer; the feature extraction layer is used to extract image features; based on the first image and the global feature extraction layer in the image feature extraction branch, the global image features corresponding to the first image are obtained; based on the first image and at least one local feature extraction layer in the image feature extraction branch, the local image features corresponding to the first image at at least one scale are obtained.

[0282] In one possible implementation, the global feature extraction layer acquires the first image, extracts the image features of the first image through the first feature extraction layer in the global feature extraction layer, and uses it as the feature map of the first image. The feature map is then passed to the next feature extraction layer so that the next feature extraction layer can extract features from the feature map. The above process is repeated until the last feature extraction layer of the global feature extraction layer extracts features from the feature map extracted by the previous feature extraction layer to obtain the global features corresponding to the sample image.

[0283] In one possible implementation, the image feature extraction branch can be a ResNet (residual network) for image recognition, wherein each feature extraction layer is a Resblock (residual module) of the ResNet, and the Resblock is used to progressively extract image features from the image through residual connections.

[0284] In one possible implementation, when the image feature extraction branch is ResNet, the number of resblocks in the global feature extraction layer of the image branch model, as well as the number of convolution kernels, kernel size, stride, and other hyperparameters of each layer, can be pre-set according to the image parameters (e.g., resolution) of the first image, in order to improve the feature extraction capability of the first image.

[0285] For example, when the resolution of the sample image is large, the stride can be increased and the number of convolution kernels can be increased to reduce the resolution of the first image while keeping the number of Resblocks in the global feature extraction layer unchanged. At the same time, the number of parameters obtained by convolution kernels can be increased to ensure that enough image features are extracted to improve the recognition ability of the first image.

[0286] In one possible implementation, based on a first specified number of feature extraction layers in the global feature extraction layer, feature extraction is performed on the first image to obtain the first layer intermediate image features corresponding to the first image; the first layer intermediate image features are segmented to obtain at least two intermediate local features corresponding to the first layer intermediate image features; the at least two intermediate local features corresponding to the first layer intermediate image features are concatenated to obtain the first layer intermediate fusion feature; the first layer intermediate fusion feature is an image feature different from the first layer intermediate image features; the first layer intermediate fusion feature has the same size as the first layer intermediate image features; based on the first layer intermediate fusion feature and at least one local feature extraction layer in the image feature extraction branch, at least one scale of local image features corresponding to the first image are obtained.

[0287] The first specified number can be preset, that is, when the first image is extracted through a preset number of feature extraction layers in the global feature extraction layer, the first intermediate image features of the first image can be obtained and passed to the local feature extraction layer; the first specified number is less than the number of feature extraction layers in the global feature extraction layer.

[0288] After the first image is processed by a first specified number of feature extraction layers, the features are then passed to the local feature extraction layer. This allows the local features obtained by the local feature extraction layer to have a certain similarity to the global features, and the local features and global features can more realistically reflect the image features of the first image.

[0289] The first-layer intermediate fusion feature is formed by stitching together at least two intermediate local features corresponding to the first-layer intermediate image feature. The first-layer intermediate fusion feature is a different image feature from the first-layer intermediate image feature. That is, after the first-layer intermediate image feature is segmented into two intermediate local features, the spatial order of the at least two intermediate local features is shuffled and then stitched together to obtain the first-layer intermediate fusion feature with the same size as the first-layer intermediate image feature but different features.

[0290] In one possible implementation, the first intermediate image features are averaged based on spatial height information to obtain a first intermediate local feature and a second intermediate local feature corresponding to the first intermediate image features, wherein the spatial height of the first intermediate local feature is greater than that of the second intermediate local feature.

[0291] In one possible implementation, the first intermediate local feature and the second intermediate local feature are randomly concatenated to obtain the intermediate fusion feature corresponding to the first layer intermediate image feature.

[0292] After the first layer intermediate image features are averaged according to height, the first intermediate local feature and the second intermediate local feature are obtained. The first intermediate local feature and the second intermediate local feature are then randomly combined according to height to obtain the intermediate fusion feature corresponding to the first layer intermediate image features. The feature size of the intermediate fusion feature is the same as the feature size of the first layer intermediate image features.

[0293] That is, the process of segmenting and randomly combining the features of the intermediate image corresponding to the first image can be as follows: Figure 7 As shown, by Figure 7 As shown, the intermediate local features are spatially segmented and then randomly combined. When the feature extraction layer extracts image features, the spatial order between the upper and lower modules has less impact on the image features. The feature extraction layer is more likely to notice local features. This reduces the influence of spatial orientation features on local feature extraction by shuffling the spatial order of the feature map while keeping the feature map size unchanged, thus improving the accuracy of local feature extraction.

[0294] The first layer of intermediate image features can also be segmented using other segmentation methods (such as vertical segmentation, region segmentation, etc.) and divided into more intermediate local features. These intermediate local features can be recombined by sequentially exchanging and combining them.

[0295] In one possible implementation, local image features of a first scale corresponding to the first image are obtained based on the first layer of intermediate image features and the first local feature extraction layer in the image feature extraction branch.

[0296] In one possible implementation, based on the second specified number of feature extraction layers in the (i-1)th local feature extraction layer, feature extraction is performed on the intermediate fusion feature of the (i-1)th layer to obtain the intermediate image feature of the i-th layer; the intermediate image feature of the i-th layer is segmented to obtain at least two intermediate local features corresponding to the intermediate image feature of the i-th layer; the at least two intermediate local features corresponding to the intermediate image feature of the i-th layer are concatenated to obtain the intermediate fusion feature of the i-th layer; the intermediate fusion feature of the i-th layer is an image feature different from the intermediate image feature of the i-th layer; the intermediate fusion feature of the i-th layer has the same size as the intermediate image feature of the i-th layer; based on the intermediate fusion feature of the i-th layer and the i-th local feature extraction layer in the image feature extraction branch, the local image feature of the i-th scale corresponding to the first image is obtained; where 2≤i≤N, and i and N are integers.

[0297] When the image feature extraction branch contains at least two local feature extraction layers, the lower local feature extraction layer (i.e., the local feature extraction layer other than the first local feature extraction layer) can input the intermediate image features corresponding to the local feature extraction layer into the local feature extraction layer to obtain the local features to be segmented corresponding to the local feature extraction layer. By segmenting the local features to be segmented corresponding to the local feature extraction layer, the local features corresponding to the local feature extraction layer can be obtained.

[0298] The intermediate image features corresponding to this local feature extraction layer are obtained by extracting features from the intermediate fusion features corresponding to the previous local feature extraction layer using a specified number of feature extraction layers. Specifically, the intermediate fusion features corresponding to the local feature extraction layer can be obtained by segmenting the intermediate image features of this layer to obtain at least two intermediate local features, and then fusing these intermediate local features.

[0299] In one possible implementation, the second specified quantity is determined based on the number of layers in the local feature extraction layer.

[0300] That is, the second specified number of local feature extraction layers in each layer can be different.

[0301] In one possible implementation, the number of feature extraction layers in the local feature extraction layer is inversely related to the number of layers in the local feature extraction layer.

[0302] The larger the number of local feature extraction layers, the more times the corresponding intermediate image features are extracted by the feature extraction layers, and therefore the smaller the image resolution. At this time, the fewer feature extraction layers are needed to extract local features from the intermediate image features.

[0303] Step 512: Input the image features of at least two scales into the second feature fusion branch of the text image matching model to obtain the second multi-scale fusion feature corresponding to the first image.

[0304] In one possible implementation, the second feature fusion branch is a fully connected layer or a deep learning model.

[0305] In one possible implementation, the first feature fusion branch and the second feature fusion branch can both be feature fusion branches with the same structure, or they can be feature fusion branches with different structures. For example, the first feature fusion branch is a fully connected layer and the second feature fusion branch is a deep learning model; or the first feature fusion branch is a deep learning model and the second feature fusion branch is a fully connected layer; or both of the first feature fusion branches are fully connected layers or deep learning models.

[0306] Step 513: Based on the first multi-scale fusion feature corresponding to the first text and the second multi-scale fusion feature corresponding to the first image, obtain the feature similarity between the first text and the first image.

[0307] In one possible implementation, the cosine distance between the first multi-scale fusion feature and the second multi-scale fusion feature is obtained, and the cosine distance is used as the feature similarity between the first multi-scale fusion feature and the second multi-scale fusion feature.

[0308] In another possible implementation, the Euclidean distance between the first multi-scale fusion feature and the second multi-scale fusion feature is obtained, and the Euclidean distance is used as the feature similarity between the first multi-scale fusion feature and the second multi-scale fusion feature.

[0309] The first multi-scale fusion feature corresponding to the first text retains the text features of the first text at different scales to a certain extent, and the second multi-scale fusion feature corresponding to the first image also retains the image features of the first image at different scales to a certain extent. Therefore, the feature similarity between the first text and the first image obtained by the first multi-scale fusion feature and the second multi-scale fusion feature is the feature similarity between the first text and the first image obtained by the influence of the similarity between the image features at different scales and the text features at different scales. This feature similarity includes the similarity relationship between text features and image features at the same scale, the similarity relationship between text features and image features at different scales, and the similarity relationship after the fusion of text features at different scales and the fusion of image features at different scales. Therefore, this feature similarity can better represent the similarity between the first text and the first image.

[0310] Step 514: Based on the feature similarity, determine the matching relationship between the first text and the first image.

[0311] In one possible implementation, all candidate images in the candidate image set are input into the image feature extraction branch to obtain the candidate image features corresponding to all candidate images. Based on the feature similarity between the candidate image features of each candidate image and the target text features, the image features are sorted according to the feature similarity. The candidate image with the highest feature similarity is obtained as the target image that matches the first text.

[0312] Please refer to Figure 9 This illustrates a schematic diagram of a text-image matching model training method according to an embodiment of this application. Figure 9As shown, the first sample image 911 is input into the image feature extraction branch 910 of the text image matching model to obtain the sample image features corresponding to the first sample image 911; the sample text 921 is input into the text feature extraction branch 920 (i.e., the BERT model) of the text image matching model, and the sample text features corresponding to the sample text 921 are input. Then, the image fusion feature obtained by fusing the sample image features through a fully connected layer and the text fusion feature obtained by fusing the sample text features through a fully connected layer are input into the loss function 903, and the backpropagation algorithm is used to update the back gradient of the text feature extraction branch 920 and the image feature extraction branch 910 according to the loss function value.

[0313] Please refer to Figure 10 This illustration shows a text-image matching application based on an embodiment of this application. Figure 10 As shown, the text-image matching method involved in this application embodiment can be applied to finding people based on text. For example, by using the information in the missing person notice 1001, the target text to be searched (i.e., the descriptive text of the person's appearance and clothing) is determined, and the target text is input into the text-image matching application 1002. At this time, the text-image matching model obtains the target text, and according to the candidate image set corresponding to the text-image matching model, performs text-image matching with the target text using the method shown in this application embodiment, and outputs the target image containing the candidate person that matches the target text.

[0314] In summary, the solution presented in this application obtains features at least two scales corresponding to the first text and the first image, fuses the text features at least two scales into a first multi-scale fusion feature, fuses the image features at least two scales into a second multi-scale fusion feature, and determines the matching relationship between the first text and the first image based on the similarity between the first multi-scale fusion feature and the second multi-scale fusion feature. This solution determines the matching relationship between text and image by using the similarity between the fusion features of text and image at multiple scales, while also considering the feature similarity between text and image at different feature scales, thus improving the matching accuracy between text and image.

[0315] Please refer to Figure 11 This is a schematic diagram illustrating a model training and text-image matching framework according to an exemplary embodiment. The model training process is executed in the model training device 1100, and the text-image matching process is executed in the text-image matching device 1110, as follows... Figure 11As shown, the first sample image 1101 is input into the image feature extraction branch 1102 of the text image matching model. Based on the global feature extraction layer and at least one local feature extraction layer in the image feature extraction branch, the first sample image 1101 is subjected to feature extraction and global average pooling to obtain the sample image feature 1103 corresponding to the first sample image 1101. The sample image feature 1103 contains the global features and local features of the first sample image. The global features and local features are fused through a fully connected layer and input into the loss function 1107.

[0316] The sample text 1104 is input into the text feature extraction branch 1105 of the text-image matching model. This text feature extraction branch can be a BERT model. The BERT model obtains sample text features 1106 based on the CLS identifier in the sample text. These sample text features 1106 include global text features, clause text features, and word text features. The global text features, clause text features, and word text features are then fused together using a fully connected layer and input into the loss function 1107. The loss function obtains feature similarity based on the fused features of the sample image features 1103 and the fused features of the sample text features 1106, and outputs a loss function value based on this feature similarity. The text-image matching model then performs backpropagation based on this loss function value to update the text feature extraction branch 1105 and the image feature extraction branch 1102.

[0317] After the text feature extraction branch and the image feature extraction branch are trained in the model training device 1100, the text image matching model 1112 is transmitted to the text image matching device 1110. When the first text 1111 is input, the text image matching model 1112 determines the matching relationship between each candidate image in the corresponding candidate image set 1113 and the first text, and selects the candidate image with the highest feature similarity to the target text 1111 as the target image 1114 for output.

[0318] Figure 12 This is a structural block diagram illustrating a text-image matching device according to an exemplary embodiment. The text-image matching device can realize the following: Figure 4 or Figure 5 The text image matching model training apparatus includes all or part of the steps in the method provided in the illustrated embodiment:

[0319] The text-image acquisition module 1201 is used to acquire the first text and the first image;

[0320] The text feature acquisition module 1202 is used to acquire text features at least two scales corresponding to the first text;

[0321] The first feature fusion module 1203 is used to fuse text features of at least two scales corresponding to the first text to obtain the first multi-scale fusion feature corresponding to the first text.

[0322] The image feature acquisition module 1204 is used to acquire image features of at least two scales corresponding to the first image;

[0323] The second feature fusion module 1205 is used to fuse the image features of at least two scales to obtain the second multi-scale fusion feature corresponding to the first image.

[0324] The similarity acquisition module 1206 is used to acquire the feature similarity between the first text and the first image based on the first multi-scale fusion feature corresponding to the first text and the second multi-scale fusion feature corresponding to the first image.

[0325] The matching relationship acquisition module 1207 is used to determine the matching relationship between the first text and the first image based on the feature similarity.

[0326] In one possible implementation, the first feature fusion module 1203 is used to input the text features of at least two scales into the first feature fusion branch of the text image matching model to obtain the first multi-scale fusion feature corresponding to the first text.

[0327] The second feature fusion module 1205 is used to input the image features of at least two scales into the second feature fusion branch of the text image matching model to obtain the second multi-scale fusion feature corresponding to the first image.

[0328] In one possible implementation, the first feature fusion branch is a fully connected layer or a deep learning model;

[0329] Alternatively, the second feature fusion branch may be the fully connected part or a deep learning model.

[0330] In one possible implementation, the image feature acquisition module 1204 is used for,

[0331] The first image is input into the image feature extraction branch of the text image matching model to obtain image features at least two scales corresponding to the first image.

[0332] In one possible implementation, the image features at least two scales include global image features and local image features at at least one scale; the image feature extraction branch includes a global feature extraction layer and at least one local feature extraction layer; the global feature extraction layer includes at least two feature extraction layers; the local feature extraction layer includes at least one feature extraction layer; the feature extraction layer is used to extract image features;

[0333] The image feature acquisition module 1204 includes:

[0334] A global image feature acquisition unit is used to acquire global image features corresponding to the first image based on the first image and the global feature extraction layer in the image feature extraction branch.

[0335] The local image feature acquisition unit is used to acquire local image features of at least one scale corresponding to the first image based on the first image and at least one local feature extraction layer in the image feature extraction branch.

[0336] In one possible implementation, the local image feature acquisition unit includes:

[0337] The first intermediate feature acquisition subunit is used to extract features from the first image based on a first specified number of feature extraction layers in the global feature extraction layer, and to obtain the first layer intermediate image features corresponding to the first image.

[0338] The intermediate local feature acquisition subunit is used to segment the intermediate image features of the first layer and obtain at least two intermediate local features corresponding to the intermediate image features of the first layer.

[0339] The local feature stitching subunit is used to stitch together at least two intermediate local features corresponding to the intermediate image features of the first layer to obtain the intermediate fusion feature of the first layer; the intermediate fusion feature of the first layer is an image feature that is different from the intermediate image features of the first layer; the size of the intermediate fusion feature of the first layer is the same as that of the intermediate image features of the first layer.

[0340] The local feature acquisition subunit is used to acquire local image features of at least one scale corresponding to the first image based on the intermediate fusion features of the first layer and at least one local feature extraction layer in the image feature extraction branch.

[0341] In one possible implementation, the image feature extraction branch includes N local feature extraction layers; the local feature acquisition subunit is used for,

[0342] Based on the intermediate image features of the first layer and the first local feature extraction layer in the image feature extraction branch, the local image features of the first scale corresponding to the first image are obtained.

[0343] Based on the second specified number of feature extraction layers in the (i-1)th local feature extraction layer, feature extraction is performed on the intermediate fusion features of the (i-1)th layer to obtain the intermediate image features of the i-th layer.

[0344] The intermediate image features of the i-th layer are segmented to obtain at least two intermediate local features corresponding to the intermediate image features of the i-th layer;

[0345] At least two intermediate local features corresponding to the intermediate image features of the i-th layer are concatenated to obtain the intermediate fusion feature of the i-th layer; the intermediate fusion feature of the i-th layer is an image feature that is different from the intermediate image features of the i-th layer; the intermediate fusion feature of the i-th layer has the same size as the intermediate image features of the i-th layer.

[0346] Based on the intermediate fusion features of the i-th layer and the i-th local feature extraction layer in the image feature extraction branch, the local image features of the i-th scale corresponding to the first image are obtained; where 2≤i≤N, and i and N are integers.

[0347] In one possible implementation, the text feature acquisition module 1202 is used to,

[0348] The first text is input into the text feature extraction branch of the text image matching model to obtain at least two scales of text features corresponding to the first text; the text feature extraction branch is a neural network model used to extract text features.

[0349] In one possible implementation, the text feature acquisition module 1202 is used to,

[0350] Obtain at least two scales of sub-text from the first text; the at least two scales of sub-text include global text and at least one scale of local text; the scale of the global text is larger than the scale of the local text;

[0351] Input at least two scales of subtext of the first text into the text feature extraction branch of the text image model to obtain at least two scales of text features corresponding to the first text.

[0352] In one possible implementation, the device further includes:

[0353] The training sample set acquisition module is used to acquire a training sample set, which includes sample text and sample images that match the sample text.

[0354] The sample text feature acquisition module 1202 is used to input the sample text into the text feature extraction branch of the text image matching model to acquire at least two scales of text features corresponding to the sample text.

[0355] The sample image feature acquisition module 1204 is used to input the first sample image into the image feature extraction branch of the text image matching model to acquire image features of at least two scales corresponding to the first sample image.

[0356] The sample text fusion module is used to fuse text features of at least two scales corresponding to the sample text to obtain the first multi-scale fusion feature corresponding to the sample text.

[0357] The sample image fusion module is used to fuse image features of at least two scales corresponding to the first sample image to obtain the second multi-scale fusion feature corresponding to the sample image.

[0358] The loss function value acquisition module is used to input the first multi-scale fusion feature corresponding to the sample text and the second multi-scale fusion feature corresponding to the first sample image into the loss function to obtain the loss function value corresponding to the sample text.

[0359] The matching model update module is used to update the text image matching model based on the loss function value corresponding to the sample text.

[0360] In summary, the solution presented in this application obtains features at least two scales corresponding to the first text and the first image, fuses the text features at least two scales into a first multi-scale fusion feature, fuses the image features at least two scales into a second multi-scale fusion feature, and determines the matching relationship between the first text and the first image based on the similarity between the first multi-scale fusion feature and the second multi-scale fusion feature. This solution determines the matching relationship between text and image by using the similarity between the fusion features of text and image at multiple scales, while also considering the feature similarity between text and image at different feature scales, thus improving the matching accuracy between text and image.

[0361] Figure 13 This is a structural block diagram illustrating a text-image matching device according to an exemplary embodiment. The text-image matching device can realize the following: Figure 2 or Figure 6 All or part of the steps in the method provided in the illustrated embodiment. The text-image matching apparatus includes:

[0362] The training sample set acquisition module 1301 is used to acquire a training sample set, which includes sample text and sample images that match the sample text.

[0363] The sample text feature acquisition module 1302 is used to input the sample text into the text feature extraction branch of the text image matching model to acquire at least two scales of text features corresponding to the sample text.

[0364] The sample image feature acquisition module 1303 is used to input the first sample image into the image feature extraction branch of the text image matching model to acquire at least two scales of image features corresponding to the first sample image.

[0365] The sample text fusion module 1304 is used to fuse text features of at least two scales corresponding to the sample text to obtain the first multi-scale fusion feature corresponding to the sample text.

[0366] The sample image fusion module 1305 is used to fuse image features of at least two scales corresponding to the first sample image to obtain a second fused feature corresponding to the sample image.

[0367] The loss function value acquisition module 1306 is used to input the second multi-scale fusion feature corresponding to the sample text and the second fusion feature corresponding to the first sample image into the loss function to obtain the loss function value corresponding to the sample text.

[0368] The matching model update module 1307 is used to update the text image matching model based on the loss function value corresponding to the sample text.

[0369] In summary, the solution presented in this application obtains features at least two scales corresponding to the first text and the first image, fuses the text features at least two scales into a first multi-scale fusion feature, fuses the image features at least two scales into a second multi-scale fusion feature, and determines the matching relationship between the first text and the first image based on the similarity between the first multi-scale fusion feature and the second multi-scale fusion feature. This solution determines the matching relationship between text and image by using the similarity between the fusion features of text and image at multiple scales, while also considering the feature similarity between text and image at different feature scales, thus improving the matching accuracy between text and image.

[0370] Figure 14This is a schematic diagram illustrating the structure of a computer device according to an exemplary embodiment. The computer device can be implemented as a model training device and / or a text-image matching device in the various method embodiments described above. The computer device 1400 includes a central processing unit (CPU) 1401, a system memory 1404 including random access memory (RAM) 1402 and read-only memory (ROM) 1403, and a system bus 1405 connecting the system memory 1404 and the central processing unit 1401. The computer device 1400 also includes a basic input / output system 1406 to facilitate information transfer between various devices within the computer, and a mass storage device 1407 for storing the operating system 1413, application programs 1414, and other program modules 1415.

[0371] The mass storage device 1407 is connected to the central processing unit 1401 via a mass storage controller (not shown) connected to the system bus 1405. The mass storage device 1407 and its associated computer-readable media provide non-volatile storage for the computer device 1400. That is, the mass storage device 1407 may include computer-readable media (not shown), such as a hard disk or a compact disc read-only memory (CD-ROM) drive.

[0372] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, flash memory or other solid-state storage technologies, CD-ROM, or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that the computer storage media are not limited to the above-mentioned types. The system memory 1404 and mass storage device 1407 described above can be collectively referred to as memory.

[0373] Computer device 1400 can be connected to the Internet or other network devices via network interface unit 1411 connected to the system bus 1405.

[0374] The memory also includes one or more programs, which are stored in the memory, and the central processing unit 1401 implements these programs by executing them. Figure 2 , Figure 4or Figure 5 All or part of the steps of the method shown.

[0375] In exemplary embodiments, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory including a computer program (instructions) that can be executed by a processor of a computer device to perform the methods shown in the various embodiments of this application. For example, the non-transitory computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0376] In an exemplary embodiment, a computer program product or computer program is also provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods shown in the various embodiments described above.

[0377] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.

[0378] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method of matching text images, characterized by, The method includes: Get the first text and the first image; Obtain at least two scales of text features corresponding to the first text; fuse the at least two scales of text features to obtain a first multi-scale fused feature corresponding to the first text; the scale of the text feature represents the text size corresponding to the feature vector extraction of the text by the deep neural network; Based on the first image and the global feature extraction layer in the image feature extraction branch of the text image matching model, global image features of the first image are obtained; the global feature extraction layer includes at least two feature extraction layers; the image feature extraction branch also includes N local feature extraction layers; based on a first specified number of feature extraction layers in the global feature extraction layer, features are extracted from the first image to obtain a first layer of intermediate image features; the first layer of intermediate image features are averaged and then randomly combined to obtain at least two intermediate local features of the first layer of intermediate image features; the at least two intermediate local features of the first layer of intermediate image features are concatenated to obtain a first layer of intermediate fusion features; based on the first layer of intermediate image features and the first local feature extraction layer, local image features of a first scale are obtained; based on the first... In the i-1 local feature extraction layers, a second specified number of feature extraction layers are used to extract features from the intermediate fusion features of the i-1th layer to obtain intermediate image features of the i-th layer; the intermediate image features of the i-th layer are averaged and then randomly combined to obtain at least two intermediate local features of the intermediate image features of the i-th layer; the at least two intermediate local features corresponding to the intermediate image features of the i-th layer are concatenated to obtain intermediate fusion features of the i-th layer; based on the intermediate fusion features of the i-th layer and the i-th local feature extraction layer, local image features of the i-th scale are obtained; where 2≤i≤N, and i and N are integers; image features of at least two scales are fused to obtain the second multi-scale fusion features corresponding to the first image; the image features of at least two scales include the global image features and the local image features of at least one scale; Based on the first multi-scale fusion feature corresponding to the first text and the second multi-scale fusion feature corresponding to the first image, the feature similarity between the first text and the first image is obtained. Based on the feature similarity, the matching relationship between the first text and the first image is determined.

2. The method according to claim 1, characterized in that, The step of fusing the text features at least two scales to obtain the first multi-scale fused feature corresponding to the first text includes: The text features at at least two scales are input into the first feature fusion branch of the text image matching model to obtain the first multi-scale fusion feature corresponding to the first text; The step of fusing the image features at least two scales to obtain the second multi-scale fused feature corresponding to the first image includes: The image features at least two scales are input into the second feature fusion branch of the text image matching model to obtain the second multi-scale fusion feature corresponding to the first image.

3. The method according to claim 2, characterized in that, The first feature fusion branch is a fully connected layer or a deep learning model; Alternatively, the second feature fusion branch may be the fully connected layer or the deep learning model.

4. The method according to claim 2 or 3, characterized in that, The step of obtaining text features at least two scales corresponding to the first text includes: The first text is input into the text feature extraction branch of the text image matching model to obtain at least two scales of text features corresponding to the first text; the text feature extraction branch is a neural network model used to extract text features.

5. The method according to claim 4, characterized in that, The step of inputting the first text into the text feature extraction branch of the text image matching model to obtain text features at least two scales corresponding to the first text includes: Obtain at least two scales of sub-text from the first text; the at least two scales of sub-text include global text and at least one scale of local text; the scale of the global text is larger than the scale of the local text; Input at least two scales of subtext of the first text into the text feature extraction branch of the text image model to obtain at least two scales of text features corresponding to the first text.

6. The method according to any one of claims 1, 2, 3, and 5, characterized in that, The method further includes: Obtain a training sample set, which contains sample text and a first sample image that matches the sample text; The sample text is input into the text feature extraction branch of the text image matching model to obtain text features at least two scales corresponding to the sample text; The first sample image is input into the image feature extraction branch of the text image matching model to obtain image features at least two scales corresponding to the first sample image; By fusing text features at least two scales corresponding to the sample text, a first multi-scale fusion feature corresponding to the sample text is obtained. The image features at least two scales corresponding to the first sample image are fused to obtain the second multi-scale fusion feature corresponding to the first sample image. Input the first multi-scale fusion feature corresponding to the sample text and the second multi-scale fusion feature corresponding to the first sample image into the loss function to obtain the loss function value corresponding to the sample text; The text-image matching model is updated based on the loss function value corresponding to the sample text.

7. A method for training a text-image matching model, characterized in that, The method includes: Obtain a training sample set, which contains sample text and a first sample image that matches the sample text; The sample text is input into the text feature extraction branch of the text image matching model to obtain text features at least two scales corresponding to the sample text; the scale of the text features represents the text size corresponding to the feature vector extraction of the text by the deep neural network; Based on the first sample image and the global feature extraction layer in the image feature extraction branch of the text image matching model, the global image features of the first sample image are obtained; the global feature extraction layer includes at least two feature extraction layers; the image feature extraction branch also includes N local feature extraction layers; based on a first specified number of feature extraction layers in the global feature extraction layer, feature extraction is performed on the first sample image to obtain the first layer of intermediate image features; the first layer of intermediate image features are averaged and then randomly combined to obtain at least two intermediate local features of the first layer of intermediate image features; the at least two intermediate local features of the first layer of intermediate image features are concatenated to obtain the first layer of intermediate fused features; based on the... The process involves: defining the first layer of intermediate image features and the first local feature extraction layer to obtain local image features at a first scale; extracting features from the intermediate fusion features of the (i-1)th layer based on the second specified number of feature extraction layers in the (i-1)th local feature extraction layer to obtain intermediate image features of the i-th layer; dividing the intermediate image features of the i-th layer into equal parts and combining them in random order to obtain at least two intermediate local features of the intermediate image features of the i-th layer; concatenating the at least two intermediate local features corresponding to the intermediate image features of the i-th layer to obtain intermediate fusion features of the i-th layer; and obtaining local image features at the i-th scale based on the intermediate fusion features of the i-th layer and the i-th local feature extraction layer; where 2≤i≤N, and i and N are integers. By fusing text features at least two scales corresponding to the sample text, a first multi-scale fusion feature corresponding to the sample text is obtained. The image features at least two scales corresponding to the first sample image are fused to obtain the second multi-scale fusion feature corresponding to the first sample image; the image features at least two scales include the global image features and the local image features at at least one scale; Input the first multi-scale fusion feature corresponding to the sample text and the second multi-scale fusion feature corresponding to the first sample image into the loss function to obtain the loss function value corresponding to the sample text; The text-image matching model is updated based on the loss function value corresponding to the sample text.

8. A text-image matching device, characterized in that, The device includes: The text-image acquisition module is used to acquire the first text and the first image; The text feature acquisition module is used to acquire at least two scales of text features corresponding to the first text; the scale of the text features represents the text size when the deep neural network extracts the feature vector of the text; The first feature fusion module is used to fuse text features of at least two scales corresponding to the first text to obtain the first multi-scale fusion feature corresponding to the first text. An image feature acquisition module is used to acquire global image features of the first image based on the first image and the global feature extraction layer in the image feature extraction branch of the text image matching model; the global feature extraction layer includes at least two feature extraction layers; the image feature extraction branch also includes N local feature extraction layers; based on a first specified number of feature extraction layers in the global feature extraction layer, feature extraction is performed on the first image to obtain a first layer of intermediate image features; the first layer of intermediate image features are averaged and then randomly combined to obtain at least two intermediate local features of the first layer of intermediate image features; the at least two intermediate local features of the first layer of intermediate image features are concatenated to obtain a first layer of intermediate fused features; based on The first layer of intermediate image features and the first local feature extraction layer are used to obtain local image features at a first scale; based on the second specified number of feature extraction layers in the (i-1)th local feature extraction layer, feature extraction is performed on the (i-1)th layer intermediate fusion features to obtain the i-th layer intermediate image features; the i-th layer intermediate image features are averaged and then randomly combined to obtain at least two intermediate local features of the i-th layer intermediate image features; the at least two intermediate local features corresponding to the i-th layer intermediate image features are concatenated to obtain the i-th layer intermediate fusion features; based on the i-th layer intermediate fusion features and the i-th local feature extraction layer, local image features at the i-th scale are obtained; where 2≤i≤N, and i and N are integers; The second feature fusion module is used to fuse the image features at least two scales to obtain the second multi-scale fusion feature corresponding to the first image; the image features at least two scales include the global image features and the local image features at at least one scale; The similarity acquisition module is used to acquire the feature similarity between the first text and the first image based on the first multi-scale fusion feature corresponding to the first text and the second multi-scale fusion feature corresponding to the first image. The matching relationship acquisition module is used to determine the matching relationship between the first text and the first image based on the feature similarity.

9. An image matching model training device, characterized in that, The device includes: The training sample set acquisition module is used to acquire a training sample set, which includes sample text and a first sample image that matches the sample text. The sample text feature acquisition module is used to input the sample text into the text feature extraction branch of the text image matching model to obtain text features at least two scales corresponding to the sample text; the scale of the text feature represents the text size corresponding to the feature vector extraction of the text by the deep neural network; The sample image feature acquisition module is used to acquire global image features of the first sample image based on the first sample image and the global feature extraction layer in the image feature extraction branch of the text image matching model; the global feature extraction layer includes at least two feature extraction layers; the image feature extraction branch also includes N local feature extraction layers; based on a first specified number of feature extraction layers in the global feature extraction layer, feature extraction is performed on the first sample image to obtain a first layer of intermediate image features; the first layer of intermediate image features are averaged and then randomly combined to obtain at least two intermediate local features of the first layer of intermediate image features; the at least two intermediate local features of the first layer of intermediate image features are concatenated to obtain a first layer of intermediate fusion. Features; Based on the first layer intermediate image features and the first local feature extraction layer, obtain local image features at a first scale; Based on the second specified number of feature extraction layers in the (i-1)th local feature extraction layer, perform feature extraction on the (i-1)th layer intermediate fusion features to obtain the i-th layer intermediate image features; After the i-th layer intermediate image features are averaged and randomly combined, obtain at least two intermediate local features of the i-th layer intermediate image features; Concatenate the at least two intermediate local features corresponding to the i-th layer intermediate image features to obtain the i-th layer intermediate fusion features; Based on the i-th layer intermediate fusion features and the i-th local feature extraction layer, obtain local image features at the i-th scale; Wherein, 2≤i≤N, and i and N are integers; The sample text fusion module is used to fuse text features of at least two scales corresponding to the sample text to obtain a first multi-scale fusion feature corresponding to the sample text; the image features of at least two scales include the global image features and the local image features of at least one scale; The sample image fusion module is used to fuse image features of at least two scales corresponding to the first sample image to obtain a second fused feature corresponding to the sample image. The loss function value acquisition module is used to input the second multi-scale fusion feature corresponding to the sample text and the second fusion feature corresponding to the first sample image into the loss function to obtain the loss function value corresponding to the sample text. The matching model update module is used to update the text image matching model based on the loss function value corresponding to the sample text.

10. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one instruction, at least one program, a code set, or an instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the text image matching method as described in any one of claims 1 to 6; or, the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the text image matching model training method as described in claim 7.

11. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or instruction set is loaded and executed by a processor to implement the text image matching method as described in any one of claims 1 to 6; or, the at least one instruction, the at least one program, the code set, or instruction set is loaded and executed by the processor to implement the text image matching model training method as described in claim 7.

12. A computer program product, characterized in that, The computer program product includes computer instructions that are loaded and executed by a processor to implement the text image matching method as described in any one of claims 1 to 6; or, the computer instructions are loaded and executed by a processor to implement the text image matching model training method as described in claim 7.

Images