Image-text matching method, product retrieval method, electronic device, and storage medium

Abstract

The application discloses an image-text matching method, a product retrieval method, an electronic device and a storage medium. The method comprises the following steps: obtaining an image to be matched and text to be matched; encoding the image to be matched and the text to be matched respectively to obtain an image hidden state of the image to be matched and a text hidden state of the text to be matched; performing attention processing on the image hidden state based on a plurality of first view encodings to obtain an image feature of the image to be matched, and performing attention processing on the text hidden state based on a plurality of second view encodings to obtain a text feature of the text to be matched; and matching the image to be matched and the text to be matched based on the image feature and the text feature to obtain a target matching result. The application solves the technical problem of low accuracy of image-text matching by using a double-flow model.

Description

Technical Field

[0001] This application relates to the field of cross-modal data processing, and more specifically, to an image-text matching method, a product retrieval method, an electronic device, and a storage medium. Background Technology

[0002] Currently, in the field of image-text matching, a two-stream model is commonly used. This model uses a single feature to encode both the image and text, and determines the matching score based on the cosine similarity or inner product of the feature vectors. However, a single representation cannot cover complex content and struggles to match multiple meanings. Therefore, the performance of the two-stream model is limited, resulting in low accuracy for image-text matching methods.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This application provides an image-text matching method, a product retrieval method, an electronic device, and a storage medium to at least solve the technical problem of low accuracy in image-text matching using a two-stream model.

[0005] According to one aspect of the embodiments of this application, an image-text matching method is provided, comprising: acquiring an image to be matched and text to be matched; encoding the image to be matched and the text to be matched respectively to obtain an image hidden state of the image to be matched and a text hidden state of the text to be matched; performing attention processing on the image hidden state based on multiple first view encodings to obtain image features of the image to be matched, and performing attention processing on the text hidden state based on multiple second view encodings to obtain text features of the text to be matched; and matching the image to be matched and the text to be matched based on the image features and the text features to obtain a target matching result.

[0006] According to another aspect of the embodiments of this application, a product retrieval method is also provided, comprising: acquiring retrieval text and multiple product images; encoding the product images and retrieval text respectively to obtain the image hiding state of the product images and the text hiding state of the retrieval text; performing attention processing on the image hiding state based on multiple first view encodings to obtain image features of the product images, and performing attention processing on the text hiding state based on multiple second view encodings to obtain text features of the retrieval text; and determining a target product image that matches the retrieval text based on the multiple image features and text features.

[0007] According to another aspect of the embodiments of this application, an image-text matching method is also provided, comprising: responding to an input command applied to an operation interface, displaying an image to be matched and text to be matched on the operation interface; responding to a matching command applied to the operation interface, displaying a target matching result on the operation interface, wherein the target matching result is obtained by matching the image to be matched and the text to be matched based on the image features of the image to be matched and the text features of the text to be matched, wherein the image features are obtained by performing attention processing on the image hidden state of the image to be matched based on multiple first view codes, and the text features are obtained by performing attention processing on the text hidden state of the text to be matched based on multiple second view codes, wherein the image hidden state and the text hidden state are respectively encoded by the image to be matched and the text to be matched.

[0008] According to another aspect of the embodiments of this application, a product retrieval method is also provided, comprising: displaying retrieval text on the operation interface in response to an input command applied to the operation interface; and displaying a target product image matching the retrieval text on the operation interface in response to a matching command applied to the operation interface, wherein the target product image is determined based on the text features of the retrieval text and the image features of multiple product images, the image features are obtained by performing attention processing on the image hiding state of the product images based on multiple first view codes, and the text features are obtained by performing attention processing on the text hiding state of the retrieval text based on multiple second view codes, wherein the image hiding state and the text hiding state are obtained by encoding the product images and the retrieval text respectively.

[0009] According to another aspect of the embodiments of this application, an image-text matching method is also provided, comprising: obtaining an image to be matched and text to be matched by calling a first interface, wherein the first interface includes a first parameter, the parameter value of the first parameter being the image to be matched and the text to be matched; encoding the image to be matched and the text to be matched respectively to obtain the image hiding state of the image to be matched and the text hiding state of the text to be matched; performing attention processing on the image hiding state based on multiple first view encodings to obtain image features of the image to be matched, and performing attention processing on the text hiding state based on multiple second view encodings to obtain text features of the text to be matched; matching the image to be matched and the text to be matched based on the image features and the text features to obtain a target matching result; and outputting the target matching result by calling a second interface, wherein the second interface includes a second parameter, the parameter value of the second parameter being the target matching result.

[0010] According to another aspect of the embodiments of this application, a product retrieval method is also provided, comprising: obtaining retrieval text by calling a first interface, wherein the first interface includes a first parameter, and the parameter value of the first parameter is the retrieval text; encoding multiple product images and the retrieval text respectively to obtain image hiding states of product images and text hiding states of retrieval text; performing attention processing on the image hiding states based on multiple first view encodings to obtain image features of product images, and performing attention processing on the text hiding states based on multiple second view encodings to obtain text features of retrieval text; determining a target product image that matches the retrieval text based on multiple image features and text features; and outputting the target product image by calling a second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the target product image.

[0011] According to another aspect of the embodiments of this application, an electronic device is also provided, including: a memory storing an executable program; and a processor for running the program, wherein the program executes the method of any one of the above embodiments when it runs.

[0012] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is running, the device where the computer-readable storage medium is located executes any of the methods described above.

[0013] In this embodiment, after obtaining the image to be matched and the text to be matched, the image to be matched and the text to be matched are encoded separately to obtain the image hidden state of the image to be matched and the text hidden state of the text to be matched. Then, attention processing is performed on the image hidden state based on multiple first-view encodings to obtain the image features of the image to be matched, and attention processing is performed on the text hidden state based on multiple second-view encodings to obtain the text features of the text to be matched. Finally, based on the image features and text features, the image to be matched and the text to be matched are matched to obtain the target matching result, thus achieving the purpose of image-text matching. It is easy to note that the image features and text features finally used for image-text matching are obtained by using multi-view encoding to perform attention processing on the hidden state. This allows the image and text to be encoded from different angles, obtaining features containing more information. This allows the matching score between the image and text to be calculated from different aspects, achieving the technical effect of improving the accuracy of image-text matching, and thus solving the technical problem of low accuracy of image-text matching using a two-stream model.

[0014] It is worth noting that the general description above and the detailed description that follow are merely for illustrative purposes and do not constitute a limitation on this application. Attached Figure Description

[0015] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0016] Figure 1 This is a hardware structure block diagram of a computer terminal for implementing an image-text matching method according to an embodiment of this application;

[0017] Figure 2 This is a schematic diagram of an optional computer terminal application scenario according to an embodiment of this application;

[0018] Figure 3 This is a flowchart of the image-text matching method according to Embodiment 1 of this application;

[0019] Figure 4 This is a flowchart of an optional image-text matching method according to an embodiment of this application;

[0020] Figure 5 This is a flowchart of the product retrieval method according to Embodiment 2 of this application;

[0021] Figure 6 This is a flowchart of the image-text matching method according to Embodiment 3 of this application;

[0022] Figure 7A This is a schematic diagram of an operation interface according to an embodiment of this application;

[0023] Figure 7B This is a schematic diagram of another user interface according to an embodiment of this application;

[0024] Figure 7C This is a schematic diagram of another user interface according to an embodiment of this application;

[0025] Figure 8 This is a flowchart of the product retrieval method according to Embodiment 4 of this application;

[0026] Figure 9 This is a flowchart of the image-text matching method according to Embodiment 5 of this application;

[0027] Figure 10 This is a flowchart of the product retrieval method according to Embodiment 6 of this application;

[0028] Figure 11 This is a schematic diagram of an image-text matching device according to Embodiment 7 of this application;

[0029] Figure 12 This is a schematic diagram of a product retrieval device according to Embodiment 8 of this application;

[0030] Figure 13 This is a schematic diagram of an image-text matching device according to Embodiment 9 of this application;

[0031] Figure 14 This is a schematic diagram of a product retrieval device according to Embodiment 10 of this application;

[0032] Figure 15 This is a schematic diagram of an image-text matching device according to Embodiment 11 of this application;

[0033] Figure 16 This is a schematic diagram of a product retrieval device according to Embodiment 12 of this application;

[0034] Figure 17 This is a structural block diagram of a computer terminal according to an embodiment of this application. Detailed Implementation

[0035] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0036] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0037] First, some nouns or terms that appear in the description of the embodiments of this application shall be interpreted as follows:

[0038] Image-text matching: also known as cross-modal image-text retrieval, which means retrieving semantically related instances in another modality from instances in one modality.

[0039] Two-stream model: A model that includes two processing modules. One processing module is used to process images, and the other processing module is used to process text. The two processing modules can be independent neural network models or they can share some parameters.

[0040] Multiple view encoding: can be features used to represent multiple different views of the same data. For example, multiple first view encoding can be features used to represent multiple first views of an image, and multiple second view encoding can be features used to represent multiple second views of text.

[0041] Attention processing: By establishing associations between different locations in the input data and all other locations, it effectively captures the global context and relational information of the input data.

[0042] ViT: Vision Transformer, a visual attention model, an attention-based model for image processing.

[0043] Faster R-CNN: Faster Region Convolutional Neural Network, can be a deep learning model used for object detection.

[0044] BERT: Bidirectional Encoder Representations from Transformers, can be a Transformer-based model pre-trained on large-scale Wikipedia and Bookcorpus datasets. It can use bidirectional contextual information to generate word representations, enabling the model to better understand semantics and sentence structure.

[0045] Example 1

[0046] According to an embodiment of this application, an image-text matching method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0047] The method embodiment provided in Embodiment 1 of this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 1 A hardware block diagram of a computer terminal (or mobile device) for implementing an image-text matching method is shown. Figure 1As shown, the computer terminal 10 (or mobile device) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (processor 102 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0048] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0049] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the image-text matching method in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the image-text matching method described above. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0050] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0051] The display can be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0052] Figure 1 The hardware structure block diagram shown can serve not only as an exemplary block diagram of the aforementioned computer terminal 10 (or mobile device), but also as an exemplary block diagram of the aforementioned server. In one optional embodiment, Figure 2 The use of the above is illustrated in a block diagram. Figure 1 The computer terminal 10 (or mobile device) shown is one embodiment of the receiving end. For example... Figure 2 As shown, computer terminal 10 (or mobile device) can be connected to one or more clients 20 via a data network connection or electronically. The data network connection can be a local area network connection, a wide area network connection, an Internet connection, or other types of data network connection.

[0053] Under the aforementioned operating environment, this application provides the following: Figure 3 The image-text matching method shown. Figure 3 This is a flowchart of the image-text matching method according to Embodiment 1 of this application. Figure 3 As shown, the method includes the following steps:

[0054] Step S302: Obtain the image to be matched and the text to be matched.

[0055] In one optional embodiment, for scenarios such as image-text synthesis and image-based question answering, users can upload images as images to be matched and text as text to be matched. The client can then upload the images and text to be matched to the server, whereby the server performs image-text matching.

[0056] In another alternative embodiment, for scenarios such as product search on e-commerce platforms and image search in search engines, users can input text to search for images as the text to be matched. The client can then upload the images to be matched to the server, and the server can use images from the image database as the images to be matched, thereby performing image-text matching.

[0057] In another alternative embodiment, for scenarios such as image translation and image annotation, users can upload images themselves as images to be matched. The client can then upload the text to be matched to the server, and the server can use the text in the text database as the text to be matched, thereby performing image-text matching.

[0058] Step S304: Encode the image to be matched and the text to be matched respectively to obtain the image hiding state of the image to be matched and the text hiding state of the text to be matched.

[0059] In an optional embodiment, since the image to be matched and the text to be matched belong to different modalities of data, in order to facilitate the server's matching, the image to be matched and the text to be matched can be encoded and converted into vectors, that is, the above-mentioned image hiding state and text hiding state are obtained.

[0060] Optionally, an image encoder and a text decoder can be pre-trained. The image encoder can employ typical image encoder structures such as ViT or Faster R-CNN, and the text encoder can employ typical image encoder structures such as BERT, but is not limited to these. The image encoder encodes different visual regions of the image to extract the image's hidden states; the text encoder encodes different words of the text to extract the text's hidden states. It should be noted that the image encoder and text encoder can employ model structures from related technologies capable of performing feature extraction from images and text; this application does not impose specific limitations on them.

[0061] For example, taking the Faster R-CNN image encoder as an example, it can convert the image to be matched into pixel-level features. For the image V to be matched, the pre-trained Faster R-CNN image encoder is first used to detect N salient image regions, and pooled ROI (Region of Interest) features are extracted for each region. In addition, location features were also considered. Location features can be normalized coordinates of the top-left and bottom-right corners of the region, as well as the region's width, height, and area. Image embedding is V. frcnn = (v1, l1), ..., (vN , l N Then, a special token [IMG] is added at the beginning to represent the entire image representation. The context representation obtained through the 12-layer Transformer network is as follows: Generally speaking, h after pooling [IMG] It can be regarded as the semantic representation of the image R v The specific formula is as follows:

[0062] H v =Transformer([[IMG]; V frcnn ]),

[0063] R v =Pool(h [IMG] ),

[0064] The pooling layer is a fully connected layer with a tanh function.

[0065] For example, taking ViT's image encoder as an example, it can convert an image into a token-level embedding. This corresponds to an input image. Where (H, W) represents the image resolution, and C represents the number of channels. First, the image can be divided into a sequence of 2D image blocks. Where N = HW / / P 2 Let P be the number of image patches, and (P, P') be the resolution of each image patch. Then, to obtain the contextual representation, a special token [IMG] can be added before each image patch. Similar to the image encoder in FRCNN, a Transformer can be used to encode the image patches to obtain the hidden states, and the final image representation is obtained by pooling the hidden states of token [IMG].

[0066] For example, using BERT as an example, the BERT Tokenizer splits the input statement T into a token sequence [CLS], t1, ..., t2. L [CLS] and [SEP] are special tokens used to mark the start and end points. The input is then encoded into L+2 hidden states using a Transformer. The final text representation R is obtained by pooling the hidden state of token [CLS]. t .

[0067] H t =Transformer([[CLS];T;[SEP]]),

[0068] R t =Pool(h [CLS] ).

[0069] It should be noted that, in this application, pooling hidden state R is not used. v Or R t Instead of using image features or text features, the following approach is adopted.

[0070] Step S306: Attention processing is performed on the hidden state of the image based on multiple first view codes to obtain the image features of the image to be matched, and attention processing is performed on the hidden state of the text based on multiple second view codes to obtain the text features of the text to be matched.

[0071] The aforementioned multiple first-view encodings can be visual representations of multiple views constructed for an image, and the aforementioned multiple second-view encodings can be textual representations of multiple views constructed for text. Here, multiple views can be understood as multiple angles from which an image or text is viewed.

[0072] In one alternative embodiment, attention processing can be performed on the image hidden state based on multiple first view encodings and on the text hidden state based on multiple second view encodings. This enables encoding of the image and text from different perspectives, rather than encoding the image and text with a single representation. As a result, the final image features and text features can contain more information and can focus on more key points.

[0073] Optionally, to enhance the representational power of the two-stream matching model and encode more information, this application proposes a multi-view attention model that encodes multimodal data from different views with multiple attention heads. In this model, images or text are represented by multiple vectors, rather than a single vector in a traditional two-stream model.

[0074] Step S308: Based on image features and text features, match the image to be matched and the text to be matched to obtain the target matching result.

[0075] In one alternative embodiment, a matching score can be obtained based on the cosine similarity or vector inner product between image features and text features. By pre-setting a threshold, if the matching score is greater than the threshold, the target matching result is determined to be a successful match between the image and the text to be matched; if the matching score is less than the threshold, the target matching result is determined to be a failed match between the image and the text to be matched.

[0076] The following is combined Figure 4 A preferred embodiment of this application will be described in detail, such as... Figure 4As shown, the processing flow of this method is as follows: The image to be matched and the text to be matched can be processed separately by two parallel modules. First, the image to be matched can be segmented to obtain multiple image blocks; then, the ViT image encoder is used to encode the multiple image blocks to obtain the hidden state H of the image to be matched. v The hidden state of the image contains sub-states of multiple image patches. It also includes the hidden state h of the special token [IMG]. [IMG] Furthermore, encoding can be based on multiple first views. For the hidden state H of the image v Attention processing is performed to obtain the multi-R image features of the image to be matched. v Similarly, the text to be matched can be segmented into multiple words; then BERT can be used to encode these multiple words to obtain the text hidden state H of the text to be matched. t The hidden text state contains sub-states for multiple words. It also includes the hidden state h of the special tokens [CLS] and [SEP]. [CLS] and h [SEP] Furthermore, encoding can be based on multiple second views. Hidden state H of text t Attention processing is performed to obtain the multi-R text features of the text to be matched. t Finally, by calculating the inner product of image features and text features, the matching score is obtained, which in turn yields the target matching result.

[0077] Optionally, after obtaining the target matching result, subsequent processes can continue based on the target matching result. For example, in an image-text synthesis scenario, if the target matching result is successful, the image and text to be matched can be synthesized, and the synthesized result can be returned to the client for the user to view; if the target matching result is unsuccessful, the synthesis process is not performed. As another example, in a product search scenario, if the target matching result is successful, the product corresponding to the image to be matched can be used as the searched product, and this product can be returned to the client for the user to view; if the target matching result is unsuccessful, the text to be matched can continue to be matched with other images. If the text to be matched fails to match with any image in the image database, the result of no product being found is returned to the client for the user to view. For example, in an image translation scenario, if the target matching result is successful, the text to be matched can be used as the image translation result and returned to the client for the user to view. If the target matching result is unsuccessful, the image to be matched will continue to be matched with other texts. If the image to be matched fails to match any text in the text database, the result of image translation failure will be returned to the client for the user to view.

[0078] The solution provided by the above embodiments of this application, after obtaining the image to be matched and the text to be matched, encodes the image to be matched and the text to be matched respectively to obtain the image hidden state of the image to be matched and the text hidden state of the text to be matched. Then, attention processing is performed on the image hidden state based on multiple first view encodings to obtain the image features of the image to be matched, and attention processing is performed on the text hidden state based on multiple second view encodings to obtain the text features of the text to be matched. Finally, based on the image features and text features, the image to be matched and the text to be matched are matched to obtain the target matching result, thus achieving the purpose of image-text matching. It is easy to note that the image features and text features finally used for image-text matching are obtained by using multi-view encoding to perform attention processing on the hidden state, which allows the image and text to be encoded from different angles to obtain features containing more information. This allows the matching score between the image and text to be calculated from different aspects, thereby achieving the technical effect of improving the accuracy of image-text matching and solving the technical problem of low accuracy of image-text matching using a two-stream model.

[0079] In the above embodiments of this application, attention processing is performed on the image hidden state based on multiple first view codes to obtain image features of the image to be matched, and attention processing is performed on the text hidden state based on multiple second view codes to obtain text features of the text to be matched. This includes: performing cross-attention processing on multiple first view codes and image hidden states to obtain multiple first attention features, and performing cross-attention processing on multiple second view codes and text hidden states to obtain multiple second attention features; performing a weighted sum of multiple first attention features and image hidden states to obtain multiple first view features, and performing a weighted sum of multiple second attention features and text hidden states to obtain multiple second view features; concatenating multiple first view features to obtain image features, and concatenating multiple second view features to obtain text features.

[0080] In one alternative embodiment, multiple first view encodings are performed. in, D is The dimension is [not specified]. Multiple first-view encodings and image hidden states can be processed using a cross-attention mechanism to obtain multiple first-attention features. in, Based on image hidden state H v The calculation formula is as follows: (This is derived from the encoding of the i-th first view.) Furthermore, the hidden state H of the image can be... v and multiple first attention features By performing a weighted sum, multiple first-view features are calculated. in, It is for H v and the i-th first attention feature The weighted sum is obtained, and the specific calculation formula is as follows:

[0081]

[0082] Finally, by concatenating the features from multiple first-view images, we can obtain the final features of the image to be matched, which is the aforementioned multi-R image feature. v The calculation formula is as follows:

[0083]

[0084] For multiple second-view encodings, the final features of the text to be matched can be calculated in a similar manner as described above, which is the aforementioned text feature multi-R. t The calculation formula is as follows:

[0085]

[0086]

[0087]

[0088] In the above embodiments of this application, cross-attention processing is performed on multiple first view codes and image hidden states to obtain multiple first attention features, and cross-attention processing is performed on multiple second view codes and text hidden states to obtain multiple second attention features, including: using the first view code as a first query, the image hidden state as a first key and a first value, and using the second view code as a second query, the text hidden state as a second key and a second value; processing the first query, the first key and the first value to obtain the first attention features, and processing the second query, the second key and the second value to obtain the second attention features.

[0089] In one alternative embodiment, multiple first views can be encoded as attention queries, and the image hidden state can be used as the key and value of attention, thereby utilizing a cross-attention mechanism to obtain first attention features. Similarly, multiple second views can be encoded as attention queries, and the text hidden state can be used as the key and value of attention, thereby utilizing a cross-attention mechanism to obtain second attention features.

[0090] In the above embodiments of this application, a weighted sum is performed on multiple first attention features and image hiding states to obtain multiple first view features, and a weighted sum is performed on multiple second attention features and text hiding states to obtain multiple second view features. This includes: weighting the first feature value contained in the first attention feature with the first sub-state contained in the image hiding state to obtain the first view feature; and weighting the second feature value contained in the second attention feature with the second sub-state contained in the text hiding state to obtain the second view feature.

[0091] In an optional embodiment, based on the calculation formula for the first view feature described above, for the i-th attention feature, each element of the i-th attention feature can be multiplied by each sub-state in the image hidden state, and then all products are summed to obtain the first view feature described above. The second view feature can be calculated in the same way.

[0092] In the above embodiments of this application, the method further includes: acquiring training samples, wherein the training samples include: training images and training text; encoding the training images and training text respectively to obtain a first hidden state of the training images and a second hidden state of the training text; performing attention processing on the first hidden state based on multiple first initial encodings to obtain a first feature of the training images, and performing attention processing on the second hidden state based on multiple second initial encodings to obtain a second feature of the training text; constructing a target loss based on the first feature, the second feature, multiple first initial encodings, and multiple second initial encodings; and adjusting the multiple first initial encodings and multiple second initial encodings based on the target loss to obtain multiple first view encodings and multiple second view encodings.

[0093] In an optional embodiment, a contrastive learning objective can be used to train two encoders. A batch of B×B image-text pairs can be pre-constructed as training samples. The matching process for the training samples is the same as the matching process for the images and text to be matched, and will not be elaborated here. To improve the accuracy of image-text matching and to ensure that image and text features contain comprehensive information and expand the diversity of multi-view attention, the view encoding can be constrained to focus on different aspects. A target loss can be constructed based on a first feature, a second feature, multiple first initial encodings, and multiple second initial encodings. After determining the target loss, it can be compared with a pre-constructed loss threshold. If the target loss is greater than the loss threshold, training needs to continue. That is, the model parameters of all the above models, as well as the multiple first view encodings and multiple second view encodings, are continuously adjusted until the target loss is less than the loss threshold.

[0094] In the above embodiments of this application, the target loss is constructed based on the first feature, the second feature, a plurality of first initial codes, and a plurality of second initial codes, including: constructing a matching loss based on the first feature and the second feature; constructing a diversity loss based on the plurality of first initial codes and a plurality of second initial codes; and weighting the matching loss and the diversity loss to obtain the target loss.

[0095] In an optional embodiment, to improve the accuracy of image-text matching, a contrastive loss can be constructed based on the first feature and the second feature. cl To constrain multiple first-view codes and multiple second-view codes, a diversity loss can be constructed based on multiple first initial codes and multiple second initial codes. div The target loss can be further calculated using the following formula:

[0096] loss = loss cl +β(loss div ), where β is the weighting coefficient.

[0097] In the above embodiments of this application, the matching loss is constructed based on the first feature and the second feature, including: constructing a first image-to-text matching loss based on the first feature and multiple second features; constructing a second text-to-image matching loss based on the second feature and multiple first features; and obtaining the mean of the first matching loss and the second matching loss to obtain the matching loss.

[0098] In an optional embodiment, since image-to-text matching can include both image-to-text matching and text-to-image matching, the image-to-text loss can be calculated separately. i2t and text-to-image loss t2i .

[0099] Specifically, for image-to-text loss, the similarity between different images and all texts can be calculated, where Score(V) = ... i T j Let represent the matching score between the i-th image and the j-th text. For the matching of the i-th image to the text, the positive sample is the i-th text, and the other texts are negative samples. Then the loss for image-to-text retrieval is:

[0100]

[0101] Similarly, the loss for text-to-image retrieval can be calculated. t2i The final matching loss is:

[0102]

[0103] In the above embodiments of this application, a diversity loss is constructed based on multiple first initial codes and multiple second initial codes, including: constructing a first matrix based on multiple first initial codes and constructing a second matrix based on multiple second initial codes; obtaining the product of the first matrix and the transpose of the first matrix to obtain a first matrix multiplication, and obtaining the product of the second matrix and the transpose of the second matrix to obtain a second matrix multiplication; obtaining the difference between the first matrix multiplication and the identity matrix to obtain a first matrix difference, and obtaining the difference between the second matrix multiplication and the identity matrix to obtain a second matrix difference; obtaining the norm of the first matrix difference and the second matrix difference to obtain a first diversity loss and a second diversity loss; and obtaining the sum of the first diversity loss and the second diversity loss to obtain a diversity loss.

[0104] In an alternative embodiment, taking an image as an example, it can be represented by a first matrix A. v Multiplying by the transpose yields the similarity between each view attention (i.e., the first matrix multiplication mentioned above), where A v yes Then, subtract the first matrix multiplication from the identity matrix to obtain...

[0105] The first matrix difference is used to calculate the first diversity loss, and the norm of the first matrix difference, such as the Frobenius norm, is used to obtain the first diversity loss. The specific calculation formula is as follows:

[0106]

[0107] Where I is an m-dimensional identity matrix used to remove the self-correspondence of the m view attentions on the diagonal of the first or second matrix multiplication; ||·|| F This represents the Frobenius norm of the matrix.

[0108] The same similar operation can be applied to text multi-view attention diversity to obtain a second diversity loss. Diversity loss div It can be calculated using the following formula:

[0109]

[0110] In the above embodiments of this application, the image to be matched and the text to be matched are encoded respectively to obtain the image hiding state of the image to be matched and the text hiding state of the text to be matched, including: using an image encoder to encode the image to be matched to obtain the image hiding state; and using a text encoder to encode the text to be matched to obtain the text hiding state.

[0111] In one alternative embodiment, a pre-trained image encoder can be used to encode the image to be matched to obtain the hidden image state, or a pre-trained text encoder can be used to encode the text to be matched to obtain the hidden text state.

[0112] In the above embodiments of this application, the image to be matched is encoded using an image encoder to obtain an image hiding state, including: segmenting the image to be matched to obtain multiple image blocks; concatenating first preset information with the multiple image blocks to obtain first concatenation information, wherein the first preset information is used to characterize the image to be matched; and encoding the first concatenation information using an image encoder to obtain an image hiding state.

[0113] In an alternative embodiment, the image to be matched can be encoded using the ViT image encoder. The specific encoding process is the same as that of the image encoder described above, and will not be repeated here.

[0114] In the above embodiments of this application, the text to be matched is encoded using a text encoder to obtain a text-hidden state, including: segmenting the text to be matched into multiple words; concatenating the second preset information, the multiple words, and the third preset information to obtain second concatenated information, wherein the second preset information is used to represent the start position of the text to be matched, and the third preset information is used to represent the end position of the text to be matched; and encoding the second concatenated information using a text encoder to obtain a text-hidden state.

[0115] In an alternative embodiment, BERT can be used to encode the text to be matched. The specific encoding process is the same as that of the text encoder described above, and will not be repeated here.

[0116] In the above embodiments of this application, based on image features and text features, the image to be matched and the text to be matched are matched to obtain a target matching result, including: obtaining the inner product of image features and text features to obtain a matching score, wherein the matching score is used to characterize the matching probability of the image to be matched and the text to be matched; comparing the matching score with a preset score to obtain a target matching result.

[0117] The preset score mentioned above can be a threshold set in advance based on the user's accuracy requirements. If the matching score is greater than the threshold, it indicates that the image and text to be matched are successfully matched. If the matching score is less than the threshold, it indicates that the image and text to be matched have failed to match.

[0118] In one optional embodiment, the matching score of an image and text can be predicted by calculating the inner product of image features and text features. A higher matching score indicates a greater probability of a match between the image and text. The target matching result can be obtained by comparing the matching score with a preset score. The formula for calculating the matching score is as follows:

[0119] Score(V,T)= <R v R t >

[0120] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0121] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0122] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, they can also be implemented by hardware. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0123] Example 2

[0124] According to an embodiment of this application, a product retrieval method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0125] Figure 5 This is a flowchart of the product retrieval method according to Embodiment 2 of this application. For example... Figure 5 As shown, the method includes the following steps:

[0126] Step S502: Obtain the search text and multiple product images.

[0127] The search text mentioned above can be text entered by the user on an e-commerce shopping platform, in which the user hopes to search for products on the platform. The multiple product images mentioned above can be images of different products from the image database corresponding to the e-commerce shopping platform.

[0128] In one alternative embodiment, when a user wishes to view or purchase a product, they can enter the search text for the product in the input box of the e-commerce shopping platform. The client can then send the search text to the server, which will match the detected text with multiple product images through image-text matching to achieve the purpose of retrieving the target product.

[0129] Step S504: Encode the product image and the search text respectively to obtain the image hiding state of the product image and the text hiding state of the search text.

[0130] In an alternative embodiment, the encoding process for the product image and the retrieval text can be implemented using the encoding process provided in Embodiment 1, which will not be described in detail here.

[0131] Step S506: Attention processing is performed on the hidden state of the image based on multiple first view codes to obtain the image features of the product image, and attention processing is performed on the hidden state of the text based on multiple second view codes to obtain the text features of the retrieved text.

[0132] In an optional embodiment, the attention processing of product images and search text can be implemented using the attention processing process provided in Embodiment 1, which will not be described in detail here.

[0133] Step S508: Based on multiple image features and text features, determine the target product image that matches the retrieved text.

[0134] In one optional embodiment, by matching image features and text features, multiple matching scores can be obtained. Then, the product image with the highest matching score is selected as the target product image, or multiple product images with the highest matching scores are selected as the target product image. Thus, the server can send the target product image and information such as the purchase link of the target product to the client, which will then display it to the user.

[0135] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0136] Example 3

[0137] According to an embodiment of this application, an image-text matching method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0138] Figure 6 This is a flowchart of the image-text matching method according to Embodiment 3 of this application. Figure 6 As shown, the method includes the following steps:

[0139] Step S602: In response to the input command applied to the operation interface, display the image to be matched and the text to be matched on the operation interface.

[0140] The aforementioned user interface can be a client-side interface provided to users for uploading images and text to be matched. Since image-text matching methods can be applied to various scenarios, the user interface can be customized to suit different applications. The input commands can be generated by the user clicking buttons or input boxes provided on the user interface.

[0141] In one optional embodiment, for scenarios such as image-text synthesis and image question answering, the client can provide users with, for example... Figure 7A The interface shown allows users to select an image for image-text matching by clicking the "Upload Image" button, or directly drag the desired image into the dotted box. Users can then enter text in the input box to obtain the text to be matched.

[0142] In another alternative embodiment, for scenarios such as product search on e-commerce platforms and image search in search engines, the client can provide users with, for example... Figure 7B The user interface shown allows users to enter text in the input box to obtain the text to be matched. The server can directly use images from the image database as the images to be matched, and then perform image-text matching.

[0143] In another alternative embodiment, for scenarios such as image translation and image annotation, the client can provide users with, for example... Figure 7C As shown in the interface, users can select the image to be matched by clicking the "Upload Image" button, or they can directly drag the image to be matched into the dotted box to obtain the image to be matched. The server can use the text in the text database as the text to be matched, and then the server will perform image-text matching.

[0144] Step S604: In response to the matching command applied to the operation interface, the target matching result is displayed on the operation interface. The target matching result is obtained by matching the image to be matched and the text to be matched based on the image features of the image to be matched and the text features of the text to be matched. The image features are obtained by performing attention processing on the image hidden state of the image to be matched based on multiple first view codes. The text features are obtained by performing attention processing on the text hidden state of the text to be matched based on multiple second view codes. The image hidden state and the text hidden state are obtained by encoding the image to be matched and the text to be matched, respectively.

[0145] The aforementioned matching instruction can be the quality generated by the user clicking a button provided on the operation interface.

[0146] In an alternative embodiment, for such Figures 7A to 7C The user interface shown allows users to generate matching commands by clicking the "Image-Text Matching" button. These commands are then forwarded to the server by the client. Upon receiving the command, the server processes the image and text to be matched, obtaining the target matching result. The server can then return the target matching result to the client, which displays it as shown in the image. Figures 7A to 7C Within the "Results Display Area" of the interface shown.

[0147] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0148] Example 4

[0149] According to an embodiment of this application, a product retrieval method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0150] Figure 8 This is a flowchart of the product retrieval method according to Embodiment 4 of this application. For example... Figure 8 As shown, the method includes the following steps:

[0151] Step S802: In response to the input command applied to the operation interface, display the search text on the operation interface.

[0152] The aforementioned user interface can be the search interface of an e-commerce shopping platform, and the aforementioned input commands can be commands generated by the user entering search text in the input box of the search interface.

[0153] Step S804: In response to the matching command applied to the operation interface, a target product image matching the search text is displayed on the operation interface. The target product image is determined based on the text features of the search text and the image features of multiple product images. The image features are obtained by performing attention processing on the image hiding state of the product images based on multiple first view codes. The text features are obtained by performing attention processing on the text hiding state of the search text based on multiple second view codes. The image hiding state and the text hiding state are obtained by encoding the product images and the search text, respectively.

[0154] The matching command mentioned above can be generated when the user clicks the "Search" button on the search interface.

[0155] In one alternative embodiment, the image of the target product may be displayed in the "search results area" on the search interface.

[0156] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0157] Example 5

[0158] According to an embodiment of this application, an image-text matching method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0159] Figure 9 This is a flowchart of the image-text matching method according to Embodiment 5 of this application. Figure 9 As shown, the method includes the following steps:

[0160] Step S902: Obtain the image to be matched and the text to be matched by calling the first interface, wherein the first interface includes a first parameter, and the parameter value of the first parameter is the image to be matched and the text to be matched.

[0161] The first interface mentioned above can be an interface for data transmission between the client and the server. Through the first interface, the client can upload the image to be matched and the text to be matched to the server.

[0162] Step S904: Encode the image to be matched and the text to be matched respectively to obtain the image hiding state of the image to be matched and the text hiding state of the text to be matched.

[0163] Step S906: Attention processing is performed on the hidden state of the image based on multiple first view codes to obtain the image features of the image to be matched, and attention processing is performed on the hidden state of the text based on multiple second view codes to obtain the text features of the text to be matched.

[0164] Step S908: Based on image features and text features, match the image to be matched and the text to be matched to obtain the target matching result.

[0165] Step S910: Output the target matching result by calling the second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the target matching result.

[0166] The second interface mentioned above can be an interface for data transmission between the server and the client. Through the second interface, the server can return the target matching result to the client.

[0167] It should be noted that the second interface can be the same as the first interface, but is not limited to this.

[0168] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0169] Example 6

[0170] According to an embodiment of this application, a product retrieval method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0171] Figure 10 This is a flowchart of the product retrieval method according to Embodiment 6 of this application. For example... Figure 10 As shown, the method includes the following steps:

[0172] Step S1002: Obtain the search text by calling the first interface, wherein the first interface includes a first parameter, and the parameter value of the first parameter is the search text.

[0173] Step S1004: Encode multiple product images and search text respectively to obtain the image hiding state of the product images and the text hiding state of the search text.

[0174] Step S1006: Attention processing is performed on the hidden state of the image based on multiple first view codes to obtain the image features of the product image, and attention processing is performed on the hidden state of the text based on multiple second view codes to obtain the text features of the retrieved text.

[0175] Step S1008: Based on multiple image features and text features, determine the target product image that matches the retrieved text.

[0176] Step S1010: Output the target product image by calling the second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the target product image.

[0177] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0178] Example 7

[0179] According to embodiments of this application, an image text matching apparatus for implementing the above-described image text matching method is also provided, such as... Figure 11As shown, the device 1100 includes: an acquisition module 1102, an encoding module 1104, an attention processing module 1106, and a matching module 1108.

[0180] The acquisition module 1102 is used to acquire the image to be matched and the text to be matched; the encoding module 1104 is used to encode the image to be matched and the text to be matched respectively to obtain the image hiding state of the image to be matched and the text hiding state of the text to be matched; the attention processing module 1106 is used to perform attention processing on the image hiding state based on multiple first view encodings to obtain the image features of the image to be matched, and to perform attention processing on the text hiding state based on multiple second view encodings to obtain the text features of the text to be matched; the matching module 1108 is used to match the image to be matched and the text to be matched based on the image features and the text features to obtain the target matching result.

[0181] It should be noted that the acquisition module 1102, encoding module 1104, attention processing module 1106, and matching module 1108 mentioned above correspond to steps S302 to S308 in Embodiment 3. The four modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules or units can be hardware or software components stored in memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n). The above modules can also be part of a device and run in the computer terminal 10 provided in Embodiment 1.

[0182] In the above embodiments of this application, the attention processing module 1006 includes: a processing unit, a first weighted sum unit, and a splicing unit.

[0183] The processing unit is used to perform cross-attention processing on multiple first view encodings and image hidden states to obtain multiple first attention features, and to perform cross-attention processing on multiple second view encodings and text hidden states to obtain multiple second attention features; the first weighted sum unit is used to perform weighted summation on multiple first attention features and image hidden states to obtain multiple first view features, and to perform weighted summation on multiple second attention features and text hidden states to obtain multiple second view features; the stitching unit is used to stitch together multiple first view features to obtain image features, and to stitch together multiple second view features to obtain text features.

[0184] In the above embodiments of this application, the processing unit is further configured to use the first view encoding as the first query, the image hiding state as the first key and the first value, and the second view encoding as the second query, the text hiding state as the second key and the second value; process the first query, the first key and the first value to obtain the first attention feature, and process the second query, the second key and the second value to obtain the second attention feature.

[0185] In the above embodiments of this application, the weighted sum unit is further configured to perform a weighted sum of the first feature value contained in the first attention feature and the first sub-state contained in the image hiding state to obtain the first view feature; and to perform a weighted sum of the second feature value contained in the second attention feature and the second sub-state contained in the text hiding state to obtain the second view feature.

[0186] In the above embodiments of this application, the device further includes: a construction module and an adjustment module.

[0187] The acquisition module is further used to acquire training samples, which include training images and training text; to encode the training images and training text respectively to obtain a first hidden state of the training images and a second hidden state of the training text; the encoding module is further used to perform attention processing on the first hidden state based on multiple first initial encodings to obtain a first feature of the training images, and to perform attention processing on the second hidden state based on multiple second initial encodings to obtain a second feature of the training text; the construction module is used to construct a target loss based on the first feature, the second feature, multiple first initial encodings, and multiple second initial encodings; the adjustment module is used to adjust the multiple first initial encodings and multiple second initial encodings based on the target loss to obtain multiple first view encodings and multiple second view encodings.

[0188] In the above embodiments of this application, the construction module includes: a first construction unit, a second construction unit, and a second weighted sum unit.

[0189] The first construction unit is used to construct a matching loss based on the first feature and the second feature; the second construction unit is used to construct a diversity loss based on multiple first initial codes and multiple second initial codes; and the second weighted sum unit is used to perform a weighted sum of the matching loss and the diversity loss to obtain the target loss.

[0190] In the above embodiments of this application, the first construction unit is further configured to construct a first image-to-text matching loss based on the first feature and a plurality of second features; construct a second text-to-image matching loss based on the second feature and a plurality of first features; and obtain the average of the first matching loss and the second matching loss to obtain the matching loss.

[0191] In the above embodiments of this application, the second construction unit is further configured to construct a first matrix based on multiple first initial codes, and construct a second matrix based on multiple second initial codes; obtain the product of the first matrix and the transpose of the first matrix to obtain a first matrix multiplication, and obtain the product of the second matrix and the transpose of the second matrix to obtain a second matrix multiplication; obtain the difference between the first matrix multiplication and the identity matrix to obtain a first matrix difference, and obtain the difference between the second matrix multiplication and the identity matrix to obtain a second matrix difference; obtain the norm of the first matrix difference and the second matrix difference to obtain a first diversity loss and a second diversity loss; and obtain the sum of the first diversity loss and the second diversity loss to obtain a diversity loss.

[0192] In the above embodiments of this application, the encoding module includes: a first encoding unit and a second encoding unit.

[0193] The first encoding unit is used to encode the image to be matched using an image encoder to obtain the hidden state of the image; the second encoding unit is used to encode the text to be matched using a text encoder to obtain the hidden state of the text.

[0194] In the above embodiments of this application, the first encoding unit is used to segment the image to be matched to obtain multiple image blocks; to concatenate the first preset information with the multiple image blocks to obtain first concatenation information, wherein the first preset information is used to characterize the image to be matched; and to encode the first concatenation information using an image encoder to obtain the image hiding state.

[0195] In the above embodiments of this application, the second encoding unit is used to segment the text to be matched into multiple words; concatenate the second preset information, the multiple words, and the third preset information to obtain the second concatenation information, wherein the second preset information is used to represent the start position of the text to be matched, and the third preset information is used to represent the end position of the text to be matched; and the second concatenation information is encoded using a text encoder to obtain the text hiding state.

[0196] In the above embodiments of this application, the matching module includes: an inner product unit and a comparison unit.

[0197] The inner product unit is used to obtain the inner product of image features and text features to get a matching score, where the matching score is used to represent the matching probability of the image and text to be matched; the matching unit is used to compare the matching score with a preset score to get the target matching result.

[0198] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0199] Example 8

[0200] According to embodiments of this application, a product retrieval device for implementing the above-described product retrieval method is also provided, such as... Figure 12 As shown, the device 1200 includes: an acquisition module 1202, an encoding module 1204, an attention processing module 1206, and a determination module 1208.

[0201] The acquisition module 1202 is used to acquire the search text and multiple product images; the encoding module 1204 is used to encode the product images and the search text respectively to obtain the image hiding state of the product images and the text hiding state of the search text; the attention processing module 1206 is used to perform attention processing on the image hiding state based on multiple first view encodings to obtain the image features of the product images, and to perform attention processing on the text hiding state based on multiple second view encodings to obtain the text features of the search text; the determination module 1208 is used to determine the target product image that matches the search text based on multiple image features and text features.

[0202] It should be noted that the acquisition module 1202, encoding module 1204, attention processing module 1206, and determination module 1208 mentioned above correspond to steps S502 to S508 in Embodiment 2. The four modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules or units can be hardware or software components stored in memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n). The above modules can also be part of a device and run in the computer terminal 10 provided in Embodiment 1.

[0203] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0204] Example 9

[0205] According to embodiments of this application, an image text matching apparatus for implementing the above-described image text matching method is also provided, such as... Figure 13 As shown, the device 1300 includes: a first display module 1302 and a second display module 1304.

[0206] The first display module 1302 is used to respond to input commands applied to the operation interface and display the image to be matched and the text to be matched on the operation interface; the second display module 1304 is used to respond to matching commands applied to the operation interface and display the target matching result on the operation interface. The target matching result is obtained by matching the image to be matched and the text to be matched based on the image features of the image to be matched and the text features of the text to be matched. The image features are obtained by attention processing of the image hidden state of the image to be matched based on multiple first view codes, and the text features are obtained by attention processing of the text hidden state of the text to be matched based on multiple second view codes. The image hidden state and the text hidden state are obtained by encoding the image to be matched and the text to be matched, respectively.

[0207] It should be noted that the first display module 1302 and the second display module 1304 mentioned above correspond to steps S602 to S604 in Embodiment 3. The two modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules or units can be hardware components or software components stored in memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n). The above modules can also be part of a device and run in the computer terminal 10 provided in Embodiment 1.

[0208] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0209] Example 10

[0210] According to embodiments of this application, a product retrieval device for implementing the above-described product retrieval method is also provided, such as... Figure 14 As shown, the device includes: a first display module 1402 and a second display module 1404.

[0211] The first display module 1402 is used to respond to input commands applied to the operation interface and display the search text on the operation interface; the second display module 1404 is used to respond to matching commands applied to the operation interface and display the target product image that matches the search text on the operation interface. The target product image is determined based on the text features of the search text and the image features of multiple product images. The image features are obtained by performing attention processing on the image hiding state of the product images based on multiple first view codes. The text features are obtained by performing attention processing on the text hiding state of the search text based on multiple second view codes. The image hiding state and the text hiding state are obtained by encoding the product images and the search text, respectively.

[0212] It should be noted that the first display module 1402 and the second display module 1404 mentioned above correspond to steps S802 to S804 in Embodiment 4. The two modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules or units can be hardware components or software components stored in memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n). The above modules can also be part of the device and run in the computer terminal 10 provided in Embodiment 1.

[0213] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0214] Example 11

[0215] According to embodiments of this application, an image text matching apparatus for implementing the above-described image text matching method is also provided, such as... Figure 15 As shown, the device 1500 includes: a first invocation module 1502, an encoding module 1504, an attention processing module 1506, a matching module 1508, and a second invocation module 1510.

[0216] The system comprises the following modules: a first calling module 1502, which obtains the image to be matched and the text to be matched by calling a first interface, wherein the first interface includes a first parameter whose value is the image to be matched and the text to be matched; an encoding module 1504, which encodes the image to be matched and the text to be matched respectively to obtain the image hiding state of the image to be matched and the text hiding state of the text to be matched; an attention processing module 1506, which performs attention processing on the image hiding state based on multiple first view encodings to obtain the image features of the image to be matched, and performs attention processing on the text hiding state based on multiple second view encodings to obtain the text features of the text to be matched; a matching module 1508, which matches the image to be matched and the text to be matched based on the image features and the text features to obtain the target matching result; and a second calling module 1510, which outputs the target matching result by calling a second interface, wherein the second interface includes a second parameter whose value is the target matching result.

[0217] It should be noted that the first invocation module 1502, encoding module 1504, attention processing module 1506, matching module 1508, and second invocation module 1510 mentioned above correspond to steps S902 to S910 in Embodiment 5. The five modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules or units can be hardware or software components stored in memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n). The above modules can also be part of a device and run in the computer terminal 10 provided in Embodiment 1.

[0218] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0219] Example 12

[0220] According to embodiments of this application, a product retrieval device for implementing the above-described product retrieval method is also provided, such as... Figure 16 As shown, the device 1600 includes: a first invocation module 1602, an encoding module 1604, an attention processing module 1606, a matching module 1608, and a second invocation module 1610.

[0221] The system comprises the following modules: a first calling module 1602, which obtains the search text by calling a first interface, wherein the first interface includes a first parameter and the value of the first parameter is the search text; an encoding module 1604, which encodes multiple product images and the search text respectively to obtain the image hiding state of the product images and the text hiding state of the search text; an attention processing module 1606, which performs attention processing on the image hiding state based on multiple first view encodings to obtain the image features of the product images, and performs attention processing on the text hiding state based on multiple second view encodings to obtain the text features of the search text; a matching module 1608, which determines the target product image that matches the search text based on multiple image features and text features; and a second calling module 1610, which outputs the target product image by calling a second interface, wherein the second interface includes a second parameter and the value of the second parameter is the target product image.

[0222] It should be noted that the first invocation module 1602, encoding module 1604, attention processing module 1606, matching module 1608, and second invocation module 1610 mentioned above correspond to steps S1002 to S1010 in Embodiment 6. The five modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules or units can be hardware or software components stored in memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n). The above modules can also be part of a device and run in the computer terminal 10 provided in Embodiment 1.

[0223] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0224] Example 13

[0225] Embodiments of this application may provide a computer terminal, which may be any computer terminal device in a group of computer terminals. Optionally, in this embodiment, the aforementioned computer terminal may also be replaced by a mobile terminal or other terminal device.

[0226] Optionally, in this embodiment, the computer terminal may be located in at least one of a plurality of network devices in a computer network.

[0227] In this embodiment, the computer terminal described above can execute the program code for the following steps in the image-text matching method: acquiring the image to be matched and the text to be matched; encoding the image to be matched and the text to be matched respectively to obtain the image hiding state of the image to be matched and the text hiding state of the text to be matched; performing attention processing on the image hiding state based on multiple first view encodings to obtain the image features of the image to be matched, and performing attention processing on the text hiding state based on multiple second view encodings to obtain the text features of the text to be matched; matching the image to be matched and the text to be matched based on the image features and the text features to obtain the target matching result.

[0228] Optionally, Figure 17 This is a structural block diagram of a computer terminal according to an embodiment of this application. Figure 17 As shown, the computer terminal A may include: one or more (only one is shown in the figure) processors 1702, memory 1704, memory controller, and peripheral interfaces, wherein the peripheral interfaces are connected to a radio frequency module, an audio module, and a display.

[0229] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the image-text matching method and apparatus in this embodiment. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby realizing the aforementioned image-text matching method. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to terminal A via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0230] The processor can invoke information and application programs stored in the memory through the transmission device to perform the following steps: acquiring the image to be matched and the text to be matched; encoding the image to be matched and the text to be matched respectively to obtain the image hiding state of the image to be matched and the text hiding state of the text to be matched; performing attention processing on the image hiding state based on multiple first view encodings to obtain the image features of the image to be matched, and performing attention processing on the text hiding state based on multiple second view encodings to obtain the text features of the text to be matched; matching the image to be matched and the text to be matched based on the image features and the text features to obtain the target matching result.

[0231] Optionally, the processor may also execute program code that performs the following steps: performing cross-attention processing on multiple first view encodings and image hidden states to obtain multiple first attention features, and performing cross-attention processing on multiple second view encodings and text hidden states to obtain multiple second attention features; performing a weighted sum on multiple first attention features and image hidden states to obtain multiple first view features, and performing a weighted sum on multiple second attention features and text hidden states to obtain multiple second view features; concatenating multiple first view features to obtain image features, and concatenating multiple second view features to obtain text features.

[0232] Optionally, the processor may also execute program code that performs the following steps: using the first view encoding as the first query, the image hiding state as the first key and the first value, and the second view encoding as the second query, the text hiding state as the second key and the second value; processing the first query, the first key and the first value to obtain the first attention feature, and processing the second query, the second key and the second value to obtain the second attention feature.

[0233] Optionally, the processor may also execute program code that performs the following steps: weighted summation of the first feature value contained in the first attention feature and the first sub-state contained in the image hidden state to obtain the first view feature; weighted summation of the second feature value contained in the second attention feature and the second sub-state contained in the text hidden state to obtain the second view feature.

[0234] Optionally, the processor may also execute program code for the following steps: acquiring training samples, wherein the training samples include: training images and training text; encoding the training images and training text respectively to obtain a first hidden state of the training images and a second hidden state of the training text; performing attention processing on the first hidden state based on multiple first initial encodings to obtain a first feature of the training images, and performing attention processing on the second hidden state based on multiple second initial encodings to obtain a second feature of the training text; constructing a target loss based on the first feature, the second feature, multiple first initial encodings, and multiple second initial encodings; and adjusting the multiple first initial encodings and multiple second initial encodings based on the target loss to obtain multiple first view encodings and multiple second view encodings.

[0235] Optionally, the processor may also execute program code that performs the following steps: constructing a matching loss based on a first feature and a second feature; constructing a diversity loss based on multiple first initial codes and multiple second initial codes; and weighting the matching loss and the diversity loss to obtain the target loss.

[0236] Optionally, the processor may also execute program code that performs the following steps: constructing a first image-to-text matching loss based on a first feature and multiple second features; constructing a second text-to-image matching loss based on the second features and multiple first features; and obtaining the mean of the first matching loss and the second matching loss to obtain the matching loss.

[0237] Optionally, the processor may also execute program code with the following steps: constructing a first matrix based on multiple first initial codes, and constructing a second matrix based on multiple second initial codes; obtaining the product of the first matrix and its transpose to obtain a first matrix multiplication, and obtaining the product of the second matrix and its transpose to obtain a second matrix multiplication; obtaining the difference between the first matrix multiplication and the identity matrix to obtain a first matrix difference, and obtaining the difference between the second matrix multiplication and the identity matrix to obtain a second matrix difference; obtaining the norm of the first matrix difference and the second matrix difference to obtain a first diversity loss and a second diversity loss; and obtaining the sum of the first diversity loss and the second diversity loss to obtain a diversity loss.

[0238] Optionally, the processor may also execute program code that performs the following steps: encodes the image to be matched using an image encoder to obtain the image hidden state; and encodes the text to be matched using a text encoder to obtain the text hidden state.

[0239] Optionally, the processor may also execute program code for the following steps: segmenting the image to be matched to obtain multiple image blocks; concatenating the first preset information with the multiple image blocks to obtain first concatenation information, wherein the first preset information is used to characterize the image to be matched; and encoding the first concatenation information using an image encoder to obtain the image hiding state.

[0240] Optionally, the processor may also execute program code with the following steps: segmenting the text to be matched into multiple words; concatenating the second preset information, the multiple words, and the third preset information to obtain second concatenated information, wherein the second preset information is used to represent the start position of the text to be matched, and the third preset information is used to represent the end position of the text to be matched; and encoding the second concatenated information using a text encoder to obtain the text hidden state.

[0241] Optionally, the processor may also execute program code that performs the following steps: obtaining the inner product of image features and text features to obtain a matching score, wherein the matching score is used to characterize the matching probability of the image to be matched and the text to be matched; comparing the matching score with a preset score to obtain the target matching result.

[0242] This application provides an image-text matching scheme. After obtaining the image to be matched and the text to be matched, the image and text are encoded separately to obtain the image hidden state of the image to be matched and the text hidden state of the text to be matched. Then, attention processing is applied to the image hidden state based on multiple first-view encodings to obtain the image features of the image to be matched, and attention processing is applied to the text hidden state based on multiple second-view encodings to obtain the text features of the text to be matched. Finally, based on the image features and text features, the image and text to be matched are matched to obtain the target matching result, thus achieving the purpose of image-text matching. It is noteworthy that the final image features and text features used for image-text matching are obtained by applying attention processing to the hidden state using multi-view encoding. This allows the image and text to be encoded from different angles, obtaining features containing more information. This enables the matching score between the image and text to be calculated from different aspects, achieving the technical effect of improving the accuracy of image-text matching, and thus solving the technical problem of low accuracy in image-text matching using a two-stream model.

[0243] The processor can invoke information and application programs stored in the memory via a transmission device to perform the following steps: acquiring search text and multiple product images; encoding the product images and search text respectively to obtain the image hiding state of the product images and the text hiding state of the search text; performing attention processing on the image hiding state based on multiple first view encodings to obtain image features of the product images, and performing attention processing on the text hiding state based on multiple second view encodings to obtain text features of the search text; and determining the target product image that matches the search text based on the multiple image features and text features.

[0244] The processor can invoke information and application programs stored in memory via a transmission device to perform the following steps: in response to an input command applied to the operation interface, displaying search text on the operation interface; in response to a matching command applied to the operation interface, displaying a target product image that matches the search text on the operation interface, wherein the target product image is determined based on the text features of the search text and the image features of multiple product images, the image features are obtained by attention processing of the image hiding state of the product images based on multiple first view codes, and the text features are obtained by attention processing of the text hiding state of the search text based on multiple second view codes, the image hiding state and the text hiding state are obtained by encoding the product images and the search text respectively.

[0245] The processor can invoke information and application programs stored in memory via a transmission device to perform the following steps: in response to an input command applied to the operation interface, displaying search text on the operation interface; in response to a matching command applied to the operation interface, displaying a target product image that matches the search text on the operation interface, wherein the target product image is determined based on the text features of the search text and the image features of multiple product images, the image features are obtained by attention processing of the image hiding state of the product images based on multiple first view codes, and the text features are obtained by attention processing of the text hiding state of the search text based on multiple second view codes, the image hiding state and the text hiding state are obtained by encoding the product images and the search text respectively.

[0246] The processor can invoke information and application programs stored in the memory via a transmission device to perform the following steps: acquiring an image to be matched and text to be matched by calling a first interface, wherein the first interface includes a first parameter whose value is the image to be matched and the text to be matched; encoding the image to be matched and the text to be matched respectively to obtain the image hiding state of the image to be matched and the text hiding state of the text to be matched; performing attention processing on the image hiding state based on multiple first view encodings to obtain image features of the image to be matched, and performing attention processing on the text hiding state based on multiple second view encodings to obtain text features of the text to be matched; matching the image to be matched and the text to be matched based on the image features and the text features to obtain a target matching result; and outputting the target matching result by calling a second interface, wherein the second interface includes a second parameter whose value is the target matching result.

[0247] The processor can invoke information and application programs stored in the memory via a transmission device to perform the following steps: obtaining search text by calling a first interface, wherein the first interface includes a first parameter, and the parameter value of the first parameter is the search text; encoding multiple product images and the search text respectively to obtain the image hiding state of the product images and the text hiding state of the search text; performing attention processing on the image hiding state based on multiple first view encodings to obtain image features of the product images, and performing attention processing on the text hiding state based on multiple second view encodings to obtain text features of the search text; determining the target product image that matches the search text based on multiple image features and text features; and outputting the target product image by calling a second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the target product image.

[0248] Those skilled in the art will understand that Figure 17The structure shown is for illustrative purposes only. The computer terminal can also be a smartphone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (MID), a PAD, and other terminal devices. Figure 17 This does not limit the structure of the aforementioned electronic device. For example, computer terminal A may also include components that are more... Figure 17 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 17 The different configurations shown.

[0249] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0250] Example 14

[0251] Embodiments of this application also provide a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the image-text matching method provided in Embodiment 1.

[0252] Optionally, in this embodiment, the storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0253] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: acquiring an image to be matched and text to be matched; encoding the image to be matched and text to be matched respectively to obtain the image hiding state of the image to be matched and the text hiding state of the text to be matched; performing attention processing on the image hiding state based on multiple first view encodings to obtain image features of the image to be matched, and performing attention processing on the text hiding state based on multiple second view encodings to obtain text features of the text to be matched; matching the image to be matched and the text to be matched based on the image features and the text features to obtain the target matching result.

[0254] Optionally, the storage medium is further configured to store program code for performing the following steps: performing cross-attention processing on multiple first view encodings and image hidden states to obtain multiple first attention features, and performing cross-attention processing on multiple second view encodings and text hidden states to obtain multiple second attention features; performing a weighted sum of multiple first attention features and image hidden states to obtain multiple first view features, and performing a weighted sum of multiple second attention features and text hidden states to obtain multiple second view features; concatenating multiple first view features to obtain image features, and concatenating multiple second view features to obtain text features.

[0255] Optionally, the storage medium is further configured to store program code for performing the following steps: encoding the first view as a first query, the image hiding state as a first key and a first value, encoding the second view as a second query, and the text hiding state as a second key and a second value; processing the first query, the first key, and the first value to obtain a first attention feature, and processing the second query, the second key, and the second value to obtain a second attention feature.

[0256] Optionally, the storage medium is further configured to store program code for performing the following steps: weighting the first feature value contained in the first attention feature with the first sub-state contained in the image hidden state to obtain a first view feature; weighting the second feature value contained in the second attention feature with the second sub-state contained in the text hidden state to obtain a second view feature.

[0257] Optionally, the aforementioned storage medium is further configured to store program code for performing the following steps: acquiring training samples, wherein the training samples include: training images and training text; encoding the training images and training text respectively to obtain a first hidden state of the training images and a second hidden state of the training text; performing attention processing on the first hidden state based on multiple first initial encodings to obtain a first feature of the training images, and performing attention processing on the second hidden state based on multiple second initial encodings to obtain a second feature of the training text; constructing a target loss based on the first feature, the second feature, the multiple first initial encodings, and the multiple second initial encodings; and adjusting the multiple first initial encodings and the multiple second initial encodings based on the target loss to obtain multiple first view encodings and multiple second view encodings.

[0258] Optionally, the storage medium is also configured to store program code for performing the following steps: constructing a matching loss based on a first feature and a second feature; constructing a diversity loss based on multiple first initial codes and multiple second initial codes; and weighting the matching loss and the diversity loss to obtain the target loss.

[0259] Optionally, the storage medium is further configured to store program code for performing the following steps: constructing a first image-to-text matching loss based on a first feature and multiple second features; constructing a second text-to-image matching loss based on the second features and multiple first features; and obtaining the mean of the first matching loss and the second matching loss to obtain the matching loss.

[0260] Optionally, the aforementioned storage medium is further configured to store program code for performing the following steps: constructing a first matrix based on multiple first initial codes, and constructing a second matrix based on multiple second initial codes; obtaining the product of the first matrix and its transpose to obtain a first matrix multiplication, and obtaining the product of the second matrix and its transpose to obtain a second matrix multiplication; obtaining the difference between the first matrix multiplication and the identity matrix to obtain a first matrix difference, and obtaining the difference between the second matrix multiplication and the identity matrix to obtain a second matrix difference; obtaining the norm of the first matrix difference and the second matrix difference to obtain a first diversity loss and a second diversity loss; and obtaining the sum of the first diversity loss and the second diversity loss to obtain a diversity loss.

[0261] Optionally, the storage medium is also configured to store program code for performing the following steps: encoding the image to be matched using an image encoder to obtain an image hidden state; and encoding the text to be matched using a text encoder to obtain a text hidden state.

[0262] Optionally, the storage medium is further configured to store program code for performing the following steps: segmenting the image to be matched to obtain multiple image blocks; concatenating the first preset information with the multiple image blocks to obtain first concatenation information, wherein the first preset information is used to characterize the image to be matched; and encoding the first concatenation information using an image encoder to obtain the image hiding state.

[0263] Optionally, the storage medium is further configured to store program code for performing the following steps: segmenting the text to be matched into multiple words; concatenating the second preset information, the multiple words, and the third preset information to obtain second concatenated information, wherein the second preset information is used to represent the start position of the text to be matched, and the third preset information is used to represent the end position of the text to be matched; and encoding the second concatenated information using a text encoder to obtain a text hidden state.

[0264] Optionally, the storage medium is also configured to store program code for performing the following steps: obtaining the inner product of image features and text features to obtain a matching score, wherein the matching score is used to characterize the matching probability of the image to be matched and the text to be matched; comparing the matching score with a preset score to obtain the target matching result.

[0265] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: acquiring retrieval text and multiple product images; encoding the product images and retrieval text respectively to obtain the image hiding state of the product images and the text hiding state of the retrieval text; performing attention processing on the image hiding state based on multiple first view encodings to obtain image features of the product images, and performing attention processing on the text hiding state based on multiple second view encodings to obtain text features of the retrieval text; and determining the target product image that matches the retrieval text based on the multiple image features and text features.

[0266] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: in response to an input command applied to the operation interface, displaying search text on the operation interface; in response to a matching command applied to the operation interface, displaying a target product image that matches the search text on the operation interface, wherein the target product image is determined based on the text features of the search text and the image features of multiple product images, the image features are obtained by performing attention processing on the image hiding state of the product images based on multiple first view codes, the text features are obtained by performing attention processing on the text hiding state of the search text based on multiple second view codes, and the image hiding state and the text hiding state are obtained by encoding the product images and the search text respectively.

[0267] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: in response to an input command applied to the operation interface, displaying search text on the operation interface; in response to a matching command applied to the operation interface, displaying a target product image that matches the search text on the operation interface, wherein the target product image is determined based on the text features of the search text and the image features of multiple product images, the image features are obtained by performing attention processing on the image hiding state of the product images based on multiple first view codes, the text features are obtained by performing attention processing on the text hiding state of the search text based on multiple second view codes, and the image hiding state and the text hiding state are obtained by encoding the product images and the search text respectively.

[0268] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: obtaining an image to be matched and text to be matched by calling a first interface, wherein the first interface includes a first parameter, the parameter value of the first parameter being the image to be matched and the text to be matched; encoding the image to be matched and the text to be matched respectively to obtain the image hiding state of the image to be matched and the text hiding state of the text to be matched; performing attention processing on the image hiding state based on multiple first view encodings to obtain image features of the image to be matched, and performing attention processing on the text hiding state based on multiple second view encodings to obtain text features of the text to be matched; matching the image to be matched and the text to be matched based on the image features and the text features to obtain a target matching result; and outputting the target matching result by calling a second interface, wherein the second interface includes a second parameter, the parameter value of the second parameter being the target matching result.

[0269] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: obtaining search text by calling a first interface, wherein the first interface includes a first parameter, and the parameter value of the first parameter is the search text; encoding multiple product images and the search text respectively to obtain the image hiding state of the product images and the text hiding state of the search text; performing attention processing on the image hiding state based on multiple first view encodings to obtain image features of the product images, and performing attention processing on the text hiding state based on multiple second view encodings to obtain text features of the search text; determining a target product image that matches the search text based on multiple image features and text features; and outputting the target product image by calling a second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the target product image.

[0270] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0271] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0272] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0273] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0274] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0275] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0276] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. An image-text matching method, characterized in that, include: Get the image and text to be matched; The image to be matched and the text to be matched are encoded respectively to obtain the image hiding state of the image to be matched and the text hiding state of the text to be matched; Attention processing is performed on the hidden state of the image based on multiple first view codes to obtain the image features of the image to be matched, and attention processing is performed on the hidden state of the text based on multiple second view codes to obtain the text features of the text to be matched. The multiple first view codes are visual representations of multiple views constructed based on the image, and the multiple second view codes are text representations of multiple views constructed based on the text. Based on the image features and the text features, the image to be matched and the text to be matched are matched to obtain the target matching result.

2. The method according to claim 1, characterized in that, Attention processing is applied to the hidden state of the image based on multiple first view encodings to obtain the image features of the image to be matched, and attention processing is applied to the hidden state of the text based on multiple second view encodings to obtain the text features of the text to be matched, including: Cross-attention processing is performed on multiple first view codes and the image hidden state to obtain multiple first attention features, and cross-attention processing is performed on multiple second view codes and the text hidden state to obtain multiple second attention features; Multiple first attention features and the image hiding state are weighted and summed to obtain multiple first view features, and multiple second attention features and the text hiding state are weighted and summed to obtain multiple second view features; The image features are obtained by stitching together multiple first view features, and the text features are obtained by stitching together multiple second view features.

3. The method according to claim 2, characterized in that, Cross-attention processing is performed on multiple first view encodings and the image hidden state to obtain multiple first attention features, and cross-attention processing is performed on multiple second view encodings and the text hidden state to obtain multiple second attention features, including: The first view encoding is used as the first query, the image hiding state is used as the first key and the first value, and the second view encoding is used as the second query, the text hiding state is used as the second key and the second value; The first query, the first key, and the first value are processed to obtain the first attention feature, and the second query, the second key, and the second value are processed to obtain the second attention feature.

4. The method according to claim 2, characterized in that, Multiple first attention features and the image hiding state are weighted and summed to obtain multiple first view features, and multiple second attention features and the text hiding state are weighted and summed to obtain multiple second view features, including: The first view feature is obtained by weighting the first feature value contained in the first attention feature and the first sub-state contained in the image hidden state; The second view feature is obtained by weighting the second feature value contained in the second attention feature with the second sub-state contained in the text hidden state.

5. The method according to claim 2, characterized in that, The method further includes: Obtain training samples, wherein the training samples include: training images and training text; The training image and the training text are encoded respectively to obtain the first hidden state of the training image and the second hidden state of the training text; Attention processing is performed on the first hidden state based on multiple first initial codes to obtain the first feature of the training image, and attention processing is performed on the second hidden state based on multiple second initial codes to obtain the second feature of the training text. Based on the first feature, the second feature, multiple first initial codes, and multiple second initial codes, a target loss is constructed; Based on the target loss, multiple first initial codes and multiple second initial codes are adjusted to obtain multiple first view codes and multiple second view codes.

6. The method according to claim 5, characterized in that, Based on the first feature, the second feature, multiple first initial codes, and multiple second initial codes, a target loss is constructed, including: Based on the first feature and the second feature, a matching loss is constructed; A diversity loss is constructed based on multiple first initial codes and multiple second initial codes; The target loss is obtained by weighting and summing the matching loss and the diversity loss.

7. The method according to claim 6, characterized in that, Based on the first feature and the second feature, a matching loss is constructed, including: Based on the first feature and multiple second features, a first image-to-text matching loss is constructed; Based on the second feature and multiple first features, a second text-to-image matching loss is constructed; The matching loss is obtained by averaging the first matching loss and the second matching loss.

8. The method according to claim 6, characterized in that, Based on multiple first initial codes and multiple second initial codes, a diversity loss is constructed, including: A first matrix is ​​constructed based on multiple first initial codes, and a second matrix is ​​constructed based on multiple second initial codes; Obtain the product of the first matrix and its transpose to get the first matrix multiplication, and obtain the product of the second matrix and its transpose to get the second matrix multiplication; Obtain the difference between the first matrix multiplied by the identity matrix to obtain the first matrix difference, and obtain the difference between the second matrix multiplied by the identity matrix to obtain the second matrix difference; Obtain the norms of the first matrix difference and the second matrix difference to obtain the first diversity loss and the second diversity loss; The diversity loss is obtained by summing the first diversity loss and the second diversity loss.

9. The method according to claim 1, characterized in that, Encode the image to be matched and the text to be matched respectively to obtain the image hiding state of the image to be matched and the text hiding state of the text to be matched, including: The image to be matched is encoded using an image encoder to obtain the hidden state of the image; The text to be matched is encoded using a text encoder to obtain the hidden state of the text.

10. The method according to claim 9, characterized in that, The image to be matched is encoded using an image encoder to obtain the hidden state of the image, including: The image to be matched is segmented to obtain multiple image blocks; The first preset information is concatenated with the plurality of image blocks to obtain the first concatenation information, wherein the first preset information is used to characterize the image to be matched; The first stitching information is encoded using the image encoder to obtain the hidden state of the image.

11. The method according to claim 9, characterized in that, The text to be matched is encoded using a text encoder to obtain the text hiding state, including: The text to be matched is segmented into multiple words; The second preset information, the multiple words, and the third preset information are concatenated to obtain the second concatenation information, wherein the second preset information is used to represent the start position of the text to be matched, and the third preset information is used to represent the end position of the text to be matched. The second concatenated information is encoded using the text encoder to obtain the text hidden state.

12. The method according to claim 1, characterized in that, Based on the image features and the text features, the image to be matched and the text to be matched are matched to obtain the target matching result, including: The inner product of the image features and the text features is obtained to get the matching score, wherein the matching score is used to characterize the matching probability between the image to be matched and the text to be matched; The matching score is compared with a preset score to obtain the target matching result.

13. A product retrieval method, characterized in that, include: Retrieve the search text and multiple product images; The product image and the search text are encoded respectively to obtain the image hiding state of the product image and the text hiding state of the search text; Attention processing is performed on the hidden state of the image based on multiple first view codes to obtain the image features of the product image, and attention processing is performed on the hidden state of the text based on multiple second view codes to obtain the text features of the retrieved text. The multiple first view codes are visual representations of multiple views constructed based on the image, and the multiple second view codes are text representations of multiple views constructed based on the text. Based on multiple image features and text features, a target product image matching the search text is determined.

14. An image-text matching method, characterized in that, include: In response to input commands applied to the user interface, the user interface displays the image and text to be matched. In response to a matching command applied to the operation interface, a target matching result is displayed on the operation interface. The target matching result is obtained by matching the image to be matched and the text to be matched based on the image features of the image to be matched and the text features of the text to be matched. The image features are obtained by performing attention processing on the image hidden state of the image to be matched based on multiple first view codes, and the text features are obtained by performing attention processing on the text hidden state of the text to be matched based on multiple second view codes. The image hidden state and the text hidden state are respectively encoded on the image to be matched and the text to be matched. The multiple first view codes are visual representations of multiple views constructed based on the image, and the multiple second view codes are text representations of multiple views constructed based on the text.

15. A product retrieval method, characterized in that, include: In response to input commands applied to the user interface, the search text is displayed on the user interface; In response to a matching command applied to the operation interface, a target product image matching the search text is displayed on the operation interface. The target product image is determined based on the text features of the search text and image features of multiple product images. The image features are obtained by performing attention processing on the image hiding state of the product images based on multiple first view codes. The text features are obtained by performing attention processing on the text hiding state of the search text based on multiple second view codes. The image hiding state and the text hiding state are respectively encoded on the product image and the search text. The multiple first view codes are visual representations of multiple views constructed based on images, and the multiple second view codes are text representations of multiple views constructed based on text.

16. An image-text matching method, characterized in that, include: The image to be matched and the text to be matched are obtained by calling a first interface, wherein the first interface includes a first parameter, and the parameter value of the first parameter is the image to be matched and the text to be matched; The image to be matched and the text to be matched are encoded respectively to obtain the image hiding state of the image to be matched and the text hiding state of the text to be matched; Attention processing is performed on the hidden state of the image based on multiple first view codes to obtain the image features of the image to be matched, and attention processing is performed on the hidden state of the text based on multiple second view codes to obtain the text features of the text to be matched. The multiple first view codes are visual representations of multiple views constructed based on the image, and the multiple second view codes are text representations of multiple views constructed based on the text. Based on the image features and the text features, the image to be matched and the text to be matched are matched to obtain the target matching result; The target matching result is output by calling the second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the target matching result.

17. A product retrieval method, characterized in that, include: The search text is obtained by calling a first interface, wherein the first interface includes a first parameter, and the value of the first parameter is the search text; Encode multiple product images and the search text respectively to obtain the image hiding state of the product images and the text hiding state of the search text; Attention processing is performed on the hidden state of the image based on multiple first view codes to obtain the image features of the product image, and attention processing is performed on the hidden state of the text based on multiple second view codes to obtain the text features of the retrieved text. The multiple first view codes are visual representations of multiple views constructed based on the image, and the multiple second view codes are text representations of multiple views constructed based on the text. Based on multiple image features and text features, a target product image matching the search text is determined; The target product image is output by calling a second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the target product image.

18. An electronic device, characterized in that, include: Memory, which stores executable programs; A processor for running the program, wherein the program, when running, performs the method according to any one of claims 1 to 17.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored executable program, wherein, when the executable program is executed, it controls the device on which the computer-readable storage medium is located to perform the method according to any one of claims 1 to 17.

Images