Search method, model training method and related devices

Abstract

This disclosure relates to a retrieval method, a model training method, and related apparatus. The retrieval method includes: acquiring current text for retrieval; generating a target text representation of the current text using a first tower model of a dual-tower model; if the current text includes visual attribute information, determining a target item from candidate items based on the target multimodal representation of candidate items and the target text representation of the current text, wherein the target multimodal representation of the candidate item is generated using a second tower model of the dual-tower model based on candidate text and candidate visual information related to the candidate item, and the target multimodal representation integrates feature information from candidate text and candidate visual information; if the current text does not include visual attribute information, determining a target item from candidate items based on the target text representation of candidate text and the target text representation of the current text, wherein the target text representation of the candidate text is generated using a second tower model based on candidate text related to the candidate item.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to retrieval methods, model training methods and related devices, electronic devices, and computer-storable media. Background Technology

[0002] In e-commerce scenarios, the main task of search or retrieval is to quickly retrieve a small number (e.g., tens of thousands) of items that are relevant to the user's search intent from a massive (e.g., billion-level) item database.

[0003] In related technologies, in response to receiving retrieval text from a user, a textual representation of the retrieval text and a representation of single-modal information related to the retrieved items are generated using the user tower and item tower of a dual-tower model, respectively. Based on the textual representation and the single-modal information representation, the target item is determined from the candidate items as the retrieval result. Summary of the Invention

[0004] In related technologies, the item tower only has single modal information of the item, which cannot meet the needs of rich semantic expression of the item tower, and the accuracy of retrieval and matching is poor.

[0005] To address the aforementioned technical issues, this disclosure proposes a solution that can more accurately match search text and improve search accuracy.

[0006] According to a first aspect of this disclosure, a retrieval method is provided, comprising: acquiring current text for retrieval; generating a target text representation of the current text using a first tower model of a dual-tower model; determining whether the current text includes visual attribute information related to an item; if the current text includes the visual attribute information, determining a target item from the candidate items as a retrieval result based on the target multimodal representation of candidate items and the target text representation of the current text, wherein the target multimodal representation of the candidate items is generated using a second tower model of the dual-tower model based on candidate text and candidate visual information related to the candidate items, and the target multimodal representation integrates feature information of the candidate text and the candidate visual information; if the current text does not include the visual attribute information, determining a target item from the candidate items as a retrieval result based on the target text representation of the candidate text and the target text representation of the current text, wherein the target text representation of the candidate text is generated using the second tower model based on candidate text related to the candidate items.

[0007] According to a second aspect of this disclosure, a model training method is provided, comprising: generating a target text representation for a first training text to be retrieved using a first tower model of a dual-tower model; generating a target multimodal representation of the training item and a target text representation of the second training text using the second tower model of the dual-tower model, based on a second training text related to a training item and training visual information, wherein the target multimodal representation integrates feature information from the second training text and the training visual information, and the training item is associated with the first training text; determining whether the first training text includes visual attribute information related to the item; if the first training text includes the visual attribute information, training the dual-tower model based on the target text representation of the first training text and the target multimodal representation of the training item; if the first training text does not include the visual attribute information, training the dual-tower model based on the target text representation of the first training text and the target text representation of the second training text.

[0008] According to a third aspect of this disclosure, a retrieval apparatus is provided, comprising: an acquisition module configured to acquire current text for retrieval; a generation module configured to generate a target text representation of the current text using a first tower model of a dual-tower model; a judgment module configured to determine whether the current text for retrieval includes visual attribute information related to an item; a first determination module configured to, when the current text includes the visual attribute information, determine a target item from the candidate items based on the target multimodal representation of candidate items and the target text representation of the current text, as a retrieval result, wherein the target multimodal representation of the candidate items is generated using the second tower model of the dual-tower model based on candidate text and candidate visual information related to the candidate items, and the target multimodal representation integrates feature information of the candidate text and the candidate visual information; and a second determination module configured to, when the current text does not include the visual attribute information, determine a target item from the candidate items based on the target text representation of the candidate text and the target text representation of the current text, as a retrieval result, wherein the target text representation of the candidate text is generated using the second tower model based on candidate text related to the candidate items.

[0009] According to a fourth aspect of this disclosure, a model training apparatus is provided, comprising: a first generation module configured to generate a target text representation for retrieving a first training text using a first tower model of a dual-tower model; a second generation module configured to generate a target multimodal representation of the training item and a target text representation of the second training text using the second tower model of the dual-tower model, based on a second training text related to a training item and training visual information, wherein the target multimodal representation integrates feature information from the second training text and the training visual information, and the training item is associated with the first training text; a judgment module configured to determine whether the first training text includes visual attribute information related to the item; and a training module configured to train the dual-tower model based on the target text representation of the first training text and the target multimodal representation of the training item when the first training text includes the visual attribute information, and to train the dual-tower model based on the target text representation of the first training text and the target text representation of the second training text when the first training text does not include the visual attribute information.

[0010] According to a fifth aspect of this disclosure, an electronic device is provided, comprising: a memory; and a processor coupled to the memory, the processor being configured to execute the model training method or retrieval method described in any of the above embodiments based on instructions stored in the memory.

[0011] According to a sixth aspect of this disclosure, a computer-storeable medium is provided that stores computer program instructions thereon, which, when executed by a processor, implement the model training method or retrieval method described in any of the above embodiments.

[0012] In the above embodiments, the search text can be matched more accurately, thereby improving the accuracy of the search. Attached Figure Description

[0013] The accompanying drawings, which form part of this specification, illustrate embodiments of this disclosure and, together with the specification, serve to explain the principles of this disclosure.

[0014] This disclosure will become clearer with reference to the accompanying drawings and the following detailed description, wherein:

[0015] Figure 1 This is a flowchart illustrating a retrieval method according to some embodiments of the present disclosure;

[0016] Figure 2 This is a schematic diagram illustrating the feature splicing process according to some embodiments of the present disclosure;

[0017] Figure 3This is a schematic diagram illustrating the generation of target text representation and target multimodal representation using a dual-tower model and the execution of adaptive modality selection according to some embodiments of the present disclosure;

[0018] Figure 4 This is a flowchart illustrating a model training method according to some embodiments of the present disclosure;

[0019] Figure 5 This is a block diagram illustrating a retrieval apparatus according to some embodiments of the present disclosure;

[0020] Figure 6 This is a block diagram illustrating a model training apparatus according to some embodiments of the present disclosure;

[0021] Figure 7 This is a block diagram illustrating an electronic device according to some embodiments of the present disclosure;

[0022] Figure 8 This is a block diagram illustrating a computer system for implementing some embodiments of the present disclosure. Detailed Implementation

[0023] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.

[0024] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0025] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0026] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0027] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0028] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0029] Figure 1 This is a flowchart illustrating a retrieval method according to some embodiments of the present disclosure.

[0030] like Figure 1 As shown, the retrieval method includes steps S110 to S150. In some embodiments, the retrieval method is performed by a retrieval device.

[0031] In step S110, the current text used for retrieval is obtained.

[0032] In some embodiments, the current text used for retrieval can be obtained in the following manner.

[0033] First, receive the search text entered by the user.

[0034] Then, in response to receiving the search text input by the user, the system obtains the user's user characteristic information. For example, user characteristic information includes, but is not limited to, personalized characteristics such as the user's gender and age. By incorporating user characteristic information, personalized search results can be provided to the user, further improving the accuracy and precision of the search.

[0035] Finally, the search text entered by the user and the user's characteristic information are determined as the current text.

[0036] In step S120, the first tower model of the dual-tower model is used to generate the target text representation of the current text. For the online retrieval service system, the first tower model of the dual-tower model is deployed online to generate the target text representation of the current text in real time based on the user's current text.

[0037] In step S130, it is determined whether the current text includes visual attribute information related to the item. For example, visual attribute information related to the item includes, but is not limited to, at least one of the item's color, shape, etc.

[0038] If the current text includes visual attribute information, step S140 is executed. In step S140, based on the target multimodal representation of the candidate items and the target text representation of the current text, the target item is determined from the candidate items as the retrieval result. The target multimodal representation of the candidate item is generated using the second tower model of a dual-tower model based on candidate text and candidate visual information related to the candidate item. The target multimodal representation fuses feature information from both candidate text and candidate visual information. The multimodal representation of the candidate item is obtained by fusing information from two different modalities, candidate text and candidate visual information, using the second tower model of a dual-tower model. For example, candidate visual information includes, but is not limited to, at least one of images and videos.

[0039] If the current text does not contain visual attribute information, proceed to step S150. In step S150, based on the target text representations of the candidate texts and the target text representation of the current text, the target item is determined from the candidate items and used as the retrieval result. The target text representations of the candidate texts are generated using the second tower model based on the candidate texts related to the candidate items.

[0040] In some embodiments, steps S140 and S150 can exist independently or simultaneously. That is, if the current text includes visual attribute information, step S140 is executed; if the current text does not include visual attribute information, step S150 is executed to determine the target item or to determine the target item using a method other than step S150. Similarly, if the current text does not include visual attribute information, step S150 is executed; if the current text includes visual attribute information, step S140 is executed to determine the target item or to determine the target item using a method other than step S140.

[0041] In e-commerce, the range of goods is vast and diverse, covering thousands of categories, and different products have varying degrees of reliance on images. Some fashion categories (such as bags and clothing) rely far more on visual information than electronics categories (such as computers and digital products). This is because fashion items have more varied styles and colors, while most computer and digital products are standardized with relatively uniform and fixed shapes. In the above embodiment, based on whether the current text includes visual attribute information, the text modality or multimodality of the item is adaptively selected for the retrieval and matching process. This addresses the issue of different types of items having varying degrees of reliance on visual attribute information, more accurately matching the user's input search text, and improving the accuracy of retrieval or recall.

[0042] For example, when the current text includes visual attribute information, a multimodal representation that integrates textual and visual feature information related to candidate items is introduced. This enriches the semantic expression of the candidate items and utilizes visual information to provide missing information or denoise redundant information in the text, thus improving the retrieval accuracy when the current text includes visual attribute information. Conversely, when the current text does not include visual attribute information, the introduction of redundant information is avoided, further improving the retrieval accuracy in this case.

[0043] For online retrieval service systems, an online vector indexing service is deployed. This service is built upon the target multimodal representation of candidate items and / or the target text representation of candidate text generated using the second tower model of a dual-tower model. In some embodiments, the vector index is a large-scale vector approximate nearest neighbor search index built using product quantization techniques.

[0044] In some embodiments, when the current text includes visual attribute information related to the item, determining the target item from the candidate items based on the target multimodal representation of the candidate item and the target text representation of the current text includes the following steps.

[0045] First, given that the current text includes visual attribute information related to the item, the similarity between the target multimodal representation of the candidate item and the target text representation of the current text is determined as the first similarity.

[0046] Then, from the candidate items, the candidate items with a first similarity greater than the similarity threshold are selected as the target items.

[0047] In some embodiments, determining the similarity between the target multimodal representation of a candidate item and the target text representation of the current text, as a first similarity, includes the following steps.

[0048] First, the target multimodal representation of the candidate item and the target text representation of the candidate text are spliced ​​together as the first splicing vector.

[0049] Figure 2 This is a schematic diagram illustrating the feature splicing process according to some embodiments of the present disclosure.

[0050] like Figure 2 As shown, taking a 128-dimensional feature representation as an example, for candidate items, the 128-dimensional target text representation I of the candidate text is used. t 128-dimensional target multimodal representation of candidate items I m The concatenation process yields a first concatenated vector of length 256, denoted as I. mt =[I t ,I m In this case, for example, for an online retrieval service system, I can be used. mt Construct vector indices for candidate items.

[0051] Then, based on the target text representation of the current text, the zero vector with the same concatenation dimension as the target text representation of the candidate text is used as the second concatenation vector, wherein the second concatenation vector has the same dimension as the first concatenation vector, and the dimension of the zero vector is the same as the dimension of the target text representation of the candidate text.

[0052] like Figure 2 As shown, when the current text includes visual attribute information related to the item, a multimodal approach is selected. In this case, a 128-dimensional zero vector is concatenated to the head of the 128-dimensional target text representation Q of the current text. We obtain a second concatenated vector of 256 dimensions, denoted as...

[0053] Finally, based on the first and second splicing vectors, the similarity between the target multimodal representation of the candidate item and the target text representation of the current text is determined as the first similarity.

[0054] For example, refer to Figure 2 The first similarity is represented as At this point, calculating the inner product of the first concatenation vector and the second concatenation vector is equivalent to calculating the inner product of the target text representation of the current text and the target text representation of the candidate item.

[0055] In some embodiments, when the current text does not include visual attribute information related to the item, determining the target item from the candidate items based on the target text representation of the candidate text and the target text representation of the current text includes: determining the similarity between the target text representation of the candidate text and the target text representation of the current text as a second similarity; and determining candidate items from the candidate items whose second similarity is greater than a similarity threshold as the target item.

[0056] In some embodiments, determining the similarity between the target text representation of the candidate text and the target text representation of the current text, as a second similarity, includes the following steps.

[0057] First, the target multimodal representation of the candidate item and the target text representation of the candidate text are concatenated to form the first concatenation vector. Related embodiments can be found in the foregoing. Figure 2 The description of the first concatenated vector.

[0058] Then, based on the target text representation of the current text, the zero vector with the same splicing dimension as the target multimodal representation of the candidate item is used as the third splicing vector, wherein the third splicing vector has the same dimension as the first splicing vector, and the dimension of the zero vector is the same as the dimension of the target multimodal representation of the candidate item.

[0059] like Figure 2 As shown, when the current text does not include visual attribute information related to the item, the text modality is selected. In this case, a 128-dimensional zero vector is appended to the end of the 128-dimensional target text representation Q of the current text. The resulting 256-dimensional third concatenated vector is denoted as...

[0060] Finally, based on the first and third concatenation vectors, the similarity between the target text representation of the candidate text and the target text representation of the current text is determined as the second similarity.

[0061] refer to Figure 2 The second similarity is represented, for example, as At this point, calculating the inner product of the first concatenation vector and the third concatenation vector is equivalent to calculating the inner product of the target text representation of the current text and the target multimodal representation of the candidate items.

[0062] Typically, retrieval service systems only support one feature representation (vector representation) for each item. In the above embodiments, this disclosure addresses the issue that an item may have both multimodal and textual feature representations during adaptive modality selection. By employing feature concatenation, these two feature representations are combined into a single feature representation, ensuring that each item corresponds to only one feature representation. This approach requires only modification to the model structure to adapt to the service system, without needing to modify the online retrieval service system, resulting in low deployment and development costs.

[0063] The following will combine Figure 3 Describes the process of generating target text representations of the current text and candidate texts, generating target multimodal representations of candidate items, and performing adaptive modality selection.

[0064] Figure 3 This is a schematic diagram illustrating the generation of target text representation and target multimodal representation using a dual-tower model and the execution of adaptive modality selection according to some embodiments of the present disclosure.

[0065] like Figure 3 As shown, the dual-tower model includes a first-tower model and a second-tower model.

[0066] In some embodiments, reference Figure 3 The first tower model includes a text feature extraction model and a dimensionality reduction module. In this case, the target text representation of the current text can be generated in the following way.

[0067] First, the text feature extraction model of the first-tower model is used to process the current text, obtaining an intermediate text representation of the current text. In some embodiments, taking the text feature extraction model of the first-tower model as a BERT (Bidirectional Encoder Representations from Transformer) language model as an example, the intermediate text representation of the current text incorporates the weight relationships between different sub-texts in the current text. For example, the BERT language model of the first-tower model has 4 layers.

[0068] For example, a subtext in the current text may include at least one of a word, a phrase, and a short sentence in the current text, and the weight relationship between different subtexts in the current text may include at least one of the following: the weight relationship between different words, the weight relationship between different phrases, the weight relationship between different short sentences, the weight relationship between words and phrases, the weight relationship between words and short sentences, and the weight relationship between phrases and short sentences.

[0069] Then, using the dimensionality reduction module of the first tower model, the intermediate text features of the current text are dimensionality reduced to obtain the target text representation of the current text.

[0070] In some embodiments, the text feature extraction model of the first tower model includes a feature representation layer and an encoding layer.

[0071] The feature representation layer of the first tower model, serving as its input layer, is configured to process the features of the current text, resulting in an embedding vector. For example, this embedding vector has a dimension of 128. Character features such as the current text undergo word segmentation, with each segmented unit corresponding to a vector. Continuous numerical features are segmented, with each segment interval represented by a vector. Discrete numerical features have each value directly corresponding to a vector.

[0072] The encoding layer of the first tower model is configured to encode the embedding vector of the current text using a deep encoder to obtain the intermediate text representation of the current text. For example, the feature length of the intermediate text representation of the current text is 768, denoted as F. q (f query ), where f query F represents the embedding vector representation of the current text. q () indicates the processing procedure of the coding layer of the first tower model.

[0073] In some embodiments, the dimensionality reduction module of the first tower model serves as the output layer of the first tower model, normalizing the intermediate text representation of the current text to obtain the target text representation of the current text. For example, the feature length of the target text representation of the current text is 128, denoted as Q = N(F q (f query )), where N() represents the processing procedure of the dimensionality reduction module of the first tower model.

[0074] In some embodiments, reference Figure 3 The second tower model includes a text feature extraction model and a dimensionality reduction module. The text feature extraction model is configured to generate intermediate text representations of candidate texts. The dimensionality reduction module is configured to perform dimensionality reduction on the intermediate text representations of the candidate texts to obtain the target text representations of the candidate texts.

[0075] In some embodiments, taking the BERT language model as an example of the text feature extraction model of the second tower model, the intermediate text representation of the candidate text incorporates the weight relationships between different sub-texts in the candidate text. For example, the BERT language model of the second tower model has 4 layers. The sub-texts in the candidate text and the weight relationships between sub-texts can be referred to the description of the sub-texts in the current text and the weight relationships between sub-texts mentioned above, and will not be repeated here.

[0076] In some embodiments, the text feature extraction model of the second tower model includes a feature representation module and an encoding module.

[0077] The feature representation module of the second tower model is configured to perform feature representation processing on the candidate text, obtaining the embedding vector of the candidate text. For example, the dimension of this embedding vector is 128. Character features such as candidate text undergo word segmentation, with each segmented unit corresponding to a vector. Continuous numerical features are segmented, with each segment interval represented by a vector. Discrete numerical features have each value directly corresponding to a vector.

[0078] The encoding module of the second tower model is configured to encode the embedding vectors of the candidate text using a deep encoder to obtain the intermediate text representation of the candidate text. For example, the feature length of the intermediate text representation of the candidate text is 768, denoted as F. text (f item-text ), where f item-text F represents the embedding vector representation of the candidate text. text () indicates the processing procedure of the encoding module of the second tower model.

[0079] In some embodiments, the feature representation module of the second tower model belongs to the input layer of the second tower model, and the encoding module of the second tower model belongs to the encoding layer of the second tower model.

[0080] In some embodiments, the dimensionality reduction module of the second tower model serves as the output layer of the second tower model, normalizing the intermediate text representations of the candidate text to obtain the target text representation of the candidate text. For example, the feature length of the target text representation of the candidate text is 128, denoted as I. t =N(F text (f item-text )), where N() represents the processing procedure of the dimensionality reduction module of the second tower model.

[0081] In some embodiments, reference Figure 3 The second tower model includes a text feature extraction model, a visual feature extraction model, a cross-attention module, and a dimensionality reduction module.

[0082] The text feature extraction model of the second tower model is configured to generate intermediate text representations of candidate texts based on the candidate texts.

[0083] The visual feature extraction model of the second tower model is configured to generate intermediate visual representations of candidate visual information based on the candidate visual information. In some embodiments, the visual feature extraction model includes a ResNet (Residual Network) model and a visual feature dimensionality reduction module. For example, taking an image as the candidate visual information, the feature size output by the ResNet model is [7, 7, 2048], denoted as F. img (f item-img ), where f item-img F represents candidate visual information. img () indicates the processing procedure of the ResNet model. After dimensionality reduction by the visual feature dimensionality reduction module, an intermediate visual representation with dimensions [49, 768] is obtained, denoted as I. img =N(F img (f item-img The ResNet model can also use other visual feature extraction models.

[0084] The cross-attention module is configured to generate intermediate multimodal representations of candidate items based on intermediate textual representations of candidate text and intermediate visual representations of candidate visual information. These intermediate multimodal representations incorporate weighted relationships between different modalities (textual and visual modalities). For example, the processing of the cross-attention module can be denoted as the fusion layer Fuse(). The intermediate multimodal representation is, for example, a 768-dimensional multimodal vector, denoted as Fuse(F... text (f item-text ), F img (f item-img )).

[0085] The dimensionality reduction module of the second tower model is configured to reduce the dimensionality of the intermediate multimodal representation to obtain the target multimodal representation of the candidate item. For example, the target multimodal representation is a 768-dimensional intermediate multimodal representation. After dimensionality reduction, it yields a 128-dimensional multimodal vector, denoted as I. m =N(Fuse(F) text (f item-text ), F img (f item-img ))).

[0086] In some embodiments, the visual feature extraction model and cross-attention module of the second tower model, as well as the encoding module of the first tower model, all belong to the encoding layer of the second tower model. The dimensionality reduction module of the second tower model serves as the output layer of the second tower model, normalizing the intermediate multimodal representations to obtain the target multimodal representation.

[0087] In some embodiments, reference Figure 3 An adaptive modality selection process can be performed based on whether the current text includes visual attribute information related to the item. If the current text includes visual attribute information related to the item, a multimodal selection process is performed, that is, selecting the target text representation of the current text and the target multimodal representation of the candidate item to determine the first similarity between them. If the current text does not include visual attribute information related to the item, a text modality selection process is performed, that is, selecting the target text representation of the current text and the target text representation of the candidate text to determine the second similarity between them. The target item can then be determined from the candidate items based on either the first or second similarity.

[0088] Figure 4 This is a flowchart illustrating a model training method according to some embodiments of the present disclosure.

[0089] like Figure 4 As shown, the model training method includes steps S410 to S450. In some embodiments, the model training method is performed by a model training device.

[0090] In step S410, the first tower model of the dual-tower model is used to generate a target text representation for the first training text to be retrieved. In some embodiments, the first tower model may be referred to as a query tower model or a user tower model.

[0091] In some embodiments, the training data for training the dual-tower model includes a first training sample and a second training sample. The first training text in the first training sample is generated based on the descriptive information of the training items, and the first training text in the second training sample includes the text used in the actual retrieval process.

[0092] In some embodiments, the first training text in the first training sample is a random substring of a specified length extracted from the description information of the training item. For example, the description information of the training item includes, but is not limited to, the title information of the training item.

[0093] In some embodiments, the text used in the actual retrieval process includes the search text entered by the user (such as query terms or search terms), and the first training text in the second training sample also includes the user's user characteristic information. For example, user characteristic information includes, but is not limited to, personalized characteristic information such as the user's gender and age. By introducing the user's user characteristic information during the training process, personalized search results can be provided to the user, further improving the accuracy and precision of the retrieval.

[0094] In step S420, based on the second training text and training visual information associated with the training item, a target multimodal representation of the training item and a target text representation of the second training text are generated using the second tower model of the dual-tower model. The target multimodal representation integrates feature information from the second training text and training visual information. The training item and the first training text are associated. In some embodiments, the second tower model can be referred to as an item tower model.

[0095] For example, based on the second training text associated with the training item, the second tower model of the dual-tower model is used to generate the target text representation of the second training text. As another example, based on the second training text associated with the training item and training visual information, the second tower model of the dual-tower model is used to generate the target multimodal representation of the training item.

[0096] In some embodiments, the second training text includes attribute features such as item title, value information, store information, and related brand information. Training visual information includes, but is not limited to, at least one of images and videos of the item. The second training text can be added to or removed depending on the scenario.

[0097] In some embodiments, taking the training data for training the dual-tower model as an example, which includes a first training sample and a second training sample, the training items corresponding to the second training sample include items for which a specified operation is performed during the actual retrieval process. For example, the specified operation includes at least one of a click operation, a browsing operation for a specified duration, and a sharing operation.

[0098] In some embodiments, in the first training sample, the second training text and training visual information related to the training item corresponding to the description information that generated the first training text are used as positive samples, and the second training text and training visual information related to other training items are used as negative samples; in the second training sample, the second training text and training visual information related to the training item to which the specified operation is performed in the actual retrieval process are used as positive samples, and the second training text and training visual information related to other training items in the actual retrieval process are used as negative samples.

[0099] Training the dual-tower model by combining positive and negative samples can improve the model's learning ability, thereby increasing the accuracy of the model's multimodal representation of candidate items and further improving the accuracy of retrieval.

[0100] In step S430, it is determined whether the first training text includes visual attribute information related to the item. For example, visual attribute information includes, but is not limited to, at least one of color description information and shape information.

[0101] If the first training text includes the aforementioned visual attribute information, step S440 is executed. In step S440, a dual-tower model is trained based on the target text representation of the first training text and the target multimodal representation of the training item.

[0102] If the first training text does not include the visual attribute information, step S450 is executed. In step S450, a dual-tower model is trained based on the target text representation of the first training text and the target text representation of the second training text.

[0103] In e-commerce scenarios, the range of goods is vast and diverse, covering thousands of categories, and different products have varying degrees of reliance on images. Some fashion categories (such as bags and clothing) rely far more on visual information than electronic categories (such as computers and digital products). This is because fashion items have greater style variations and a wider range of colors, while most computer and digital products are standardized products with relatively uniform and fixed shapes. In the above embodiment, based on whether the first training text contains visual attribute information, the text modality or multimodal information of the item is adaptively selected for training, solving the problem of different types of items having different degrees of reliance on visual information and improving the accuracy of retrieval.

[0104] For example, in training the second tower model of the dual-tower model, when the first training text includes visual attribute information, a multimodal representation that integrates textual and visual feature information related to the item is introduced. This enriches the semantic expression of the item on the second tower model side and utilizes visual information to provide missing information or denoise redundant information in the text. It can learn the semantic relationship between the visual and textual information of the retrieval text and the item from massive user retrieval data. This allows the trained dual-tower model to more accurately match the user's input retrieval text when the retrieval text includes visual attribute information, improving the accuracy of retrieval or recall when the retrieval text does not include visual attribute information. Conversely, when the first training text does not include visual attribute information, the introduction of redundant information is avoided, improving the accuracy of retrieval when the retrieval text does not include visual attribute information.

[0105] In the above embodiments, the dual-tower structure is designed primarily due to the sheer number of candidate items. In e-commerce search systems, for a given search text, relevant items need to be retrieved from millions or even hundreds of millions of candidate items. Therefore, considering computational costs and online efficiency, this model cannot be used for online scoring. By separating the first-tower model and the second-tower model, feature representations of relevant information can be calculated separately, enabling rapid online indexing and retrieval using Faiss (Facebook AI Similarity Search). A high-quality semantic understanding model helps the retrieval platform better understand user needs, recalling a richer variety of items more relevant to the user's request and better matching the search text, significantly improving the platform's service quality and user experience.

[0106] In some embodiments, the trained first tower model is configured to generate a target text representation of the current text for retrieval; the trained second tower model is configured to generate a target multimodal representation of the candidate items based on candidate text and candidate visual information associated with the candidate items. When the current text includes visual attribute information associated with the item, the target text representation of the current text and the target multimodal representation of the candidate items are used to determine the target item from the candidate items as the retrieval result. When the current text does not include visual attribute information associated with the item, the target text representation of the current text and the target text representation of the candidate text are used to determine the target item from the candidate items as the retrieval result.

[0107] In some embodiments, there are multiple first training texts, and each first training text, as well as the second training text and training visual information related to the training items associated with each first training text, belong to the same training sample. Multiple training samples are used to train the dual-tower model.

[0108] In this case, for each training sample, if the first training text includes visual attribute information, the similarity between the target text representation of the first training text and the target multimodal representation of the training item is determined as the third similarity; for each training sample, if the first training text does not include visual attribute information, the similarity between the target text representation of the first training text and the target text representation of the second training text is determined as the fourth similarity; the dual-tower model is constructed based on the third and fourth similarities determined from multiple training samples.

[0109] In some embodiments, the similarity between the target text representation of the first training text and the target multimodal representation of the training item can be determined as a third similarity in the following manner.

[0110] First, the target multimodal representation of the training item is concatenated with the target text representation of the second training text to form the fourth concatenation vector. For example, the generation of the fourth concatenation vector can refer to... Figure 2 The first concatenated vector.

[0111] Then, based on the target text representation of the first training text, a zero vector with the same dimension as the target text representation of the second training text is concatenated to form the fifth concatenated vector. The fifth concatenated vector has the same dimension as the fourth concatenated vector, and the dimension of the zero vector is the same as the dimension of the target text representation of the second training text. For example, the generation of the fifth concatenated vector can refer to... Figure 2 The second concatenated vector.

[0112] Finally, based on the fourth and fifth concatenation vectors, the similarity between the target text representation of the first training text and the target multimodal representation of the training item is determined as the third similarity. For example, the representation of the third similarity can refer to the representation of the first similarity.

[0113] In some embodiments, the similarity between the target text representation of the first training text and the target text representation of the second training text can be determined as a fourth similarity in the following manner.

[0114] First, the target multimodal representation of the training item is spliced ​​with the target text representation of the second training text to form the fourth splicing vector.

[0115] Then, based on the target text representation of the first training text, the zero vector with the same dimension as the target multimodal representation of the training item is concatenated and used as the sixth concatenation vector. The sixth concatenation vector has the same dimension as the fourth concatenation vector, and the dimension of the zero vector is the same as the dimension of the target multimodal representation of the training item. For example, the generation of the sixth concatenation vector can refer to... Figure 2 The third concatenated vector.

[0116] Finally, based on the fourth and sixth concatenation vectors, the similarity between the target text representations of the first training text and the target text representations of the second training text is determined as the fourth similarity. For example, the representation of the fourth similarity can refer to the representation of the second similarity.

[0117] In some embodiments, the dual-tower model can be implemented by using a third similarity and a fourth similarity determined based on multiple training samples.

[0118] First, the loss value is determined based on the third and fourth similarities determined from multiple training samples.

[0119] Then, based on the loss value, train the dual-tower model. For example, it can be trained using the gradient backpropagation algorithm, the Adam optimizer (adaptive moment estimator), and a learning rate of 5e-5. The data size for each optimization batch can be 256, and the optimization can be iterated 1 million times.

[0120] In some embodiments, taking the example that both the first and second training samples include positive and negative samples, the InfoNce (info Noise Contrastive Estimation) loss function can be used to determine the loss value.

[0121] For example, for the second training sample, user query information and product information with click relationships in the same optimization batch constitute positive sample pairs, while those without click relationships constitute negative sample pairs. First, the target text representation of the first training text, identified by 'i', is determined. The target multimodal representation of the training items denoted by j The similarity between them is denoted as s(Q i ,I i ), where s is the similarity calculation function between Q and I (e.g., an inner product function). When i = j, Q i and I j Data from the same click record is used as a positive sample. When i≠j, Q i with I j Different click data are used as negative samples. Then, the determined similarity s(Q) is... i ,I i Taking the exponent with base e, we obtain the exponential similarity, denoted as: Then, for all values ​​of j, calculate the sum of all exponential similarities. Next, calculate the ratio between the exponential similarity of the positive sample and the sum of all exponential similarities. Then, take the logarithm of the comparison value. Next, take the negative value of the logarithm to obtain the loss value corresponding to the first training text i. Finally, for all values ​​of i, calculate the sum of the loss values, which is taken as the total loss value, denoted as L. Similarly, the total loss value corresponding to the first training sample can also be calculated.

[0122] In the above embodiments, when the retrieved text includes visual attribute information related to the item, it indicates that the item retrieved by the user is closely related to the visual information. By using target multimodal representation to determine the third similarity when the first training text includes visual attribute information related to the item, not only can the visual features in the multimodal representation help reduce redundant text information caused by word stuffing in the item's text information, avoiding semantic shifts that lead to retrieved items that do not match the user's true intent, but the visual features in the multimodal representation can also supplement the item's text information, thereby improving the accuracy and precision of retrieving such items. By using the target text representation of the second training text to determine the fourth similarity when the first training text does not include visual attribute information related to the item, redundant information brought by visual features can be avoided, improving the accuracy of retrieval.

[0123] Furthermore, typically, retrieval service systems only support one feature representation (vector representation) for each item. In implementing adaptive modality selection, this disclosure addresses the issue that an item may have both multimodal and textual feature representations. In the above embodiments, feature concatenation is used to combine the two feature representations into a single feature representation, ensuring that each item corresponds to only one feature representation. This approach requires only modification to the model structure to adapt to the service system, without needing to modify the online retrieval service system, resulting in low deployment and development costs.

[0124] In some embodiments, taking the training data for training the dual-tower model as including a first training sample and a second training sample as an example, the model training method further includes: performing a first training on the dual-tower model based on the target text representation of the first training text in the first training sample and the target multimodal representation of the training item corresponding to the first training sample.

[0125] In some embodiments, the similarity between the target text representation of the first training text in the first training sample and the target multimodal representation of the training item corresponding to the first training sample is determined, and then a loss value is determined based on the similarity, and the loss value is used to perform the first training on the dual-tower model.

[0126] In this case, after the first training of the dual-tower model, it is determined whether the first training text in the second training sample includes visual attribute information related to the item. If the first training text in the second training sample includes visual attribute information, the dual-tower model is trained a second time based on the target text representation of the first training text in the second training sample and the target multimodal representation of the training item corresponding to the second training sample. If the first training text in the second training sample does not include visual attribute information, the dual-tower model is trained a second time based on the target text representation of the first training text in the second training sample and the target text representation of the second training text in the second training sample.

[0127] In the above embodiments, by synthesizing the first training text based on the description information of the training items, a large number of first training samples can be obtained, ensuring the diversity and richness of training instances. The dual-tower model is then trained using a larger number of first training samples, and subsequently trained a second time using second training samples that better reflect the actual retrieval scenario. By employing a pre-training combined with fine-tuning approach, the dual-tower model learns deeper modal alignment and fusion in the first training phase, fully exploring the semantic expression of the retrieval text and items, improving the model's generalization to mid-to-long-tail query terms, and enhancing the accuracy of the model's multimodal representations. In the second training phase, an adaptive modality selection process is incorporated to learn precise matching between the retrieval text and items. User retrieval data is used to fit the real data distribution, making the retrieved items more consistent with the retrieval text, and fully exploring the semantic association between the retrieval text and items during the adaptive modality selection process. This approach can further improve the accuracy of the retrieval matching of the trained dual-tower model under different retrieval text conditions, thereby further improving retrieval accuracy.

[0128] In some embodiments, the dual-tower model includes a first-tower model and a second-tower model, wherein the first-tower model includes a text feature extraction model and a dimensionality reduction module. In this case, the target text representation for the first training text used for retrieval can be generated in the following manner.

[0129] First, the text feature extraction model of the first tower model is used to process the first training text to obtain an intermediate text representation of the first training text. In some embodiments, taking the text feature extraction model of the first tower model as a BERT (Bidirectional Encoder Representations from Transformer) language model as an example, the intermediate text representation of the first training text incorporates the weight relationships between different sub-texts in the first training text. For example, the BERT language model of the first tower model has 4 layers.

[0130] For example, a subtext in the first training text may include at least one of a word, a phrase, and a short sentence in the first training text, and the weight relationship between different subtexts in the first training text may include at least one of the following: the weight relationship between different words, the weight relationship between different phrases, the weight relationship between different short sentences, the weight relationship between words and phrases, the weight relationship between words and short sentences, and the weight relationship between phrases and short sentences.

[0131] Then, using the dimensionality reduction module of the first tower model, the intermediate text features of the first training text are dimensionality reduced to obtain the target text representation of the first training text.

[0132] In some embodiments, the second tower model includes a text feature extraction model and a dimensionality reduction module. In this case, the target text representation of the second training text can be generated using the second tower model in the following manner.

[0133] First, based on the second training text, an intermediate text representation of the second training text is generated using the text feature extraction model of the second tower model. In some embodiments, taking the text feature extraction model of the second tower model as a BERT language model as an example, the intermediate text representation of the second training text incorporates the weight relationships between different sub-texts in the second training text. For example, the BERT language model of the second tower model has 4 layers.

[0134] Then, the dimensionality reduction module of the second tower model is used to reduce the dimensionality of the intermediate text representation of the second training text to obtain the target text representation of the second training text.

[0135] In some embodiments, the second tower model includes a text feature extraction model, a visual feature extraction model, a cross-attention module, and a dimensionality reduction module. In this case, the target multimodal representation of the training item can be generated in the following manner.

[0136] First, based on the second training text and the training visual information, the text feature extraction model and the visual feature extraction model of the second tower model are used to generate intermediate text representations of the second training text and intermediate visual representations of the training visual information, respectively.

[0137] Then, based on the intermediate text representation of the second training text and the intermediate visual representation of the training visual information, an intermediate multimodal representation of the training item is generated using a cross-attention module. The intermediate multimodal representation incorporates the weight relationships between different modalities.

[0138] Finally, the dimensionality reduction module of the second tower model is used to reduce the dimensionality of the intermediate multimodal representation to obtain the target multimodal representation of the training item.

[0139] For descriptions of relevant parts such as model structure in the model training method and retrieval training method, please refer to the various embodiments of the retrieval method, which will not be repeated here.

[0140] Figure 5 This is a block diagram illustrating a retrieval apparatus according to some embodiments of the present disclosure.

[0141] like Figure 5 As shown, the retrieval device 5 includes an acquisition module 51, a generation module 52, a judgment module 53, a first determination module 54, and a second determination module 55.

[0142] The acquisition module 51 is configured to acquire the current text for retrieval, for example, by performing an operation such as... Figure 1 The step S110 is shown.

[0143] In some embodiments, the acquisition module 51 is configured to acquire the current text for retrieval, including: receiving the retrieval text input by the user; in response to receiving the retrieval text input by the user, acquiring the user's user feature information; and determining the retrieval text input by the user and the user feature information as the current text.

[0144] The generation module 52 is configured to generate a target text representation of the current text using the first tower model of the dual-tower model, for example, by performing... Figure 1 The step S120 shown.

[0145] In some embodiments, the first tower model includes a text feature extraction model and a dimensionality reduction processing module. The generation module 52 is configured to use the text feature extraction model of the first tower model to process the current text and obtain an intermediate text representation of the current text; and to use the dimensionality reduction processing module of the first tower model to perform dimensionality reduction processing on the intermediate text features of the current text and obtain a target text representation of the current text.

[0146] The judgment module 53 is configured to determine whether the current text used for retrieval includes visual attribute information related to the item, such as performing an action like... Figure 1 The step S130 shown.

[0147] The first determining module 54 is configured to, when the current text includes visual attribute information, determine the target item from the candidate items based on the target multimodal representation of the candidate items and the target text representation of the current text, as the retrieval result. The target multimodal representation of the candidate item is generated using the second tower model of a dual-tower model based on candidate text and candidate visual information related to the candidate item. The target multimodal representation integrates feature information from both candidate text and candidate visual information, for example, by performing... Figure 1 Step S140 is shown.

[0148] In some embodiments, the first determining module 54 is configured to determine, when the current text includes visual attribute information, the similarity between the target multimodal representation of the candidate item and the target text representation of the current text as a first similarity; and from the candidate items, determine the candidate items whose first similarity is greater than a similarity threshold as the target items.

[0149] In some embodiments, the first determining module 54 is configured to concatenate the target multimodal representation of the candidate item and the target text representation of the candidate text as a first concatenation vector; based on the target text representation of the current text, concatenate a zero vector with the same dimension as the target text representation of the candidate text as a second concatenation vector, wherein the second concatenation vector has the same dimension as the first concatenation vector, and the dimension of the zero vector is the same as the dimension of the target text representation of the candidate text; and determine the similarity between the target multimodal representation of the candidate item and the target text representation of the current text based on the first concatenation vector and the second concatenation vector as a first similarity.

[0150] The second determining module 55 is configured to, when the current text does not include visual attribute information, determine the target item from the candidate items based on the target text representation of the candidate text and the target text representation of the current text, as the retrieval result. The target text representation of the candidate text is generated using a second tower model based on candidate text related to the candidate item, for example, by performing... Figure 1 The step S150 shown.

[0151] In some embodiments, the second determining module 55 is configured to determine the similarity between the target text representation of the candidate text and the target text representation of the current text, as a second similarity; and from the candidate items, determine the candidate items whose second similarity is greater than a similarity threshold, as the target items.

[0152] In some embodiments, the second determining module 55 is configured to concatenate the target multimodal representation of the candidate item and the target text representation of the candidate text as a first concatenation vector; based on the target text representation of the current text, concatenate a zero vector with the same dimension as the target multimodal representation of the candidate item as a third concatenation vector, wherein the third concatenation vector has the same dimension as the first concatenation vector, and the dimension of the zero vector is the same as the dimension of the target multimodal representation of the candidate item; and determine the similarity between the target text representation of the candidate text and the target text representation of the current text based on the first concatenation vector and the third concatenation vector as a second similarity.

[0153] In some embodiments, the second tower model includes a text feature extraction model and a dimensionality reduction processing module; the text feature extraction model of the second tower model is configured to generate an intermediate text representation of the candidate text based on the candidate text; the dimensionality reduction processing module of the second tower model is configured to perform dimensionality reduction processing on the intermediate text representation of the candidate text to obtain the target text representation of the candidate text.

[0154] In some embodiments, the second tower model includes a text feature extraction model, a visual feature extraction model, a cross-attention module, and a dimensionality reduction module. The text feature extraction model of the second tower model is configured to generate an intermediate text representation of the candidate text based on the candidate text. The visual feature extraction model of the second tower model is configured to generate an intermediate visual representation of the candidate visual information based on the candidate visual information. The cross-attention module is configured to generate an intermediate multimodal representation of the candidate item based on the intermediate text representation of the candidate text and the intermediate visual representation of the candidate visual information, wherein the intermediate multimodal representation incorporates the weight relationship between different modalities. The dimensionality reduction module of the second tower model is configured to perform dimensionality reduction processing on the intermediate multimodal representation to obtain the target multimodal representation of the candidate item.

[0155] Figure 6 This is a block diagram illustrating a model training apparatus according to some embodiments of the present disclosure.

[0156] like Figure 6 As shown, the model training device 6 includes a first generation module 61, a second generation module 62, a judgment module 63, and a training module 64.

[0157] The first generation module 61 is configured to utilize the first tower model of the dual-tower model to generate a target text representation for retrieving the first training text, for example, by performing actions such as... Figure 4 Step S410 is shown.

[0158] In some embodiments, the first tower model includes a text feature extraction model and a dimensionality reduction processing module. The first generation module 61 is configured to use the text feature extraction model of the first tower model to process the first training text to obtain an intermediate text representation of the first training text; and to use the dimensionality reduction processing module of the first tower model to perform dimensionality reduction processing on the intermediate text features of the first training text to obtain a target text representation of the first training text.

[0159] The second generation module 62 is configured to generate a target multimodal representation of the training item and a target text representation of the second training text based on the second training text and training visual information related to the training item, using the second tower model of the dual-tower model. The target multimodal representation integrates feature information from the second training text and training visual information. The training item and the first training text are associated, for example, by performing actions such as... Figure 4 Step S420 is shown.

[0160] In some embodiments, the second tower model includes a text feature extraction model and a dimensionality reduction processing module. The second generation module 62 is configured to generate an intermediate text representation of the second training text based on the text feature extraction model of the second tower model; and to perform dimensionality reduction processing on the intermediate text representation of the second training text using the dimensionality reduction processing module of the second tower model to obtain the target text representation of the second training text.

[0161] In some embodiments, the second tower model includes a text feature extraction model, a visual feature extraction model, a cross-attention module, and a dimensionality reduction module. The second generation module 62 is configured to generate intermediate text representations of the second training text and intermediate visual representations of the training visual information using the text feature extraction model and visual feature extraction model of the second tower model, respectively, based on the second training text and the training visual information; generate intermediate multimodal representations of the training item using the cross-attention module based on the intermediate text representations of the second training text and the intermediate visual representations of the training visual information, wherein the intermediate multimodal representations incorporate the weight relationships between different modalities; and perform dimensionality reduction processing on the intermediate multimodal representations using the dimensionality reduction module of the second tower model to obtain the target multimodal representation of the training item.

[0162] The judgment module 63 is configured to determine whether the first training text includes visual attribute information related to the item, for example, by performing the following: Figure 4 The step S430 shown.

[0163] Training module 64 is configured to train a dual-tower model based on the target text representation of the first training text and the target multimodal representation of the training item when the first training text includes visual attribute information, and to train a dual-tower model based on the target text representation of the first training text and the target text representation of the second training text when the first training text does not include visual attribute information, for example, by performing... Figure 4 Steps S440 to S450 are shown.

[0164] In some embodiments, there are multiple first training texts, and each first training text, as well as the second training text and training visual information related to the training items associated with each first training text, belong to the same training sample. Multiple training samples are used to train the dual-tower model.

[0165] In this case, training module 64 includes a first determining unit configured to, for each training sample, determine the similarity between the target text representation of the first training text and the target multimodal representation of the training item, as a third similarity, provided that the first training text includes visual attribute information. Training module 64 includes a second determining unit configured to, for each training sample, determine the similarity between the target text representation of the first training text and the target text representation of the second training text, as a fourth similarity, provided that the first training text does not include visual attribute information. Training module 64 includes a training unit configured to train the dual-tower model based on the third and fourth similarities determined from multiple training samples.

[0166] In some embodiments, the first determining unit is configured to concatenate the target multimodal representation of the training item with the target text representation of the second training text as a fourth concatenation vector; based on the target text representation of the first training text, concatenate a zero vector with the same dimension as the target text representation of the second training text as a fifth concatenation vector, wherein the fifth concatenation vector has the same dimension as the fourth concatenation vector, and the dimension of the zero vector is the same as the dimension of the target text representation of the second training text; and determine the similarity between the target text representation of the first training text and the target multimodal representation of the training item based on the fourth and fifth concatenation vectors as a third similarity.

[0167] In some embodiments, the second determining unit is configured to concatenate the target multimodal representation of the training item with the target text representation of the second training text as a fourth concatenation vector; based on the target text representation of the first training text, concatenate a zero vector with the same dimension as the target multimodal representation of the training item as a sixth concatenation vector, wherein the sixth concatenation vector has the same dimension as the fourth concatenation vector, and the dimension of the zero vector is the same as the dimension of the target multimodal representation of the training item; and determine the similarity between the target text representation of the first training text and the target text representation of the second training text based on the fourth concatenation vector and the sixth concatenation vector as a fourth similarity.

[0168] In some embodiments, the training data for training the dual-tower model includes a first training sample and a second training sample. The first training text in the first training sample is generated based on the descriptive information of the training items. The first training text in the second training sample includes text used in the actual retrieval process. The training items corresponding to the second training sample include items for which a specified operation is performed during the actual retrieval process. In this case, the model training device 6 further includes a third training module configured to perform a first training on the dual-tower model based on the target text representation of the first training text in the first training sample and the target multimodal representation of the training items corresponding to the first training sample.

[0169] The judgment module 63 is configured to, after performing the first training on the dual-tower model, determine whether the first training text in the second training sample includes visual attribute information related to the item. The training module 64 is configured to, after performing the first training on the dual-tower model, if the first training text in the second training sample includes visual attribute information, perform a second training on the dual-tower model based on the target text representation of the first training text in the second training sample and the target multimodal representation of the training item corresponding to the second training sample; if the first training text in the second training sample does not include visual attribute information, perform a second training on the dual-tower model based on the target text representation of the first training text in the second training sample and the target text representation of the second training text in the second training sample.

[0170] In some embodiments, in the first training sample, the second training text and training visual information related to the training item corresponding to the description information that generated the first training text are used as positive samples, and the second training text and training visual information related to other training items are used as negative samples; in the second training sample, the second training text and training visual information related to the training item to which the specified operation is performed in the actual retrieval process are used as positive samples, and the second training text and training visual information related to other training items in the actual retrieval process are used as negative samples.

[0171] In some embodiments, the text used in the actual retrieval process includes the retrieval text entered by the user, and the first training text in the second training sample also includes the user's user feature information.

[0172] In some embodiments, the trained first tower model is configured to generate a target text representation of the current text for retrieval; the trained second tower model is configured to generate a target multimodal representation of the candidate item and a target text representation of the candidate text based on candidate text and candidate visual information associated with the candidate item; wherein, when the current text includes visual attribute information associated with the item, the target text representation of the current text and the target multimodal representation of the candidate item are used to determine the target item from the candidate items as the retrieval result, and when the current text does not include visual attribute information associated with the item, the target text representation of the current text and the target text representation of the candidate text are used to determine the target item from the candidate items as the retrieval result.

[0173] Figure 7 This is a block diagram illustrating an electronic device according to some embodiments of the present disclosure.

[0174] like Figure 7As shown, the electronic device 7 includes a memory 71 and a processor 72 coupled to the memory 71. The memory 71 is used to store instructions for executing embodiments of model training methods or retrieval methods. The processor 72 is configured to execute model training methods or retrieval methods in any of the embodiments of this disclosure based on the instructions stored in the memory 71.

[0175] Figure 8 This is a block diagram illustrating a computer system for implementing some embodiments of the present disclosure.

[0176] like Figure 8 As shown, the computer system 80 can be represented in the form of a general computing device. The computer system 80 includes a memory 810, a processor 820, and a bus 800 connecting different system components.

[0177] The memory 810 may include, for example, system memory, non-volatile storage media, etc. The system memory may store, for example, an operating system, application programs, a boot loader, and other programs. The system memory may include volatile storage media, such as random access memory (RAM) and / or cache memory. The non-volatile storage media may store, for example, instructions for executing corresponding embodiments of model training or retrieval methods. Non-volatile storage media include, but are not limited to, disk storage, optical storage, flash memory, etc.

[0178] The processor 820 can be implemented using a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete hardware components such as discrete gates or transistors. Accordingly, each module, such as the decision module and the determination module, can be implemented by executing instructions in the central processing unit (CPU) memory to perform the corresponding steps, or by implementing dedicated circuitry to perform the corresponding steps.

[0179] Bus 800 can use any of the various bus architectures. For example, bus architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MCA) bus, and the Peripheral Component Interconnect (PCI) bus.

[0180] The computer system 80 may also include an input / output interface 830, a network interface 840, and a storage interface 850. These interfaces 830, 840, and 850, as well as the memory 810 and processor 820, can be connected via a bus 800. The input / output interface 830 provides a connection interface for input / output devices such as a monitor, mouse, and keyboard. The network interface 840 provides a connection interface for various networked devices. The storage interface 850 provides a connection interface for external storage devices such as floppy disks, USB flash drives, and SD cards.

[0181] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus, and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations thereof, can be implemented by computer-readable program instructions.

[0182] These computer-readable program instructions are provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable device to produce a machine, such that execution of the instructions by the processor produces means for implementing the functions specified in one or more boxes of the flowchart and / or block diagram.

[0183] These computer-readable program instructions may also be stored in a computer-readable storage medium. These instructions cause a computer to work in a particular manner to produce an article of manufacture, including instructions that implement the functions specified in one or more boxes in a flowchart and / or block diagram.

[0184] This disclosure may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects.

[0185] It should be noted that the collection, gathering, updating, analysis, processing, use, transmission, and storage of user personal information involved in this disclosed technical solution all comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken to prevent unauthorized access to user personal information data and to safeguard user personal information security, network security, and national security.

[0186] By using the retrieval method, model training method, and related devices, electronic devices, and computer storage media described in the above embodiments, the retrieval text can be matched more accurately, thereby improving the accuracy of the retrieval.

[0187] The model training method, retrieval method, and related apparatus, electronic devices, and computer-storable media according to this disclosure have been described in detail. To avoid obscuring the concept of this disclosure, some details known in the art have not been described. Those skilled in the art can fully understand how to implement the technical solutions disclosed herein based on the above description.

Claims

1. A retrieval method, comprising: Retrieve the current text used for retrieval; The target text representation of the current text is generated using the first tower model of the dual-tower model. Determine whether the current text includes visual attribute information related to the item; When the current text includes the visual attribute information, a target item is determined from the candidate items based on the target multimodal representation of the candidate items and the target text representation of the current text, as the retrieval result. The target multimodal representation of the candidate items is generated using the second tower model of the dual-tower model based on the candidate text and candidate visual information related to the candidate items. The target multimodal representation integrates the feature information of the candidate text and the candidate visual information. If the current text does not include the visual attribute information, a target item is determined from the candidate items based on the target text representation of the candidate text and the target text representation of the current text, and is used as the retrieval result. The target text representation of the candidate text is generated using the second tower model based on the candidate text related to the candidate item.

2. The retrieval method according to claim 1, wherein, Based on the target multimodal representation of the candidate items and the target text representation of the current text, the target items are determined from the candidate items, including: When the current text includes the visual attribute information, the similarity between the target multimodal representation of the candidate item and the target text representation of the current text is determined as the first similarity; From the candidate items, select the candidate items with a first similarity greater than a similarity threshold as the target items.

3. The retrieval method according to claim 2, wherein, Determining the similarity between the target multimodal representation of the candidate item and the target text representation of the current text, as a first similarity, includes: The target multimodal representation of the candidate item and the target text representation of the candidate text are concatenated to form a first concatenation vector; Based on the target text representation of the current text, a zero vector with the same concatenation dimension as the target text representation of the candidate text is used as the second concatenation vector, wherein the second concatenation vector has the same dimension as the first concatenation vector, and the dimension of the zero vector is the same as the dimension of the target text representation of the candidate text. Based on the first concatenation vector and the second concatenation vector, the similarity between the target multimodal representation of the candidate item and the target text representation of the current text is determined as the first similarity.

4. The retrieval method according to any one of claims 1-3, wherein, Based on the target text representation of the candidate text and the target text representation of the current text, the target items are determined from the candidate items, including: The similarity between the target text representation of the candidate text and the target text representation of the current text is determined as a second similarity. From the candidate items, select the candidate items with a second similarity greater than the similarity threshold as the target items.

5. The retrieval method according to claim 4, wherein, Determining the similarity between the target text representation of the candidate text and the target text representation of the current text, as a second similarity, includes: The target multimodal representation of the candidate item and the target text representation of the candidate text are concatenated to form a first concatenation vector; Based on the target text representation of the current text, a zero vector with the same splicing dimension as the target multimodal representation of the candidate item is used as the third splicing vector, wherein the third splicing vector has the same dimension as the first splicing vector, and the dimension of the zero vector is the same as the dimension of the target multimodal representation of the candidate item. Based on the first concatenation vector and the third concatenation vector, the similarity between the target text representation of the candidate text and the target text representation of the current text is determined as the second similarity.

6. The retrieval method according to any one of claims 1-3, wherein, The second tower model includes a text feature extraction model and a dimensionality reduction processing module; The text feature extraction model of the second tower model is configured to generate an intermediate text representation of the candidate text based on the candidate text. The dimensionality reduction module of the second tower model is configured to perform dimensionality reduction processing on the intermediate text representation of the candidate text to obtain the target text representation of the candidate text.

7. The retrieval method according to any one of claims 1-3, wherein, The second tower model includes a text feature extraction model, a visual feature extraction model, a cross-attention module, and a dimensionality reduction module; The text feature extraction model of the second tower model is configured to generate an intermediate text representation of the candidate text based on the candidate text. The visual feature extraction model of the second tower model is configured to generate an intermediate visual representation of the candidate visual information based on the candidate visual information. The cross-attention module is configured to generate an intermediate multimodal representation of the candidate item based on the intermediate text representation of the candidate text and the intermediate visual representation of the candidate visual information, wherein the intermediate multimodal representation incorporates the weight relationship between different modalities. The dimensionality reduction module of the second tower model is configured to perform dimensionality reduction on the intermediate multimodal representation to obtain the target multimodal representation of the candidate item.

8. The retrieval method according to any one of claims 1-3, wherein, The first tower model includes a text feature extraction model and a dimensionality reduction processing module, and the target text representation of the current text is generated as follows: The text feature extraction model of the first tower model is used to process the current text to obtain the intermediate text representation of the current text. The dimensionality reduction processing module of the first tower model is used to perform dimensionality reduction processing on the intermediate text features of the current text to obtain the target text representation of the current text.

9. The retrieval method according to any one of claims 1-3, wherein, Retrieving the current text used for retrieval includes: Receive the search text input by the user; In response to receiving the search text input by the user, the user's user characteristic information is obtained; The search text input by the user and the user feature information are determined as the current text.

10. A model training method, comprising: Using the first tower model of the dual-tower model, a target text representation for the first training text used for retrieval is generated; Based on the second training text and training visual information related to the training item, the second tower model of the dual-tower model is used to generate the target multimodal representation of the training item and the target text representation of the second training text. The target multimodal representation integrates the feature information of the second training text and the training visual information. The training item is associated with the first training text. Determine whether the first training text includes visual attribute information related to the item; When the first training text includes the visual attribute information, the dual-tower model is trained based on the target text representation of the first training text and the target multimodal representation of the training item. If the first training text does not include the visual attribute information, the dual-tower model is trained based on the target text representation of the first training text and the target text representation of the second training text.

11. The model training method according to claim 10, wherein, There are multiple first training texts. Each first training text, along with second training texts and training visual information related to training items associated with each first training text, belongs to the same training sample. Multiple training samples are used to train the dual-tower model. For each training sample, if the first training text includes the visual attribute information, the similarity between the target text representation of the first training text and the target multimodal representation of the training item is determined as a third similarity. For each training sample, if the first training text does not include the visual attribute information, the similarity between the target text representation of the first training text and the target text representation of the second training text is determined as a fourth similarity. The dual-tower model is trained based on the third and fourth similarities determined from the multiple training samples.

12. The model training method according to claim 11, wherein, Determining the similarity between the target text representation of the first training text and the target multimodal representation of the training visual information, as a third similarity, includes: The target multimodal representation of the training item is concatenated with the target text representation of the second training text to form a fourth concatenation vector; Based on the target text representation of the first training text, a zero vector with the same concatenation dimension as the target text representation of the second training text is used as the fifth concatenation vector, wherein the fifth concatenation vector has the same dimension as the fourth concatenation vector, and the dimension of the zero vector is the same as the dimension of the target text representation of the second training text. Based on the fourth and fifth splicing vectors, the similarity between the target text representation of the first training text and the target multimodal representation of the training item is determined as the third similarity.

13. The model training method according to claim 11, wherein, Determining the similarity between the target text representation of the first training text and the target text representation of the second training text, as a fourth similarity, includes: The target multimodal representation of the training item is concatenated with the target text representation of the second training text to form a fourth concatenation vector; Based on the target text representation of the first training text, a zero vector with the same splicing dimension as the target multimodal representation of the training item is used as the sixth splicing vector, wherein the sixth splicing vector has the same dimension as the fourth splicing vector, and the dimension of the zero vector is the same as the dimension of the target multimodal representation of the training item. Based on the fourth concatenation vector and the sixth concatenation vector, the similarity between the target text representation of the first training text and the target text representation of the second training text is determined as the fourth similarity.

14. The model training method according to any one of claims 10-13, wherein, The training data for training the dual-tower model includes a first training sample and a second training sample. The first training text in the first training sample is generated based on the description information of the training items. The first training text in the second training sample includes text used in the actual retrieval process. The training items corresponding to the second training sample include items for which a specified operation is performed in the actual retrieval process. The model training method further includes: The dual-tower model is first trained based on the target text representation of the first training text in the first training sample and the target multimodal representation of the training item corresponding to the first training sample. Specifically, after performing the first training on the dual-tower model, it is determined whether the first training text in the second training sample includes visual attribute information related to the item. If the first training text in the second training sample includes the visual attribute information, the dual-tower model is trained a second time based on the target text representation of the first training text in the second training sample and the target multimodal representation of the training item corresponding to the second training sample. If the first training text in the second training sample does not include the visual attribute information, the dual-tower model is trained a second time based on the target text representation of the first training text in the second training sample and the target text representation of the second training text in the second training sample.

15. The model training method according to claim 14, wherein, In the first training sample, the second training text and training visual information related to the training item corresponding to the description information that generates the first training text are positive samples, and the second training text and training visual information related to other training items are negative samples. In the second training sample, the second training text and training visual information related to the training item in which the specified operation is performed during the actual retrieval process are used as positive samples, and the second training text and training visual information related to other training items in the actual retrieval process are used as negative samples.

16. The model training method according to claim 14, wherein, The text used in the actual retrieval process includes the retrieval text entered by the user, and the first training text in the second training sample also includes the user's user feature information.

17. The model training method according to any one of claims 10-13, wherein, The first tower model after training is configured to generate a target text representation of the current text for retrieval; The trained second tower model is configured to generate a target multimodal representation of the candidate item and a target text representation of the candidate text based on candidate text and candidate visual information associated with the candidate item. Wherein, when the current text includes visual attribute information related to the item, the target text representation of the current text and the target multimodal representation of the candidate item are used to determine the target item from the candidate items as the retrieval result; when the current text does not include visual attribute information related to the item, the target text representation of the current text and the target text representation of the candidate text are used to determine the target item from the candidate items as the retrieval result.

18. The model training method according to any one of claims 10-13, wherein, The second tower model includes a text feature extraction model and a dimensionality reduction module. Based on the second training text, the intermediate text representation of the second training text is generated using the text feature extraction model of the second tower model. The intermediate text representation of the second training text is reduced in dimensionality using the dimensionality reduction module of the second tower model to obtain the target text representation of the second training text.

19. The model training method according to any one of claims 10-13, wherein, The second tower model includes a text feature extraction model, a visual feature extraction model, a cross-attention module, and a dimensionality reduction module. Based on the second training text and the training visual information, the text feature extraction model and visual feature extraction model of the second tower model are used to generate intermediate text representations of the second training text and intermediate visual representations of the training visual information, respectively. Based on the intermediate text representation of the second training text and the intermediate visual representation of the training visual information, the intermediate multimodal representation of the training item is generated using the cross-attention module, wherein the intermediate multimodal representation incorporates the weight relationship between different modalities. The intermediate multimodal representation is reduced in dimensionality using the dimensionality reduction module of the second tower model to obtain the target multimodal representation of the training item.

20. The model training method according to any one of claims 10-13, wherein, The first tower model includes a text feature extraction model and a dimensionality reduction module, which generate a target text representation for the first training text used for retrieval, including: Using the text feature extraction model of the first tower model, the first training text is processed to obtain the intermediate text representation of the first training text. The dimensionality reduction processing module of the first tower model is used to perform dimensionality reduction processing on the intermediate text features of the first training text to obtain the target text representation of the first training text.

21. A retrieval device, comprising: The retrieval module is configured to retrieve the current text used for retrieval; The generation module is configured to generate a target text representation of the current text using the first tower model of the dual-tower model. The judgment module is configured to determine whether the current text used for retrieval includes visual attribute information related to the item; The first determining module is configured to, when the current text includes the visual attribute information, determine a target item from the candidate items based on the target multimodal representation of the candidate items and the target text representation of the current text, as the retrieval result, wherein the target multimodal representation of the candidate item is generated using the second tower model of the dual-tower model based on the candidate text and candidate visual information related to the candidate item, and the target multimodal representation integrates the feature information of the candidate text and the candidate visual information; The second determining module is configured to, when the current text does not include the visual attribute information, determine a target item from the candidate items based on the target text representation of the candidate text and the target text representation of the current text, as the result of the retrieval, wherein the target text representation of the candidate text is generated using the second tower model based on the candidate text related to the candidate item.

22. A model training device, comprising: The first generation module is configured to use the first tower model of the dual-tower model to generate a target text representation for the first training text to be retrieved. The second generation module is configured to generate a target multimodal representation of the training item and a target text representation of the second training text based on the second training text and training visual information related to the training item, using the second tower model of the dual-tower model. The target multimodal representation integrates feature information from the second training text and the training visual information, and the training item is associated with the first training text. The judgment module is configured to determine whether the first training text includes visual attribute information related to the item; The training module is configured to train the dual-tower model based on the target text representation of the first training text and the target multimodal representation of the training item when the first training text includes the visual attribute information, and to train the dual-tower model based on the target text representation of the first training text and the target text representation of the second training text when the first training text does not include the visual attribute information.

23. An electronic device, comprising: Memory; as well as A processor coupled to the memory, the processor being configured to execute the retrieval method as described in any one of claims 1 to 9 or the model training method as described in any one of claims 10 to 20, based on instructions stored in the memory.

24. A computer-storeable medium having stored thereon computer program instructions that, when executed by a processor, implement the retrieval method as described in any one of claims 1 to 9 or the model training method as described in any one of claims 10 to 20.

Images