Characterization method, search method, model training method, device and storage medium

Abstract

Provided in the embodiments of the present description are a characterization method, a search method, a model training method, a device and a storage medium. The characterization method comprises: acquiring a text description of a search object, and an image of the search object; dividing the image of the search object into a plurality of image blocks and, by means of an image feature extraction model, performing feature extraction on the plurality of image blocks to obtain a feature sequence of the plurality of image blocks; and extracting pixel-level features from the image of the search object by means of a text feature extraction model, and determining a characterization vector of the search object on the basis of the feature sequence, the pixel-level features and the image of the search object. An extracted feature sequence and extracted pixel-level features represent high-level and low-level features of an image, respectively; therefore, finally-determined features of an image of a search object are more comprehensive and accurate, and then a constructed characterization vector of the search object is also more accurate.

Description

Characterization method, search method, model training method, device and storage medium

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on September 27, 2023, with application number 202311268796.4 and application name “Characterization method, search method, model training method, device and storage medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The embodiments of this specification relate to the field of artificial intelligence technology, and in particular to a characterization method, a search method, a model training method, a device, and a storage medium. Background Art

[0003] In various search scenarios, after obtaining the search terms input by the user, the search system can obtain candidate objects that are highly correlated with the search terms from a large number of search objects stored in the database, and then display the relevant information of these candidate objects to the user. Currently, in the recall stage of search scenarios, most of the search objects that are highly correlated with the search terms are determined by judging the similarity between the search terms and the text descriptions of the search objects (i.e., by matching between the texts). Since the text descriptions of the search objects and the search terms are constructed by different users, there are large differences in structure, sentence length, description habits, etc. If only the matching of search terms and text descriptions is emphasized, there may be scenarios where the image of the search object that is finally recalled does not match the search terms, which seriously affects the user experience.

[0004] Summary of the Invention

[0005] To overcome the problems existing in the related art, the embodiments of this specification provide a characterization method, a search method, a model training method, a device and a storage medium.

[0006] According to a first aspect of an embodiment of this specification, a method for representing a search object is provided, the method comprising:

[0007] Obtaining a text description of the search object and an image of the search object;

[0008] Dividing the image of the search object into a plurality of image blocks, and performing feature extraction on each of the plurality of image blocks using an image feature extraction model to obtain feature sequences of the plurality of image blocks; wherein the image feature extraction model is pre-trained using a general image dataset;

[0009] Pixel-level features are extracted from the image of the search object through a text feature extraction model, and a representation vector of the search object is determined based on the feature sequence, the pixel-level features, and the image of the search object through the text feature extraction model; wherein the text feature extraction model is trained based on historical search data.

[0010] According to a second aspect of the embodiments of this specification, a search method is provided, the method comprising:

[0011] Obtaining a search term input by a user and representation vectors of multiple search objects, wherein the representation vectors of the search objects are obtained using the representation method mentioned in the first aspect above;

[0012] Determining a representation vector of the search term using a text extraction feature model;

[0013] Based on the similarity between the representation vector of the search term and the representation vectors of the multiple search objects, a candidate object is determined from the multiple search objects, so as to present the attribute data of the candidate object to the user.

[0014] According to a third aspect of the embodiments of this specification, a model training method is provided, the method comprising:

[0015] Obtain an image of a sample search object;

[0016] Dividing the image of the sample search object into a plurality of image blocks, and performing feature extraction on each of the plurality of image blocks using an image feature extraction model to obtain sample feature sequences of the plurality of image blocks, wherein the image feature extraction model is pre-trained using a general image dataset;

[0017] Extracting pixel-level features from the image of the sample search object through an encoder of a preset initial model;

[0018] Reconstructing the image of the sample search object based on the sample feature sequence and the pixel-level features through the decoder of the initial model to obtain a reconstructed image;

[0019] adjusting model parameters of the initial model based on a difference between the reconstructed image and the image of the sample search object to pre-train the initial model;

[0020] The pre-trained initial model is fine-tuned based on historical search data to obtain a text feature extraction model.

[0021] According to a fourth aspect of the embodiments of this specification, a model training method is provided, the method comprising:

[0022] Obtaining the hot search terms corresponding to the sample search object, the text description of the sample search object, and the image of the sample search object;

[0023] splicing the hot search terms of the sample search object and the text description of the sample search object to obtain a spliced ​​text;

[0024] Dividing the image of the sample search object into a plurality of image blocks, and performing feature extraction on each of the plurality of image blocks using an image feature extraction model to obtain sample feature sequences of the plurality of image blocks, wherein the image feature extraction model is pre-trained using a general image dataset;

[0025] Inputting the sample feature sequence, the image of the sample search object, and the spliced ​​text into a preset initial model, and outputting a prediction result through the initial model, the prediction result being used to indicate whether the spliced ​​text and the image of the sample search object match;

[0026] Adjusting the model parameters of the initial model based on the deviation between the output prediction result and the actual result to pre-train the initial model;

[0027] The pre-trained initial model is fine-tuned based on historical search data to obtain a text feature extraction model.

[0028] According to a fifth aspect of the embodiments of this specification, a model training method is provided, the method comprising:

[0029] Pre-train the preset initial model based on historical search data;

[0030] Acquire sample data, wherein the sample data includes a sample search term, an image of a sample search object, and a text description of the sample search object;

[0031] Dividing the image of the sample search object into a plurality of image blocks, and performing feature extraction on each of the plurality of image blocks using the image feature extraction model to obtain sample feature sequences of the plurality of image blocks, wherein the image feature extraction model is pre-trained using a general image dataset;

[0032] Inputting the sample search term, the image of the sample search object, the text description of the sample search object, and the sample feature sequence into the pre-trained initial model, and outputting a representation vector of the sample search term and a representation vector of the sample search object through the pre-trained initial model;

[0033] Inputting the representation vector of the sample search term and the representation vector of the sample search object into a preset discriminator respectively, and using the discriminator to determine the type of the input representation vector, the type including search term or search object;

[0034] Determine the target loss based on the difference between the discriminator's judgment result and the true result;

[0035] The model parameters of the pre-trained initial model are adjusted according to the target loss to train the text feature extraction model.

[0036] According to the sixth aspect of the embodiments of this specification, an electronic device is provided, which includes a processor, a memory, and a computer program stored in the memory and executable by the processor, and when the computer program is executed, the method mentioned in the first aspect above is implemented.

[0037] According to the seventh aspect of the embodiments of this specification, a computer storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method mentioned in the first, second, third, fourth and / or fifth aspects above is implemented.

[0038] The beneficial effects of the embodiments of this specification: The present disclosure is a solution of an embodiment. When determining the representation vector of the search object, the image of the search object can be first divided into blocks to obtain multiple image blocks of the search object, and then a pre-trained image feature extraction model is used to extract features from the multiple image blocks to obtain a feature sequence of the image blocks. By dividing an entire image into multiple image blocks, it is easier for the image feature extraction model to understand the image and extract more fine-grained features. Since the image feature extraction model is a model specifically used for image feature extraction, it can extract some implicit, high-order image block features from the image blocks. At the same time, considering that the image feature extraction model is pre-trained using a general image dataset, there is a certain deviation between these image datasets and the image of the search object, resulting in the extracted image block feature sequence may not be comprehensive and accurate. Therefore, a text feature extraction model can be further used to extract features from the original image of the search object to extract some low-order pixel-level features. Then, the text feature extraction model can combine the pixel-level features, the feature sequence of the image blocks, and the text features extracted from the text description of the search object to obtain the representation vector of the search object. Since the feature sequence extracted by the image feature extraction model and the pixel-level features extracted by the text feature extraction model represent the high-level and low-level features of the image respectively, the features of the image of the search object finally determined are more comprehensive and accurate, and the representation vector of the search object constructed based on the image features and the text features of the text description is also more accurate.

[0039] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory and are not restrictive of the embodiments of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings herein are incorporated in and constitute a part of the embodiments of this specification, illustrate embodiments consistent with the embodiments of this specification, and together with the description, serve to explain the principles of the embodiments of this specification.

[0041] FIG1 is a schematic diagram of recalling candidate objects based only on text matching in the related art;

[0042] FIG2 is a schematic diagram showing a mismatch between an image of a candidate object and a search term in a text matching recall according to an exemplary embodiment of this specification;

[0043] FIG3 is a schematic diagram of determining a representation vector of a search object according to an exemplary embodiment of this specification;

[0044] FIG4 is a schematic diagram of determining a representation vector of a search object according to an exemplary embodiment of this specification;

[0045] FIG5 is a schematic diagram of determining a representation vector of a search object according to an exemplary embodiment of this specification;

[0046] FIG6 is a schematic diagram of a search scenario shown in an exemplary embodiment of this specification;

[0047] FIG7 is a flowchart of a method for representing a search object according to an exemplary embodiment of this specification;

[0048] FIG8 is a schematic diagram illustrating a method of extracting a foreground object from an image of a search object and dividing the image into blocks according to an exemplary embodiment of this specification;

[0049] FIG9 is a schematic diagram of pre-training an initial model according to an exemplary embodiment of this specification;

[0050] FIG10 is a schematic diagram of pre-training an initial model according to an exemplary embodiment of this specification;

[0051] FIG11 is a schematic diagram of pre-training an initial model according to an exemplary embodiment of this specification;

[0052] FIG12 is a schematic diagram of a training process of a generative adversarial model according to an exemplary embodiment of this specification;

[0053] FIG13 is a schematic diagram of fine-tuning a pre-trained initial model according to an exemplary embodiment of this specification;

[0054] FIG14 is a schematic diagram of fine-tuning a pre-trained initial model according to an exemplary embodiment of this specification;

[0055] FIG15 is a logic block diagram of an electronic device according to an exemplary embodiment of this specification. DETAILED DESCRIPTION

[0056] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all possible implementations consistent with the embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with certain aspects of the embodiments of this specification, as detailed in the appended claims.

[0057] The terms used in the embodiments of this specification are for the purpose of describing specific embodiments only and are not intended to limit the embodiments of this specification. The singular forms "a," "the," and "the" used in the embodiments of this specification and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0058] It should be understood that although the terms first, second, third, etc. may be used to describe various information in the embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the embodiments of this specification, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein can be interpreted as "when..." or "when..." or "in response to determining."

[0059] In various search scenarios, after obtaining the search term input by the user, the search system can obtain candidate objects that are highly correlated with the search term from a large number of search objects stored in the database, and then display the relevant information of these candidate objects to the user. Generally, each search object includes attribute data, which can be used to describe the characteristics of the search object. When the search system recalls candidate objects that are highly correlated with the search term, it can determine the degree of correlation between the search object and the search term based on the attribute data of the search object. Generally, the attribute data used to describe the characteristics of the search object includes data of multiple modalities. For example, the attribute data can be a text description of the search object, an image of the search object, or other types of data.

[0060] At present, in the recall stage of the search scenario, the search objects with a greater correlation with the search terms are mostly determined by determining the similarity between the search terms and the text descriptions of the search objects (i.e., by matching the texts). For example, as shown in FIG1 , the search system generally includes a text feature extraction model 11, which can be any type of model suitable for processing natural language, such as a BERT model. The text extraction feature model can determine the representation vectors of each search object based on the text descriptions of each search object, as well as the representation vectors of the search terms input by the user, that is, map the two to the same vector space, and then determine the search objects with a greater correlation with the search terms based on the similarity between the two, and present them to the user. Since the text descriptions of the search objects and the search terms are constructed by different users, there are large differences in structure, sentence length, description habits, etc. If only the matching of the search terms and text descriptions is emphasized, there may be a scenario where the image of the search object finally recalled does not match the search terms, which seriously affects the user experience.

[0061] For example, as shown in Figure 2, in an e-commerce product recall scenario, if a user enters the search term "red dress," and recalling related products solely relies on a match between the search term and the title (i.e., a text match), a product title containing the words "red dress" might appear to the user, while the actual product image is a red high heel. For example, a product titled "Fashionable red high heels, perfect with a red dress!" would also be recalled and displayed to the user based solely on text matching. Clearly, the recalled product does not match the user's search term, severely impacting the user experience.

[0062] In order to avoid the above problems, one way that can be easily thought of is to match the text description of the search object and the image of the search object as a whole with the search term. For example, the text description of the search object and the image of the search object can be combined to determine the representation vector corresponding to the search object, and then matched with the representation vector of the search term to determine the search object with a higher similarity. However, since the representation models used in current search scenarios are mostly text feature extraction models 11, this type of model is good at feature extraction of text and can accurately obtain the representation vector corresponding to the text, but is not good at feature extraction of images. As shown in Figure 3, if the text description of the search object and the image of the search object are input into the text feature extraction model 11 to determine the representation vector of the search object, since the extracted image features are inaccurate, the representation vector of the search object finally determined is also inaccurate.

[0063] In order to accurately extract features from images without the need for additional training of a model specifically used for feature extraction of images of search objects. As shown in FIG4 , another approach that is easily conceivable is to use some open source pre-trained image feature extraction models 12 (e.g., the Resent model) to extract features from images. The text feature extraction model 11 can use the image feature extraction model 12 to extract image features from the image of the search object and the text description of the search object to determine the representation vector of the search object, and then match it with the representation vector of the search term. However, the applicant found that the accuracy of the search objects recalled using the above method was not high enough, and the matching degree with the search term was not good enough.

[0064] According to the applicant's analysis, the main reasons for the above problems are as follows: Usually, the pre-trained image feature extraction model 12 is trained using a general image data set, and the images in the search scenario have their own particularities. For example, taking the product images in the e-commerce scenario as an example, this type of image is usually the main picture of the product, so the number of objects in the image is often relatively single, for example, only including products, and the images in the general image data set used for pre-training the image feature extraction model 12 cover various types of images, which does not have the above characteristics, that is, there is a certain difference between the data set used in the pre-training stage of the image feature extraction model 12 and the data set used in the application stage, which may cause the performance of the pre-trained image feature extraction model 12 to decline, and the extracted features are not comprehensive and accurate enough.

[0065] In order to obtain a more accurate representation vector of the search object, the embodiment of the present disclosure considers combining the above two methods, as shown in FIG5 , that is, when determining the representation vector of the search object, the image of the search object can be first divided into blocks to obtain multiple image blocks of the search object, and then the pre-trained image feature extraction model 12 is used to extract features from the multiple image blocks to obtain a feature sequence of the image blocks. By dividing an entire image into multiple image blocks, it is easier for the image feature extraction model 12 to understand the image and extract more fine-grained features. Moreover, since the image feature extraction model 12 is a model specifically used for image feature extraction, it can extract some implicit, high-order image block features from the image blocks. At the same time, considering that the image feature extraction model 12 is pre-trained using a general image dataset, there are certain deviations between these image datasets and the image of the search object, resulting in the feature sequence of the extracted image block being not comprehensive and accurate enough. Therefore, the text feature extraction model 11 can be further used to extract features from the original image of the search object, extracting some low-order original pixel-level features therefrom, and then the text feature extraction model 11 can combine the pixel-level features, the feature sequence of the image block, and the text features extracted from the text description of the search object to obtain a representation vector of the search object. Since the feature sequence extracted by the image feature extraction model 12 and the pixel-level features extracted by the text feature extraction model 11 represent high-level and low-level features of the image respectively, the features of the image of the search object that are finally determined are more comprehensive and accurate, and thus the representation vector of the search object constructed based on the features of the image and the text features of the text description is also more accurate.

[0066] As shown in Figure 6, it is a schematic diagram of a search application scenario. The search system may include a client 21 and a server 22. The client 21 can be set on a user's mobile phone, computer or other terminal device, and the server 22 can be set in a server or server cluster. For certain specific search scenarios (the figure takes the product search in the e-commerce scenario as an example), a large amount of attribute data of the product will be pre-stored in the server 22, such as the text description of the product and the image of the product. In order to improve the efficiency of responding to the user's search request, usually, for each product, the server 22 will pre-construct and store the representation vector of the product. When the user enters the search term, the representation vector of the search term can be determined, and then the representation vector of the search term is matched with the stored representation of the product to determine the product with a greater correlation with the search term and return it to the user.

[0067] The representation method of the embodiment of the present disclosure can be executed by the server 22, that is, the representation vector of each search object is constructed by the server 22. Specifically, as shown in FIG7 , the representation method of the search object may include the following steps:

[0068] S702, obtaining a text description of the search object and an image of the search object;

[0069] In step S702, for each search object to be characterized, a text description and an image of the search object may be obtained. The text description may include a title of the search object, an introduction to the search object, or other textual descriptions of the search object's characteristics. The image of the search object may be one or more frames containing the search object, for example, a main image of the search object, which typically only includes the search object.

[0070] In S704, the image of the search object is divided into a plurality of image blocks, and features are extracted from each of the plurality of image blocks using an image feature extraction model 12 to obtain feature sequences of the plurality of image blocks; wherein the image feature extraction model 12 is pre-trained using a general image dataset;

[0071] Similar to text processing, considering that directly inputting a long text into the model may be difficult for the model to understand, segmenting the text into multiple words can facilitate model understanding and extract more useful information. A similar method can also be used for image processing, that is, the image can be divided into multiple image blocks, and then the image blocks are input into the model respectively, and the model performs feature extraction on the image blocks to extract more fine-grained useful information. Therefore, in step S704, the image of the search object can be first segmented to obtain multiple image blocks. In order to obtain some implicit and high-order features of the image, the pre-trained image feature extraction model 12 can be used to extract features from the multiple image blocks respectively to obtain feature sequences of multiple image blocks. Among them, the features of each image block can be represented by a vector or a vector group, and the feature sequence can be a sequence composed of multiple vectors or vector groups.

[0072] The image feature extraction model 12 can be obtained by pre-training using a general image dataset. The image feature extraction model 12 can be some open source models, for example, a ResNet model.

[0073] S706. Extract pixel-level features from the image of the search object through the text feature extraction model 11, and determine the representation vector of the search object based on the feature sequence, the pixel-level features, and the image of the search object through the text feature extraction model 11; wherein, the text feature extraction model 11 is trained based on historical search data.

[0074] In step S706, in the search scenario, in order to determine the representation vector of the search object and the representation vector of the search term so as to match the two, a text feature extraction model 11 is usually pre-trained. The text feature extraction model 11 can be trained using historical search data, and can map the search object and the search term to the same vector space to compare whether the two are close. The historical search data can include search terms that the user has entered in the past, search objects that the user has clicked or browsed based on the search terms, and text descriptions, image data, etc. of the search objects.

[0075] Considering that the image feature extraction model 12 is trained using a general image dataset, and the image of the search object has its own particularity, there is a certain deviation between it and the general image dataset. Therefore, the feature sequence extracted by the image feature extraction model 12 may not be accurate and comprehensive. To alleviate the above problem, when using the text feature extraction model 11 to determine the representation vector of the search object, the image of the search object can also be input into the text feature extraction model 11. Since the image of the search object retains the original features of the image, the text feature extraction model 11 can extract some low-order, pixel-level features from it, thereby compensating for the defects of the extracted high-order image block feature sequence. Then, the text feature extraction model 11 can combine the feature sequence and the pixel-level features to obtain more accurate and comprehensive features of the image of the search object. At the same time, the text feature extraction model 11 can extract features from the text description of the search object, map the features of the image of the search object and the features of the text of the search object into the same vector space, and then combine the two to determine the representation vector used to describe the characteristics of the search object from multiple dimensions.

[0076] Considering that in addition to the search object, the image may also include some meaningless backgrounds, when the image feature extraction model 12 extracts features from the search object, these meaningless backgrounds may cause certain interference to the image feature extraction model 12, affecting both the processing efficiency of the image feature extraction model 12 and the accuracy of the extracted features.

[0077] Therefore, in some embodiments, as shown in FIG8 , before the image blocks of the search object are divided into blocks, the foreground object can be first extracted from the image of the search object. Since the main body of the image is usually the search object, the extracted foreground object is likely to be the search object. The image area corresponding to the foreground object can then be divided into multiple image blocks. The size of the image blocks can be set based on actual conditions. The multiple image blocks are then input into the image feature extraction model 12 , which performs feature extraction on each of the multiple image blocks to determine the features of each of the multiple image blocks and obtain a feature sequence for the multiple image blocks. For each image block, its features include both the contour, color, and other features of the image block extracted by the image feature model, as well as features used to represent the position of the image block in the image.

[0078] In some embodiments, considering that the text description of the search object is often a long text, in order to allow the text feature extraction model 11 to better understand the text semantics, the text description of the search object can be first segmented, and then the text feature extraction model 11 can be used to extract features. For example, the text description of the search object can be segmented to obtain multiple words, and then the representation vectors of the multiple words can be determined. For each word, its representation vector can be obtained based on the word vector of the word itself and the position vector representing the position of the word in the text description. The word vector of the word can be determined with the help of a pre-constructed word list, and its function is to convert the word into information that the model can understand. After determining the representation vectors of the multiple words, the representation vectors of the multiple words, the feature sequence obtained above, and the image of the search object can be input into the text feature extraction model 11, and the text feature extraction model 11 extracts pixel-level features from the image of the search object, and then determines the representation vector of the search object based on the representation vectors of the multiple words, the feature sequence obtained above, and the pixel-level features.

[0079] In the related art, when training a text feature extraction model 11, it usually includes two stages, one is a pre-training stage and the other is a fine-tuning stage. In the pre-training stage, a large amount of corpus data can be used to train the initial model 13 (for example, the Bert model). This process is usually unsupervised training, so that the pre-trained initial model 14 has a better understanding of the text. The fine-tuning stage is mainly to let the pre-trained model learn a specific task on a specific data set, so that the model has a better performance on the specific task. The fine-tuning stage is usually supervised training, and it is necessary to pre-construct labeled sample data. For example, the specific data set can be sample data constructed based on historical search data. The sample data may include attribute data of the search object, as well as search terms that match or do not match the search object, etc. The specific task may be to let the model predict whether the search object and the search term match. Since the pre-training stage usually does not require labeled samples, through this pre-training + fine-tuning method, even when the number of labeled samples is small, a model with better performance can be trained.

[0080] Similarly, the embodiment of the present disclosure may also include two stages when training the text feature extraction model 11, namely a pre-training stage and a fine-tuning stage. The pre-training stage may pre-train the preset initial model 13, and the fine-tuning stage may further train the pre-trained initial model 14 using a sample data set constructed from historical search data to obtain the text feature extraction model 11.

[0081] In the embodiment of the present disclosure, it is considered that the text feature extraction model 11 not only needs to process the text and extract the features of the text, but also needs to process the image of the search object to obtain comprehensive and accurate image features. Therefore, in some embodiments, during the pre-training stage of the text feature extraction model 11, a large number of images of search objects (hereinafter referred to as sample search objects) can be obtained from the database of the search system. The images of the large number of sample search objects are used to perform unsupervised training on the preset initial model 13 to improve the ability of the pre-trained initial model 14 to extract features from the image. For example, as shown in Figure 9, the initial model 13 may include an encoder 131 and a decoder 132. When pre-training the initial model 13, a large number of sample search object images can be obtained from the search system. For each frame of the sample search object image, it can be divided into multiple image blocks. The image feature extraction model 12 is used to extract features from the multiple image blocks to obtain feature sequences of the multiple image blocks (hereinafter referred to as sample feature sequences of the image blocks extracted during the training stage). The image of the sample search object can then be input into the encoder 131, and the encoder 131 extracts pixel-level features from the image of the sample search object. The decoder 132 can reconstruct the image of the sample search object based on the sample feature sequence and the pixel-level features to obtain a reconstructed image. The model parameters of the initial model 13 are then adjusted based on the difference between the reconstructed image and the image of the sample search object to pre-train the initial model 13. Pre-training the initial model 13 in the above manner can ensure that the pre-trained initial model 14 has better image feature extraction performance, allowing the pre-trained initial model 14 to obtain more accurate image features.

[0082] In some embodiments, it is considered that the task of the text feature extraction model 11 is either to extract features from text or to extract features from images. The aforementioned pre-training process can improve the performance of the model in extracting features from images. In order to further improve the performance of the model in extracting features from text, a large number of text descriptions of sample search objects can be obtained from the database of the search system, and then these text descriptions can be used as corpus data to perform unsupervised training on the preset initial model 13. For example, in the process of pre-training the initial model 13, for a text, a certain word in the text can be masked, and then the initial model 13 can be used to predict what the masked word is based on the context of the text, and the model parameters of the initial model 13 are adjusted based on the difference between the prediction result of the initial model 13 and the actual result to pre-train the initial model. By using a large number of text descriptions of sample search objects as corpus data to train the initial model 13, the ability of the initial model 13 to understand the text can be improved.

[0083] In search scenarios, the text description of the search object and the search terms often differ significantly in terms of structure and expression. For example, the text description of the search object is often long text with specific sentence structure and expression, while the search terms are often short and concise. Taking product search in e-commerce scenarios as an example, the product title is the seller's language, while the search terms are the user's language. Clearly, there are often significant semantic differences between the search terms and the product title. Related techniques have directly used the same model to map the search terms and the text description of the search object into the same vector space, ignoring the semantic differences between the two, which may result in inaccurate representation vectors. To mitigate the semantic differences between the text description of the search object and the search terms, in some embodiments, for each sample search object, the most popular search terms for that sample search object can be obtained. These popular search terms are then combined with the sample search object's text description to pre-train the initial model 13, allowing the initial model 13 to better learn the differences between the search terms and the sample search object's standard text description. The most popular search terms for a sample search object can be one or more search terms with a high click-through rate for that sample search object. For example, you can count the number of times a sample search object is clicked after a user enters a search term, and then use the top N search terms with the most clicks on the sample search object as the hot search terms for the sample search object. For each sample search object, you can determine the hot search term corresponding to the sample search object based on historical search data.

[0084] As shown in FIG10 , when pre-training the initial model 13, the hot search terms corresponding to the sample search object, the text description of the sample search object, and the image of the sample search object can be obtained. The text description of the sample search object can then be concatenated with a hot search term of the sample search object to obtain a concatenated text. The image of the sample search object is divided into multiple image blocks, and the image feature extraction model 12 performs feature extraction on each of the multiple image blocks to obtain a sample feature sequence for the multiple image blocks. The sample feature sequence, the image of the sample search object, and the concatenated text are input into the initial model 13. The initial model 13 can then extract pixel-level features from the image of the search object and text features from the concatenated text. A representation vector for the image of the sample search object can then be determined based on the sample feature sequence and pixel-level features, and a representation vector for the concatenated text can then be determined based on the text features. This means that the concatenated text and the image of the sample search object are mapped into the same vector space. A prediction can then be made as to whether the concatenated text and the image of the sample search object match. For example, if the representation vectors of the two are close, i.e., if the similarity of the representation vectors of the two is greater than a preset similarity, then the two are considered to match. Then, the model parameters of the initial model 13 can be adjusted based on the deviation between the predicted result and the actual result to pre-train the initial model 13. Since the concatenated text is the text obtained by concatenating the text description of the search object and the popular search term, it can be considered to match the search object, and the actual result is a match between the two.

[0085] During the pre-training process, the text description of the sample search object and the popular search terms of the sample search object are fused together and then matched with the image of the sample search object. This allows the model to better learn the semantic differences between the text description of the sample search object and the search terms during training, thereby alleviating the problem of mismatch between the final recalled search object and the search terms caused by the difference between the two.

[0086] Of course, when actually pre-training the initial model 13, one or more of the three pre-training processes may be included. When the three pre-training processes are included at the same time, the three pre-training processes may be performed separately or simultaneously, and the present disclosure does not limit this.

[0087] For example, in some scenarios, in order to make the pre-trained initial model 14 have better performance, the pre-training process can include the above three pre-training processes at the same time, and the three pre-training processes can be carried out simultaneously, that is, the initial model 13 is pre-trained in a joint training manner.

[0088] For example, as shown in FIG11 , the image of the sample search object can be first divided into blocks to obtain multiple image blocks, and then the image feature extraction model 12 can be used to extract features from the multiple images to obtain a sample feature sequence. The text description of the sample search object, the hot search terms of the sample search object, the image of the sample search object, and the sample feature sequence can then be simultaneously input into the initial model 13. The initial model 13 can simultaneously perform the above three prediction tasks based on the input data, namely, the task of predicting the masked words in combination with the text context, the task of reconstructing the image of the sample search object, and the task of determining whether the concatenated text of the text description of the sample search object and the hot search terms matches the image of the sample search object. The initial model 13 can then output the prediction results corresponding to each of the above three prediction tasks, and determine the target loss based on the difference between the predicted results and the actual results of the above three tasks, and adjust the model parameters of the initial model 13 based on the target loss.

[0089] After pre-training the initial model 13 in the aforementioned pre-training phase, the pre-trained initial model 14 can already effectively integrate information from different modalities, such as the image and text of the search object, to obtain a relatively accurate representation vector for the search object. However, during the recall phase of the search scenario, since the search object integrates information from different modalities, such as text and images, it will differ significantly from the search term. If the pre-trained model is fine-tuned using the methods provided in the related art, a text feature extraction model 11 is obtained. In the application phase, after the final text feature extraction model 11 maps the search object and the search term to the same vector space, the representation vectors of the two will be severely split. That is, the representation vector of the search term will be distributed in one region of the space, while the representation vector of the search object will be distributed in another region of the space. Since the recall phase typically uses the KNN (K-Nearest Neighbor) algorithm to search for the N nearest representation vectors of the search object to the representation vector of the search term, and returns these N representation vectors of the search object to the user, it is generally desirable that the distribution difference between the representation vector of the search term and the representation vector of the search object in the vector space is not too large, and that they should be within a common subspace as much as possible.

[0090] Therefore, in some embodiments, in the process of fine-tuning the pre-trained initial model 13, an adversarial learning mechanism can be introduced to eliminate the large difference in the distribution of the representation vectors of the search object and the search term in the vector space after the search object and the search term are finally mapped to the same vector space due to the inclusion of information of different modalities.

[0091] As shown in Figure 12, a Generative Adversarial Network (GAN) 15 generally consists of a generator 151 and a discriminator 152. The main idea is to train the model through the continuous competition between the generator 151 and the discriminator 152 during the training process. For example, using the GAN 15 to generate a similar representation vector for a search object and a similar representation vector for a search term, the generator 151 can generate a representation vector for the search object and a representation vector for the search term based on the attribute data of the search term and the search object, respectively. The discriminator 152 then distinguishes which is the representation vector for the search object and which is the representation vector for the search term. Based on the judgment results, the model parameters of the generator 151 and the discriminator 152 can be adjusted to train both. The goal of the training process is to make the representation vector for the search object generated by the generator 151 as close as possible to the representation vector for the search term, so that the discriminator 152 cannot distinguish between the two. At the same time, the discriminator 152 must be able to determine as accurately as possible which is the representation vector for the search object and which is the representation vector for the search term. The entire training process is a process of continuous confrontation between the generator 151 and the discriminator 152, so that the representation vector of the search term generated by the generator 151 after training is not too different from the representation vector of the search object, and they are as close as possible to the same common subspace.

[0092] In the embodiment of the present disclosure, as shown in FIG13 , the pre-trained initial model 14 can be used as the generator 151 of the generative adversarial model 15 . During the fine-tuning process of the pre-trained initial model 13 , sample data can be obtained, which includes a sample search term, an image of the sample search object, and a text description of the sample search object. The image of the sample search object can then be divided into multiple image blocks, and the aforementioned image feature extraction model 12 can be used to extract features from each of the multiple image blocks to obtain sample feature sequences for the multiple image blocks. The sample search term, the image of the sample search object, the text description of the sample search object, and the sample feature sequence can then be input into the pre-trained initial model 14 . The pre-trained initial model 14 can extract pixel-level features from the image of the sample search object, determine the image features of the sample search object based on the pixel-level features and the sample feature sequence, and then determine the text features of the sample search object based on the text description of the sample search object. The image features and text features are combined to obtain a representation vector for the sample search object. The representation vector for the sample search term can then be determined.

[0093] After obtaining the representation vector of the sample search object and the representation vector of the sample search term, the representation vectors of the two can be input into the preset discriminator 152, and the discriminator 152 determines the type of the input representation vector, that is, the discriminator 152 determines whether the input representation vector is the representation vector of the sample search object or the representation vector of the search term. Then, the target loss can be determined based on the difference between the determination result of the discriminator 152 and the true result, and the model parameters of the pre-trained initial model 14 can be adjusted according to the target loss to train the text feature extraction model 11. For example, if the representation vector input to the discriminator 152 is the representation vector of the sample search object, but the discriminator 152 predicts it to be the representation vector of the sample search term, that is, the model's prediction result is inaccurate.

[0094] Among them, in the process of adversarial learning, the purpose of training the generator 151 (i.e., the initial model 14 after pre-training) is to make the representation vector of the sample search object and the representation vector of the search term generated by it as close as possible, so that the discriminator 152 cannot accurately distinguish between the two. The purpose of training the discriminator 152 is to enable the discriminator 152 to accurately distinguish between the two. By competing with each other, after the generator 151 (i.e., the initial model 14 after pre-training) maps the search object and the search term to the same vector space, the distribution of the representation vectors of the two will be closer, and they can be in a common subspace as much as possible to facilitate subsequent recall.

[0095] In some embodiments, during the fine-tuning phase, in addition to determining a loss based on the difference between the discriminator 152's discrimination result and the true result, and adjusting the parameters of the pre-trained initial model 14, the pre-trained initial model 14 can also simultaneously perform a prediction task of whether a sample search object and a sample search term in the sample data match, and adjust the parameters of the pre-trained initial model 14 based on the difference between the predicted result of this prediction task and the true result. That is, the loss function can be comprehensively determined by combining the processing results of the pre-trained initial model 14 for the above two tasks to adjust the model parameters, thereby training a text feature extraction model 11 with good performance and higher accuracy.

[0096] For example, sample data can also carry a label that indicates whether the sample search term and the sample search object match. A match between the sample search term and the sample search object means that the sample search object is the object pointed to by the sample search term. For example, if the sample search term is "dress" and the sample search object is "dress," then the two match. However, if the sample search object is "shoes," then the two do not match. Furthermore, a match can mean that both the text description and the image of the search object match the search term.

[0097] As shown in FIG14 , when determining the target loss based on the difference between the determination result of the discriminator 152 and the true result, a first loss can be first determined based on the difference between the determination result of the discriminator 152 and the true result. Then, the sample search term, the image of the sample search object, the text description of the sample search object, and the sample feature sequence can be input into the pre-trained initial model 14, and the pre-trained initial model 14 outputs a prediction result, which is used to indicate whether the sample search term and the sample search object match. Furthermore, a second loss can be determined based on the difference between the determination result of the discriminator 152 and the true result indicated by the label. The target loss is obtained based on the first loss and the second loss, and the model parameters of the pre-trained initial model 14 are adjusted based on the target loss to train the text feature extraction model 11. For example, the two losses can be weighted according to a certain weight and then added together to obtain the target loss.

[0098] In some embodiments, the training of the above-mentioned generative adversarial model 15 can be divided into two stages. In the first stage, the model parameters of the discriminator 152 are fixed and no gradient update is performed. Only the model parameters of the pre-trained initial model 14 are updated. In this stage, the model parameters of the pre-trained initial model 14 can be adjusted based on two losses. One is a loss determined based on the task of the discriminator 152 to distinguish whether the input representation vector is the representation vector of the sample search term or the representation vector of the sample search object, and the other is a loss determined based on the task of the pre-trained initial model 14 to predict whether the sample search term and the sample search object match. In the second stage, the model parameters of the pre-trained initial model 14 are fixed, and only the parameters of the discriminator 152 are gradient updated. In this stage, the model parameters of the discriminator 152 are adjusted based only on one loss, which is a loss determined based on the task of the discriminator 152 to distinguish whether the input representation vector is the representation vector of the sample search term or the representation vector of the sample search object. The above two stages can be repeated continuously throughout the training process until convergence.

[0099] In some embodiments, in order to ensure the accuracy of the ultimately trained text feature extraction model 11, the diversity of the sample data can be ensured as much as possible when constructing the sample data. For example, the sample data can include positive samples and negative samples, wherein the sample search terms in the positive samples match the sample search objects, while the sample search terms in the negative samples do not match the search objects.

[0100] In some embodiments, when constructing the sample data, historical search data may be obtained, and the search terms used by the user may be determined based on the historical search data. After returning search objects based on the search terms, the search objects clicked by the user may be determined from the returned search objects. The positive sample may then be constructed using the search terms as the sample search terms in the positive sample, and any search objects clicked by the user as the sample search objects in the positive sample. The negative sample may also be constructed using the search terms as the sample search terms in the negative sample, and any search objects not clicked by the user as the sample search objects in the negative sample.

[0101] Similarly, in the stage of training the text feature extraction model 11 (including the pre-training stage and the fine-tuning stage), in order to eliminate the interference of the background part that has no practical meaning in the image of the sample search object and to extract more effective information from the image, the foreground object can also be extracted from the image of the sample search object first, and the image area corresponding to the foreground object can be divided into multiple image blocks; the multiple image blocks are respectively input into the image feature extraction model 12, and the image feature extraction model 12 performs feature extraction on the multiple image blocks respectively to obtain a sample feature sequence.

[0102] Similarly, the text description of the sample search object and the processing of the above-mentioned spliced ​​text are similar. They can also be segmented first to obtain the representation vector of each word, and then input into the initial model 13 or the pre-trained initial model 14.

[0103] In some embodiments, the initial model may be a BERT model, the method may be used for product search in an e-commerce scenario, and the search object may be a product in an e-commerce scenario.

[0104] Furthermore, the embodiments of the present disclosure also provide a search method that can be used in various search scenarios, for example, it can be used for product search in e-commerce scenarios. The specific process of the method is as follows:

[0105] The search term input by the user can be obtained, and then the representation vector of the search term can be determined using the aforementioned text extraction feature model. The text feature extraction model can be trained using the method described in the aforementioned embodiment, which will not be repeated here. The representation vectors of multiple search objects can then be obtained, wherein the representation vectors of the search objects can be obtained by the methods described in the various embodiments of the aforementioned representation method. For details, please refer to the description in the aforementioned embodiment, which will not be repeated here. After obtaining the representation vectors of the search term and the multiple search objects, candidate objects can be determined from the multiple search objects based on the similarity between the representation vector of the search term and the respective representation vectors of the multiple search objects, so as to display the attribute data of the candidate objects to the user. For example, the KNN algorithm can be used to determine the N search objects that are closest to the representation vector of the search term from the representation vectors of the multiple search objects, and then the attribute data such as images and text descriptions of these N search objects can be displayed to the user.

[0106] Furthermore, the present disclosure also provides a model training method, which may include the following steps:

[0107] Obtain an image of a sample search object;

[0108] Dividing the image of the sample search object into a plurality of image blocks, and performing feature extraction on each of the plurality of image blocks using an image feature extraction model to obtain sample feature sequences of the plurality of image blocks, wherein the image feature extraction model is pre-trained using a general image dataset;

[0109] Extracting pixel-level features from the image of the sample search object through an encoder of a preset initial model;

[0110] Reconstructing the image of the sample search object based on the sample feature sequence and the pixel-level features through the decoder of the initial model to obtain a reconstructed image;

[0111] adjusting model parameters of the initial model based on a difference between the reconstructed image and the image of the sample search object to pre-train the initial model;

[0112] The pre-trained initial model is fine-tuned based on historical search data to obtain a text feature extraction model.

[0113] Among them, the specific implementation details of the model training can be referred to the description in the above embodiment and will not be repeated here.

[0114] Furthermore, the present disclosure also provides a model training method, which may include the following steps:

[0115] Obtaining the hot search terms corresponding to the sample search object, the text description of the sample search object, and the image of the sample search object;

[0116] splicing the hot search terms of the sample search object and the text description of the sample search object to obtain a spliced ​​text;

[0117] Dividing the image of the sample search object into a plurality of image blocks, and performing feature extraction on each of the plurality of image blocks using an image feature extraction model to obtain sample feature sequences of the plurality of image blocks, wherein the image feature extraction model is pre-trained using a general image dataset;

[0118] Inputting the sample feature sequence, the image of the sample search object, and the spliced ​​text into a preset initial model, and outputting a prediction result through the initial model, the prediction result being used to indicate whether the spliced ​​text and the image of the sample search object match;

[0119] Adjusting the model parameters of the initial model based on the deviation between the output prediction result and the actual result to pre-train the initial model;

[0120] The pre-trained initial model is fine-tuned based on historical search data to obtain a text feature extraction model.

[0121] Among them, the specific implementation details of the model training can be referred to the description in the above embodiment and will not be repeated here.

[0122] Furthermore, the present disclosure also provides a model training method, which may include the following steps:

[0123] Pre-train the preset initial model based on historical search data;

[0124] Acquire sample data, wherein the sample data includes a sample search term, an image of a sample search object, and a text description of the sample search object;

[0125] Dividing the image of the sample search object into a plurality of image blocks, and performing feature extraction on each of the plurality of image blocks using the image feature extraction model to obtain sample feature sequences of the plurality of image blocks, wherein the image feature extraction model is pre-trained using a general image dataset;

[0126] Inputting the sample search term, the image of the sample search object, the text description of the sample search object, and the sample feature sequence into the pre-trained initial model, and outputting a representation vector of the sample search term and a representation vector of the sample search object through the pre-trained initial model;

[0127] Inputting the representation vector of the sample search term and the representation vector of the sample search object into a preset discriminator respectively, and using the discriminator to determine the type of the input representation vector, the type including search term or search object;

[0128] Determine the target loss based on the difference between the discriminator's judgment result and the true result;

[0129] The model parameters of the pre-trained initial model are adjusted according to the target loss to train the text feature extraction model.

[0130] Among them, the specific implementation details of the model training can be referred to the description in the above embodiment and will not be repeated here.

[0131] Corresponding to the embodiments of the search object characterization method, search method, and model training method provided in the embodiments of this specification, this specification also provides a search object characterization device, a search device, and a model training device. The implementation process of the functions and effects of each unit in the above-mentioned device is specifically detailed in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0132] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the embodiment scheme of this specification. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0133] From the hardware level, as shown in Figure 15, it is a hardware structure diagram of the device where the search object characterization device, search device and / or model training device of the embodiment of this specification are located. In addition to the processor 1502 and memory 1504 shown in Figure 15, the device can generally also include other hardware, such as a forwarding chip responsible for processing messages; from the hardware structure, the device may also be a distributed device, which may include multiple interface cards to expand message processing at the hardware level. The memory 1504 stores computer instructions, and when the processor 1502 executes the computer instructions, it implements the method mentioned in any of the above embodiments.

[0134] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0135] Since the portion of the embodiment of this specification that contributes to the prior art or the entire or partial technical solution can be embodied in the form of a software product, the computer software product is stored in a storage medium and includes a number of instructions for causing a terminal device to execute all or part of the steps of each method of the embodiment of this specification. The aforementioned storage medium includes: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc., various media that can store program code.

[0136] The above description is only a preferred embodiment of the embodiments of this specification and is not intended to limit the embodiments of this specification. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the embodiments of this specification should be included in the scope of protection of the embodiments of this specification.

Claims

1. A method for representing a search object, characterized in that: The method comprises: Obtaining a text description of the search object and an image of the search object; Dividing the image of the search object into multiple image blocks, and performing feature extraction on the multiple image blocks respectively through an image feature extraction model to obtain feature sequences of the multiple image blocks; wherein the image feature extraction model is pre-trained using a general image data set; Pixel-level features are extracted from the image of the search object through a text feature extraction model, and a representation vector of the search object is determined based on the feature sequence, the pixel-level features, and the image of the search object through the text feature extraction model; wherein the text feature extraction model is trained based on historical search data.

2. The characterization method according to claim 1, characterized in that The step of dividing the image of the search object into a plurality of image blocks comprises: extracting a foreground object from the image of the search object; The image area corresponding to the foreground object is divided into a plurality of image blocks.

3. The characterization method according to claim 1, characterized in that The determining the representation vector of the search object based on the feature sequence, the pixel-level features, and the text description of the search object by the text feature extraction model includes: Performing word segmentation processing on the text description of the search object to obtain multiple words; Determine a representation vector for each of the plurality of words, wherein the representation vector is obtained based on a word vector of each word and a position vector representing a position of each word in the text description; The representation vector of the search object is determined by the text feature extraction model based on the feature sequence, the pixel-level features, and the representation vectors of each of the multiple words.

4. The characterization method according to claim 1, characterized in that The text feature extraction model is trained based on historical search data and includes: Pre-train the preset initial model; Fine-tuning the pre-trained initial model based on historical search data to obtain the text feature extraction model; The process of pre-training the preset initial model includes: Acquire an image of a sample search object; divide the image of the sample search object into a plurality of image blocks, respectively extract features from the plurality of image blocks through the image feature extraction model, and obtain sample feature sequences of the plurality of image blocks; extract pixel-level features from the image of the sample search object through the encoder of the initial model, and reconstruct the image of the sample search object based on the sample feature sequence and the pixel-level features through the decoder of the initial model to obtain a reconstructed image; adjust the model parameters of the initial model based on the difference between the reconstructed image and the image of the sample search object, so as to pre-train the initial model; and / or Obtain the hot search words corresponding to the sample search object, the text description of the sample search object, and the image of the sample search object; splice the hot search words of the sample search object and the text description of the sample search object, Obtaining a concatenated text; dividing the image of the sample search object into a plurality of image blocks, respectively extracting features from the plurality of image blocks through the image feature extraction model, and obtaining a sample feature sequence of the plurality of image blocks; inputting the sample feature sequence, the image of the sample search object, and the concatenated text into the initial model, and outputting a prediction result through the initial model, the prediction result being used to indicate whether the concatenated text and the image of the sample search object match; adjusting a model parameter of the initial model based on a deviation between the output prediction result and a true result, so as to pre-train the initial model; and / or The text descriptions of multiple sample search objects are obtained as corpus data, and the initial model is trained unsupervisedly using the corpus data.

5. The characterization method according to any one of claims 1 to 4, characterized in that: The text feature extraction model is trained based on historical search data and includes: Pre-train the preset initial model based on historical search data; Acquire sample data, wherein the sample data includes a sample search term, an image of a sample search object, and a text description of the sample search object; Dividing the image of the sample search object into a plurality of image blocks, and performing feature extraction on the plurality of image blocks respectively by the image feature extraction model to obtain sample feature sequences of the plurality of image blocks; Inputting the sample search term, the image of the sample search object, the text description of the sample search object, and the sample feature sequence into the pre-trained initial model, and outputting a representation vector of the sample search term and a representation vector of the sample search object through the pre-trained initial model; Inputting the representation vector of the sample search word and the representation vector of the sample search object into a preset discriminator respectively, and using the discriminator to determine the type of the input representation vector, the type including the search word or the search object; Determine the target loss based on the difference between the discriminator's judgment result and the true result; The model parameters of the pre-trained initial model are adjusted according to the target loss to train the text feature extraction model.

6. The characterization method according to claim 5, characterized in that: The sample data carries a label, and the label is used to indicate whether the sample search term matches the sample search object; the determining the target loss based on the difference between the determination result of the discriminator and the true result includes: Determine a first loss based on the difference between the determination result of the discriminator and the true result; Inputting the sample search term, the image of the sample search object, the text description of the sample search object, and the sample feature sequence into the pre-trained initial model, and outputting a prediction result through the pre-trained initial model, the prediction result is used to indicate whether the sample search term and the sample search object match; Determine a second loss based on the difference between the determination result of the discriminator and the true result indicated by the label; The target loss is obtained according to the first loss and the second loss.

7. The characterization method according to claim 6, characterized in that: The sample data includes positive samples and negative samples. The sample search terms and sample search objects in the positive samples match, while the sample search terms and search objects in the negative samples do not match.

8. The characterization method according to claim 7, characterized in that: The sample data is obtained based on the following method: Get historical search data; Determining the search terms used by the user based on the historical search data, and determining the search objects clicked by the user from the search objects recalled by the search terms; The positive sample is constructed by using the search term as a sample search term in the positive sample and any search object clicked by the user as a sample search object in the positive sample; The negative sample is constructed by using the search term as a sample search term in the negative sample and using any search object that has not been clicked by the user as a sample search object in the negative sample.

9. A search method, characterized in that: The method comprises: Obtaining a search term input by a user and a representation vector of a plurality of search objects, wherein the representation vector of the search object is obtained by the representation method according to any one of claims 1 to 8; Determine the representation vector of the search term using a text extraction feature model; Based on the similarity between the representation vector of the search term and the representation vectors of each of the multiple search objects, a candidate object is determined from the multiple search objects to display the attribute data of the candidate object to the user.

10. A model training method, characterized in that: The method comprises: Obtain an image of a sample search object; Dividing the image of the sample search object into a plurality of image blocks, performing feature extraction on the plurality of image blocks respectively through an image feature extraction model to obtain sample feature sequences of the plurality of image blocks, wherein the image feature extraction model is pre-trained using a general image data set; Extracting pixel-level features from the image of the sample search object through an encoder of a preset initial model; Reconstructing the image of the sample search object based on the sample feature sequence and the pixel-level features through the decoder of the initial model to obtain a reconstructed image; adjusting a model parameter of the initial model based on a difference between the reconstructed image and the image of the sample search object to pre-train the initial model; The pre-trained initial model is fine-tuned based on historical search data to obtain a text feature extraction model.

11. A model training method, characterized in that: The method comprises: Obtaining the hot search terms corresponding to the sample search object, the text description of the sample search object, and the image of the sample search object; splicing the hot search term of the sample search object and the text description of the sample search object to obtain a spliced ​​text; The image of the sample search object is divided into multiple image blocks, and the image feature extraction model is used to extract the image features of the image blocks. Extracting features from the plurality of image blocks to obtain sample feature sequences of the plurality of image blocks, wherein the image feature extraction model is pre-trained using a general image data set; Inputting the sample feature sequence, the image of the sample search object, and the spliced ​​text into a preset initial model, and outputting a prediction result through the initial model, the prediction result being used to indicate whether the spliced ​​text and the image of the sample search object match; Adjusting the model parameters of the initial model based on the deviation between the output prediction result and the actual result to pre-train the initial model; The pre-trained initial model is fine-tuned based on historical search data to obtain a text feature extraction model.

12. A model training method, characterized in that: The method comprises: Pre-train the preset initial model based on historical search data; Acquire sample data, wherein the sample data includes a sample search term, an image of a sample search object, and a text description of the sample search object; Dividing the image of the sample search object into a plurality of image blocks, performing feature extraction on the plurality of image blocks respectively by the image feature extraction model, and obtaining sample feature sequences of the plurality of image blocks, wherein the image feature extraction model is pre-trained using a general image data set; Inputting the sample search term, the image of the sample search object, the text description of the sample search object, and the sample feature sequence into the pre-trained initial model, and outputting a representation vector of the sample search term and a representation vector of the sample search object through the pre-trained initial model; Inputting the representation vector of the sample search word and the representation vector of the sample search object into a preset discriminator respectively, and using the discriminator to determine the type of the input representation vector, the type including the search word or the search object; Determine the target loss based on the difference between the discriminator's judgment result and the true result; The model parameters of the pre-trained initial model are adjusted according to the target loss to train the text feature extraction model.

13. An electronic device, characterized in that: The electronic device comprises a processor, a memory, and a computer program stored in the memory and executable by the processor, and the processor implements the method described in any one of claims 1 to 12 when executing the computer program.

14. A computer storage medium, characterized in that: The computer storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 12 is implemented.

Images