Commodity object search method and its device, equipment, medium, and product

Abstract

The present application discloses a method for searching commodity objects and its device, equipment, medium, and product. The method includes: obtaining a comprehensive feature vector of each commodity object in a commodity database, the comprehensive feature vector including the image semantic features and text semantic features of the commodity object; the image semantic features are generated by extracting the semantic features of the multiple segmented graph elements of the commodity image of the commodity object and then performing feature interaction; constructing a feature index matrix for storing similarity data between the comprehensive feature vectors of each commodity object; obtaining the visited commodity objects in the user behavior message and determining them as the commodity objects to be searched; and querying multiple target commodity objects that meet the similarity matching conditions with the commodity objects to be searched from the feature index matrix. The present application realizes similarity calculation by performing feature interaction at a deep semantic level on the commodity images and summary texts of the commodity objects, so that similar commodity matching is more accurate.

Description

Technical Field

[0001] The present application relates to the field of e-commerce information technology, and in particular to a commodity object search method and its corresponding device, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] Nowadays, major e-commerce platforms will recommend some related products based on the user's browsing history to achieve the purpose of product promotion. On the one hand, it can improve the user's shopping experience, and on the other hand, it can help increase product sales. The most important part of this is the search for similar products on the site.

[0003] When searching within the site, on the one hand, considering that product information generally has two forms, text information (title text, detailed content) and image information (main image, detailed image), these two types of information describe the product from different dimensions. The existing technology generally uses a single information or a combination of the two for search and matching, but the effect is not obvious in practice.

[0004] The reason is that in the existing technology, similar images are recognized based on the semantic feature information of product images and text information. However, since the representation learning of the semantic feature information is relatively superficial, the obtained semantic representation lacks the ability to comprehensively describe the characteristics of the product object. Therefore, corresponding improvement solutions need to be explored. Summary of the invention

[0005] The primary purpose of the present application is to solve at least one of the above problems and to provide a commodity object search method and its corresponding device, computer equipment, computer-readable storage medium, and computer program product.

[0006] In order to meet the various objectives of this application, this application adopts the following technical solutions:

[0007] A method for searching for commodity objects provided to meet one of the purposes of the present application comprises the following steps:

[0008] Obtaining a comprehensive feature vector of each commodity object in a commodity database, wherein the comprehensive feature vector includes a picture semantic feature and a text semantic feature of the commodity object; the picture semantic feature is generated by extracting semantic features of a plurality of segmented graph elements of a commodity image of the commodity object and then performing feature interaction;

[0009] Construct a feature index matrix to store the similarity data between the comprehensive feature vectors of two commodity objects;

[0010] Obtain the visited product object in the user behavior message and determine it as the product object to be searched;

[0011] A plurality of target commodity objects that meet similarity matching conditions with the commodity object to be searched are queried from the feature index matrix.

[0012] In a further embodiment, obtaining a comprehensive feature vector of each commodity object in a commodity database includes the following steps:

[0013] Calling a pre-trained image feature extraction model to extract image semantic features of product images of each product object in the product database;

[0014] Calling a pre-trained text feature extraction model to extract text semantic features of summary texts of each product object in the product database;

[0015] The image semantic features and text semantic features of each product object are concatenated to obtain the corresponding comprehensive feature vector.

[0016] In a specific embodiment, calling a pre-trained image feature extraction model to extract image semantic features of product images of each product object in a product database includes the following steps:

[0017] The product image of each product object is divided into multiple segmented graphics elements;

[0018] For each segmented image primitive, a first image feature extraction model is used to extract a corresponding single image feature vector;

[0019] Expand the single image feature vector into a single image embedding vector represented by a single row vector, and concatenate all the single image embedding vectors into a picture encoding vector, which carries the position information and distinguishing information of each segmentation element in the product picture;

[0020] A second image feature extraction model based on a multi-head attention mechanism is used to perform feature interaction on the image encoding vector to obtain image semantic features that realize deep semantic interaction of the single image feature vector.

[0021] In a specific embodiment, calling a pre-trained text feature extraction model to extract text semantic features of summary texts of each commodity object in a commodity database includes the following steps:

[0022] Segment the summary text of each product object;

[0023] Encoding is performed according to the word segmentation to obtain corresponding text embedding vectors, position embedding vectors, and sentence embedding vectors;

[0024] A text feature extraction model is used to extract features based on the text embedding vector, position embedding vector, and sentence embedding vector to obtain corresponding text semantic features.

[0025] In a specific embodiment, a second image feature extraction model based on a multi-head attention mechanism is used to perform feature interaction on the image encoding vector to obtain image semantic features that realize deep semantic interaction, including the following steps performed by at least one level encoder in the second image feature extraction model:

[0026] Construct a query vector, a key vector and a value vector corresponding to each row vector in the current input image encoding vector;

[0027] Perform weighted summation on the value vectors corresponding to all row vectors, where the weight of each value vector is the normalized score of the dot product of the key vector corresponding to it and the query vector corresponding to other row vectors;

[0028] Multiply the weighted sum result by the conversion weight matrix to obtain its corresponding intermediate information matrix;

[0029] The intermediate information matrix is ​​subjected to feature extraction by a multi-layer perceptron to obtain picture semantic features, which are used as picture coding vectors required by the next encoder or directly output in the last encoder.

[0030] In a preferred embodiment, the second image feature extraction model is a DeiT model based on the Vision Transformer architecture, which is pre-trained to a convergence state.

[0031] In a further embodiment, searching for multiple target commodity objects that meet similarity matching conditions with the commodity object to be searched from the feature index matrix includes the following steps:

[0032] Querying all commodity objects having similarity data mapping relationships with the commodity object to be searched from the feature index matrix;

[0033] Selecting a predetermined number of commodity objects with the largest similarity data as target commodity objects that meet the similarity data matching condition;

[0034] Constructing a search result list containing the target product object;

[0035] The search result list is pushed in response to the commodity object search request.

[0036] A commodity object search device provided to meet one of the purposes of the present application includes: a feature summary module, a matrix construction module, a search start module, and a matching feedback module, wherein the feature summary module is used to obtain a comprehensive feature vector of each commodity object in a commodity database, and the comprehensive feature vector includes a picture semantic feature and a text semantic feature of the commodity object; the picture semantic feature is generated by extracting the semantic features of multiple segmented graphics elements of the commodity image of the commodity object and then performing feature interaction; the matrix construction module is used to construct a feature index matrix for storing similarity data between the comprehensive feature vectors of each commodity object; the search start module is used to obtain the visited commodity objects in the user behavior message and determine them as the commodity objects to be searched; the matching feedback module is used to query multiple target commodity objects that meet the similarity matching conditions with the commodity objects to be searched from the feature index matrix.

[0037] In a further embodiment, the feature summary module includes: an image extraction submodule, which is used to call a pre-trained image feature extraction model to extract image semantic features of product images of each product object in the product database; a text extraction submodule, which is used to call a pre-trained text feature extraction model to extract text semantic features of summary texts of each product object in the product database; and a feature splicing submodule, which is used to perform feature splicing on the image semantic features and text semantic features of each product object to obtain its corresponding comprehensive feature vector.

[0038] In a specific embodiment, the image extraction submodule includes: a primitive segmentation unit, which is used to divide the product image of each product object into multiple segmented primitives; a primitive extraction unit, which is used to extract the corresponding single image feature vector for each segmented primitive using a first image feature extraction model; a splicing encoding unit, which is used to expand the single image feature vector into a single image embedding vector represented by a single row vector, and splice all single image embedding vectors into an image encoding vector, which carries the position information and distinguishing information of each segmented primitive in the product image; a primitive interaction unit, which is used to perform feature interaction on the image encoding vector using a second image feature extraction model based on a multi-head attention mechanism, and obtain image semantic features that realize deep semantic interaction of the single image feature vector.

[0039] In a specific embodiment, the text extraction submodule includes: a text segmentation unit, which is used to segment the summary text of each commodity object; a text encoding unit, which is used to encode according to the segmentation to obtain corresponding text embedding vectors, position embedding vectors and sentence embedding vectors; a text extraction unit, which is used to use a text feature extraction model to perform feature extraction according to the text embedding vectors, position embedding vectors and sentence embedding vectors to obtain corresponding text semantic features.

[0040] In a specific embodiment, the encoder in the second image feature extraction model adopted in the primitive interaction unit includes: a vector mapping module, which is used to construct a query vector, a key vector and a value vector corresponding to each row vector in the currently input picture coding vector; an interactive operation module, which is used to perform weighted summation on the value vectors corresponding to all row vectors, and the corresponding weight of each value vector is the normalized score of the dot product of the corresponding key vector and the query vector corresponding to other row vectors; a weight conversion module, which is used to multiply the weighted summation result by the conversion weight matrix to obtain the corresponding intermediate information matrix; a perceptual extraction module, which is used to perform feature extraction on the intermediate information matrix through a multi-layer perceptron to obtain picture semantic features as the picture coding vector required by the next encoder, or directly output in the last encoder.

[0041] In a preferred embodiment, the second image feature extraction model is a DeiT model based on the Vision Transformer architecture, which is pre-trained to a convergence state.

[0042] In a further embodiment, the matching feedback module includes: a feature query submodule, used to query all product objects that have a similarity data mapping relationship with the product object to be searched from the feature index matrix; a target optimization submodule, used to select a predetermined number of product objects with the largest similarity data as target product objects that meet the similarity data matching conditions; a list construction submodule, used to construct a search result list containing the target product object; and a result push submodule, used to push the search result list in response to the product object search request.

[0043] A computer device provided to meet one of the purposes of the present application includes a central processing unit and a memory, wherein the central processing unit is used to call and run a computer program stored in the memory to execute the steps of the commodity object search method described in the present application.

[0044] A computer-readable storage medium is provided to meet another purpose of the present application, which stores a computer program implemented according to the commodity object search method in the form of computer-readable instructions. When the computer program is called and executed by a computer, the steps included in the method are executed.

[0045] A computer program product provided to meet another purpose of the present application includes a computer program / instruction, which, when executed by a processor, implements the steps of the method described in any embodiment of the present application.

[0046] Compared with the prior art, the advantages of this application are as follows:

[0047] After the product image of the product object is deeply interacted at the feature level to obtain the image semantic features, the application splices them with the text semantic features of the summary text of the product object to form a comprehensive feature vector. On this basis, the similarity data between the two product objects is calculated to construct a feature index matrix. Subsequently, the relevant target product objects that meet the similarity matching conditions can be queried from the feature index matrix according to the product object to be searched. Among them, the image semantic features are generated by extracting the semantic features of the multiple segmented graphics elements of the product image of the product object and then performing feature interaction. Therefore, the product image has fully interacted at the semantic feature level, so that the image semantic features obtained by representation learning of the product image, and the comprehensive feature vector obtained by splicing it with the text semantic features, have achieved good representation learning of the product image. Based on this, similarity calculation can obtain more accurate similarity data. When further searching for products based on this, the matching accuracy of similar product objects can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0049] Figure 1 A flowchart of a typical embodiment of the commodity object search method of the present application;

[0050] Figure 2 Schematic diagram of the process of obtaining the comprehensive feature vector in the embodiment of the present application;

[0051] Figure 3 Schematic diagram of the process of acquiring the semantic features of an image in an embodiment of the present application;

[0052] Figure 4 Schematic diagram of the process of acquiring text semantic features in the embodiment of the present application;

[0053] Figure 5 A flowchart of a process of encoding based on a multi-head attention mechanism in an embodiment of the present application;

[0054] Figure 6 A schematic diagram of the process of matching similar commodity objects in an embodiment of the present application;

[0055] Figure 7 This is a functional block diagram of the commodity object search device of the present application;

[0056] Figure 8 A schematic diagram of the structure of a computer device used in this application. DETAILED DESCRIPTION

[0057] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.

[0058] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0059] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0060] It will be understood by those skilled in the art that the "client", "terminal" and "terminal device" used herein include both devices with wireless signal receivers, which are devices with only wireless signal receivers without transmission capabilities, and devices with receiving and transmitting hardware, which are devices with receiving and transmitting hardware capable of two-way communication on a two-way communication link. Such devices may include: cellular or other communication devices such as personal computers, tablet computers, which have single-line displays or multi-line displays or cellular or other communication devices without multi-line displays; PCS (Personal Communications Service, personal communication system), which can combine voice, data processing, fax and / or data communication capabilities; PDA (Personal Digital Assistant, personal digital assistant), which may include a radio frequency receiver, pager, Internet / intranet access, web browser, notepad, calendar and / or GPS (Global Positioning System, global positioning system) receiver; conventional laptop and / or palmtop computers or other devices, which have and / or include a conventional laptop and / or palmtop computer or other device with and / or including a radio frequency receiver. The "client", "terminal" and "terminal device" used herein may be portable, transportable, installed in a vehicle (air, sea and / or land), or suitable for and / or configured to run locally, and / or in a distributed form, at any other location on the earth and / or in space. The "client", "terminal" and "terminal device" used herein may also be a communication terminal, an Internet terminal, a music / video playing terminal, for example, a PDA, a MID (Mobile Internet Device) and / or a mobile phone with a music / video playing function, or a smart TV, a set-top box and other devices.

[0061] The hardware referred to by the names such as "server", "client", and "service node" in this application is essentially an electronic device with the equivalent capabilities of a personal computer. It is a hardware device with the necessary components revealed by the von Neumann principle, such as a central processing unit (including an arithmetic unit and a controller), a memory, an input device, and an output device. The computer program is stored in its memory, and the central processing unit calls the program stored in the external memory into the internal memory for execution, executes the instructions in the program, and interacts with the input and output devices to complete specific functions.

[0062] It should be pointed out that the concept of "server" referred to in this application can also be extended to the case of server clusters. According to the network deployment principle understood by those skilled in the art, the servers should be logically divided. In physical space, these servers can be independent of each other but can be called through interfaces, or integrated into a physical computer or a set of computer clusters. Those skilled in the art should understand this flexibility, and should not use it to restrict the implementation of the network deployment method of this application.

[0063] Unless expressly specified, one or more technical features of the present application can be deployed on a server for implementation and accessed by a client through a remote call to obtain an online service interface provided by the server, or can be directly deployed and run on a client for access.

[0064] The neural network models referenced or may be referenced in this application, unless expressly specified, can be deployed on a remote server and remotely called on the client, or can be deployed and directly called on a client with sufficient device capabilities. In some embodiments, when it runs on the client, its corresponding intelligence can be obtained through transfer learning to reduce the requirements for the client's hardware operating resources and avoid excessive occupation of the client's hardware operating resources.

[0065] Unless explicitly specified, the various data involved in this application can be stored remotely on a server or on a local terminal device, as long as it is suitable for being called by the technical solution of this application.

[0066] Those skilled in the art should be aware that, although the various methods of the present application are described based on the same concept and thus present commonality to each other, unless otherwise specified, these methods can be independently executed. Similarly, for each embodiment disclosed in the present application, they are all proposed based on the same inventive concept, therefore, concepts with the same expression, and concepts that are appropriately changed for convenience despite different expressions, should be understood as equivalent.

[0067] Unless the mutually exclusive relationship between the embodiments to be disclosed in this application is explicitly stated, the relevant technical features involved in each embodiment can be cross-combined to flexibly construct a new embodiment, as long as such combination does not deviate from the creative spirit of this application and can meet the needs of the prior art or solve certain deficiencies in the prior art. Those skilled in the art should be aware of this flexibility.

[0068] A commodity object search method of the present application can be programmed as a computer program product and deployed in a client or server for execution. For example, in the e-commerce platform application scenario of the present application, it is generally deployed in a server for implementation, whereby the method can be executed by accessing an interface opened after the computer program product is running and performing human-computer interaction with the process of the computer program product through a graphical user interface.

[0069] One application scenario of the present application is an application in an e-commerce platform based on independent stations. Each independent station is a merchant instance of the e-commerce platform and has an independent access domain name. Its actual owner is responsible for publishing and updating products.

[0070] The merchant instance of the independent station puts each product online. After the e-commerce platform obtains the information related to the product, it constructs the corresponding product object for data storage. The information of the product object mainly includes text information and image information, among which the text information includes but is not limited to the title information of the product object for prominent display, the content information for introducing the product details, and the attribute information for describing the product characteristics.

[0071] In order to implement the technical solution of the present application, summary text and product images of the commodity object can be collected, wherein the summary text is mainly composed of the title information, content information and one or more attribute information in the commodity object, and the corresponding data composition can be flexibly determined by those skilled in the art; the product image can be the main image of the product appearance of the commodity object. Generally speaking, summary text and product images can be obtained according to preset quantity and content requirements. For example, the summary text can be specified to include the title information of the commodity object and the attribute information of all its attribute items, and the product image can only select a single image for default display. Of course, those skilled in the art can flexibly adjust and process on this basis.

[0072] See also Figure 1 In a typical embodiment, the commodity object search method of the present application includes the following steps:

[0073] Step S1100: Obtain a comprehensive feature vector of each commodity object in a commodity database, wherein the comprehensive feature vector includes a picture semantic feature and a text semantic feature of the commodity object; the picture semantic feature is generated by extracting the semantic features of the multiple segmented graph elements of the commodity image of the commodity object and then performing feature interaction:

[0074] In order to enable the commodity objects in the commodity data of the merchant instance to realize feature-level similarity calculation, it is necessary to perform feature-level preprocessing on each of the commodity objects, use artificial neural networks to perform representation learning on each commodity object, obtain the corresponding comprehensive feature vector of each commodity object, and then calculate the similarity between the commodity objects based on the comprehensive feature vectors of each commodity object.

[0075] Among them, the comprehensive feature vector is the result of representation learning of the commodity information of the commodity object. It can be understood that the comprehensive feature vector should contain and express the deep semantic features of the commodity information of the commodity object as richly as possible. Corresponding to the two types of information in the commodity information, namely image information and text information, in this application, for each commodity object, the commodity image and summary text are obtained respectively, wherein the commodity image preferably adopts the default main image of the commodity object, and the summary text preferably adopts the title text of the commodity object, or further enhances individual key attribute data and label data as needed.

[0076] The comprehensive feature vector is a vector that has been simply concatenated, and can be specifically formed by concatenating two types of features corresponding to the product image and the summary text. It is not difficult to understand that by using the product image to extract image features, its corresponding image semantic features can be obtained; by using the summary text to extract text features, its corresponding text semantic features can be obtained. Then, the image semantic features and text semantic features are concatenated at the feature level to obtain the comprehensive feature vector.

[0077] The corresponding comprehensive feature vector can be obtained for each commodity object, so the feature similarity between two commodity objects can be calculated based on each other's comprehensive feature vectors.

[0078] In the present application, the image semantic features and text semantic features fully consider the contextual information on the sequence and realize deep feature interaction. Specifically, the image semantic features are generated by extracting the semantic features of multiple segmented primitives of the corresponding product image respectively, and then performing deep semantic interaction on the feature level for all segmented primitives with reference to the position information and distinction information between these primitives. Therefore, the image semantic features finally obtained can more comprehensively reflect the deep semantics of the entire product image, and it is not easy to ignore the detailed information in the product image; the text semantic features are generated by performing deep semantic interaction on the feature level according to the sequence reflected by the relative position relationship between its segmented words and the sentence information of the corresponding summary text, with full reference to the contextual information. Therefore, the text semantic features finally obtained can also more effectively reflect the deep semantics of the entire summary text. Preferably, the image semantic features and text semantic features can finally be unified to the same scale by technical personnel in this field for splicing.

[0079] In an alternative embodiment, the encoding architecture of the Transformer model composed of multiple encoders can be used as the preferred model, which is pre-trained to a convergent state and put into use for deep semantic interaction at the feature level for multiple segmented primitives of the product image, thereby obtaining the image semantic features. Such a model can be a model based on ViT (Vision Transformer) and transformed by applying the DeiT idea. Among them, ViT itself can be trained using a self-supervised learning training method. During training, a student network and a teacher network are constructed. The ViT architecture in the two networks is completely consistent. The output of the teacher network guides the learning of the student network. The weight of the teacher network is updated by the sliding average of the student network weights. This is the application of the DeiT structure, which becomes a DeiT model. The idea of ​​knowledge distillation is introduced into the training of ViT. This training strategy uses a convolutional network as a teacher network for distillation, which can achieve better results than using a network with a Transformer architecture as a teacher network.

[0080] In another embodiment, a pre-trained Bert model is used to extract text semantic features from the summary text to obtain corresponding text semantic features. It can be understood that since the Bert model is also developed based on the Transformer architecture, deep interaction of the semantic features of the summary text can also be achieved based on the Transformer encoder principle.

[0081] From this, it can be understood that for a commodity object, whether it is its commodity image or its summary text, in the process of extracting semantic features, sufficient semantic interaction is carried out between the feature reference contexts of its internal units. The image semantic features and text semantic features obtained in this way are concatenated to obtain a comprehensive feature vector. This comprehensive feature vector can more comprehensively represent the deep semantics of the commodity information of the commodity object.

[0082] Step S1200: construct a feature index matrix for storing similarity data between comprehensive feature vectors of two commodity objects:

[0083] In order to facilitate the calculation of similarity data between two product objects, a recommended method is to use the Faiss architecture to construct a feature index matrix. In this process, the comprehensive feature vector is used to calculate the similarity, generate the similarity data between two product objects, and store it in the feature index matrix for subsequent calls.

[0084] In another alternative embodiment, the feature index matrix can be constructed by itself. Specifically, any known distance similarity algorithm such as the Euclidean distance algorithm, the Jaccard distance algorithm, the cosine similarity algorithm, etc. can be used to calculate the similarity between the two commodity objects based on the comprehensive feature vector. In a simplified case, according to the principle of the cosine similarity algorithm, each comprehensive feature vector can be expanded into a single row vector, and then the single row vectors of the two commodity objects are multiplied to obtain the corresponding similarity. The feature index matrix is ​​constructed as a square matrix, in which the row coordinates and the column coordinates correspond to the various commodity objects in a unified order, and then, for the two commodity objects, their similarity data is stored in the elements pointed to by the commodity objects pointed to by the row coordinates and the commodity objects pointed to by the column coordinates. When the similarity data of all commodity objects are stored in the feature index matrix, the construction of the feature index matrix is ​​completed, and it can be used for similarity query.

[0085] Step S1300: Obtain the visited product object in the user behavior message and determine it as the product object to be searched:

[0086] In combination with the exemplary application scenarios of the present application, when a consumer user on the site of a merchant instance is browsing the product objects on its website, the user behavior message of the user can be obtained and the user's behavior events can be learned by pre-installing the embedding code in the web page of the site in the user's terminal device or in the application that accesses the site, or by analyzing the web page visited by the user on the background server. When the product object that the user has visited is known through the behavior event, the background server can query the feature index matrix accordingly to query the corresponding similar product object for the product object.

[0087] The logic of triggering the backend server to query the feature index matrix can also be initiated when the user clicks on a specific entrance of the independent site's web page. For example, when the user enters the "Guess You Like" page of the independent site, it is triggered to match similar products based on a product object that the user has just visited.

[0088] In specific implementation, when a user visits a commodity object, a user behavior message can be constructed through the user's terminal device or the backend server. The user behavior message encapsulates the event of the user clicking on a commodity object, or other behavior events, such as ordering events, adding to a shopping cart events, and order payment events, so that the backend server can learn the commodity object that the user has visited based on the user behavior message. Then, it is determined as a commodity object to be searched, so as to start a query for similar commodity objects.

[0089] Step S1400: Query from the feature index matrix a plurality of target product objects that meet similarity matching conditions with the product object to be searched:

[0090] As mentioned above, there is similarity data between every two commodity objects in the feature index matrix, and its row coordinates and column coordinates correspond to the same commodity object, presenting a square matrix structure. It can be understood that each row vector (or column vector) includes multiple elements, and each element stores the similarity data of the commodity object corresponding to the row vector mapped to the commodity object pointed to by each column coordinate. In other words, it is only necessary to obtain the row vector (or column vector) corresponding to the commodity object to be searched, so as to obtain all commodity objects represented by similarity data with the commodity object to be searched.

[0091] Since the similarity data of the commodity objects in the row vector (or column vector) are different, in order to facilitate screening, each element of the row vector (or column vector) can be sorted in reverse order according to the size of the similarity data, so that the sorting between each similar correspondence presents regularity.

[0092] Then, according to the preset similarity matching conditions, several top-ranked product objects are selected from the sorted row vector (or column vector) as multiple target product objects that meet the similarity matching conditions with the product object to be searched, and these target product objects can be pushed to the user of the terminal device.

[0093] Among them, the similarity matching conditions can be flexibly determined by those skilled in the art based on the principles disclosed in this application. For example, the Top_K algorithm can be used, where K indicates the selection of the K elements with the greatest similarity as pointers to the target product object, and the specific value of K can be determined by those skilled in the art. For another example, according to the principle of proportion, the top 10% of the elements in the row vector (or column vector) can be selected as pointers to the target product object. It can be seen that in any case, the target product object can be obtained in the end.

[0094] After the target product object is pushed to the user, the user can see other product objects similar to the product object he has just visited in the graphical user interface of his web page or application. Since the targeted product object is the one he has just visited, and the similarity calculation principle in the background is determined based on the comprehensive feature vector obtained from the deep semantic interaction of the product images and summary texts between the two product objects, the user can be quickly provided with similar products he needs at the time, which can increase the user's click-through rate and purchase rate for similar product objects.

[0095] Through the description of this typical embodiment, it can be seen that after the present application obtains the image semantic features through deep interaction at the feature level of the product image of the product object, it is spliced ​​with the text semantic features of the summary text of the product object to form a comprehensive feature vector, and on this basis, the similarity data between the two product objects is calculated to construct a feature index matrix, and then the relevant target product objects that meet the similarity matching conditions can be queried from the feature index matrix according to the product object to be searched. Among them, the image semantic features are generated by extracting the semantic features of the multiple segmented graphics elements of the product image of the product object and then performing feature interaction. Therefore, the product image has fully interacted at the semantic feature level, so that the image semantic features obtained by representation learning of the product image, and the comprehensive feature vector obtained by splicing it with the text semantic features, realize good representation learning of the product image, and similarity calculation based on this can obtain more accurate similarity data, and further when performing product search based on this, the matching accuracy of similar product objects can be improved.

[0096] See also Figure 2 In a further embodiment, the step S1100, obtaining the comprehensive feature vector of each commodity object in the commodity database, comprises the following steps:

[0097] Step S1110: Call the pre-trained image feature extraction model to extract the image semantic features of the product images of each product object in the product database:

[0098] In this embodiment, following the idea of ​​the typical embodiment of the present application, a pre-trained DeiT model based on the ViT architecture is used as an image feature extraction model to perform image semantic features on the product images of each product object in the product database. Of course, the generation of the image semantic features is still generated by deep feature interaction based on multiple segmented graphics elements of the product image.

[0099] The product image adopts the default main image of the product object, which is the main image used to summarize the product object. It is usually used in the product object display list, or after entering the product object's details page, the default main image becomes the first image displayed in a prominent position.

[0100] Step S1120: Call the pre-trained text feature extraction model to extract the text semantic features of the summary text of each commodity object in the commodity database:

[0101] In this embodiment, a pre-trained Bert model is used as the text feature extraction model to extract the text semantic features of the summary text of each commodity object. The summary text, in this embodiment, can only select the title text of the commodity object. It can be understood that since the Bert model is also implemented based on the Transformer encoding architecture, it can also achieve deep semantic interaction of the features of the title file, fully consider the contextual information of the title text, and obtain the text semantic features.

[0102] Step S1130: perform feature splicing on the image semantic features and text semantic features of each commodity object to obtain the corresponding comprehensive feature vector:

[0103] At this point, for each commodity object, its image semantic features and text semantic features are obtained respectively, and the two features can be simply spliced. Of course, the two semantic features can be unified to the same scale first, and then spliced ​​into a single comprehensive feature vector to facilitate subsequent calculations. In addition, corresponding processing can also be performed later.

[0104] In this embodiment, by extracting semantic features from product images and summary texts respectively, the processing of information in the two modes is decoupled, and the two can be modeled separately, thereby improving the flexibility of the technical solution of the present application in the programming development stage.

[0105] See also Figure 3 In a specific embodiment, the step S1110, calling a pre-trained image feature extraction model to extract image semantic features of the product images of each product object in the product database, includes the following steps:

[0106] Step S1111: Segment the product image of each product object into a plurality of segmented graphic elements:

[0107] In order to realize the integration of semantic features within the product image, the product image may be preprocessed in advance. Usually, when the product image is uploaded to the e-commerce platform, it has been preprocessed accordingly, including scaling and image standardization, etc. If it has not been processed, it can also be preprocessed accordingly to meet the needs of this application. For this, those skilled in the art can flexibly implement it as needed.

[0108] Based on the standardized product image, it is further divided into multiple segmented graphics elements. These segmented graphics elements can be, for example, 16*16 pixels in size, so that these segmented graphics elements can be processed into sequence information in order and by row. In addition to obtaining its image information (Token Embedding), it is also convenient to obtain its corresponding position information (Position Embedding) and distinction information (Class Embedding).

[0109] Step S1112: for each segmented primitive, use the first image feature extraction model to extract the corresponding single image feature vector:

[0110] For each of the segmented primitives, a first image feature extraction model may be applied to perform preliminary feature extraction thereon.

[0111] In this embodiment, the first image feature extraction model uses an image feature extractor based on the Resnet network architecture to extract features from each segmented primitive. It can use a network architecture based on residual convolutional neural networks such as Resnet50, Resnet101, IResNet, etc., or EfficientNet, etc. In practice, the application of these architectures can achieve better results. Of course, the first image feature extraction model implemented by other excellent network architectures can also be used in this embodiment.

[0112] The first image feature extraction model performs feature extraction for each segmentation primitive respectively, so a single image feature vector can be obtained for each segmentation primitive, and multiple single image feature vectors can be obtained for the product image.

[0113] In the process of extracting image features from segmentation primitives by the first image feature extraction model, the segmentation primitives are input according to three channels, and the intermediate feature vectors of the three channels generated in the intermediate process can be pooled through an average pooling layer to obtain the single image feature vector.

[0114] Step S1113: Expand the single image feature vector into a single image embedding vector represented by a single row vector, and concatenate all the single image embedding vectors into a picture encoding vector, which carries the position information and distinguishing information of each segmentation primitive in the product picture:

[0115] In this step, the output of the first image feature extraction model is converted into a high-dimensional single-row vector, and the corresponding position information and distinguishing information of the corresponding segmentation primitives in the product image are superimposed in the high-dimensional single-row vector. Finally, these single-row vectors are spliced ​​into an image coding vector, which carries the position information of each segmentation primitive in the product image and the distinguishing information for distinguishing different segmentation primitives.

[0116] Specifically, the single image feature vector is a two-dimensional matrix. In order to meet the needs of the neural network model based on the multi-head attention mechanism of this application, it is necessary to reduce its dimension, specifically, to expand the two-dimensional matrix on a one-dimensional vector, that is, to sequentially splice the row vectors in the two-dimensional matrix to obtain a single row vector, that is, the single image embedding vector. Assuming that the single image feature vector is of N*D scale, the single image embedding vector obtained after its expansion becomes a scale of 1*(N*D). The row vector dimension (N*D) corresponding to this scale is pre-specified with the text feature vector in this application so that the two can be directly spliced ​​later.

[0117] In the specific process, each single image feature vector is finally expanded into the single row vector. On this basis, for the same product image, all its single image embedding vectors can be spliced ​​into a matrix of size M*(N*D) through a convolution layer, where M is the number of segmented graphics elements. This matrix is ​​the image encoding vector of the product image.

[0118] It should be pointed out that each segmentation primitive has a relative position relationship in the product image. In order to match the neural network model based on the multi-head attention mechanism adopted in this application, the position information of each segmentation primitive and the distinction information used to distinguish different primitives are further encoded, and then these codes are also added to the corresponding single image embedding vectors, so that the image encoding vector carries the corresponding position information and distinction information of each primitive.

[0119] Step S1114: Use a second image feature extraction model based on a multi-head attention mechanism to perform feature interaction on the image encoding vector to obtain image semantic features that implement deep semantic interaction of the single image feature vector:

[0120] As mentioned above, in this application, it is recommended to adopt a neural network model based on a multi-head attention mechanism to implement it. Therefore, this embodiment constructs a second image feature extraction model with the help of the specific architecture of ViT, and the encoding process of ViT actually inherits the inherent architecture of Transformer. Therefore, the encoder used in this embodiment is the encoder in the inherent architecture of Transformer.

[0121] The Transformer inherent architecture includes an encoding path consisting of multiple encoders, each of which uses the same structure, including a self-attention layer and a multi-layer perceptron (MLP), and the self-attention layer applies a multi-head attention mechanism.

[0122] Accordingly, the self-attention layer of each encoder performs deep feature interaction on the image coding vector it receives based on the algorithm inherent in the multi-head attention mechanism, and then passes the result obtained after the feature interaction to its multi-layer perceptron for feature extraction, and then the multi-layer perceptron provides the feature extraction result to the next encoder, and so on. The self-attention layer of the subsequent encoder takes the output of the multi-layer perceptron of the previous encoder as input and processes it according to the same logic. In each encoder, the output result of the self-attention layer is extracted and output by the multi-layer perceptron at a high level, until the output obtained by the last encoder can be used as the semantic feature of the image.

[0123] This embodiment pre-processes the product image, divides it into multiple segmented graph elements, extracts image features from each segmented graph element and converts them into a single-image embedding vector represented by a single-row vector, integrates the corresponding position information and distinction information of each segmented graph element in the single-image embedding vector, and then converts these single-image embedding vectors into image coding vectors. At this point, the image feature information of a product image is converted into a format with sequence features, and the neural network model based on the multi-head attention mechanism is an excellent model for processing sequence information. Therefore, the problems that previously relied on convolutional neural network processing are converted into problems suitable for RNN processing, which not only enables the advantages of the multi-head attention mechanism to be reflected in image recognition, but also makes it easier to achieve deep interaction within the image feature information.

[0124] See also Figure 4 In a specific embodiment, the step S1120, calling a pre-trained text feature extraction model to extract text semantic features of the summary text of each commodity object in the commodity database, includes the following steps:

[0125] Step S1121: Segment the summary text of each product object:

[0126] To extract features from the summary text of the commodity object, the summary text may be first segmented, and various mature pre-trained models may be used for the segmentation, including but not limited to N-gram, WordPrice, Word2Vec, and the like.

[0127] Step S1122: Encode according to the word segmentation to obtain the corresponding text embedding vector, position embedding vector and sentence embedding vector:

[0128] On the basis of word segmentation, the preset dictionary combined with one-hot encoding can be used to vectorize the word segmentation results. According to the preset dictionary, each word segment is converted into a corresponding text embedding vector (Token Embedding). Similarly, the position of the word segment is encoded to obtain a position embedding vector (Position Embedding), and the sentence embedding vector (Segment Embedding) that distinguishes different sentences is encoded. Then, these vectors are summed and fused and passed to the text feature extraction model for deep semantic information extraction, so as to obtain the corresponding text semantic features.

[0129] Step S1123: Use a text feature extraction model to perform feature extraction based on the text embedding vector, position embedding vector, and sentence embedding vector to obtain corresponding text semantic features:

[0130] In this embodiment, it is recommended to use a pre-trained Bert model as a text feature extraction model to adapt to the characteristics of the Bert model, and take the three encoded vectors as input, namely, the text embedding vector (Token Embedding) used to represent each word segment, the position embedding vector (PositionEmbedding) used to represent the position information of each word segment, and the sentence embedding vector (Segment Embedding) used to represent the distinction information between sentences. Those skilled in the art can implement this flexibly.

[0131] It should be noted that the text semantic features output by the Bert model are also a two-dimensional matrix, which is unified into the same scale as M*(N*D) in the previous embodiment so as to achieve splicing with the image semantic features.

[0132] In this embodiment, the encoding process of the summary text of the commodity object is further disclosed, and the use of the Bert model is recommended to realize text semantic feature extraction, making full use of the excellent achievements in the prior art to make the encoding efficiency higher, and the obtained text semantic features have a better understanding of the deep semantics of the summary text.

[0133] See also Figure 5 In a specific embodiment, the step S1114, using a second image feature extraction model based on a multi-head attention mechanism to perform feature interaction on the image encoding vector to obtain image semantic features that implement deep semantic interaction, includes the following steps performed by at least one encoder in the second image feature extraction model:

[0134] Step S2100: construct a query vector, a key vector and a value vector corresponding to each row vector in the currently input picture encoding vector:

[0135] Still taking the inherent architecture of Transformer as an example, when the self-attention layer in each encoder receives an input of a picture coding vector, for example, the first encoder takes the picture coding vector as its input, and the subsequent picture coding vectors take the output of the multi-layer perceptron of the previous encoder as input, and obtain the input picture coding vector. According to the inherent architectural principles, the corresponding query vector (Q), key vector (K), and value vector (V) can be constructed for each row vector therein. Constructing Q, K, and V vectors is the basic principle of the multi-head attention mechanism and the basic means to achieve deep interaction at the feature level.

[0136] Step S2200: Perform a weighted sum of the value vectors corresponding to all row vectors, where the weight corresponding to each value vector is the normalized score of the dot product of the key vector corresponding to it and the query vector corresponding to other row vectors:

[0137] According to the inherent algorithm of the self-attention mechanism, for each row vector, the dot product of its key vector and the query vector corresponding to other row vectors is first calculated, and then the dot product result is normalized using the Softmax function so that the sum of multiple normalized dot products is 1, and the normalized value is used as the weight corresponding to each row vector. On this basis, the value vectors corresponding to all row vectors are multiplied and added with their corresponding weights to achieve weighted summation and obtain a sum matrix after weighted summation.

[0138] Step S2300: multiply the weighted sum result by the conversion weight matrix to obtain its corresponding intermediate information matrix:

[0139] In order to standardize the output scale, the weighted summation result, i.e., the summation matrix, is multiplied by a conversion weight matrix to obtain an intermediate information matrix of the same scale as the original image and text splicing vector. It can be understood that after operations between Q, K, and V, deep semantic interaction between each row vector has been realized in the intermediate information matrix.

[0140] Step S2400: extract features from the intermediate information matrix through a multi-layer perceptron to obtain picture semantic features as picture coding vectors required by the next encoder, or directly output in the last encoder.

[0141] The intermediate information matrix is ​​output to the multilayer perceptron of the feedforward neural network as an encoder for high-level semantic feature extraction to obtain image semantic features that realize semantic interaction. Then, it can be output to the next encoder. For the next encoder, the image semantic features output by the previous encoder are regarded as image encoding vectors and the same operation is performed. If the encoder that outputs the image semantic features is the last encoder, the image semantic features are directly output for splicing with the text semantic features.

[0142] In this embodiment, it is disclosed in detail how to combine the multi-head attention mechanism to achieve deep interaction between the features of multiple segmented graphics elements of the product image of the product object of the present application and realize deep semantic fusion. It can be seen that the present application effectively utilizes the existing technical architecture and obtains the desired results. It plays a key contribution to the full interactive fusion of the features of the product image and lays a solid technical foundation for realizing similarity calculation based on the image and text information of the product object.

[0143] See also Figure 6 In a further embodiment, step S1400, searching from the feature index matrix for multiple target commodity objects that meet similarity matching conditions with the commodity object to be searched, comprises the following steps:

[0144] Step S1410: Query all commodity objects that have similarity data mapping relationships with the commodity object to be searched from the feature index matrix:

[0145] As described in the previous related embodiments, the feature index matrix stores the similarity data between two commodity objects, and for each commodity object to be searched, there are multiple commodity objects with similarity data mapping relationships therewith, and according to the structure of the matrix, all these commodity objects with similarity associations can be obtained by extracting the row vector (or column vector) of the commodity object to be searched, and it can be seen that the retrieval effect is very efficient. In the case of applying the Faiss framework, the interface extracted by the Faiss framework can be directly used to implement this query.

[0146] Step S1420: Select a predetermined number of commodity objects with the largest similarity data as target commodity objects that meet the similarity data matching condition:

[0147] In this embodiment, the matching condition satisfying the similarity data matching is set to filter out the first several, for example, ten target commodity objects from the row vector, and therefore, the Top_K algorithm can be applied for selection, where K is correspondingly set to 10. Of course, before screening, all elements in the row vector need to be sorted in reverse order according to the similarity data, which should be understood by those skilled in the art.

[0148] Step S1430: construct a search result list containing the target product object:

[0149] Furthermore, considering the need to push corresponding target product objects to the user who triggered the user behavior message, these target product objects can be constructed as a search result list. This search result list can be further formatted, for example, by constructing summary information of each target product object, providing product images, title texts, price information, etc. of the target product object. In this regard, technical personnel in this field can flexibly implement it.

[0150] Step S1440: Push the search result list to respond to the commodity object search request:

[0151] To adapt to the different operations of users of the terminal device, the search result list can be pushed to them in different ways. For example, when the user enters the "Guess You Like" page of the aforementioned example, a similar product search request will be initiated to the background server. The background server will respond and execute the relevant steps of this application to obtain the corresponding search result list, and then push the search result list directly to the user's terminal device for display, in order to answer the corresponding similar product search request.

[0152] In another implementation, the user does not actively enter a dedicated page, but when he subsequently enters a page loaded with an advertisement column, the backend server obtains the search result list when providing the page, and inserts the search result list into the page for display.

[0153] This embodiment further improves the interaction process between the terminal device and the background server of this application, ensuring that users can timely obtain similar product objects that are suitable for their immediate needs based on their own user behavior messages, increase the probability of similar product objects being clicked, meet user needs, and improve the user's experience of perceiving the accuracy of matching similar product objects.

[0154] See also Figure 7 A commodity object search device provided to meet one of the purposes of the present application is a functional embodiment of the commodity object search method of the present application, and the device includes: a feature summary module 1100, a matrix construction module 1200, a search start module 1300, and a matching feedback module 1400, wherein the feature summary module 1100 is used to obtain a comprehensive feature vector of each commodity object in a commodity database, and the comprehensive feature vector includes a picture semantic feature and a text semantic feature of the commodity object; the picture semantic feature is generated by extracting the semantic features of multiple segmented graphics elements of the commodity image of the commodity object and then performing feature interaction; the matrix construction module 1200 is used to construct a feature index matrix for storing similarity data between the comprehensive feature vectors of two commodity objects; the search start module 1300 is used to obtain the visited commodity objects in the user behavior message and determine them as the commodity objects to be searched; the matching feedback module 1400 is used to query multiple target commodity objects that meet the similarity matching conditions with the commodity objects to be searched from the feature index matrix.

[0155] In a further embodiment, the feature aggregation module 1100 includes: an image extraction submodule, which is used to call a pre-trained image feature extraction model to extract image semantic features of product images of each product object in the product database; a text extraction submodule, which is used to call a pre-trained text feature extraction model to extract text semantic features of summary texts of each product object in the product database; and a feature splicing submodule, which is used to perform feature splicing on the image semantic features and text semantic features of each product object to obtain its corresponding comprehensive feature vector.

[0156] In a specific embodiment, the image extraction submodule includes: a primitive segmentation unit, which is used to divide the product image of each product object into multiple segmented primitives; a primitive extraction unit, which is used to extract the corresponding single image feature vector for each segmented primitive using a first image feature extraction model; a splicing encoding unit, which is used to expand the single image feature vector into a single image embedding vector represented by a single row vector, and splice all single image embedding vectors into an image encoding vector, which carries the position information and distinguishing information of each segmented primitive in the product image; a primitive interaction unit, which is used to perform feature interaction on the image encoding vector using a second image feature extraction model based on a multi-head attention mechanism, and obtain image semantic features that realize deep semantic interaction of the single image feature vector.

[0157] In a specific embodiment, the text extraction submodule includes: a text segmentation unit, which is used to segment the summary text of each commodity object; a text encoding unit, which is used to encode according to the segmentation to obtain corresponding text embedding vectors, position embedding vectors and sentence embedding vectors; a text extraction unit, which is used to use a text feature extraction model to perform feature extraction according to the text embedding vectors, position embedding vectors and sentence embedding vectors to obtain corresponding text semantic features.

[0158] In a specific embodiment, the encoder in the second image feature extraction model adopted in the primitive interaction unit includes: a vector mapping module, which is used to construct a query vector, a key vector and a value vector corresponding to each row vector in the currently input picture coding vector; an interactive operation module, which is used to perform weighted summation on the value vectors corresponding to all row vectors, and the corresponding weight of each value vector is the normalized score of the dot product of the corresponding key vector and the query vector corresponding to other row vectors; a weight conversion module, which is used to multiply the weighted summation result by the conversion weight matrix to obtain the corresponding intermediate information matrix; a perceptual extraction module, which is used to perform feature extraction on the intermediate information matrix through a multi-layer perceptron to obtain picture semantic features as the picture coding vector required by the next encoder, or directly output in the last encoder.

[0159] In a preferred embodiment, the second image feature extraction model is a DeiT model based on the Vision Transformer architecture, which is pre-trained to a convergence state.

[0160] In a further embodiment, the matching feedback module 1400 includes: a feature query submodule, used to query all product objects that have a similarity data mapping relationship with the product object to be searched from the feature index matrix; a target optimization submodule, used to select a predetermined number of product objects with the largest similarity data as target product objects that meet the similarity data matching conditions; a list construction submodule, used to construct a search result list containing the target product object; and a result push submodule, used to push the search result list in response to the product object search request.

[0161] In order to solve the above technical problems, the present application also provides a computer device. Figure 8 As shown, a schematic diagram of the internal structure of a computer device. The computer device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. Among them, the computer-readable storage medium of the computer device stores an operating system, a database, and computer-readable instructions. The database may store a control information sequence. When the computer-readable instructions are executed by the processor, the processor may implement a commodity object search method. The processor of the computer device is used to provide computing and control capabilities to support the operation of the entire computer device. The memory of the computer device may store computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor may execute the commodity object search method of the present application. The network interface of the computer device is used to connect and communicate with a terminal. Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0162] In this embodiment, the processor is used to execute Figure 7 The memory stores the program code and various data required to execute the above modules or submodules. The network interface is used to transmit data between user terminals or servers. The memory in this embodiment stores the program code and data required to execute all modules / submodules in the commodity object search device of this application, and the server can call the program code and data of the server to execute the functions of all submodules.

[0163] The present application also provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the commodity object search method of any embodiment of the present application.

[0164] The present application also provides a computer program product, including a computer program / instruction, which implements the steps of the method described in any embodiment of the present application when executed by one or more processors.

[0165] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments of the present application can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a computer-readable storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0166] In summary, the present application can perform deep semantic interaction on the product images and summary texts of product objects at the feature level, deepen the fusion of the two types of information, and calculate similarity based on this. The similarity data can be used to match similar products, and similar product objects can be obtained with higher accuracy. It has a strong foundation and broad application prospects.

[0167] Those skilled in the art will appreciate that the various operations, methods, steps, measures, and schemes in the processes discussed in this application may be alternated, altered, combined, or deleted. Further, other steps, measures, and schemes in the various operations, methods, and processes discussed in this application may also be alternated, altered, rearranged, decomposed, combined, or deleted. Further, the steps, measures, and schemes in the prior art that are similar to those disclosed in this application may also be alternated, altered, rearranged, decomposed, combined, or deleted.

[0168] The above description is only a partial implementation method of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A commodity object search method, characterized in that: The steps include: Obtaining a comprehensive feature vector of each commodity object in a commodity database includes: calling a pre-trained image feature extraction model to extract the image semantic features of the commodity images of each commodity object in the commodity database; calling a pre-trained text feature extraction model to extract the text semantic features of the summary text of each commodity object in the commodity database; performing feature splicing on the image semantic features and text semantic features of each commodity object to obtain its corresponding comprehensive feature vector, wherein the comprehensive feature vector includes the image semantic features and text semantic features of the commodity object; the image semantic features are generated by extracting the semantic features of the image primitives of a plurality of segmented image primitives of the commodity image of the commodity object and then performing feature interaction; wherein calling a pre-trained image feature extraction model to extract the image semantic features of the commodity image of each commodity object and then performing feature interaction on the image semantic features of the commodity image of each commodity object; The feature extraction model extracts the image semantic features of the product images of each product object in the product database, including the following steps: dividing the product image of each product object into a plurality of segmented graph elements; for each segmented graph element, using the first image feature extraction model to extract the corresponding single-image feature vector; expanding the single-image feature vector into a single-image embedding vector represented by a single-row vector, and splicing all the single-image embedding vectors into an image encoding vector, which carries the position information and distinguishing information of each segmented graph element in the product image; using the second image feature extraction model based on the multi-head attention mechanism to perform feature interaction on the image encoding vector, and obtaining the image semantic features that realize the deep semantic interaction of the single-image feature vector; Construct a feature index matrix to store the similarity data between the comprehensive feature vectors of two commodity objects; Obtain the visited product object in the user behavior message and determine it as the product object to be searched; A plurality of target commodity objects that meet similarity matching conditions with the commodity object to be searched are queried from the feature index matrix.

2. The commodity object search method according to claim 1, characterized in that: Calling the pre-trained text feature extraction model to extract the text semantic features of the summary text of each product object in the product database includes the following steps: Segment the summary text of each product object; Encoding is performed according to the word segmentation to obtain corresponding text embedding vectors, position embedding vectors, and sentence embedding vectors; A text feature extraction model is used to extract features based on the text embedding vector, position embedding vector, and sentence embedding vector to obtain corresponding text semantic features.

3. The commodity object search method according to claim 1, characterized in that: The second image feature extraction model based on the multi-head attention mechanism is used to perform feature interaction on the image encoding vector to obtain image semantic features that realize deep semantic interaction, including the following steps performed by at least one level encoder in the second image feature extraction model: Construct a query vector, a key vector and a value vector corresponding to each row vector in the current input image encoding vector; Perform a weighted summation of the value vectors corresponding to all row vectors, where the weight corresponding to each value vector is the normalized score of the dot product of its corresponding key vector and the query vector corresponding to other row vectors; Multiply the weighted sum result by the conversion weight matrix to obtain its corresponding intermediate information matrix; The intermediate information matrix is ​​subjected to feature extraction by a multi-layer perceptron to obtain picture semantic features, which are used as picture coding vectors required by the next encoder or directly output in the last encoder.

4. The commodity object search method according to any one of claims 1 to 3, characterized in that: The second image feature extraction model is a De i T model based on the Vision Transformer architecture, which is pre-trained to a convergence state.

5. The commodity object search method according to any one of claims 1 to 3, characterized in that: Querying multiple target product objects that meet similarity matching conditions with the product object to be searched from the feature index matrix includes the following steps: Querying all commodity objects having similarity data mapping relationships with the commodity object to be searched from the feature index matrix; Selecting a predetermined number of commodity objects with the largest similarity data as target commodity objects that meet the similarity data matching condition; Constructing a search result list containing the target product object; The search result list is pushed in response to the commodity object search request.

6. A computer device comprising a central processing unit and a memory, characterized in that: The central processing unit is used to call and run the computer program stored in the memory to execute the steps of the method according to any one of claims 1 to 5.

7. A computer-readable storage medium, characterized in that: It stores a computer program implemented according to the method described in any one of claims 1 to 5 in the form of computer-readable instructions, and when the computer program is called and executed by a computer, the steps included in the corresponding method are executed.

8. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method described in any one of claims 1 to 5 are implemented.

Images