Object detection method, product detection method, and similarity prediction model training method

Abstract

Embodiments of the present specification provide an object detection method, a commodity detection method and a similarity prediction model training method, wherein the object detection method comprises: obtaining description texts and object images of at least two to-be-detected objects; performing feature extraction on the description texts and the object images of each to-be-detected object to obtain text features, image features and image-text features of each to-be-detected object; determining a similarity between a first to-be-detected object and a second to-be-detected object based on the text features, the image features and the image-text features of the first to-be-detected object and the image-text features of the second to-be-detected object, wherein the first to-be-detected object and the second to-be-detected object are any two to-be-detected objects; and determining a target object in the at least two to-be-detected objects according to the similarity and processing the target object. The accuracy of the similarity can be effectively improved, and the efficiency of object detection is improved.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of computer technology, and in particular to an object detection method. Background Technology

[0002] The development of computer and internet technologies has spurred the growth of the e-commerce industry, leading to an increasing number of people choosing online shopping. However, e-commerce platforms often feature products that are essentially the same but have different descriptions (such as titles, images, and keywords).

[0003] For users, the desire is for e-commerce platforms to offer a feature recommending similar products, allowing them to quickly find comparable items when interested in a particular product, and compare prices and performance. For merchants, in order to gain more exposure and transactions, they may repeatedly display the same product using different images and titles, ensuring users see their products when searching with different keywords. However, this duplicate listing behavior severely impacts platform health and user experience. Therefore, detecting similar products is particularly important in the e-commerce search and recommendation field. Summary of the Invention

[0004] In view of the above, embodiments of this specification provide an object detection method. One or more embodiments of this specification also relate to a product detection method, a similarity prediction model training method, an object detection device, a product detection device, a similarity prediction model training device, a computing device, a computer-readable storage medium, and a computer program, to address the technical deficiencies existing in the prior art.

[0005] According to a first aspect of the embodiments of this specification, an object detection method is provided, comprising:

[0006] Obtain descriptive text and object images for at least two objects to be detected;

[0007] Feature extraction is performed on the descriptive text and object image of each object to be detected to obtain the text features, image features and image-text features of each object to be detected.

[0008] Based on the text features, image features, and image-text features of the first object to be detected, and the image-text features of the second object to be detected, the similarity between the first object to be detected and the second object to be detected is determined, wherein the first object to be detected and the second object to be detected are any two objects to be detected.

[0009] The target object is determined from the at least two objects to be detected based on the similarity, and the target object is then processed.

[0010] According to a second aspect of the embodiments of this specification, a product detection method is provided, comprising:

[0011] Retrieve the product title and product image of the queried product and at least one candidate product;

[0012] Feature extraction is performed on the product title and the product image to obtain the text features, image features, and image-text features of the queried product and each candidate product;

[0013] The similarity between the query product and each candidate product is determined by calculating the similarity between the text features, image features, and image-text features of the query product and the image-text features of each candidate product.

[0014] The target product is determined from the at least one candidate product based on the similarity, and the target product is then processed.

[0015] According to a third aspect of the embodiments of this specification, a method for training a similarity prediction model is provided, comprising:

[0016] Obtain a training set, wherein the training set contains at least one sample pair, the sample pair comprising sample text and sample image of two sample objects;

[0017] Input the sample text and sample image of each sample object in each sample pair into the feature extraction layer of the model to be trained to obtain the predicted text features, predicted image features and predicted image-text features of each sample object.

[0018] For any sample pair corresponding to the first sample object and the second sample object, the predicted text features, predicted image features, and predicted image-text features of the first sample object and the predicted image-text features of the second sample object are input into the similarity determination layer of the model to be trained to obtain the predicted similarity between the first sample object and the second sample object.

[0019] The total loss value is determined based on the predicted similarity, and the model parameters of the feature extraction layer and the similarity determination layer are adjusted based on the total loss value. The step of obtaining the training set is then performed, and training is stopped when the preset training stopping condition is met, thereby obtaining the similarity prediction model.

[0020] According to a fourth aspect of the embodiments of this specification, an object detection apparatus is provided, comprising:

[0021] The first acquisition module is configured to acquire descriptive text and object images of at least two objects to be detected.

[0022] The first extraction module is configured to extract features from the descriptive text and object image of each object to be detected, and obtain the text features, image features and image-text features of each object to be detected.

[0023] The first similarity determination module is configured to determine the similarity between the first object to be detected and the second object to be detected based on the text features, image features, and image-text features of the first object to be detected and the image-text features of the second object to be detected, wherein the first object to be detected and the second object to be detected are any two objects to be detected;

[0024] The target object determination module is configured to determine the target object among the at least two objects to be detected based on the similarity, and to process the target object.

[0025] According to a fifth aspect of the embodiments of this specification, a commodity detection apparatus is provided, comprising:

[0026] The second acquisition module is configured to acquire the product title and product image of the queried product and at least one candidate product;

[0027] The second extraction module is configured to extract features from the product title and the product image to obtain the text features, image features, and image-text features of the queried product and each candidate product;

[0028] The second similarity determination module is configured to calculate the similarity between the text features, image features, and image-text features of the queried product and the image-text features of each candidate product, thereby determining the similarity between the queried product and each candidate product.

[0029] The target product determination module is configured to determine the target product among the at least one candidate products based on the similarity, and to process the target product.

[0030] According to a sixth aspect of the embodiments of this specification, a similarity prediction model training apparatus is provided, comprising:

[0031] The third acquisition module is configured to acquire a training set, wherein the training set contains at least one sample pair, the sample pair including sample text and sample image of two sample objects;

[0032] The first input module is configured to input the sample text and sample image of each sample object in each sample pair into the feature extraction layer of the model to be trained, so as to obtain the predicted text features, predicted image features and predicted image-text features of each sample object.

[0033] The second input module is configured to, for any sample pair corresponding to a first sample object and a second sample object, input the predicted text features, predicted image features, and predicted image-text features of the first sample object and the predicted image-text features of the second sample object into the similarity determination layer of the model to be trained, so as to obtain the predicted similarity between the first sample object and the second sample object.

[0034] The adjustment module is configured to determine the total loss value based on the predicted similarity, and adjust the model parameters of the feature extraction layer and the similarity determination layer based on the total loss value, continue to execute the step of obtaining the training set, and stop training when the preset training stopping condition is reached, thereby obtaining the similarity prediction model.

[0035] According to a seventh aspect of the embodiments of this specification, a computing device is provided, comprising:

[0036] Memory and processor;

[0037] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, they implement the steps of the above-described object detection method, product detection method, or similarity prediction model training method.

[0038] According to an eighth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores computer-executable instructions, which, when executed by a processor, implement the steps of the object detection method, the product detection method, or the similarity prediction model training method described above.

[0039] According to a ninth aspect of the embodiments of this specification, a computer program is provided, wherein when the computer program is executed in a computer, it causes the computer to perform the steps of the object detection method, the product detection method, or the similarity prediction model training method described above.

[0040] This specification provides an object detection method that acquires descriptive text and object images of at least two objects to be detected; extracts features from the descriptive text and object images of each object to obtain text features, image features, and image-text features for each object; determines the similarity between the first and second objects to be detected based on the text features, image features, and image-text features of the first object to be detected, and the image-text features of the second object to be detected, wherein the first and second objects to be detected are any two objects; determines a target object among the at least two objects to be detected based on the similarity, and processes the target object. By extracting features from the descriptive text and object images of the objects to be detected, obtaining the text features, image features, and image-text features representing the text and image, and then calculating the similarity between the objects to be detected based on the text features, image features, and image-text features, the accuracy of the similarity can be effectively improved, thereby improving the efficiency of object detection. Attached Figure Description

[0041] Figure 1AThis is a flowchart of an object detection system provided in one embodiment of this specification.

[0042] Figure 1B This is a schematic diagram of a system architecture applied in a product recommendation scenario provided by one embodiment of this specification;

[0043] Figure 2 This is a flowchart of an object detection method provided in one embodiment of this specification;

[0044] Figure 3A This is a schematic diagram of the structure of a similarity prediction model in an object detection method provided in one embodiment of this specification;

[0045] Figure 3B This is a schematic diagram illustrating the determination of the total loss value in an object detection method provided in one embodiment of this specification;

[0046] Figure 3C This is a schematic diagram illustrating the determination of the total loss value in an object detection method provided in another embodiment of this specification;

[0047] Figure 3D This is a schematic diagram illustrating the effect of constructing samples in an object detection method provided in one embodiment of this specification;

[0048] Figure 3E This is a heterogeneous graph in an object detection method provided in one embodiment of this specification;

[0049] Figure 3F This is a schematic diagram illustrating the effect of applying an object detection method provided in one embodiment of this specification to a search engine;

[0050] Figure 4 This is a flowchart of a product testing method provided in one embodiment of this specification;

[0051] Figure 5 This is a flowchart illustrating a similarity prediction model training method provided in one embodiment of this specification;

[0052] Figure 6 This is a schematic diagram of the structure of an object detection device provided in one embodiment of this specification;

[0053] Figure 7 This is a schematic diagram of the structure of a commodity detection device provided in one embodiment of this specification;

[0054] Figure 8 This is a schematic diagram of the structure of a similarity prediction model training device provided in one embodiment of this specification;

[0055] Figure 9This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0056] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0057] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0058] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0059] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0060] Same Style Products Retrieval (SSPR) refers to identifying a group of products in an e-commerce system that are essentially the same but contain different titles, images, and keywords.

[0061] Multimodal refers to different modalities of data such as images, text, and speech. A model that can process data from these modalities simultaneously and generate a representation that includes information from these modalities is said to be a multimodal model.

[0062] Visual-language representation refers to the use of deep learning methods to numerically represent images and text of products, facilitating data processing in numerical dimensions.

[0063] Visual-Language Pre-training (VLP) models refer to pre-training methods that use basic self-attention mechanisms and Bidirectional Encoder Representation from Transformers (BERT) to process text and images in a unified manner. The goal is to align the representations of images and text, which belong to two different modalities.

[0064] The image-text matching model (Fashionbert) is a VLP model suitable for e-commerce vector recall and is widely used in the multimodal representation of a large number of search systems.

[0065] This specification provides an object detection method, and also relates to a product detection method, a similarity prediction model training method, an object detection device, a product detection device, a similarity prediction model training device, a computing device, a computer-readable storage medium, and a computer program, which will be described in detail in the following embodiments.

[0066] The development of computer and internet technologies has spurred the growth of the e-commerce industry, leading to an increasing number of people choosing online shopping. However, e-commerce platforms often feature products that are essentially the same but have different descriptions (such as titles, images, and keywords).

[0067] For users, the desire is for e-commerce platforms to offer a feature recommending similar products, allowing them to quickly find comparable items when interested in a particular product, and compare prices and performance. For merchants, in order to gain more exposure and transactions, they may repeatedly display the same product using different images and titles, ensuring users see their products when searching with different keywords. However, this duplicate listing behavior severely impacts platform health and user experience. Therefore, detecting similar products is particularly important in the e-commerce search and recommendation field.

[0068] In existing technologies, there are two main methods for detecting identical products: one is to use an image-to-image matching model to score the similarity of images of all candidate products, and then select products with scores greater than a certain threshold as identical products. This method mainly considers visual attributes such as shape, style, and color of the products, while ignoring textual attributes contained in the product title and keywords, such as specific styles, materials, and brands. This image-to-image matching model is generally ineffective for products in industries where images are not so important (such as machinery, hardware tools, and electronic components). The other method is to supplement the first method with additional text-to-text matching models to further filter products with irrelevant attributes. However, merchants may add irrelevant keywords to the title to gain more exposure, which can mislead the model to make incorrect judgments.

[0069] Therefore, this specification provides an object detection method that acquires descriptive text and object images of at least two objects to be detected; extracts features from the descriptive text and object images of each object to obtain text features, image features, and image-text features of each object; determines the similarity between the first object and the second object based on the text features, image features, and image-text features of the first object and the second object, wherein the first object and the second object are any two objects to be detected; determines the target object among the at least two objects to be detected based on the similarity, and processes the target object. By extracting features from the descriptive text and object images of the objects to be detected, obtaining the text features, image features, and image-text features representing the text and image, and then calculating the similarity between the objects to be detected based on the text features, image features, and image-text features, the accuracy of the similarity can be effectively improved, thereby improving the efficiency of object detection.

[0070] See Figure 1A , Figure 1A A flowchart illustrating an object detection process in an object detection system according to an embodiment of this specification is shown. Figure 1A As shown, the object detection system includes a client and a server.

[0071] The client is configured to send at least two descriptive texts and object images of the objects to be detected to the server;

[0072] The server is configured to acquire descriptive text and object images of at least two objects to be detected; extract features from the descriptive text and object images of each object to obtain text features, image features, and image-text features of each object; determine the similarity between the first object to be detected and the second object to be detected based on the text features, image features, and image-text features of the first object to be detected and the image-text features of the second object to be detected, wherein the first object to be detected and the second object to be detected are any two objects to be detected; determine the target object among the at least two objects to be detected based on the similarity, and process the target object.

[0073] In practical applications, the server is further configured to feed back the processing results of the target object to the client.

[0074] This specification provides an object detection method that acquires descriptive text and object images of at least two objects to be detected; extracts features from the descriptive text and object images of each object to obtain text features, image features, and image-text features for each object; determines the similarity between the first and second objects to be detected based on the text features, image features, and image-text features of the first object to be detected, and the image-text features of the second object to be detected, wherein the first and second objects to be detected are any two objects; determines a target object among the at least two objects to be detected based on the similarity, and processes the target object. By extracting features from the descriptive text and object images of the objects to be detected, obtaining the text features, image features, and image-text features representing the text and image, and then calculating the similarity between the objects to be detected based on the text features, image features, and image-text features, the accuracy of the similarity can be effectively improved, thereby improving the efficiency of object detection.

[0075] The embodiments provided in this specification can be applied to various recommendation scenarios, such as product recommendations, restaurant recommendations, user recommendations, and partner recommendations. In a practical application, it is particularly suitable for product recommendation scenarios where similar products are searched based on images or keywords, i.e., similar product retrieval scenarios. Similar product retrieval, as a basic requirement, is widely used in two tasks: recommending similar products from different merchants and deduplicating similar products from the same merchant.

[0076] See Figure 1B , Figure 1BThis specification illustrates a system architecture diagram for a product recommendation scenario based on an embodiment of this specification. The system may include a server 102 and multiple clients 104. The multiple clients 104 can establish communication connections through the server 102. In the product recommendation scenario, the server 102 is used to provide object (product) detection services among the multiple clients 104. The multiple clients 104 can act as either senders or receivers, and real-time processing is achieved through the server 102.

[0077] Users can interact with server 102 through client 104 to receive data sent by other clients 104, or send data to other clients 104, etc. In the product recommendation scenario, users can send product keywords or images to server 102 through client 104. Server 104 generates the processing result of the target object (product) based on the product keywords or images, and pushes the processing result to other clients that have established communication.

[0078] In this setup, client 104 and server 102 establish a connection via a network. The network provides the medium for the communication link between the client and the server. The network can include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0079] Client 104 can be a browser, an app (application), a web application such as an H5 (HyperText Markup Language 5) application, a lightweight application (also known as a mini-program), or a cloud application. Client 104 can be based on the corresponding service SDK (Software Development Kit) provided by the server, such as RTCSDK. Client 104 can be deployed on electronic devices, requiring the device to run or certain apps on the device to function. Electronic devices may have displays and support information browsing, such as personal mobile terminals like smartphones, tablets, and personal computers. Various other types of applications can also be configured on electronic devices, such as human-computer interaction applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, and social media platforms.

[0080] Server 102 may include servers that provide various services, such as servers that provide object detection services for multiple clients, servers that support the models used on the clients for background training, and servers that process keywords or images of products sent by the clients.

[0081] It should be noted that server 102 can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. The server can also be a server in a distributed system, or a server integrated with blockchain. The server can also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0082] It is worth noting that the object detection method provided in the embodiments of this specification is generally executed by the server. However, in other embodiments of this specification, the client may also have similar functionality to the server, thereby executing the object detection method provided in the embodiments of this specification. In other embodiments, the object detection method provided in the embodiments of this specification may also be executed jointly by the client and the server.

[0083] See Figure 2 , Figure 2 A flowchart of an object detection method provided in one embodiment of this specification is shown, which specifically includes the following steps.

[0084] Step 202: Obtain description text and object images for at least two objects to be detected.

[0085] Specifically, "object" refers to the objects that the recommendation system can recommend, such as products, restaurants, music, merchants, and videos; "object to be detected" refers to the objects that need to be detected, such as detecting whether there are similar objects; "descriptive text" refers to the text that describes the attributes (color, size, material, style, etc.), structure, etc. of the object to be detected, such as topic, summary, keywords, and title; and "object image" refers to the pictures and photos corresponding to the object to be detected.

[0086] In practical applications, at least two objects to be detected can be identified first. For example, the client sends the identifiers (names, images, etc.) or links of each object to the server, and the server identifies each object based on the identifiers or links. Alternatively, the server receives a search term from the client and then searches the object database for at least two objects associated with the search term as objects to be detected. Further, based on each object to be detected, descriptive text and object images are obtained; for example, descriptive text and object images can be extracted from the display page corresponding to the object.

[0087] For example, a user enters "red summer dress" in the search box of a shopping platform, and then the executing entity searches for products related to "red summer dress" in the product database, i.e., the objects to be detected. Then, it retrieves the product title (description text) and the homepage product image (object image) from the details page of each product.

[0088] Step 204: Extract features from the descriptive text and object image of each object to be detected, and obtain the text features, image features, and image-text features of each object to be detected.

[0089] Specifically, features refer to the characteristics of things, which can be feature vectors; text features refer to the features corresponding to descriptive text; image features refer to the features corresponding to object images; and image-text features refer to the features corresponding to the combination of descriptive text and object images.

[0090] In practical applications, for any one of the objects to be detected, a preset feature extraction method can be used to extract features from the descriptive text and image of that object, resulting in text features, image features, and image-text features. This process is repeated for each object to be detected to obtain its text features, image features, and image-text features.

[0091] In one optional embodiment of this specification, for any descriptive text and object image of a target object, a preset feature extraction tool can be used to directly extract features from the descriptive text and object image of the target object, obtaining text features corresponding to the descriptive text, image features corresponding to the object image, and image-text features corresponding to both the descriptive text and object image. By traversing the descriptive text and object image of each target object, text features, image features, and image-text features of each target object are obtained. Thus, using a preset feature extraction tool to directly extract features can improve the efficiency of obtaining text features, image features, and image-text features, thereby improving the efficiency of object detection.

[0092] In another optional embodiment of this specification, to improve the accuracy of text features, image features, and image-text features, for any descriptive text and object image of a target object, the descriptive text and object image are segmented to obtain finer-grained sub-descriptive text and sub-object images. Then, feature extraction is performed on the sub-descriptive text and sub-object images, and text features, image features, and image-text features are determined based on the extracted features. That is, the specific implementation process of extracting features from the descriptive text and object image of each target object to obtain the text features, image features, and image-text features of each target object can be as follows:

[0093] For any one of the at least two objects to be detected, the description text of the object to be detected is split into multiple sub-description texts, and the object image of the object to be detected is split into multiple sub-object images;

[0094] Feature extraction is performed on each sub-description text to obtain the sub-text features of each sub-description text, and feature extraction is performed on each sub-object image to obtain the sub-image features of each sub-object image;

[0095] Based on the sub-text features and the sub-image features, the text features, image features, and image-text features of the object to be detected are obtained.

[0096] Specifically, sub-description text refers to the unit that constitutes the description text, that is, the description text after being split. For example, if the description text is "withered vines, old trees, and crows at dusk", then "withered vines", "old trees", and "crows at dusk" obtained after splitting the description text are all sub-description texts; sub-object image refers to the unit that constitutes the object image, that is, the object image after being split.

[0097] In practical applications, for any descriptive text and object image of a target object, the descriptive text can be split into multiple sub-descriptive texts, and the object image can be split into multiple sub-object images. It should be noted that the splitting of the descriptive text and the splitting of the object image can be performed simultaneously, or the descriptive text can be split first, followed by the object image, or vice versa.

[0098] Optionally, the number of text units in the description text can be determined first, and then the text can be split according to the number of text units, that is, the description text can be split into individual text units, which are also called sub-description text. The object image can then be split into multiple sub-object images of a specified size. For example, if the description text is "I love my motherland" and has 6 text units, it can be split into 6 sub-description texts: "I", "love", "I", "of", "motherland", and "country". If the object image size is 4*4 and the specified size is 2*2, then the object image can be split into 4 sub-object images of size 2*2.

[0099] Optionally, a first preset value and a second preset value can be preset to split each descriptive text into the first preset value of sub-descriptive texts and each object image into the second preset value of sub-object images. Here, the preset value refers to a pre-set value, generally a positive integer, such as 2, 30, 41, etc.; the first preset value is the text splitting value, that is, the number of times the descriptive text is split; for example, if the descriptive text is split into 10, then 10 is the first preset value; the second preset value is the image splitting value, that is, the number of times the object image is split.

[0100] For example, if the description text of an object to be detected is "winter couple gloves" and the object image is a glove picture, with a first preset value of 6 and a second preset value of 2*2=4, then "winter couple gloves" will be split into 6 sub-description texts: "winter", "season", "love", "couple", "hands", and "gloves". The glove picture will be split into 4 sub-glove pictures in a 2*2 format.

[0101] In addition, the description text can be split into a first preset number of sub-description texts. This can be done by splitting them evenly, for example, if the description text has 10 characters and the first preset number is 5, then each sub-description text will have 2 characters. Alternatively, it can be split randomly, for example, if the description text has 10 characters and the first preset number is 4, then the number of characters in the 4 sub-description texts will be 1, 2, 3, and 4 respectively.

[0102] Furthermore, a text feature extraction method is employed to extract features from each sub-descriptive text, yielding sub-text features for each sub-descriptive text. Similarly, an image feature extraction method is employed to extract features from each sub-object image, yielding sub-image features for each sub-object image. Then, the sub-text features and sub-image features are processed to obtain the text features, image features, and image-text features of the object to be detected. Following the above method, the descriptive text and object images of each object to be detected are traversed to obtain the text features, image features, and image-text features of each object to be detected.

[0103] Following the previous example, text features are extracted from "winter," "season," "affection," "partner," "hand," and "glove," resulting in six sub-text features. Image features are extracted from four sub-glove images, resulting in four sub-image features. Then, based on the six sub-text features and four sub-image features, the text features, image features, and image-text features of the object to be detected are obtained.

[0104] In this way, by splitting the descriptive text and object image, we obtain more granular sub-descriptive text and sub-object image. Then, we extract features from the sub-descriptive text and sub-object image, making the obtained sub-text features and sub-image features more accurate. Based on the determination of text features, image features, and graphic-text features according to the sub-text features and sub-image features, we further improve the accuracy and reliability of text features, image features, and graphic-text features.

[0105] In one optional embodiment of this specification, the process of obtaining the text features, image features, and image-text features of the object to be detected based on the sub-text features and the sub-image features can be implemented as follows:

[0106] The text features of the object to be detected are obtained based on the sub-text features;

[0107] The image features of the object to be detected are obtained based on the sub-image features;

[0108] The image and text features of the object to be detected are obtained based on the sub-text features and the sub-image features.

[0109] In practical applications, the descriptive text and image of a target object are split into multiple sub-descriptive texts and multiple sub-object images. Features are extracted from each sub-descriptive text and each sub-object image. The resulting sub-text features are then weighted and summed to obtain the text features of the target object. Similarly, the resulting sub-image features are weighted and summed to obtain the image features of the target object. Finally, the sub-text features and sub-image features are weighted and summed to obtain the image-text features of the target object. This process improves the efficiency of determining text features, image features, and image-text features.

[0110] Following the previous example, we calculate the average of the 6 sub-text features to obtain the text features of the object to be detected, calculate the average of the 4 sub-image features to obtain the image features of the object to be detected, and calculate the average of the 6 sub-text features and the 4 sub-image features to obtain the image and text features of the object to be detected.

[0111] It should be noted that when determining the three features—text features, image features, and text-image features—the process can be carried out simultaneously, sequentially, or in other ways; this specification does not limit this.

[0112] Step 206: Based on the text features, image features, and image-text features of the first object to be detected and the image-text features of the second object to be detected, determine the similarity between the first object to be detected and the second object to be detected, wherein the first object to be detected and the second object to be detected are any two objects to be detected.

[0113] Based on the feature extraction of the descriptive text and object image of each object to be detected, and the resulting text features, image features, and image-text features, the similarity between the objects to be detected is calculated.

[0114] In practical applications, after determining the text features, image features, and image-text features of each object to be detected, for any two objects to be detected, the similarity between the text features, image features, and image-text features of the first object to be detected and the image-text features of the second object to be detected can be calculated to determine the similarity between the first and second objects to be detected. This process is repeated for all pairs of objects to be detected to determine the similarity between them.

[0115] In one optional embodiment of this specification, for any two of at least two objects to be detected, the text features, image features, and image-text features of the first object to be detected are weighted and summed to obtain a comprehensive feature. Then, the similarity between this comprehensive feature and the image-text features of the second object to be detected is calculated, and the resulting value is used as the similarity between the first and second objects to be detected. Following this method, all objects to be detected are traversed to obtain the similarity between each object. Thus, by determining the comprehensive feature through weighted summation and then calculating the similarity, the efficiency of determining the similarity can be improved, thereby improving the object detection efficiency.

[0116] In another optional embodiment of this specification, for any two of the at least two objects to be detected, the text features, image features, and image-text features of the first object to be detected are compared with the image-text features of the second object to be detected, respectively, to calculate three similarities—text similarity, image similarity, and image-text similarity. That is, the similarity includes text similarity, image similarity, and image-text similarity. Accordingly, based on the text features, image features, and image-text features of the first object to be detected and the image-text features of the second object to be detected, the similarity between the first object to be detected and the second object to be detected is determined. The specific implementation process can be as follows:

[0117] The text features of the first object to be detected and the image features of the second object to be detected are used to calculate the similarity, and the text similarity between the first object to be detected and the second object to be detected is obtained.

[0118] The similarity between the image features of the first object to be detected and the text features of the second object to be detected is calculated to obtain the text similarity between the first object to be detected and the second object to be detected.

[0119] The similarity between the image and text features of the first object to be detected and the image and text features of the second object to be detected is calculated to obtain the image and text similarity between the first object to be detected and the second object to be detected.

[0120] Specifically, text similarity refers to the similarity between text features and image features; image similarity refers to the similarity between image features and text / image features; and text / image similarity refers to the similarity between image features and text / image features.

[0121] In practical applications, the text features of the first object to be detected can be compared with the image and text features of the second object to obtain text similarity; the image features of the first object to be detected can be compared with the image and text features of the second object to obtain image similarity; and the image and text features of the first object to be detected can be compared with the image and text features of the second object to obtain image and text similarity. By calculating the similarity between the text features, image features, and image and text features of the first object to be detected and the image and text features of the second object to be detected, and determining the similarity from three dimensions, the accuracy of the similarity calculation can be improved, thereby improving the accuracy of object detection.

[0122] It should be noted that the calculation of text similarity, text similarity, and image-text similarity can be performed one by one or simultaneously; this specification does not impose any limitation on this. Here, similarity refers to the degree of similarity or distance between two features. It can be Euclidean distance (the similarity calculated using the Euclidean distance algorithm), Manhattan distance (the similarity calculated using the Manhattan distance algorithm), Minkowski distance (the similarity calculated using the Minkowski distance algorithm), or cosine similarity (the similarity calculated using the cosine similarity algorithm); this specification does not impose any limitation on this.

[0123] Step 208: Determine the target object among the at least two objects to be detected based on the similarity, and process the target object.

[0124] In practical applications, after determining the similarity between each object to be detected, the similarity can be compared with preset similarity detection conditions. The similarity that meets the similarity detection conditions is determined as the target similarity, and the object to be detected corresponding to the target similarity is determined as the target object. For example, if the similarity detection condition is greater than a preset similarity threshold, then the similarity greater than the similarity threshold is determined as the target similarity, and the two objects to be detected corresponding to each target similarity are determined as the target objects.

[0125] Furthermore, after identifying the target object, the target object is processed. When the similarity detection condition is greater than a preset similarity threshold, for any target similarity corresponding to a target object that is the same type of object (i.e., the target object consists of two identical objects to be detected): the target objects among the at least two objects to be detected can be deduplicated, and then the deduplicated objects to be detected can be recommended to the user; alternatively, the target object can be directly recommended to the user.

[0126] For example, when a user views products in a store on a shopping platform, to avoid the store displaying the same product with multiple links, all products linked to in that store are considered as potential products. Following the method described above, duplicate products within the potential product pair are identified, and then deduplicated. The deduplicated products are then recommended to the user, ensuring that no duplicate products are shown to the user. Similarly, if a user wants to find a product identical to a picture, the specific product in the picture and all products on the shopping platform are identified as potential products. Products on the shopping platform with a similarity greater than a similarity threshold to the specified product are identified as target products and recommended to the user, meaning that identical products to the specified product are recommended to the user.

[0127] In one or more optional embodiments of this specification, a pre-trained similarity prediction model can also be obtained. Using this model, the similarity between each object to be detected is determined based on its descriptive text and object image. That is, before extracting features from the descriptive text and object image of each object to obtain its text features, image features, and image-text features, the method further includes:

[0128] Obtain a pre-trained similarity prediction model, wherein the similarity prediction model includes a feature extraction layer and a similarity determination layer;

[0129] Accordingly, the step of extracting features from the descriptive text and object image of each object to be detected, to obtain the text features, image features, and image-text features of each object to be detected, includes:

[0130] The descriptive text and object image of each object to be detected are input into the feature extraction layer for feature extraction, thereby obtaining the text features, image features and image-text features of each object to be detected;

[0131] Accordingly, determining the similarity between the first object to be detected and the second object to be detected based on the text features, image features, and image-text features of the first object to be detected, and the image-text features of the second object to be detected, includes:

[0132] The text features, image features, and image-text features of the first object to be detected, as well as the image-text features of the second object to be detected, are input into the similarity determination layer to calculate the similarity between the first object to be detected and the second object to be detected.

[0133] Specifically, a similarity prediction model refers to a model that takes descriptive text and object images as input and outputs similarity. It can be a pre-trained neural network model or an algorithm with tuned parameters. The feature extraction layer refers to the unit that encodes or processes the text or image to extract features. The similarity determination layer refers to the unit that performs similarity calculation.

[0134] In practical applications, a pre-trained similarity prediction model containing a feature extraction layer and a similarity determination layer can be obtained first. Then, the descriptive text and object images of all objects to be detected are input to the feature extraction layer for feature extraction. The feature extraction layer extracts features from the descriptive text and object images of each object to obtain text features, image features, and image-text features for each object. Further, the text features, image features, and image-text features of each object are input to the similarity determination layer. The similarity determination layer calculates the similarity between each pair of objects based on the text features, image features, and image-text features of the first object, and the image-text features of the second object. Thus, using a pre-trained similarity prediction model for feature extraction and similarity calculation not only improves processing efficiency but also enhances the accuracy of feature and similarity calculations.

[0135] For example, there are three objects to be detected: object A, object B, and object C. The descriptive texts for objects A, B, and C are text a1, text b1, and text c1, respectively. The object images for objects A, B, and C are image a2, image b2, and image c2, respectively. Text a1, text b1, and text c1, along with images a2, b2, and c2, are input into a feature extraction layer for feature extraction, resulting in text features a3, image features a4, and image-text features a5 for object A; text features b3, image features b4, and image-text features b5 for object B; and text features c3, image features c4, and image-text features c5 for object C. Next, text features a3-c3, image features a4-c4, and image-text features a5-c5 are input into the similarity determination layer for similarity calculation: similarity is calculated based on text features a3, image features a4, image-text features a5, and image-text features b5 to obtain the similarity between object A and object B; similarity is calculated based on text features b3, image features b4, image-text features b5, and image-text features c5 to obtain the similarity between object B and object C; similarity is calculated based on text features c3, image features c4, image-text features c5, and image-text features a5 to obtain the similarity between object C and object A.

[0136] It should be noted that before obtaining a pre-trained similarity prediction model, the model to be trained needs to be trained to obtain a similarity prediction model with feature extraction and similarity calculation capabilities. In other words, before obtaining a pre-trained similarity prediction model, it is necessary to train the similarity prediction model so that it can output accurate similarity scores. The training process of the similarity prediction model can be as follows:

[0137] Obtain a training set, wherein the training set contains at least one sample pair, the sample pair comprising sample text and sample image of two sample objects;

[0138] Input the sample text and sample image of each sample object in each sample pair into the feature extraction layer of the model to be trained to obtain the predicted text features, predicted image features and predicted image-text features of each sample object.

[0139] For any sample pair corresponding to the first sample object and the second sample object, the predicted text features, predicted image features, and predicted image-text features of the first sample object and the predicted image-text features of the second sample object are input into the similarity determination layer of the model to be trained to obtain the predicted similarity between the first sample object and the second sample object.

[0140] The total loss value is determined based on the predicted similarity, and the model parameters of the feature extraction layer and the similarity determination layer are adjusted based on the total loss value. The step of obtaining the training set is then performed, and training is stopped when the preset training stopping condition is met, thereby obtaining the similarity prediction model.

[0141] Specifically, the training set refers to the collection of samples used to train the model; a sample pair refers to the samples used to train the model, with each sample pair containing two sample texts and two sample images: a first sample text, a second sample text, a first sample image, and a second sample image. The first sample text and the first sample image are the descriptive text and image of the first sample object, and the second sample text and the second sample image are the descriptive text and image of the second sample object. Any two sample objects can be either the first or second sample object. The model to be trained refers to a pre-specified pre-trained neural network model. Predicted text features refer to the features corresponding to the sample text; predicted image features refer to the features corresponding to the sample image; predicted image-text features refer to the features corresponding to the combination of the sample text and the sample image; predicted similarity refers to the degree of similarity between the first and second sample objects; the training stopping condition can be that the loss value is less than or equal to a preset threshold, the number of training iterations reaches a preset iteration value, or the loss value converges, meaning the loss value no longer decreases with continued training.

[0142] In practical applications, there are various ways to obtain training sets. For example, operators can send training instructions for the model to be trained, or instructions to obtain the training set, to the execution entity. Upon receiving these instructions, the execution entity then begins obtaining the training set. Alternatively, the server can automatically obtain the training set at preset intervals. For instance, after a preset interval, a server with model training capabilities can automatically obtain the training set from a specified access area; or after a preset interval, a terminal with model training capabilities can automatically obtain the training set stored locally. This manual does not impose any limitations on the method of obtaining the training set.

[0143] See Figure 3A , Figure 3A This document illustrates a schematic diagram of the structure of a similarity prediction model in an object detection method according to an embodiment of this specification, which is also a schematic diagram of the structure of the model to be trained. The feature extraction layer of the model to be trained includes a feature extraction layer and a similarity determination layer. After acquiring a training set containing at least one sample pair, the sample text and sample image of each sample object in each sample pair are input into the feature extraction layer of the model to be trained. The feature extraction layer extracts features from each sample text and sample image, outputting predicted text features, predicted image features, and predicted image-text features for each sample object. These predicted text features, predicted image features, and predicted image-text features are then input into the similarity determination layer of the model to be trained. The similarity determination layer determines the predicted similarity between the two sample objects corresponding to each sample pair based on the predicted text features, predicted image features, and predicted image-text features. That is, for any first sample object and second sample object corresponding to a sample pair, the predicted text features, predicted image features, and predicted image-text features of the first sample object and the predicted image-text features of the second sample object are input into the similarity determination layer of the model to be trained to obtain the predicted similarity between the first sample object and the second sample object. Then, each sample pair is traversed to obtain the predicted similarity corresponding to each sample pair, that is, the predicted similarity between the two sample objects corresponding to each sample pair. Next, based on the predicted similarities and the preset loss function, the total loss value is determined. If the preset training stopping condition is not met, the model parameters of the model to be trained—that is, the model parameters of the feature extraction layer and the similarity determination layer—are adjusted according to the total loss value. Then, a training set is obtained, and the next round of training is performed. Training stops when the preset training stopping condition is met, and the trained model is designated as the similarity prediction model. In this way, training the model to be trained using the training set improves the accuracy and speed of feature extraction and similarity determination in the similarity prediction model, thereby enhancing its robustness.

[0144] In one or more optional embodiments of this specification, the similarity determination layer can be divided into three sub-layers: a text similarity determination sub-layer, an image similarity determination sub-layer, and a text-image similarity determination sub-layer. For any sample pair corresponding to two sample objects, the predicted text features of the first sample object and the predicted text-image features of the second sample object are input into the text similarity determination sub-layer; the predicted image features of the first sample object and the predicted text-image features of the second sample object are input into the image similarity determination sub-layer; and the predicted text-image features of the first sample object and the predicted text-image features of the second sample object are input into the text-image similarity determination sub-layer for similarity calculation, resulting in three similarities—text similarity, image similarity, and text-image similarity. That is, the predicted similarity includes predicted text similarity, predicted image similarity, and image-text predicted similarity. The similarity determination layer includes a text similarity determination sublayer, an image similarity determination sublayer, and an image-text similarity determination sublayer. Accordingly, the predicted text features, predicted image features, and predicted image-text features of the first sample object, along with the predicted image-text features of the second sample object, are input into the similarity determination layer of the model to be trained to obtain the predicted similarity between the first sample object and the second sample object. The specific implementation process can be as follows:

[0145] The predicted text features of the first sample object and the predicted image-text features of the second sample object are input into the text similarity determination sub-layer to obtain the predicted text similarity between the first sample object and the second sample object;

[0146] The predicted image features of the first sample object and the predicted image-text features of the second sample object are input into the image similarity determination sub-layer to obtain the predicted image similarity between the first sample object and the second sample object;

[0147] The predicted image-text features of the first sample object and the predicted image-text features of the second sample object are input into the image-text similarity determination sub-layer to obtain the predicted image-text similarity between the first sample object and the second sample object.

[0148] Specifically, predicted text similarity refers to the similarity between predicted text features and predicted image features; predicted image similarity refers to the similarity between predicted image features and predicted image-text features; and predicted image-text similarity refers to the similarity between predicted image features and predicted image-text features.

[0149] In practical applications, see Figure 3AThe similarity determination layer includes three sub-layers: text similarity determination, image similarity determination, and image-text similarity determination. For any pair of samples corresponding to two sample objects, the predicted text features of the first sample object and the predicted image-text features of the second sample object are input into the text similarity determination sub-layer for similarity calculation, yielding the predicted text similarity. Similarly, the predicted image features of the first sample object and the predicted image-text features of the second sample object are input into the image similarity determination sub-layer for similarity calculation, yielding the predicted image similarity. Likewise, the predicted image-text features of the first sample object and the predicted image-text features of the second sample object are input into the image-text similarity determination sub-layer for similarity calculation, yielding the predicted image-text similarity. By calculating the similarity between the predicted text features, predicted image features, and predicted image-text features of the first sample object and the predicted image-text features of the second sample object, respectively, the predicted similarity is determined from three dimensions. This improves the accuracy of the predicted similarity, thereby enhancing model training efficiency and object detection accuracy.

[0150] To align the features (representations) of the first and second sample objects into the same embedding space, assuming the first and second sample objects are of the same type, the multimodal image + text features (image-text features) of either the first or second sample object should be highly similar to those of the second or first sample object. Furthermore, the sample image and descriptive text of either the first or second sample object should also be aligned with the multimodal embedding of either the second or first sample object, ensuring that key features in the image and text dominate the generation of the embedding features. Specifically, during the training (fine-tuning) phase, it is necessary to calculate the predicted text similarity, predicted image similarity, and image-text predicted similarity between the first and second sample objects, as detailed in Equation 1:

[0151]

[0152] In Equation 1, i and j represent the i-th and j-th sample objects, respectively; and These represent the predicted image features, predicted image features, and predicted text features of the first sample object, respectively. This represents the predicted image and text features of the second sample object respectively; and Predicted image-text similarity, predicted image similarity, and predicted text similarity between the first sample object and the second sample object, respectively.

[0153] It should be noted that predicted similarity refers to the degree of similarity or distance between two features. It can be Euclidean distance, which is the similarity calculated using the Euclidean distance algorithm; Manhattan distance, which is the similarity calculated using the Manhattan distance algorithm; Minkowski distance, which is the similarity calculated using the Minkowski distance algorithm; or cosine similarity, which is the similarity calculated using the cosine similarity algorithm. This specification does not limit the specific method used.

[0154] In one or more optional embodiments of this specification, the feature extraction layer can be divided into a splitting sub-layer, a feature extraction sub-layer, and a feature fusion sub-layer. For any sample text and sample image of a sample object, the sample text and sample image are segmented to obtain more fine-grained sub-sample text and sub-sample images. Then, feature extraction is performed on the sub-sample text and sub-sample images, and sample text features, sample image features, and sample image-text features are determined based on the extracted features. That is, when the feature extraction layer includes a splitting sub-layer, a feature extraction sub-layer, and a feature fusion sub-layer, the specific implementation process of inputting the sample text and sample image of each sample object in each sample pair into the feature extraction layer of the model to be trained to obtain the predicted text features, predicted image features, and predicted image-text features of each sample object can be as follows:

[0155] For any sample object, the sample text and sample image of the sample object are input into the splitting sub-layer to obtain multiple sub-sample texts and multiple sub-sample images;

[0156] Each subsample text and each subsample image are input into the feature extraction sublayer to obtain the predicted subtext features of each subsample text and the predicted subimage features of each subsample image.

[0157] Each predicted sub-text feature and each predicted sub-image feature are input into the feature fusion sub-layer to obtain the predicted text features, predicted image features, and predicted image-text features of the sample object.

[0158] Specifically, subsample text refers to the unit that makes up the sample text, that is, the sample text after being split; subsample image refers to the unit that makes up the sample image, that is, the sample image after being split.

[0159] In practical applications, see Figure 3AThe feature extraction layer includes a splitting sub-layer, a feature extraction sub-layer, and a feature fusion sub-layer. For any sample object, the sample text and sample image can be input into the splitting sub-layer. The splitting sub-layer splits the sample text into multiple sub-sample texts and the sample image into multiple sub-sample images. It should be noted that the splitting of the sample text and the splitting of the sample image can be performed simultaneously within the splitting sub-layer, or the sample text can be split first within the splitting sub-layer, followed by the sample image, or vice versa.

[0160] Optionally, the number of text units in the sample text can be determined first, and then the text can be split according to the number of text units, that is, the sample text can be split into individual text units, which are also called sub-sample texts. For example, if the sample text contains 6 English words, then the sample text can be split into 6 sub-sample texts, each containing 1 English word. The sample image can also be split into multiple sub-sample images of a specified size.

[0161] Optionally, a third and fourth preset values ​​can be pre-set to further split each sample text into a first preset value of sub-sample texts and each sample image into a third preset value of sub-sample images. Here, the preset values ​​are pre-set values, generally positive integers such as 2, 30, 41, etc.; the third preset value is the text splitting value, i.e., the number of times the sample text is split; and the fourth preset value is the image splitting value, i.e., the number of times the sample images are split. Furthermore, the splitting of the sample text can be either average or random.

[0162] Furthermore, the split sub-sample text and sub-sample image are input into the feature extraction sub-layer, which extracts features from each sub-sample text to obtain the predicted sub-text features of each sub-sample text, and extracts features from each sub-sample image to obtain the predicted sub-image features of each sub-sample image.

[0163] Next, the predicted sub-text features and sub-image features are input into the sub-image features, and the sub-image features are fused with the predicted sub-text features and sub-image features to obtain the predicted text features, predicted image features, and predicted image-text features for the sample object. Following the above method, the sample text and sample images of each sample object are traversed to obtain the predicted text features, predicted image features, and predicted image-text features for each sample object.

[0164] In this way, by splitting the sample text and sample image, we obtain more granular sub-sample text and sub-sample images. Then, we extract features from the sub-sample text and sub-sample images, making the predicted sub-text features and predicted sub-image features more accurate. Based on the predicted sub-text features and predicted sub-image features, we further improve the accuracy and reliability of the predicted text features, predicted image features, and predicted image-text features, thereby improving the model training efficiency and the robustness of the similarity prediction model.

[0165] In one or more optional embodiments of this specification, the feature fusion sublayer includes a text feature fusion sublayer, an image feature fusion sublayer, and a text-image feature fusion sublayer. The step of inputting each predicted sub-text feature and each predicted sub-image feature into the feature fusion sublayer to obtain the predicted text features, predicted image features, and predicted text-image features of the sample object can be implemented as follows:

[0166] Each predicted sub-text feature is input into the text feature fusion sub-layer to obtain the predicted text features of the sample object;

[0167] Each predicted sub-image feature is input into the image feature fusion sub-layer to obtain the predicted image features of the sample object;

[0168] Each predicted sub-text feature and each predicted sub-image feature are input into the image-text feature fusion sub-layer to obtain the predicted image-text features of the sample object.

[0169] Specifically, the text feature fusion sublayer refers to the part of the similarity prediction model that determines text features (predicts text features); the image feature fusion sublayer refers to the part of the similarity prediction model that determines image features (predicts image features); and the image-text feature fusion sublayer refers to the part of the similarity prediction model that determines image-text features (predicts image-text features).

[0170] In practical applications, see Figure 3AThe feature extraction layer includes a text feature fusion sublayer, an image feature fusion sublayer, an image-text feature fusion sublayer, and a similarity determination layer. After inputting the split sub-sample text and sub-sample images into the feature extraction sublayer, it extracts features from each sub-sample text to obtain predicted sub-text features. Similarly, it extracts features from each sub-sample image to obtain predicted sub-image features. These predicted sub-text features are then input into the text feature fusion sublayer, which performs a weighted sum to obtain the predicted text features for the sample object. Likewise, the predicted sub-image features are input into the image feature fusion sublayer, which performs a weighted sum to obtain the predicted image features for the sample object. Finally, the predicted sub-text features and predicted image features are input into the image-text feature fusion sublayer, which performs a weighted sum to obtain the predicted image-text features for the sample object. This can improve the efficiency of determining predicted text features, predicted image features, and predicted image-text features.

[0171] It should be noted that when determining the three features of predicted text, predicted image, and predicted text-image, they can be done simultaneously, one by one, or in other ways. This manual does not limit this.

[0172] In one or more optional implementations of this specification, the feature extraction layer includes a text feature extraction unit, an image feature extraction unit, and a feature determination unit. Correspondingly, the step of inputting the sample text and sample image of each sample object in each sample pair into the feature extraction layer of the model to be trained to obtain the predicted text features, predicted image features, and predicted image-text features of each sample object can be implemented as follows:

[0173] For any sample object, the sample text of the sample object is input into the text feature extraction unit to obtain the initial text features of the sample object;

[0174] The sample image of the sample object is input into the image feature extraction unit to obtain the initial image features of the sample object;

[0175] The initial text features and the initial image features are input into the feature determination unit to obtain the predicted text features, predicted image features, and predicted image-text features of the sample object.

[0176] Specifically, the text feature extraction unit refers to the part of the model to be trained that extracts features from sample text (descriptive text); the image feature extraction unit refers to the part of the model to be trained that extracts features from sample images (object images); and the feature determination unit refers to the part that processes the features proposed in the model to be trained to obtain the target features (predicted text features, predicted image features, and predicted image-text features, or text features, image features, and image-text features).

[0177] In practical applications, for any given sample object, the sample text is input into the text feature extraction unit, which extracts features from the text to obtain the initial text features of the sample object. The sample image is then input into the image feature extraction unit, which extracts features from the image to obtain the initial image features of the sample object. These initial text and image features are then input into the feature determination unit, which processes them to obtain the target features, namely the predicted text features, predicted image features, and predicted text-image features of the sample object. This separate processing of text and image not only improves the model's processing efficiency but also enhances the accuracy of the predicted text, image, and text-image features.

[0178] In one or more optional embodiments of this specification, the total loss value can be determined based on the predicted similarity of positive sample pairs and the predicted similarity of negative sample pairs. The similarity prediction process performs sample processing on each positive and negative sample pair, ensuring that the output value (predicted similarity) of the positive sample pair is as large as possible compared to the output value (predicted similarity) of the negative sample pair. Preferably, the predicted similarity of the positive sample pair is made as close to 1 as possible, and the predicted similarity of the negative sample pair is made as close to 0 as possible. Then, the total loss value is determined for each predicted similarity.

[0179] See Figure 3B , Figure 3BThis diagram illustrates the determination of the total loss value in an object detection method according to an embodiment of this specification: for example, q1 and t1, q2 and t2, and q3 and t3 are positive sample pairs, and q1 and t2, q1 and t3, q2 and t1, q2 and t3, q3 and t1, and q3 and t2 are negative sample pairs. The predicted similarity of the sample pairs is qitj. When i = j, it is the predicted similarity of the positive sample pairs, i.e., the predicted similarity of the positive sample pairs is qiti. When i ≠ j, it is the predicted similarity of the negative sample pairs, i.e., the predicted similarity of the negative sample pairs is qitj and i ≠ j. A matrix [qiti] corresponding to the positive samples is established, consisting of q1t1, q2t2, and q3t3. A matrix [qitj] corresponding to the samples is established, consisting of q1t1, q1t2, q1t3, q2t1, q2t2, q2t3, q3t1, q3t2, and q3t3.

[0180] There is also a pre-defined 3x3 matrix m, where the elements corresponding to positive sample pairs are 0, and the elements corresponding to negative sample pairs are m. The matrix corresponding to the total loss value is as follows:

[0181] In one or more optional embodiments of this specification, to further improve the robustness of the similarity prediction model, the total loss value can be determined from multiple dimensions, and then the total loss value can be determined based on each loss value. That is, the specific implementation process of determining the total loss value based on the predicted similarity can be as follows:

[0182] The predicted image-text similarity, predicted image similarity, and predicted text similarity corresponding to the positive sample pair are respectively determined as the specified predicted image-text similarity, specified predicted image similarity, and specified predicted text similarity, wherein the positive sample pair is a sample pair carrying a positive sample identifier;

[0183] A first loss value is determined based on the predicted image-text similarity, the predicted image similarity, the predicted text similarity, the specified predicted image-text similarity, the specified predicted image similarity, and the specified predicted text similarity;

[0184] A second loss value is determined based on the predicted image-text similarity, the specified predicted text similarity, and the specified predicted image similarity;

[0185] A third loss value is determined based on the predicted image similarity, the predicted text similarity, and the predicted image-text similarity;

[0186] The total loss value is determined based on the first loss value, the second loss value, and the third loss value.

[0187] Specifically, a positive sample pair refers to a sample pair consisting of sample text and sample image of two identical sample objects, such as a sample pair consisting of the title and product image of two identical products; a positive sample identifier is an identifier that indicates that a sample pair is a positive sample pair; the first loss value, also known as the matching loss value, refers to the loss value obtained after matching each predicted image-text similarity with the specified predicted image similarity, each predicted text similarity with the specified predicted image similarity, and each predicted image-text similarity with the specified predicted text similarity; the second loss value, also known as the contrast loss value, refers to the loss value obtained by comparing the predicted image-text similarity with the specified predicted text similarity, and the predicted image-text similarity with the specified predicted image similarity; the third loss value, also known as the consistency loss value, refers to the loss value obtained by comparing the consistency of the predicted image-text similarity, the specified predicted text similarity, and the specified predicted image similarity.

[0188] In practical applications, after determining the predicted image-text similarity, predicted image similarity, and predicted text similarity for each sample pair, the predicted image-text similarity, predicted image similarity, and predicted text similarity for sample pairs carrying positive sample identifiers are first used as the predicted image-text similarity, specified predicted image similarity, and specified predicted text similarity. Further, each predicted image-text similarity, predicted image similarity, predicted text similarity, specified predicted image-text similarity, specified predicted image similarity, and specified predicted text similarity is calculated according to a first loss function to determine a first loss value; each predicted image-text similarity, each specified predicted text similarity, and each specified predicted image similarity is calculated according to a second loss function to determine a second loss value; and each predicted image-text similarity, each predicted text similarity, and each predicted image-text similarity are calculated according to a third loss function to determine a third loss value. Then, the first loss value, the second loss value, and the third loss value are weighted and summed to determine the total loss value. Preferably, the average of the first loss value, the second loss value, and the third loss value is used as the total loss value. See Equation 2.

[0189] Total loss value = 1 / 3 (first loss value + second loss value + third loss value) (Equation 2)

[0190] In this way, by calculating the loss value from three different perspectives and combining the obtained loss values ​​to determine the total loss value, the model to be trained can be optimized and its parameters adjusted based on the total loss value. This can help improve the training efficiency of the model and the robustness of the similarity prediction model.

[0191] For example, the sample image and sample text are first split into multiple sub-sample images and multiple sub-sample texts. Each sub-sample image and each sub-sample text are then encoded, i.e., feature extraction is performed, resulting in multiple predicted sub-image features and multiple predicted sub-text features. Further, based on each predicted sub-image feature and each predicted sub-text feature, predicted text features, predicted image features, and predicted image-text features are determined. Then, based on the predicted text features, predicted image features, and predicted image-text features, a first loss value, a second loss value, and a third loss value are determined. The total loss value is then determined based on the first loss value, the second loss value, and the model parameters are adjusted according to the total loss value.

[0192] It should be noted that the determination of the first loss value, the second loss value, and the third loss value can be carried out simultaneously or in a certain order. This manual does not limit this process.

[0193] Furthermore, the total loss during the model training phase (fine-tuning phase) includes three different loss values: the first loss value, also known as the matching loss (PPM, product / item to product / item matching losses), guides the comparison of objects of the same style (positive sample pairs) with samples of different styles (negative sample pairs) to have a greater similarity value, and improves the representation consistency of object images, descriptive text, and object image + descriptive text; the second loss value (PDC, product / item self-distinctiveness contrastive losses), ensures that the distance between the three modal embeddings should be less than the distance between the multimodal embeddings of the first and second objects to be detected; and the third loss value (PLC, product / item localityconsistencylosses), attempts to align image features, text features, and image-text features to reduce interference from irrelevant attributes. These three loss values ​​aim to learn cross-modal aligned representations from different perspectives and all help to model object images, descriptive text, and object image + descriptive text into a joint embedding space, thereby achieving accurate object detection.

[0194] Optionally, a first loss value is determined based on the predicted image-text similarity, the predicted image similarity, the predicted text similarity, the specified predicted image-text similarity, the specified predicted image similarity, and the specified predicted text similarity. The specific implementation process can be as follows:

[0195] The image-text loss value is determined based on the predicted image-text similarity and the specified predicted image-text similarity.

[0196] The image loss value is determined based on the predicted image similarity and the specified predicted image similarity;

[0197] The text loss value is determined based on the predicted text similarity and the specified predicted text similarity.

[0198] Based on the image loss value, the text loss value, and the image loss value, a first loss value is obtained.

[0199] Specifically, image loss value refers to the loss value determined based on predicted image-text similarity; text loss value refers to the loss value determined based on predicted image similarity; and text loss value refers to the loss value determined based on predicted text similarity.

[0200] In practical applications, the image-text loss value can be determined based on the difference between each predicted image-text similarity and a specified predicted image-text similarity; the image loss value can be determined based on the difference between each predicted image similarity and a specified predicted image similarity; and the text loss value can be determined based on the difference between each predicted text similarity and a specified predicted similarity. Furthermore, the image-text loss value, image loss value, and text loss value are processed according to a preset processing method to obtain the first loss value. This improves the efficiency and accuracy of determining the first loss value.

[0201] Preferably, the first loss value, also known as the matching loss (PPM, product / item to product / item matching losses), can use hinge loss to guide that sample pairs of the same style (positive sample pairs) have a greater similarity value than sample pairs of different styles (negative sample pairs), as shown in Equation 3.

[0202]

[0203] In Equation 3, L PPM α1 is the first loss value; N represents the training set size; α1 is the margin value, which is preset. and These are respectively predicting image-text similarity, predicting image similarity, and predicting text similarity. and These are specified as predicted image-text similarity, predicted image similarity, and predicted text similarity, respectively. This allows for a tighter cross-modal embedding of positive sample pairs compared to negative sample pairs.

[0204] It should be noted that the determination of the image loss value, text loss value, and image loss value can be performed simultaneously or sequentially in a certain order; this manual does not impose any restrictions on this.

[0205] Optionally, the step of determining the second loss value based on the predicted image-text similarity, the specified predicted text similarity, and the specified predicted image similarity can be implemented as follows:

[0206] The first sub-loss value is determined based on the predicted image-text similarity and the specified predicted text similarity.

[0207] The second sub-loss value is determined based on the predicted image-text similarity and the specified predicted image similarity.

[0208] Based on the first sub-loss value and the second sub-loss value, the second loss value is obtained.

[0209] Specifically, the first sub-loss value refers to the loss value determined based on the similarity between the predicted image and text and the specified predicted text; the second sub-loss value refers to the loss value determined based on the similarity between the predicted image and text and the specified predicted image.

[0210] In practical applications, a first sub-loss value can be determined based on the difference between each predicted image-text similarity and the specified predicted text similarity; a second sub-loss value can be determined based on the difference between each predicted image-text similarity and the specified predicted image similarity. Furthermore, the first and second sub-loss values ​​are processed according to a preset method to obtain the second loss value. This improves the efficiency and accuracy of determining the second loss value.

[0211] Preferably, the second loss value, also known as the contrastive loss value (PDC, product / item self-distinctiveness contrastive losses), is mainly to ensure that the distance between the three features of a sample object should be less than the distance between the multi-modal embeddings (image and text features) between the first and second sample objects. In other words... and Should be greater than and The second loss value is calculated as follows:

[0212]

[0213] In Equation 4, L PDC This is the second loss value; N represents the training set size; α2 is the margin value, which is preset. To predict image-text similarity, and Specify the predicted image similarity and the predicted text similarity, respectively. This effectively reduces the expression differences among the three features.

[0214] It should be noted that the first sub-loss value and the second sub-loss value can be determined simultaneously, or the first sub-loss value can be determined first and then the second sub-loss value, or the second sub-loss value can be determined first and then the first sub-loss value. This specification does not limit this.

[0215] Optionally, the process of determining the third loss value based on the predicted image similarity, the predicted text similarity, and the predicted image-text similarity can be implemented as follows:

[0216] The third sub-loss value is determined based on the predicted image similarity and the predicted image-text similarity.

[0217] The fourth sub-loss value is determined based on the predicted text similarity and the predicted image-text similarity.

[0218] The fifth sub-loss value is determined based on the predicted image similarity and the predicted text similarity.

[0219] The third loss value is obtained based on the third sub-loss value, the fourth sub-loss value, and the fifth sub-loss value.

[0220] Specifically, the third sub-loss value refers to the loss value determined based on the predicted image similarity and the predicted image-text similarity; the fourth sub-loss value refers to the loss value determined based on the predicted image-text similarity and the predicted text similarity; and the fifth sub-loss value refers to the loss value determined based on the predicted image similarity and the predicted text similarity.

[0221] In practical applications, a third sub-loss value can be determined based on the difference between predicted image similarity and predicted image-text similarity; a fourth sub-loss value can be determined based on the difference between predicted text similarity and predicted image-text similarity; and a fifth sub-loss value can be determined based on the difference between predicted image similarity and predicted text similarity. Furthermore, the third, fourth, and fifth sub-loss values ​​are processed according to a preset method to obtain the third loss value. This improves the efficiency and accuracy of determining the third loss value.

[0222] Preferably, the third loss value, also known as the consistency loss value (PLC, product / item locality consistency losses), attempts to align image features, text features, and graphic-text features to reduce interference from irrelevant attributes.

[0223] In other words, the three features of a sample object should be as consistent as possible, which can be described as the mean squared error (MSE) of the similarity value. It should be less than a certain specific value, see Equation 5.

[0224]

[0225] In Equation 5, L PLC This is the third loss value; N represents the training set size; α3 is the margin value, which is preset. and These are the predictions for image-text similarity, image similarity, and text similarity, respectively. Furthermore, due to the large computational cost, the top 10 sample pairs can be calculated for each first sample object, with the ranking list of sample pairs arranged in order of... From largest to smallest.

[0226] It should be noted that the determination of the third, fourth, and fifth sub-loss values ​​can be carried out simultaneously or sequentially in a certain order; this manual does not impose any restrictions on this.

[0227] See Figure 3C See Figure 3C This is a schematic diagram illustrating the determination of the total loss value in an object detection method provided in another embodiment of this specification: the left and right items are sample objects, and the descriptive text and object image of both the left and right items constitute a sample pair. The total loss value is determined by a first loss value, a second loss value, and a third loss value; the first loss value is used as an example for explanation here. The descriptive text and object image of the left item are input into the FashionBert model for feature extraction to obtain the text-image features of the left item; the descriptive text and Mash object image (i.e., an empty image) of the left item are input into the FashionBert model for feature extraction to obtain the text features of the left item; the Mash descriptive text (i.e., empty text) and object image (i.e., an empty image) of the left item are input into the FashionBert model for feature extraction to obtain the image features of the left item; the descriptive text and object image of the right item are input into the FashionBert model for feature extraction to obtain the text-image features of the right item. Image-text similarity is calculated based on the image-text features of the left and right items, and an image-text loss value is calculated based on the image-text similarity. Similarly, text similarity is calculated based on the text features of the left and right items, and a text loss value is calculated based on the text similarity. Image similarity is calculated based on the image features of the left and right items, and an image loss value is calculated based on the image similarity. Finally, a first loss value is determined based on the image-text loss value, text loss value, and image loss value.

[0228] Furthermore, the sample objects corresponding to the same training set should maintain the same type but belong to different styles. This operation prompts the similarity prediction model to pay attention to difficult samples and learn the fine-grained differences between different sample objects.

[0229] In one or more optional embodiments of this specification, obtaining the training set can also involve querying information from historical objects to determine each sample object, and then constructing sample pairs based on the sample text and sample image of each sample object to obtain the training set. That is, the specific implementation process for obtaining the training set can be as follows:

[0230] Obtain historical object query information, wherein the historical object query information includes multiple query texts and at least one hit object corresponding to each query text, wherein the hit object refers to the object found and viewed through the query text;

[0231] For any given query text, the correlation between the query text and the corresponding target object is determined based on the operation information of each target object.

[0232] Based on the correlation, multiple sample objects are determined from each hit object;

[0233] Multiple sample pairs are constructed based on sample text and sample images of multiple sample objects to obtain a training set.

[0234] Specifically, historical object query information refers to the information composed of the user's query history for objects; query text refers to the search text, that is, through the query text, users can find the objects they need; hit objects refer to the objects that the user views among all the objects found through the query text. For example, on a shopping platform, if multiple socks are found by searching for "red socks," the socks that the user clicks to view on the details page are the hit objects, including images, titles, keywords, brands, etc.; operation information refers to information such as viewing, favorites, comments, purchases, and additions; relevance refers to the degree of correlation between two physical objects.

[0235] In practical applications, historical query texts containing multiple query texts and at least one corresponding target object can be obtained. Then, for each query text, the operation information of each target object corresponding to the query text is determined from the historical object query information, and the correlation between the query text and each target object corresponding to the query text is calculated based on the operation information and a preset correlation calculation function.

[0236] Furthermore, the hit objects with a relevance greater than a preset relevance threshold can be identified as sample objects, or the hit objects of the query text object can be arranged in descending order of relevance, and the top N hit objects can be identified as sample objects, where N is a preset positive integer.

[0237] Next, a negative sample pair is formed by combining the sample text and sample image of one of the sample objects with those of other sample objects. This creates a negative sample pair, carrying a negative sample representation. Then, a positive sample pair is formed by combining the sample text and sample image of that sample object with the sample text and sample image of that object. This creates a positive sample pair, carrying a positive sample representation. For example, given two sample objects, D1 and D2, the sample text and sample image of object D1, along with the sample text and sample image of object D2, form a positive sample. The sample text and sample image of object D2, along with the sample text and sample image of object D1, form a negative sample. This improves the reliability of the training set, thereby enhancing the robustness and accuracy of the similarity prediction model.

[0238] See Figure 3D , Figure 3D This is a schematic diagram illustrating the effect of constructing samples in an object detection method provided in one embodiment of this specification: two sample objects of the same style are treated as positive sample pairs, such as Left prod1 and Right prod1, Left prod2 and Right prod2, Left prod3 and Right prod3, Left prod4 and Right prod4, and the positive sample is identified as 1; while any sample object and the corresponding sample object from other sample objects in the same batch are combined to form a negative sample pair, such as Left prod3 and Right prod1, Left prod3 and Right prod2, Left prod3 and Right prod4, and the negative sample is identified as 0.

[0239] A major drawback in training an efficient similarity prediction model for object detection is the lack of highly reliable training data: manually labeled sample pairs are very accurate but expensive, while publicly available samples online, such as this type of data, are noisy and cannot cover all categories. Since queries reflect user intent, and their click behavior indicates their interest in different objects, reliable sample objects can be collected from user query click logs (historical object query information), and the accuracy of sample pairs can be ensured based on relevance constraints. Two objects with a deep click level (action information) in the same query should have a high degree of similarity.

[0240] Optionally, in a product recommendation scenario, user actions (click behaviors) can be categorized into five levels: page click, adding to cart, contacting the supplier, placing an order, and payment. Then, a score is determined for each action, i.e., the action score. For any given action, the score is represented as an action coefficient multiplied by the number of actions, where the action coefficient is pre-set. The sum of the scores for different actions is then used as the relevance (weight). See Equation 6.

[0241] Relevance degree = λ1*n1 + λ2*n2 + λ3*n3 + λ4*n4 + λ5*n5 (Equation 6)

[0242] In Equation 6, λ1 to λ5 are the operation coefficients corresponding to page click, add to cart, contact supplier, place order and make payment, respectively, and n1 to n5 are the number of operations corresponding to page click, add to cart, contact supplier, place order and make payment, respectively.

[0243] Furthermore, for more in-depth operational information (click behavior), the corresponding operation coefficient should be set to a larger value to mitigate the impact of accidental clicks by users.

[0244] In one or more optional embodiments of this specification, a heterogeneous graph can be constructed based on the correlation degree, and then sample objects can be determined based on the heterogeneous graph. That is, the specific implementation process of determining multiple sample objects from each hit object based on the correlation degree can be as follows:

[0245] A heterogeneous graph is constructed using each query text and each hit object as nodes and the aforementioned relevance as edges;

[0246] Based on the heterogeneous graph, determine the connection relationships between each hit object;

[0247] Based on the connection relationship, multiple sample objects are determined from each hit object.

[0248] Specifically, a heterogeneous graph refers to a graph structure that includes multiple types of graph structures, that is, a graph structure that combines data or substances with different attributes. In this specification, a heterogeneous graph is a graph structure with query text and hit objects as nodes; the connection relationship refers to whether hit objects are connected through the nodes corresponding to the query text.

[0249] In practical applications, after obtaining the query text and hit objects, a heterogeneous graph can be constructed by treating each query text and each hit object as nodes and the relevance as edges. For example, a click graph (heterogeneous graph) can be constructed using search terms and products. The nodes corresponding to search terms contain the search term text, while the nodes corresponding to products consist of images, titles, keywords, brands, etc. Then, based on the heterogeneous graph, the connection relationship between each pair of hit objects is determined, and multiple sample objects are identified from each hit object based on the connection relationship. In this way, by constructing a heterogeneous graph, identifying the connection relationship between hit objects, and then determining sample objects based on the connection relationship, the reliability of sample objects can be improved, thereby improving the reliability of the training set and thus enhancing the robustness of the similarity prediction model.

[0250] See Figure 3E , Figure 3E This specification provides an embodiment of a heterogeneous graph in an object detection method: The query text has five elements q1-q5, where q1 corresponds to objects t1 and t2, q2 corresponds to objects t2, t3, and t4, q3 corresponds to object t3, q4 corresponds to object t5, and q5 corresponds to object t4. A heterogeneous graph is constructed using the query texts q1-q5 and the corresponding objects t1-t5 as nodes, with the aforementioned correlation degree as edges. Then, based on the connection relationships between the corresponding objects in the heterogeneous graph, sample objects t2, t1, t3, and t4 are selected.

[0251] It should be noted that after constructing the click graph, a weighted sampling strategy can be used to determine the sample objects. Collect no more than four objects that are hit in a single query based on their relevance from highest to lowest, and construct sample pairs accordingly.

[0252] In addition, three constraints can be set to ensure the accuracy of the sample pairs:

[0253] 1) The query text should contain at least a first preset value of text units and a second preset value of core keywords. Some users do not have a clear search intent and will enter semantically ambiguous query texts, such as "dress" or "sneakers." These query texts cover a relatively wide range of categories, and the objects hit are also very scattered, so it is necessary to restrict the query text. Preferably, the query text should contain at least two words and one core keyword. Among them, the core keyword is the noun corresponding to the entity.

[0254] 2) The two matched objects must belong to the same category: Object detection platforms have a hierarchical, fine-grained category structure, with each category containing dozens of subcategories. For example, "women's dress" contains 20 leaf categories, such as "evening dress," "business attire," "casual wear," and "casual wear." A query for "V-neck red maxi dress" will retrieve objects from these categories. Therefore, to collect matched objects of the same type, two matched objects from the same leaf category should be retained. This improves the robustness of the similarity prediction model.

[0255] 3) The similarity between the object image and description text of the hit object is greater than or equal to a preset similarity, meaning the object image and description text of the hit object have a high degree of similarity. In a single query, the operation limits the relevance of the hit objects, but cannot completely guarantee that two hit objects are of the same type. Therefore, when determining sample objects, it is necessary to determine the similarity between the object image and description text of each hit object and filter out hit samples with similarity values ​​lower than the preset similarity. Preferably, the preset similarity is 0.7.

[0256] After performing the above three types of constraint sampling, the matching product pairs containing the same core keywords are selected as the final sample set, which consists of sample objects. This reduces sampling error.

[0257] It should be noted that after training the similarity prediction model, it can also be tested. That is, in the testing phase, the similarity prediction model is used to predict the text features, image features, and image-text features of each predicted object. Based on these features, the similarity between each predicted object is calculated. Then, the objects are ranked in descending order of similarity value, and it is checked whether any of the top N most similar predicted objects are the same as those manually evaluated.

[0258] This specification presents an object detection method based on unified modeling of visual and textual representations. It accurately represents an object (the object to be detected) using both descriptive text and object images (visual content). The method comprises two main components: First, to address the lack of training samples for object detection (similar product retrieval), a sample pair construction method with category and relevance constraints is proposed, i.e., the construction of a high-confidence training data sample. This method can construct high-quality sample pairs from the messy data of user search history (historical object query information), providing a rich and reliable data foundation for training the similarity prediction model. Second, three types of losses (first loss value, second loss value, and third loss value) are designed to reduce the representational differences between images (object images), text (descriptive text), and image-text (object image + descriptive text) from different perspectives. The ensemble objective function unit formed by combining these losses can effectively model these three representations into the same embedding space, thereby fully utilizing image-text features to accurately represent a product. Furthermore, these three types of losses can be directly used in the fine-tuning stage of various pre-trained models, enabling rapid transfer to various task applications.

[0259] The object detection method provided in this specification was extensively evaluated offline on 5,200 manually labeled identical data entries in an e-commerce product pool. Compared with traditional text-to-text matching models, the retrieval accuracy was improved by 68.62%; compared with general image-to-image matching models, the retrieval accuracy was improved by 6.69%, which fully demonstrates the effectiveness of this object detection method.

[0260] It is worth noting that applying this object detection method to a similar-product search engine and then applying this engine to multiple applications such as search and recommendation has yielded significant results: 1) The coverage rate of similar products has increased to over 50%, and the accuracy rate of similar products has increased by 66.16%, achieving full coverage where similar products are available and recommending highly similar products where no similar products are available; 2) In terms of conversion efficiency, L-AB3 in search scenarios has increased by 0.74% overall, and L-AB3 in recommendation scenarios has increased by 1.58%, resulting in significant task conversion; 3) In terms of distribution efficiency, the average number of exposures per user has increased by 0.42%, the average number of clicks per user has increased by 0.21%, and the number of buyers and sellers who have communicated with users has increased by 0.89%, giving users more opportunities to make choices; 4) In terms of merchant health, the number of exposed products has increased by 5.46%, the number of exposed merchants has increased by 2.88%, the number of clicked products has increased by 1.49%, and the number of clicked merchants has increased by 1.44%, significantly improving merchants' product exposure and conversion. Meanwhile, the similar-product search engine based on this object detection method has achieved significant results in various scenarios.

[0261] See Figure 3F , Figure 3FThis diagram illustrates the effect of an object detection method provided in this specification applied to a search engine, according to one embodiment: Users can search for similar or identical products using text, images, or links. For example, they can search for products by entering "new women's clothing" on a shopping platform, or by clicking the "camera" icon on the page to search for products using images. The search page will display product images and information associated with "new women's clothing," and the search results can be displayed in different ways, such as "All," "Customized," "In Stock," and "Stores." Users can then access a similar product display page by clicking or long-pressing an image on the product display page. This page displays product images, product information, a "Customer Service" component for communication, and a "View Details" component that leads to the product's details page. The similar product display page has two options: "Other Stores" and "Similar Products." Selecting "Other Stores" displays similar products from other stores; selecting "Similar Products" displays products similar to the selected product. Furthermore, when displaying similar products, it also shows product images, product information, and a "customer service" component for communication. Additionally, the object detection method provided in this manual demonstrates excellent performance in image-plus-text transfer retrieval, showcasing its feasibility in user-interactive search and style transfer applications. For example, if a user searches for "cup" and a particular cup appears, using a saved search and adding the description "rabbit" will bring up cups related to rabbits. Similarly, if a user searches for "dress" and a black dress appears, using a saved search and adding the description "white" will bring up white dresses similar to the black dress.

[0262] This specification provides an object detection method that acquires descriptive text and object images of at least two objects to be detected; extracts features from the descriptive text and object images of each object to obtain text features, image features, and image-text features for each object; determines the similarity between the first and second objects to be detected based on the text features, image features, and image-text features of the first object to be detected, and the image-text features of the second object to be detected, wherein the first and second objects to be detected are any two objects; determines a target object among the at least two objects to be detected based on the similarity, and processes the target object. By extracting features from the descriptive text and object images of the objects to be detected, obtaining the text features, image features, and image-text features representing the text and image, and then calculating the similarity between the objects to be detected based on the text features, image features, and image-text features, the accuracy of the similarity can be effectively improved, thereby improving the efficiency of object detection.

[0263] See Figure 4 , Figure 4 A flowchart of a commodity testing method provided in one embodiment of this specification is shown, which specifically includes the following steps.

[0264] Step 402: Obtain the product title and product image of the query product and at least one candidate product.

[0265] Specifically, "search product" refers to the product being searched for, such as a product appearing on a shopping platform. By using the "search similar products" function, one can find products similar to this product, and that product is the search product. "Candidate products" refers to products provided in the database. "Product title" refers to the product's name, keywords, etc. "Product image" refers to the corresponding picture or photo of the product, generally referring to the product's homepage image.

[0266] In practical applications, the product to be searched can be determined first: The product is determined based on the user's actions; for example, if the executing entity detects that a user performs a similar product search, then that product is identified as the search product. Further, products provided by the platform are identified as candidate products. Then, the product title and image of the search product, as well as the product titles and images of each candidate product, are obtained, for example, product titles and images captured from the corresponding display page.

[0267] Step 404: Extract features from the product title and the product image to obtain the text features, image features, and image-text features of the queried product and each candidate product.

[0268] Specifically, features refer to the characteristics of things, which can be feature vectors; text features refer to the features corresponding to the product title; image features refer to the features corresponding to the product image; and image-text features refer to the features corresponding to the combination of the product title and the product image.

[0269] In practical applications, for any product in the query and candidate products, a preset feature extraction method can be used to extract features from the product title and image, resulting in text features, image features, and text-image features. This process is repeated for each product to obtain the text features, image features, and text-image features of both the query product and each candidate product.

[0270] Step 406: Calculate the similarity between the text features, image features, and image-text features of the queried product and the image-text features of each candidate product to determine the similarity between the queried product and each candidate product.

[0271] In practical applications, after determining the text features, image features, and image-text features of the query product and the image-text features of each candidate product, for any candidate product, the similarity between the text features, image features, and image-text features of the query product and the image-text features of the candidate product can be calculated to determine the similarity between the query product and the candidate product. This process is repeated for each candidate product to determine the similarity between the query product and each candidate product.

[0272] Step 408: Determine the target product among the at least one candidate products based on the similarity, and process the target product.

[0273] In practical applications, after determining the similarity between the query product and each candidate product, the similarity can be compared with preset similarity detection conditions. The similarity that meets the detection conditions is determined as the target similarity, and the candidate products corresponding to the target similarity are determined as the target products. For example, if the similarity detection condition is greater than a preset similarity threshold, then the similarity greater than the threshold is determined as the target similarity, and the candidate products corresponding to each target similarity are determined as the target products.

[0274] Furthermore, after identifying the target product, it is processed. When the similarity detection condition is greater than a preset similarity threshold, for any target similarity corresponding to a given target product, they are considered the same product (i.e., the query product and the target product are two identical products). Alternatively, the query product and the target product can be deduplicated, and the deduplicated product can be recommended to the user; or the target product can be recommended directly to the user.

[0275] The above is an illustrative scheme of a product detection method according to this embodiment. It should be noted that the technical solution of this product detection method and the technical solution of the object detection method described above belong to the same concept. For details not described in detail in the technical solution of the product detection method, please refer to the description of the technical solution of the object detection method described above.

[0276] This specification provides a product detection method, which involves acquiring the product titles and images of a query product and at least one candidate product; extracting features from the product titles and images to obtain text features, image features, and image-text features of the query product and each candidate product; calculating the similarity between the text features, image features, and image-text features of the query product and the image-text features of each candidate product to determine the similarity between the query product and each candidate product; determining the target product among the at least one candidate product based on the similarity, and processing the target product. By extracting features from the product titles and images of the query product and each candidate product to obtain the text features, image features, and image-text features representing the text and image of the query product and each candidate product, and then calculating the similarity between the query product and each candidate product based on the text features, image features, and image-text features, the accuracy of the similarity can be effectively improved, thereby improving the efficiency of product detection.

[0277] See Figure 5 , Figure 5 The flowchart of a similarity prediction model training method provided in one embodiment of this specification is shown, which specifically includes the following steps.

[0278] Step 502: Obtain a training set, wherein the training set contains at least one sample pair, the sample pair including sample text and sample image of two sample objects.

[0279] Step 504: Input the sample text and sample image of each sample object in each sample pair into the feature extraction layer of the model to be trained to obtain the predicted text features, predicted image features and predicted image-text features of each sample object.

[0280] Step 506: For any sample pair corresponding to the first sample object and the second sample object, input the predicted text features, predicted image features, and predicted image-text features of the first sample object and the predicted image-text features of the second sample object into the similarity determination layer of the model to be trained, and obtain the predicted similarity between the first sample object and the second sample object.

[0281] Step 508: Determine the total loss value based on the predicted similarity, and adjust the model parameters of the feature extraction layer and the similarity determination layer based on the total loss value. Continue to execute the step of obtaining the training set. Stop training when the preset training stopping condition is met, and obtain the similarity prediction model.

[0282] Optionally, the predicted similarity includes predicted text similarity, predicted image similarity, and image-text predicted similarity, and the similarity determination layer includes a text similarity determination sublayer, an image similarity determination sublayer, and an image-text similarity determination sublayer;

[0283] Accordingly, the step of inputting the predicted text features, predicted image features, and predicted image-text features of the first sample object and the predicted image-text features of the second sample object into the similarity determination layer of the model to be trained to obtain the predicted similarity between the first sample object and the second sample object includes:

[0284] The predicted text features of the first sample object and the predicted image-text features of the second sample object are input into the text similarity determination sub-layer to obtain the predicted text similarity between the first sample object and the second sample object;

[0285] The predicted image features of the first sample object and the predicted image-text features of the second sample object are input into the image similarity determination sub-layer to obtain the predicted image similarity between the first sample object and the second sample object;

[0286] The predicted image-text features of the first sample object and the predicted image-text features of the second sample object are input into the image-text similarity determination sub-layer to obtain the predicted image-text similarity between the first sample object and the second sample object.

[0287] Optionally, determining the total loss value based on the predicted similarity includes:

[0288] The predicted image-text similarity, predicted image similarity, and predicted text similarity corresponding to the positive sample pair are respectively determined as the specified predicted image-text similarity, specified predicted image similarity, and specified predicted text similarity, wherein the positive sample pair is a sample pair carrying a positive sample identifier;

[0289] A first loss value is determined based on the predicted image-text similarity, the predicted image similarity, the predicted text similarity, the specified predicted image-text similarity, the specified predicted image similarity, and the specified predicted text similarity;

[0290] A second loss value is determined based on the predicted image-text similarity, the specified predicted text similarity, and the specified predicted image similarity;

[0291] A third loss value is determined based on the predicted image similarity, the predicted text similarity, and the predicted image-text similarity;

[0292] The total loss value is determined based on the first loss value, the second loss value, and the third loss value.

[0293] Optionally, determining the second loss value based on the predicted image-text similarity, the specified predicted text similarity, and the specified predicted image similarity includes:

[0294] The first sub-loss value is determined based on the predicted image-text similarity and the specified predicted text similarity.

[0295] The second sub-loss value is determined based on the predicted image-text similarity and the specified predicted image similarity.

[0296] Based on the first sub-loss value and the second sub-loss value, the second loss value is obtained.

[0297] Optionally, the feature extraction layer includes a splitting sub-layer, a feature extraction sub-layer, and a feature fusion sub-layer;

[0298] Accordingly, the step of inputting the sample text and sample image of each sample object in each sample pair into the feature extraction layer of the model to be trained to obtain the predicted text features, predicted image features, and predicted image-text features of each sample object includes:

[0299] For any sample object, the sample text and sample image of the sample object are input into the splitting sub-layer to obtain multiple sub-sample texts and multiple sub-sample images;

[0300] Each subsample text and each subsample image are input into the feature extraction sublayer to obtain the predicted subtext features of each subsample text and the predicted subimage features of each subsample image.

[0301] The predicted sub-text features and the predicted sub-image features are input into the feature fusion sub-layer to obtain the predicted text features, predicted image features, and predicted image-text features of the sample object.

[0302] Optionally, obtaining the training set includes:

[0303] Obtain historical object query information, wherein the historical object query information includes multiple query texts and at least one hit object corresponding to each query text, wherein the hit object refers to the object found and viewed through the query text;

[0304] For any given query text, the correlation between the query text and the corresponding target object is determined based on the operation information of each target object.

[0305] Based on the correlation, multiple sample objects are determined from each hit object;

[0306] Multiple sample pairs are constructed based on sample text and sample images of multiple sample objects to obtain a training set.

[0307] Optionally, determining multiple sample objects from each hit object based on the correlation degree includes:

[0308] A heterogeneous graph is constructed using each query text and each hit object as nodes and the aforementioned relevance as edges;

[0309] Based on the heterogeneous graph, determine the connection relationships between each hit object;

[0310] Based on the connection relationship, multiple sample objects are determined from each hit object.

[0311] This specification provides a method for training a similarity prediction model. The method involves obtaining a training set containing at least one sample pair, where each sample pair includes sample text and sample images of two sample objects. The sample text and sample images of each sample object in each sample pair are input into the feature extraction layer of the model to be trained, yielding predicted text features, predicted image features, and predicted image-text features for each sample object. For any sample pair corresponding to a first and second sample object, the predicted text features, predicted image features, and predicted image-text features of the first sample object, along with the predicted image-text features of the second sample object, are input into the similarity determination layer of the model to be trained, yielding a predicted similarity between the first and second sample objects. A total loss value is determined based on the predicted similarity, and the model parameters of the feature extraction layer and the similarity determination layer are adjusted based on the total loss value. The method continues to execute the steps of obtaining the training set, and training is stopped when a preset training stopping condition is met, thus obtaining a similarity prediction model. By extracting features from the sample text and sample images of each sample object, we obtain the predicted text features, predicted image features, and predicted image-text features representing the text and image of each sample object. Then, based on the predicted text features, predicted image features, and predicted image-text features, we calculate the predicted similarity of each sample pair. This can effectively improve the accuracy of the predicted similarity, thereby improving the reliability and accuracy of the total loss value determined based on the predicted similarity. This, in turn, improves the training efficiency and robustness of the similarity prediction model, and ultimately improves the efficiency of object detection.

[0312] The above is an illustrative scheme of a similarity prediction model training method according to this embodiment. It should be noted that the technical solution of this similarity prediction model training method belongs to the same concept as the technical solution of the object detection method described above. For details not described in detail in the technical solution of the similarity prediction model training method, please refer to the description of the technical solution of the object detection method described above.

[0313] Corresponding to the above method embodiments, this specification also provides embodiments of object detection devices. Figure 6 A schematic diagram of an object detection device according to one embodiment of this specification is shown. Figure 6 As shown, the device includes:

[0314] The first acquisition module 602 is configured to acquire at least two descriptive texts and object images of objects to be detected;

[0315] The first extraction module 604 is configured to extract features from the descriptive text and object image of each object to be detected, and obtain the text features, image features and image-text features of each object to be detected.

[0316] The first similarity determination module 606 is configured to determine the similarity between the first object to be detected and the second object to be detected based on the text features, image features, and image-text features of the first object to be detected and the image-text features of the second object to be detected, wherein the first object to be detected and the second object to be detected are any two objects to be detected;

[0317] The target object determination module 608 is configured to determine the target object among the at least two objects to be detected based on the similarity, and to process the target object.

[0318] Optionally, the similarity includes text similarity, image similarity, and image-text similarity;

[0319] Accordingly, the first similarity determination module 606 is further configured as follows:

[0320] The text features of the first object to be detected and the image features of the second object to be detected are used to calculate the similarity, and the text similarity between the first object to be detected and the second object to be detected is obtained.

[0321] The image features of the first object to be detected and the image and text features of the second object to be detected are compared to calculate the similarity, and the image similarity between the first object to be detected and the second object to be detected is obtained.

[0322] The similarity between the image and text features of the first object to be detected and the image and text features of the second object to be detected is calculated to obtain the image and text similarity between the first object to be detected and the second object to be detected.

[0323] Optionally, the first extraction module 604 is further configured to:

[0324] For any one of the at least two objects to be detected, the description text of the object to be detected is split into multiple sub-description texts, and the object image of the object to be detected is split into multiple sub-object images;

[0325] Feature extraction is performed on each sub-description text to obtain the sub-text features of each sub-description text, and feature extraction is performed on each sub-object image to obtain the sub-image features of each sub-object image;

[0326] Based on the sub-text features and the sub-image features, the text features, image features, and image-text features of the object to be detected are obtained.

[0327] Optionally, the device further includes a training module configured to:

[0328] Obtain a pre-trained similarity prediction model, wherein the similarity prediction model includes a feature extraction layer and a similarity determination layer;

[0329] Accordingly, the first extraction module 604 is further configured as follows:

[0330] The descriptive text and object image of each object to be detected are input into the feature extraction layer for feature extraction, thereby obtaining the text features, image features and image-text features of each object to be detected;

[0331] Accordingly, the first similarity determination module 606 is further configured as follows:

[0332] The text features, image features, and image-text features of the first object to be detected, as well as the image-text features of the second object to be detected, are input into the similarity determination layer to calculate the similarity between the first object to be detected and the second object to be detected.

[0333] This specification provides an object detection device that acquires descriptive text and object images of at least two objects to be detected; extracts features from the descriptive text and object images of each object to obtain text features, image features, and image-text features for each object; determines the similarity between the first object and the second object based on the text features, image features, and image-text features of the first object and the second object, wherein the first object and the second object are any two objects to be detected; determines a target object among the at least two objects to be detected based on the similarity, and processes the target object. By extracting features from the descriptive text and object images of the objects to be detected, obtaining the text features, image features, and image-text features representing the text and image, and then calculating the similarity between the objects to be detected based on the text features, image features, and image-text features, the accuracy of the similarity can be effectively improved, thereby improving the efficiency of object detection.

[0334] The above is a schematic scheme of an object detection device according to this embodiment. It should be noted that the technical solution of this object detection device and the technical solution of the object detection method described above belong to the same concept. For details not described in detail in the technical solution of the object detection device, please refer to the description of the technical solution of the object detection method described above.

[0335] Corresponding to the above method embodiments, this specification also provides embodiments of a commodity detection device. Figure 7 A schematic diagram of a commodity detection device according to one embodiment of this specification is shown. Figure 7As shown, the device includes:

[0336] The second acquisition module 702 is configured to acquire the product title and product image of the queried product and at least one candidate product;

[0337] The second extraction module 704 is configured to extract features from the product title and the product image to obtain the text features, image features and image-text features of the queried product and each candidate product;

[0338] The second similarity determination module 706 is configured to calculate the similarity between the text features, image features, and image-text features of the query product and the image-text features of each candidate product, thereby determining the similarity between the query product and each candidate product.

[0339] The target product determination module 708 is configured to determine the target product among the at least one candidate products based on the similarity and to process the target product.

[0340] This specification provides a product detection device that extracts features from the product titles and images of the query product and each candidate product to obtain text features, image features, and graphic features representing the text and images of the query product and each candidate product. Then, based on the text features, image features, and graphic features, the similarity between the query product and each candidate product is calculated, which can effectively improve the accuracy of similarity and thus improve the efficiency of product detection.

[0341] The above is a schematic diagram of a commodity detection device according to this embodiment. It should be noted that the technical solution of this commodity detection device and the technical solution of the commodity detection method described above belong to the same concept. For details not described in detail in the technical solution of the commodity detection device, please refer to the description of the technical solution of the commodity detection method described above.

[0342] Corresponding to the above method embodiments, this specification also provides embodiments of a similarity prediction model training device. Figure 8 This diagram illustrates the structure of a similarity prediction model training device according to one embodiment of this specification. Figure 8 As shown, the device includes:

[0343] The third acquisition module 802 is configured to acquire a training set, wherein the training set contains at least one sample pair, the sample pair including sample text and sample image of two sample objects;

[0344] The first input module 804 is configured to input the sample text and sample image of each sample object in each sample pair into the feature extraction layer of the model to be trained, so as to obtain the predicted text features, predicted image features and predicted image-text features of each sample object.

[0345] The second input module 806 is configured to, for any sample pair corresponding to a first sample object and a second sample object, input the predicted text features, predicted image features, and predicted image-text features of the first sample object and the predicted image-text features of the second sample object into the similarity determination layer of the model to be trained, so as to obtain the predicted similarity between the first sample object and the second sample object.

[0346] The adjustment module 808 is configured to determine the total loss value based on the predicted similarity, and adjust the model parameters of the feature extraction layer and the similarity determination layer based on the total loss value, continue to execute the step of obtaining the training set, stop training when the preset training stopping condition is reached, and obtain the similarity prediction model.

[0347] Optionally, the predicted similarity includes predicted text similarity, predicted image similarity, and image-text predicted similarity, and the similarity determination layer includes a text similarity determination sublayer, an image similarity determination sublayer, and an image-text similarity determination sublayer;

[0348] Accordingly, the second input module 806 is further configured as follows:

[0349] The predicted text features of the first sample object and the predicted image-text features of the second sample object are input into the text similarity determination sub-layer to obtain the predicted text similarity between the first sample object and the second sample object;

[0350] The predicted image features of the first sample object and the predicted image-text features of the second sample object are input into the image similarity determination sub-layer to obtain the predicted image similarity between the first sample object and the second sample object;

[0351] The predicted image-text features of the first sample object and the predicted image-text features of the second sample object are input into the image-text similarity determination sub-layer to obtain the predicted image-text similarity between the first sample object and the second sample object.

[0352] Optionally, the adjustment module 808 is further configured to:

[0353] The predicted image-text similarity, predicted image similarity, and predicted text similarity corresponding to the positive sample pair are respectively determined as the specified predicted image-text similarity, specified predicted image similarity, and specified predicted text similarity, wherein the positive sample pair is a sample pair carrying a positive sample identifier;

[0354] A first loss value is determined based on the predicted image-text similarity, the predicted image similarity, the predicted text similarity, the specified predicted image-text similarity, the specified predicted image similarity, and the specified predicted text similarity;

[0355] A second loss value is determined based on the predicted image-text similarity, the specified predicted text similarity, and the specified predicted image similarity;

[0356] A third loss value is determined based on the predicted image similarity, the predicted text similarity, and the predicted image-text similarity;

[0357] The total loss value is determined based on the first loss value, the second loss value, and the third loss value.

[0358] Optionally, the adjustment module 808 is further configured to:

[0359] The first sub-loss value is determined based on the predicted image-text similarity and the specified predicted text similarity.

[0360] The second sub-loss value is determined based on the predicted image-text similarity and the specified predicted image similarity.

[0361] Based on the first sub-loss value and the second sub-loss value, the second loss value is obtained.

[0362] Optionally, the feature extraction layer includes a splitting sub-layer, a feature extraction sub-layer, and a feature fusion sub-layer;

[0363] Accordingly, the first input module 804 is further configured as follows:

[0364] For any sample object, the sample text and sample image of the sample object are input into the splitting sub-layer to obtain multiple sub-sample texts and multiple sub-sample images;

[0365] Each subsample text and each subsample image are input into the feature extraction sublayer to obtain the predicted subtext features of each subsample text and the predicted subimage features of each subsample image.

[0366] The predicted sub-text features and the predicted sub-image features are input into the feature fusion sub-layer to obtain the predicted text features, predicted image features, and predicted image-text features of the sample object.

[0367] Optionally, the third acquisition module 802 is further configured to:

[0368] Obtain historical object query information, wherein the historical object query information includes multiple query texts and at least one hit object corresponding to each query text, wherein the hit object refers to the object found and viewed through the query text;

[0369] For any given query text, the correlation between the query text and the corresponding target object is determined based on the operation information of each target object.

[0370] Based on the correlation, multiple sample objects are determined from each hit object;

[0371] Multiple sample pairs are constructed based on sample text and sample images of multiple sample objects to obtain a training set.

[0372] Optionally, the third acquisition module 802 is further configured to:

[0373] A heterogeneous graph is constructed using each query text and each hit object as nodes and the aforementioned relevance as edges;

[0374] Based on the heterogeneous graph, determine the connection relationships between each hit object;

[0375] Based on the connection relationship, multiple sample objects are determined from each hit object.

[0376] This specification provides a similarity prediction model training device. By extracting features from the sample text and sample images of each sample object, it obtains the predicted text features, predicted image features, and predicted image-text features representing the text and image of each sample object. Then, based on the predicted text features, predicted image features, and predicted image-text features, it calculates the predicted similarity of each sample pair. This can effectively improve the accuracy of the predicted similarity, thereby improving the reliability and accuracy of the total loss value determined based on the predicted similarity, thus improving the training efficiency and robustness of the similarity prediction model, and ultimately improving the efficiency of object detection.

[0377] The above is a schematic scheme of a similarity prediction model training device according to this embodiment. It should be noted that the technical solution of this similarity prediction model training device and the technical solution of the above-described similarity prediction model training method belong to the same concept. For details not described in detail in the technical solution of the similarity prediction model training device, please refer to the description of the technical solution of the above-described similarity prediction model training method.

[0378] Figure 9 A structural block diagram of a computing device 900 according to one embodiment of this specification is shown. The components of the computing device 900 include, but are not limited to, a memory 910 and a processor 920. The processor 920 is connected to the memory 910 via a bus 930, and a database 950 is used to store data.

[0379] The computing device 900 also includes an access device 940, which enables the computing device 900 to communicate via one or more networks 960. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 940 may include one or more of any type of wired or wireless network interface (e.g., Network Interface Controller (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0380] In one embodiment of this specification, the aforementioned components of the computing device 900 and Figure 9 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 9 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0381] The computing device 900 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or PCs. The computing device 900 can also be a mobile or stationary server.

[0382] The processor 920 is used to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above-mentioned object detection method, commodity detection method, or similarity prediction model training method.

[0383] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device belongs to the same concept as the technical solutions of the object detection method, the product detection method, or the similarity prediction model training method described above. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solutions of the object detection method, the product detection method, or the similarity prediction model training method described above.

[0384] An embodiment of this specification also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the object detection method, the product detection method, or the similarity prediction model training method described above.

[0385] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium belongs to the same concept as the technical solutions of the object detection method, the product detection method, or the similarity prediction model training method described above. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solutions of the object detection method, the product detection method, or the similarity prediction model training method described above.

[0386] An embodiment of this specification also provides a computer program, wherein when the computer program is executed in a computer, it causes the computer to perform the steps of the above-described object detection method, product detection method, or similarity prediction model training method.

[0387] The above is an illustrative example of a computer program according to this embodiment. It should be noted that the technical solution of this computer program belongs to the same concept as the technical solutions of the object detection method, product detection method, or similarity prediction model training method described above. Details not described in detail in the technical solution of the computer program can be found in the descriptions of the technical solutions of the object detection method, product detection method, or similarity prediction model training method described above.

[0388] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0389] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0390] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0391] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0392] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. An object detection method, comprising: Obtain descriptive text and object images of at least two objects to be detected, as well as a pre-trained similarity prediction model; The similarity prediction model is used to extract features from the descriptive text and object images of each object to be detected, resulting in text features, image features, and image-text features for each object. The image-text features are obtained by weighted summation of multiple sub-text features corresponding to the descriptive text and multiple sub-image features corresponding to the object image. During training, the similarity prediction model adjusts its parameters based on the total loss value, which is determined based on a first loss value, a second loss value, and a third loss value. The first loss value guides positive samples to have a greater similarity value compared to negative sample pairs, where positive sample pairs are pairs of the same type of object and negative sample pairs are pairs of different types of object samples. The second loss value ensures that the distance between the three features of a sample should be less than the distance between the multi-modal embeddings of the first and second sample objects. The third loss value is used to align the image features, text features, and image-text features. Based on the text features, image features, and image-text features of the first object to be detected, and the image-text features of the second object to be detected, the similarity between the first object to be detected and the second object to be detected is determined. The first object to be detected and the second object to be detected are any two objects to be detected. The similarity includes text similarity, image similarity, and image-text similarity. The target object is determined from the at least two objects to be detected based on the similarity, and the target object is then processed.

2. The method according to claim 1, wherein determining the similarity between the first object to be detected and the second object to be detected based on the text features, image features, and image-text features of the first object to be detected, and the image-text features of the second object to be detected, comprises: The text features of the first object to be detected and the image features of the second object to be detected are used to calculate the similarity, and the text similarity between the first object to be detected and the second object to be detected is obtained. The image features of the first object to be detected and the image and text features of the second object to be detected are compared to calculate the similarity, and the image similarity between the first object to be detected and the second object to be detected is obtained. The similarity between the image and text features of the first object to be detected and the image and text features of the second object to be detected is calculated to obtain the image and text similarity between the first object to be detected and the second object to be detected.

3. The method according to claim 1, wherein feature extraction is performed on the descriptive text and object image of each object to be detected to obtain text features, image features, and image-text features of each object to be detected, comprising: For any one of the at least two objects to be detected, the description text of the object to be detected is split into multiple sub-description texts, and the object image of the object to be detected is split into multiple sub-object images; Feature extraction is performed on each sub-description text to obtain the sub-text features of each sub-description text, and feature extraction is performed on each sub-object image to obtain the sub-image features of each sub-object image; Based on the sub-text features and the sub-image features, the text features, image features, and image-text features of the object to be detected are obtained.

4. The method according to any one of claims 1-3, wherein the similarity prediction model comprises a feature extraction layer and a similarity determination layer; Accordingly, the step of extracting features from the descriptive text and object image of each object to be detected, to obtain the text features, image features, and image-text features of each object to be detected, includes: The descriptive text and object image of each object to be detected are input into the feature extraction layer for feature extraction, thereby obtaining the text features, image features and image-text features of each object to be detected; Accordingly, determining the similarity between the first object to be detected and the second object to be detected based on the text features, image features, and image-text features of the first object to be detected, and the image-text features of the second object to be detected, includes: The text features, image features, and image-text features of the first object to be detected, as well as the image-text features of the second object to be detected, are input into the similarity determination layer to calculate the similarity between the first object to be detected and the second object to be detected.

5. A method for detecting a commodity, comprising: Obtain the product title and product image of the query product and at least one candidate product, as well as a pre-trained similarity prediction model; The similarity prediction model is used to extract features from the product title and the product image to obtain text features, image features, and image-text features of the query product and each candidate product. The image-text features are obtained by weighted summation of multiple sub-text features corresponding to the product title and multiple sub-image features corresponding to the product image. During training, the similarity prediction model adjusts its parameters based on the total loss value, which is determined based on a first loss value, a second loss value, and a third loss value. The first loss value guides positive samples to have a greater similarity value compared to negative sample pairs. The positive sample pairs are samples of the same product, and the negative sample pairs are samples of different products. The second loss value ensures that the distance between the three features of a sample should be less than the distance between the multi-modal embeddings of the first and second sample objects. The third loss value is used to align the image features, text features, and image-text features. The similarity between the text features, image features, and image-text features of the query product and the image-text features of each candidate product is calculated to determine the similarity between the query product and each candidate product. The similarity includes text similarity, image similarity, and image-text similarity. The target product is determined from the at least one candidate product based on the similarity, and the target product is then processed.

6. A method for training a similarity prediction model, comprising: Obtain a training set, wherein the training set contains at least one sample pair, the sample pair comprising sample text and sample image of two sample objects; The sample text and sample image of each sample object in each sample pair are input into the feature extraction layer of the model to be trained to obtain the predicted text features, predicted image features and predicted image-text features of each sample object. The predicted image-text features are obtained by weighted summation of multiple predicted sub-text features corresponding to the sample text and multiple predicted sub-image features corresponding to the sample image. For any sample pair corresponding to a first sample object and a second sample object, the predicted text features, predicted image features, and predicted image-text features of the first sample object and the predicted image-text features of the second sample object are input into the similarity determination layer of the model to be trained to obtain the predicted similarity between the first sample object and the second sample object, wherein the predicted similarity includes predicted text similarity, predicted image similarity, and predicted image-text similarity; The total loss value is determined based on the predicted similarity, and the model parameters of the feature extraction layer and the similarity determination layer are adjusted based on the total loss value. The step of obtaining the training set is continued, and training is stopped when the preset training stopping condition is met, thus obtaining the similarity prediction model. The total loss value is determined based on a first loss value, a second loss value, and a third loss value. The first loss value is used to guide positive samples to have a greater similarity value compared to negative sample pairs. The positive sample pairs are sample pairs of the same type of object, and the negative sample pairs are sample pairs of sample pairs of different types of objects. The second loss value is used to ensure that the distance between the three features of a sample should be less than the distance of the multi-modal embedding between the first sample object and the second sample object. The third loss value is used to align image features, text features, and image-text features.

7. The method according to claim 6, wherein the predicted similarity includes predicted text similarity, predicted image similarity, and image-text predicted similarity, and the similarity determination layer includes a text similarity determination sublayer, an image similarity determination sublayer, and an image-text similarity determination sublayer; Accordingly, the step of inputting the predicted text features, predicted image features, and predicted image-text features of the first sample object and the predicted image-text features of the second sample object into the similarity determination layer of the model to be trained to obtain the predicted similarity between the first sample object and the second sample object includes: The predicted text features of the first sample object and the predicted image-text features of the second sample object are input into the text similarity determination sub-layer to obtain the predicted text similarity between the first sample object and the second sample object; The predicted image features of the first sample object and the predicted image-text features of the second sample object are input into the image similarity determination sub-layer to obtain the predicted image similarity between the first sample object and the second sample object; The predicted image-text features of the first sample object and the predicted image-text features of the second sample object are input into the image-text similarity determination sub-layer to obtain the predicted image-text similarity between the first sample object and the second sample object.

8. The method according to claim 7, wherein determining the total loss value based on the predicted similarity comprises: The predicted image-text similarity, predicted image similarity, and predicted text similarity corresponding to the positive sample pair are respectively determined as the specified predicted image-text similarity, specified predicted image similarity, and specified predicted text similarity, wherein the positive sample pair is a sample pair carrying a positive sample identifier; A first loss value is determined based on the predicted image-text similarity, the predicted image similarity, the predicted text similarity, the specified predicted image-text similarity, the specified predicted image similarity, and the specified predicted text similarity; A second loss value is determined based on the predicted image-text similarity, the specified predicted text similarity, and the specified predicted image similarity; A third loss value is determined based on the predicted image similarity, the predicted text similarity, and the predicted image-text similarity; The total loss value is determined based on the first loss value, the second loss value, and the third loss value.

9. The method according to claim 8, wherein determining the second loss value based on the predicted image-text similarity, the specified predicted text similarity, and the specified predicted image similarity comprises: The first sub-loss value is determined based on the predicted image-text similarity and the specified predicted text similarity. The second sub-loss value is determined based on the predicted image-text similarity and the specified predicted image similarity. Based on the first sub-loss value and the second sub-loss value, the second loss value is obtained.

10. The method according to claim 6, wherein the feature extraction layer comprises a splitting sub-layer, a feature extraction sub-layer, and a feature fusion sub-layer; Accordingly, the step of inputting the sample text and sample image of each sample object in each sample pair into the feature extraction layer of the model to be trained to obtain the predicted text features, predicted image features, and predicted image-text features of each sample object includes: For any sample object, the sample text and sample image of the sample object are input into the splitting sub-layer to obtain multiple sub-sample texts and multiple sub-sample images; Each subsample text and each subsample image are input into the feature extraction sublayer to obtain the predicted subtext features of each subsample text and the predicted subimage features of each subsample image. The predicted sub-text features and the predicted sub-image features are input into the feature fusion sub-layer to obtain the predicted text features, predicted image features, and predicted image-text features of the sample object.

11. The method according to claim 6, wherein obtaining the training set comprises: Obtain historical object query information, wherein the historical object query information includes multiple query texts and at least one hit object corresponding to each query text, wherein the hit object refers to the object found and viewed through the query text; For any given query text, the correlation between the query text and the corresponding target object is determined based on the operation information of each target object. Based on the correlation, multiple sample objects are determined from each hit object; Multiple sample pairs are constructed based on sample text and sample images of multiple sample objects to obtain a training set.

12. The method according to claim 11, wherein determining multiple sample objects from each hit object based on the correlation degree includes: A heterogeneous graph is constructed using each query text and each hit object as nodes and the aforementioned relevance as edges; Based on the heterogeneous graph, determine the connection relationships between each hit object; Based on the connection relationship, multiple sample objects are determined from each hit object.

13. A computing device, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1-4, 5, or 6-12.

14. A computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1-4, 5, or 6-12.

15. A computer program product, characterized in that, Includes computer instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1-4, 5, or 6-12.

Images