A combined product retrieval method and system based on multimodal pre-training model

Abstract

The invention discloses a combined commodity retrieval method and system based on a multimodal pre-training model, wherein the method comprises the following steps: dividing commodity images into single-product images and combined commodity images; training a combined commodity image detector; acquiring and combining feature coding, position coding and segment coding of text modalities and picture modules in the combined commodity images, learning embedding representation, and inputting the constructed multimodal pre-training model; using bounding boxes and bounding box features extracted by the commodity detector as image features, combining with text features, and inputting the multimodal pre-training model for self-supervised training; using the multimodal pre-training model to extract retrieval features of picture modalities and text modalities of single-product images, and storing them in a retrieval library; the multimodal pre-training model extracts retrieval features of image and text fusion according to the bounding boxes and bounding box features of each target commodity in the combined commodity image, calculating the pre-distance between the combined commodity features and the single-product features in the retrieval library as commodity similarity, and selecting the most similar single product as the result to return.

Description

Technical Field

[0001] The present invention relates to the technical field of product retrieval, and more specifically, to a combined product retrieval method and system based on a multimodal pre-training model. Background Art

[0002] The development of Internet technology has led to the rapid expansion of online e-commerce platforms. Due to their convenience, e-commerce platforms are becoming more and more popular. The variety of products in the e-commerce field and the shopping needs of users have greatly increased. Online products are diverse, and more products are presented in the form of sets, that is, multiple different products are combined in a set product. At the same time, when users browse a set product, they may need to query the corresponding single product of the set in order to compare prices or purchase them separately. In real scenarios where the data scale is large and lacks annotations, how to perform multi-modal combination product retrieval is a practical but unresolved problem.

[0003] Combined product retrieval has high practical value and application prospects in the field of e-commerce. First, it is conducive to improving the accuracy of product search and helping online users search for more accurate and specific products; second, it can be used to build e-commerce knowledge graphs and mine product relationships; third, the product relationships obtained through combined product retrieval can be used for product recommendations, improving the recommendation effect of shopping platforms.

[0004] However, in the field of product retrieval, existing methods all input single-modal data, such as a text or an image, and then perform matching searches in the retrieval library. However, in the field of e-commerce, both images and texts are widely present in the retrieval library. Due to the lack of full utilization of multi-modal data, the current retrieval method greatly limits the actual usage scenarios. More importantly, existing methods mainly focus on relatively simple cases, such as image-level retrieval. Image-level retrieval does not determine whether there are multiple objects in the product image and distinguish these objects. Instance-level product retrieval is to retrieve all items in the combined product, and this retrieval method has not been explored yet. In addition, previous methods rely on annotation information for training, which lacks generalization when used in large-scale real-scene datasets. Summary of the invention

[0005] In order to solve the problem of low accuracy caused by current product retrieval relying on single-modal data and image-level retrieval, the present invention provides a combined product retrieval method and system based on a multimodal pre-training model, which has the advantages of high generalization, high availability and high accuracy.

[0006] In order to achieve the above-mentioned purpose of the present invention, the technical scheme adopted is as follows:

[0007] A combined product retrieval method based on a multimodal pre-training model, the method comprising the following steps:

[0008] S1: Divide the product image into a single product image and a combination product image, wherein the single product image indicates that there is only one product, and the combination product image indicates that there are multiple independent products;

[0009] S2: Train a combined product image detector to detect each independent product in the combined product image;

[0010] S3: Obtain and combine the feature encoding, position encoding, and segment encoding of the text modality and image module in the combined product image to learn the embedded representation;

[0011] S4: Build a multimodal pre-training model and use the learned embedding representation as the input of the multimodal pre-training model;

[0012] S5: The bounding box and bounding box features extracted by the product detector are used as image features, combined with text features, and input into the multimodal pre-training model for self-supervised training;

[0013] S6: Input the image modality and text modality of the single product image into the multimodal pre-training model to extract retrieval features, and store the retrieval features in the retrieval library;

[0014] S7: Extract the bounding box and bounding box features of each target product in the combination product image, input the multimodal pre-trained model trained in step S5, extract the retrieval features of image and text fusion, calculate the cosine distance between the combination product features and the single product features in the retrieval library as the feature similarity, and select the most similar single product as the result and return it.

[0015] Preferably, step S1 uses a rule-based method to divide product images into single product images and combination product images, and classifies product images with product titles such as "set", "combination", "set", and "piece set" as combination product images, otherwise they are classified as single product images.

[0016] Furthermore, a data enhancement method is used to train the combined product image detector, as follows:

[0017] First, calculate the color area ratio in the single product image, select the image with a white area ratio greater than 60% as the white background image, use the GrabCut segmentation method to obtain the foreground of the single product image, and then select a natural image as the background. Use the "copy and paste" method to combine multiple foreground objects and background images into a picture with bounding box annotations;

[0018] According to the position where the foreground object is pasted on the background image, the bounding box coordinates of the foreground object are selected as the bounding box annotation for image detector training. Using the two categories of foreground and background, a combined product image detector is trained to detect each independent product in the combined product.

[0019] Furthermore, for image modal data, a bottom-up-attention network is used to extract the features of the main target box and its coordinate position.

[0020] Furthermore, for the image frames and frame features output by the bottom-up-attention network, a 5-dimensional vector is used to calculate the position information of each frame, including the coordinates of the upper left corner and the lower right corner of the frame, and the size ratio of the frame to the entire image. This 5-dimensional vector is passed into the linear fully connected layer to obtain the position code; 0 is used as the segmentation information and passed into the linear fully connected layer to obtain the segmentation code; the frame features are passed into the linear fully connected layer to obtain the encoding of the frame features; finally, the position code, segmentation code and feature code are added together to obtain the embedded representation of the image modality.

[0021] Furthermore, for text modal data, the Bert network is used to extract the relational features between different tokens.

[0022] Furthermore, for text sequences, use an increasing natural number sequence to represent their position information, and pass it into the linear fully connected layer to obtain the position code; use 1 as the segmentation information and pass it into the linear fully connected layer to obtain the segmentation code; pass the text into the linear fully connected layer to obtain the feature code of the text; finally, add the position code, segmentation code and feature code to obtain the embedded representation of the text.

[0023] Furthermore, the multimodal pre-training model uses three Transformer network layers to extract retrieval features that integrate images and texts, including:

[0024] The text-visual Transformer network is used to extract shallow features of text and shallow features of images based on the input image embedding representation and text embedding representation;

[0025] The cross Transformer network is used to correlate the image modality and the text modality based on the line-level features of the image and the line-level features of the text, and obtain the features after the two modalities are correlated;

[0026] The public Transformer network is used to extract image features that fully integrate text and images.

[0027] Furthermore, in step S5, the self-supervised training is specifically as follows:

[0028] By masking the words in the title text, the text sequence with the masked words is input into the trained multimodal pre-trained model. During the training process, the masked words are learned to be restored, thereby extracting a feature representation with text information.

[0029] By masking the bounding box features in the image, the masked image box feature sequence is input into the multimodal pre-trained model, and the masked bounding box features are restored during the training process to extract a feature representation with visual information;

[0030] The network is trained using the contrastive learning loss function. For paired image and text pairs, their distance is shortened during the training process; for unpaired image and text pairs, their distance is increased during the training process, so as to learn discriminative image and text features.

[0031] A system for a combined product retrieval method based on the multimodal pre-training model, the system comprising a sample construction module, an image detector training module, a learning embedding representation module, a multimodal pre-training model module, a single product feature extraction module, and a combined product feature extraction module; wherein:

[0032] The sample construction module is used to divide the commodity images into single-product images and combination-product images;

[0033] The image detector training module is used to train an image detector for detecting each independent commodity in the combined commodity image;

[0034] The learning embedding representation module is used to acquire and combine the feature coding, position coding and segment coding of the text modality and the picture module in the combined product image to learn the embedding representation;

[0035] The multimodal pre-training model module is used to construct a multimodal pre-training model, and use the bounding box and bounding box features extracted by the product detector as image features, combined with text features, and input them into the multimodal pre-training model for self-supervised training;

[0036] The single product feature extraction module is used to input the picture modality and text modality of the single product image into the multimodal pre-training model to extract retrieval features, and store the retrieval features in the retrieval library;

[0037] The combination product feature extraction module is used to extract the bounding box and bounding box features of each target product in the combination product image and input them into the multimodal pre-training model, calculate the cosine distance between the combination product features and the single product features in the retrieval library as the product similarity based on the retrieval features of the image and text, and select the most similar single product as the result to be returned.

[0038] The beneficial effects of the present invention are as follows:

[0039] Compared with the supervised learning image retrieval method, the present invention uses self-supervised learning for training, relying only on naturally existing pictures and title information without relying on any manually annotated category information. Therefore, it is easy to expand to large-scale data, learn a more deterministic feature representation, ensure the realization of high-quality instance-level product retrieval tasks, and have strong generalization.

[0040] Compared with using only a single modality of information for retrieval, the present invention uses information from both image and text modalities, can effectively utilize the information between the two modalities to complement each other, fuse image features and text features, extract more distinctive image and text retrieval features, and make up for the problem of incomplete single modality information.

[0041] Compared with most multimodal pre-trained models that use a trained natural image detector to extract image features for training, the present invention uses data enhancement to train a target detector for a product image, which solves the problem of data distribution differences between natural image and product image data, avoids the problem that the multimodal pre-trained model is affected by the upstream detector, improves the feature representation effect of the multimodal pre-trained model, and is conducive to improving the accuracy of combined product retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a flowchart of the steps of the combined product search method described in Example 1.

[0043] Figure 2 Schematic diagram of the network framework of the combined product retrieval method described in Example 1.

[0044] Figure 3 It is a schematic diagram of the framework of the combination product retrieval system described in Example 2. DETAILED DESCRIPTION

[0045] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0046] Example 1

[0047] like Figure 1 As shown, a combined product retrieval method based on a multimodal pre-training model, the method comprising the following steps:

[0048] S1: Divide the product image into a single product image and a combination product image, wherein the single product image indicates that there is only one product, and the combination product image indicates that there are multiple independent products;

[0049] S2: Train a combined product image detector to detect each independent product in the combined product image;

[0050] S3: Obtain and combine the feature encoding, position encoding, and segment encoding of the text modality and image module in the combined product image to learn the embedded representation;

[0051] S4: Build a multimodal pre-training model and use the learned embedding representation as the input of the multimodal pre-training model;

[0052] S5: The bounding box and bounding box features extracted by the product detector are used as image features, combined with text features, and input into the multimodal pre-training model for self-supervised training;

[0053] S6: Input the image modality and text modality of the single product image into the multimodal pre-training model to extract retrieval features, and store the retrieval features in the retrieval library;

[0054] S7: Extract the bounding box and bounding box features of each target product in the combination product image, input the multimodal pre-trained model trained in step S5, extract the retrieval features of image and text fusion, calculate the cosine distance between the combination product features and the single product features in the retrieval library as the product similarity, and select the most similar single product as the result and return it.

[0055] In a specific embodiment, step S1, for all product data, a rule-based method is used to divide them into single product images and combination product images, and product images with product titles such as "set", "combination", "set", and "piece set" are divided into combination product images, otherwise they are divided into single product images.

[0056] Specifically, each product (I, T) consists of an image I and a corresponding title T. Based on whether the product title contains words such as "set", "combination", "set", or "piece set", the products are divided into a combination product set and a single product set. Each sample in the single product set contains only one product, which is recorded as Each sample in the combined product set contains multiple products, denoted as

[0057] In a specific embodiment, a data enhancement method is used to train a combined commodity image detector, as follows:

[0058] First, calculate the color area ratio in the single product image, select the image with a white area ratio greater than 60% as the white background image, use the GrabCut segmentation method to obtain the foreground of the single product image, and then select a natural image as the background. Use the "copy and paste" method to combine multiple foreground objects and background images into a picture with bounding box annotations;

[0059] According to the position where the foreground object is pasted on the background image, the bounding box coordinates of the foreground object are selected as the bounding box annotation for image detector training. Using the two categories of foreground and background, a combined product image detector is trained to detect each independent product in the combined product.

[0060] In a specific embodiment, a bottom-up-attention network is used to extract the features of the main target box and its coordinate position for the image modality data.

[0061] In a specific embodiment, for the image frame B output by the bottom-up-attention network = (b 0 ,b 1 ,b 2 ,…,b K ) and frame feature F=(f 0 ,f 1 ,f 2 ,…,f K ) as image feature input I=((b 0 ,f 0 ),(b 1 ,f 1 ),(b 2 ,f 2 ),…,(b K ,f K )), use the 5-dimensional vector to calculate the position information of each box, including the coordinates of the upper left corner and the lower right corner of the box, and the size ratio of the box to the entire image. Pass this 5-dimensional vector into the linear fully connected layer to obtain the position code; use 0 as the segmentation information to pass it into the linear fully connected layer to obtain the segmentation code; pass the box feature into the linear fully connected layer to obtain the box feature code; finally, add the position code, segmentation code and feature code to obtain the embedded representation E of the image modality img .

[0062] Specifically, Figure 2 As shown, the bounding box feature F is passed through the fully connected layer to obtain the encoding of the box feature The calculation formula is: Among them, w 1 and b 1 are the parameters of the fully connected layer and σ is the activation function.

[0063] Bounding box extracted by bottom-up-attention network Calculate the area ratio of each box to the entire image and construct a 5-dimensional vector After the fully connected layer, the position encoding vector is output The calculation formula is: where w 2 and b 2are the parameters of the fully connected layer and σ is the activation function.

[0064] The integer 0 is used as the segmentation information S of the image mode img After the fully connected layer, the segment coding vector is obtained The calculation formula is: where w 3 and b 3 are the parameters of the fully connected layer and σ is the activation function.

[0065] Add the feature encoding vector, position encoding vector and segment encoding vector to get the embedded representation E of the image modality img ,

[0066] In a specific embodiment, for a product title T=(t 0 ,t 1 ,t 2 ,…,t L ), that is, the text modal data uses the Bert network to extract the relationship features between different tokens.

[0067] In a specific embodiment, for a text sequence, an increasing natural number sequence is used to represent their position information, which is passed into a linear fully connected layer to obtain a position code; 1 is used as segmentation information and passed into a linear fully connected layer to obtain a segmentation code; the text is passed into a linear fully connected layer to obtain a feature code of the text; finally, the position code, segmentation code and feature code are added together to obtain an embedding representation E of the text. txt The details are as follows:

[0068] The product title T is passed through the embedding layer to obtain the feature encoding vector The calculation formula is: Among them, w 4 and b 4 are the parameters of the fully connected layer and σ is the activation function.

[0069] The position information (natural number order) of the words in the title is passed through the fully connected layer to obtain the position encoding vector The calculation formula is: where w 5 and b 5 are the parameters of the fully connected layer and σ is the activation function.

[0070] The integer 1 is used as the segment information S of the text mode txt After the fully connected layer, the segment coding vector is obtained The calculation formula is: where w 6 and b 6 are the parameters of the fully connected layer and σ is the activation function.

[0071] Add the feature encoding vector, position encoding vector and segment encoding vector to get the embedding representation E of the text modality txt ,

[0072] In a specific embodiment, the picture is embedded in the representation E img and text embedding representation E txt The multimodal pre-training model is input to extract the retrieval feature H of the fusion of image and text. The multimodal pre-training model uses three Transformer network layers to extract the retrieval features of the fusion of image and text, including:

[0073] The text-visual Transformer network is used to extract shallow features of text and images based on the input image embedding representation and text embedding representation; the details are as follows:

[0074] First, use the image Transformer and text Transformer to embed the image into representation E img and text embedding feature E txt Encode to obtain the line-level features of the image modality and text modality and The image transformer and text transformer each have four layers, and the calculation formula for each layer is:

[0075]

[0076]

[0077]

[0078]

[0079] Among them, t-1 and t are Transformer layer numbers; LN is the LayerNorm layer for feature normalization; MLP is the fully connected layer; MSA is the multi-head attention layer, and its calculation formula is as follows:

[0080]

[0081] Head i =Attention(HW i Q ,HW i K ,HW i V )

[0082] MSA(H)=Concat(Head 1,...,Head h )W O .

[0083] The text-visual Transformer network layer is repeated L=4 times and then passed to the next network.

[0084] Cross Transformer network, used to classify the line features of the image and line-level features of text Realize the correlation between the image modality and the text modality, and obtain the features after the two modalities are correlated and Specifically, the cross Transformer network includes two independent cross multi-head self-attention networks, which are realized by exchanging Q and K in the image modality and the text modality; for text, the cross Transformer network calculates the attention weight of the image to obtain the text features after cross attention; for image, the cross Transformer network calculates the attention weight of the text to obtain the image features after cross attention; this layer is repeated M = 4 times and then passed to the next network; specifically

[0085] The calculation formula is as follows:

[0086]

[0087]

[0088]

[0089]

[0090] Among them, CMSA is a cross-modal cross-head attention network, and its calculation formula is as follows:

[0091] cMSA(H img ,H txt )=Concat(Head 1 (H img ,H txt ),…,Head n (H img ,H txt ))

[0092] CMSA(H txt ,H img )=Concat(Head 1 (H txt ,H img ),…,Head n (H txt ,Himg ))

[0093] Head i (H img ,H txt )=Attention(H img W i Q ,H txt W i K ,H txt W i V )

[0094] Head i (H txt ,H img )=Attention(H txt W i Q ,H img W i K ,H img W i V )

[0095] The public Transformer network is used to extract the image features of the comprehensive fusion of text and image. In the public Transformer network, the text features and image features are spliced, and the weight of each vector focusing on all features is calculated using Q and K, and then multiplied by V to obtain the feature representation of the text and the feature representation of the image, where Q, K, and V are obtained by the features of the two modalities spliced ​​together; for text, a multi-head attention mechanism is used to calculate the attention weights for all features of the two modalities of text and image, so as to obtain the fully fused text features; for images, a multi-head attention mechanism is used to calculate the attention weights for all features of the two modalities of image and text, so as to obtain the fully fused image features; this layer is repeated for H = 4 times. The details are as follows:

[0096] Image features from the cross Transformer network and text features The data is passed to the public Transformer to perform a more comprehensive mutual attention between the image and the text, that is, each area in the image modality pays attention to the features of other areas and the features of all texts, and each word in the text modality pays attention to the features of other words and the features of all images. The calculation formula is as follows:

[0097]

[0098]

[0099]

[0100] In a specific embodiment, in step S5, the self-supervised training is specifically as follows:

[0101] By masking the words in the title text, the text sequence with the masked words is input into the trained multimodal pre-trained model. During the training process, the masked words are learned to be restored, thereby extracting a feature representation with text information.

[0102] By masking the bounding box features in the image, the masked image box feature sequence is input into the multimodal pre-trained model, and the masked bounding box features are restored during the training process to extract a feature representation with visual information;

[0103] The network is trained using the contrastive learning loss function. For paired image and text pairs, their distance is shortened during the training process; for unpaired image and text pairs, their distance is increased during the training process, so as to learn discriminative image and text features.

[0104] Specifically, three pre-training tasks are used to train the above-mentioned multimodal pre-training model, including a text masking task, an image region masking task, and a cross-modal contrastive learning task.

[0105] Specifically, for each image-text pair (I, T), I = {I 1, I 2, ,I 3 ,,…,I K,}, T={T 1, T 2, ,T 3 ,,…,T L, The text masking task is to replace the input word with "[MASK]" with a probability of 15%. The multimodal pre-trained model predicts the masked word based on the remaining words and pictures, and its loss function is:

[0106]

[0107] The image region masking task is to replace the input image frame feature with a 0 vector with a probability of 15%. The multimodal pre-training model predicts the masked image region feature based on the remaining image region and sentence words, and its loss function is:

[0108]

[0109] In addition to learning different modal features, the multimodal pre-training model needs to ensure the consistency between different input modalities in order to learn the correspondence between images and texts. Therefore, a cross-modal contrast learning task is used to align the image modality and the text modality. For N image-text pairs in a training batch, there are 2N data in total. For each sample, the corresponding data of the other modality is regarded as a positive sample pair, and the remaining samples are regarded as negative sample pairs. For the input image-text pair ((I i ,T i ), each pair of image-text features output by the text transformer and the image transformer Its loss function is:

[0110]

[0111] Where sim(u,v)=u T v / ||u||||v|| is used to calculate the similarity between the image and text pairs u and v, τ is the temperature adjustment parameter, is a binary indicator that returns 1 if and only if i! = j. This contrast loss function will make paired image-text vectors closer and unpaired image-text vectors farther apart.

[0112] Example 2

[0113] Based on the combination product retrieval method of the multimodal pre-training model described in Example 1, this embodiment also provides a combination product retrieval system of the multimodal pre-training model, the system comprising a sample construction module, an image detector training module, a learning embedding representation module, a multimodal pre-training model module, a single product feature extraction module, and a combination product feature extraction module; wherein,

[0114] The sample construction module is used to divide the commodity images into single-product images and combination-product images;

[0115] The image detector training module is used to train an image detector for detecting each independent commodity in the combined commodity image;

[0116] The learning embedding representation module is used to acquire and combine the feature coding, position coding and segment coding of the text modality and the picture module in the combined product image to learn the embedding representation;

[0117] The multimodal pre-training model module is used to construct a multimodal pre-training model, and use the bounding box and bounding box features extracted by the product detector as image features, combined with text features, and input them into the multimodal pre-training model for self-supervised training;

[0118] The single product feature extraction module is used to input the picture modality and text modality of the single product image into the multimodal pre-training model to extract retrieval features, and store the retrieval features in the retrieval library;

[0119] The combination product feature extraction module is used to extract the bounding box and bounding box features of each target product in the combination product image and input them into the multimodal pre-training model, calculate the cosine distance between the combination product features and the single product features in the retrieval library as the product similarity based on the retrieval features of the image and text, and select the most similar single product as the result to be returned.

[0120] In a specific embodiment, the multimodal pre-training model module also includes

[0121] The text-visual Transformer network module is used to extract shallow features of text and images, using Q 1 and K 1 Calculate the attention weight and multiply it by V 1 Get the feature representation of text and the feature representation of picture, where Q 1 , K 1 、V 1 From the image and text’s own modality;

[0122] The cross Transformer network module is used to extract the features of mutual attention between text and image. The cross Transformer network includes two independent cross multi-head self-attention networks. 1 and K 1 To achieve this; for text, the cross Transformer network calculates the attention weight of the image, so as to obtain the text features after cross attention; for images, the cross Transformer network calculates the attention weight of the text, so as to obtain the image features after cross attention;

[0123] The public Transformer network module is used to extract the features of the comprehensive fusion of text and pictures. In the public Transformer network, text features and picture features are spliced ​​and Q 2 and K 2 Calculate the weight of each vector focusing on all features and multiply it by V 2 Get the feature representation of text and the feature representation of picture, where Q 2 , K 2 、V 2It is obtained by concatenating the features of the two modalities. For text, a multi-head attention mechanism is used to calculate the attention weights for all features of the two modalities, text and image, so as to obtain the fully integrated text features. For images, a multi-head attention mechanism is used to calculate the attention weights for all features of the two modalities, image and text, so as to obtain the fully integrated image features.

[0124] Example 3

[0125] A computer system includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following method steps are implemented:

[0126] S1: Divide the product image into a single product image and a combination product image, wherein the single product image indicates that there is only one product, and the combination product image indicates that there are multiple independent products;

[0127] S2: Train a combined product image detector to detect each independent product in the combined product image;

[0128] S3: Obtain and combine the feature encoding, position encoding, and segment encoding of the text modality and image module in the combined product image to learn the embedded representation;

[0129] S4: Build a multimodal pre-training model and use the learned embedding representation as the input of the multimodal pre-training model;

[0130] S5: The bounding box and bounding box features extracted by the product detector are used as image features, combined with text features, and input into the multimodal pre-training model for self-supervised training;

[0131] S6: Input the image modality and text modality of the single product image into the multimodal pre-training model to extract retrieval features, and store the retrieval features in the retrieval library;

[0132] S7: Extract the bounding box and bounding box features of each target product in the combination product image, input the multimodal pre-trained model trained in step S5, extract the retrieval features of image and text fusion, calculate the cosine distance between the combination product features and the single product features in the retrieval library as the product similarity, and select the most similar single product as the result and return it.

[0133] Example 4

[0134] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following method steps are implemented:

[0135] S1: Divide the product image into a single product image and a combination product image, wherein the single product image indicates that there is only one product, and the combination product image indicates that there are multiple independent products;

[0136] S2: Train a combined product image detector to detect each independent product in the combined product image;

[0137] S3: Obtain and combine the feature encoding, position encoding, and segment encoding of the text modality and image module in the combined product image to learn the embedded representation;

[0138] S4: Build a multimodal pre-training model and use the learned embedding representation as the input of the multimodal pre-training model;

[0139] S5: The bounding box and bounding box features extracted by the product detector are used as image features, combined with text features, and input into the multimodal pre-training model for self-supervised training;

[0140] S6: Input the image modality and text modality of the single product image into the multimodal pre-training model to extract retrieval features, and store the retrieval features in the retrieval library;

[0141] S7: Extract the bounding box and bounding box features of each target product in the combination product image, input the multimodal pre-trained model trained in step S5, extract the retrieval features of image and text fusion, calculate the cosine distance between the combination product features and the single product features in the retrieval library as the product similarity, and select the most similar single product as the result and return it.

[0142] Obviously, the above embodiments of the present invention are only examples for clearly explaining the present invention, and are not intended to limit the implementation methods of the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention shall be included in the protection scope of the claims of the present invention.

Claims

1. A combined product retrieval method based on a multimodal pre-training model, characterized in that: The method comprises the following steps: S1: Divide the product image into a single product image and a combination product image, wherein the single product image indicates that there is only one product, and the combination product image indicates that there are multiple independent products; S2: Train a combined product image detector to detect each independent product in the combined product image; S3: Obtain and combine the feature encoding, position encoding, and segment encoding of the text modality and image module in the combined product image to learn the embedded representation; S4: Build a multimodal pre-training model and use the learned embedding representation as the input of the multimodal pre-training model; S5: The bounding box and bounding box features extracted by the combined product image detector are used as image features, combined with text features, and input into the multimodal pre-training model for self-supervised training; S6: Input the image modality and text modality of the single product image into the multimodal pre-training model to extract retrieval features, and store the retrieval features in the retrieval library; S7: Extract the bounding box and bounding box features of each target product in the combination product image, input the multimodal pre-trained model trained in step S5, extract the retrieval features of image and text fusion, calculate the cosine distance between the combination product features and the single product features in the retrieval library as the product similarity, and select the most similar single product as the result and return it.

2. The combined product retrieval method based on the multimodal pre-training model according to claim 1 is characterized in that: Step S1, using a rule-based method to divide into single product images and combination product images, classifying product images with product titles such as "set", "combination", "set", and "piece set" as combination product images, otherwise they are classified as single product images.

3. The combined product retrieval method based on the multimodal pre-training model according to claim 2 is characterized in that: The data enhancement method is used to train the combined product image detector as follows: First, calculate the color area ratio in the single product image, select the image with a white area ratio greater than 60% as the white background image, use the GrabCut segmentation method to obtain the foreground of the single product image, and then select the natural image as the background. Use the "copy and paste" method to combine multiple foreground objects and background images into a picture with bounding box annotations; According to the position where the foreground object is pasted on the background image, the bounding box coordinates of the foreground object are selected as the bounding box annotation for image detector training. Using the two categories of foreground and background, a combined product image detector is trained to detect each independent product in the combined product.

4. The combined product retrieval method based on the multimodal pre-training model according to claim 3 is characterized in that: For image modal data, the bottom-up-attention network is used to extract the features of the main target box and its coordinate position.

5. The combined product retrieval method based on the multimodal pre-training model according to claim 4 is characterized in that: For the image frames and frame features output by the bottom-up-attention network, a 5-dimensional vector is used to calculate the position information of each frame, including the coordinates of the upper left corner and the lower right corner of the frame, and the size ratio of the frame to the entire image. This 5-dimensional vector is passed into the linear fully connected layer to obtain the position code; 0 is used as the segmentation information and passed into the linear fully connected layer to obtain the segmentation code; the frame features are passed into the linear fully connected layer to obtain the encoding of the frame features; finally, the position code, segmentation code and feature code are added together to obtain the embedded representation of the image modality.

6. The combined product retrieval method based on a multimodal pre-training model according to claim 4, characterized in that: For text modal data, the Bert network is used to extract the relationship features between different tokens.

7. The combined product retrieval method based on a multimodal pre-training model according to claim 6, characterized in that: For text sequences, use an increasing natural number sequence to represent their position information, and pass it into the linear fully connected layer to obtain the position encoding; use 1 as the segmentation information and pass it into the linear fully connected layer to obtain the segmentation encoding; pass the text into the linear fully connected layer to obtain the feature encoding of the text; finally, add the position encoding, segmentation encoding and feature encoding to obtain the embedded representation of the text.

8. The combined product retrieval method based on a multimodal pre-training model according to claim 6, characterized in that: The multimodal pre-training model uses three Transformer network layers to extract retrieval features that integrate images and texts, including: The text-visual Transformer network is used to extract shallow features of text and shallow features of images based on the input image embedding representation and text embedding representation; The cross Transformer network realizes the correlation between the image modality and the text modality based on the line-level features of the image and the line-level features of the text, and obtains the features after the two modalities are correlated; The public Transformer network is used to extract image features that fully integrate text and images.

9. The combined product retrieval method based on a multimodal pre-training model according to claim 6, characterized in that: Step S5, the self-supervised training is specifically as follows: By masking the words in the title text, the text sequence with the masked words is input into the trained multimodal pre-trained model. During the training process, the masked words are learned to be restored, thereby extracting a feature representation with text information. By masking the bounding box features in the image, the masked image box feature sequence is input into the multimodal pre-trained model, and the masked bounding box features are restored during the training process to extract a feature representation with visual information; Use contrastive learning loss function to train the network, and shorten the distance between paired images and text during training. For unpaired image-text pairs, the distance between them is increased during the training process to learn discriminative image-text features.

10. A system for a combined product retrieval method based on the multimodal pre-training model according to any one of claims 1 to 9, characterized in that: The system includes a sample construction module, an image detector training module, a learning embedding representation module, a multimodal pre-training model module, a single product feature extraction module, and a combination product feature extraction module; wherein, The sample construction module is used to divide the commodity images into single-product images and combination-product images; The image detector training module is used to train an image detector for detecting each independent commodity in the combined commodity image; The learning embedding representation module is used to acquire and combine the feature coding, position coding and segment coding of the text modality and the picture module in the combined product image to learn the embedding representation; The multimodal pre-training model module is used to construct a multimodal pre-training model, and use the bounding box and bounding box features extracted by the image detector as image features, combined with text features, and input them into the multimodal pre-training model for self-supervised training; The single product feature extraction module is used to input the picture modality and text modality of the single product image into the multimodal pre-training model to extract retrieval features, and store the retrieval features in the retrieval library; The combination product feature extraction module is used to extract the bounding box and bounding box features of each target product in the combination product image and input them into the multimodal pre-training model, calculate the cosine distance between the combination product features and the single product features in the retrieval library as the product similarity based on the retrieval features of the image and text, and select the most similar single product as the result to be returned.

Images