Multimodal retrieval model training and multimodal retrieval method, electronic device, and storage medium

Abstract

The present application relates to the technical field of computer, specifically relates to a kind of multimodal retrieval model training and multimodal retrieval method, electronic equipment and storage medium.The present application includes: obtaining the training image and training text corresponding to multiple training commodities respectively in training data;Multiple training commodities corresponding training image and training text are respectively input into preset multimodal retrieval network, and the training image feature corresponding to training image and the training text feature corresponding to training text are output;According to the relationship between training image feature and training text feature, the parameters of preset multimodal retrieval network are updated, and the target preset multimodal retrieval model after parameter updating is obtained.The above method, training image feature and training text feature are fused, and more distinctive image-text retrieval features are extracted, so that the target preset multimodal retrieval model is suitable for image retrieval and text retrieval, and the problem of incomplete single modal information is solved.

Description

Technical Field

[0001] This invention relates to the field of computer technology, specifically to a multimodal retrieval model training method, a multimodal retrieval method, an electronic device, and a storage medium. Background Technology

[0002] The development of internet technology has led to the rapid expansion of online e-commerce platforms, which are increasingly favored by users due to their convenience. The variety of goods available in the e-commerce sector and users' shopping demands have increased dramatically, making product retrieval based on images or text information a popular research area in artificial intelligence.

[0003] In the field of artificial intelligence, building suitable deep learning models to establish the corresponding relationship between vision and natural language is a long-term goal. Vision-language pre-trained models, through offline training strategies, are pre-trained on large-scale image-text datasets and then fine-tuned for specific downstream tasks, showing promising practical application prospects and broad exploration space.

[0004] However, existing technologies typically separate image training and text training, using only information from a single modality for product retrieval, resulting in a model with limited functionality and incomplete information. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a multimodal retrieval model training method, electronic device and storage medium, which aim to solve the problem that in the prior art, image training and text training are usually separated, and only single-modal information is used for product retrieval, resulting in a single-function model and incomplete information.

[0006] According to a first aspect, embodiments of the present invention provide a multimodal retrieval model training method, characterized in that it includes:

[0007] Obtain the training images and training texts corresponding to multiple training products in the training data; wherein, the product identification information corresponding to the training images and training texts of the same training product is the same.

[0008] The training images and training texts corresponding to multiple training products are respectively input into a preset multimodal retrieval network, and the training image features corresponding to the training images and the training text features corresponding to the training texts are output.

[0009] Based on the relationship between training image features and training text features, the parameters of the preset multimodal retrieval network are updated to obtain the target preset multimodal retrieval model with updated parameters.

[0010] The multimodal retrieval model training method provided in this invention acquires training images and training text corresponding to multiple training items from the training data. These training images and text are then input into a preset multimodal retrieval network, outputting training image features corresponding to the training images and training text features corresponding to the training text, ensuring the accuracy of the output training image features and training text features. Then, based on the relationship between the training image features and training text features, the parameters of the preset multimodal retrieval network are updated to obtain the target preset multimodal retrieval model with updated parameters, ensuring the accuracy of the obtained target preset multimodal retrieval model with updated parameters. The above method inputs training images and training text corresponding to multiple training products into a preset multimodal retrieval network, and outputs training image features corresponding to the training images and training text features corresponding to the training text. This enables simultaneous training of training images and training text, effectively utilizing the complementary information between the two modalities, fusing training image features and training text features, and extracting more discriminative image-text retrieval features. As a result, the final trained target preset multimodal retrieval model is applicable to both image retrieval and text retrieval, making up for the problem of incomplete information in a single modality.

[0011] In conjunction with the first aspect, in the first embodiment of the first aspect, training images and training texts corresponding to multiple training items are respectively input into a preset multimodal retrieval network, and training image features corresponding to the training images and training text features corresponding to the training texts are output, including:

[0012] The training images and training texts corresponding to multiple training products are respectively input into the embedding representation layer of the preset multimodal retrieval network, and the image embedding representations corresponding to the training images and the text embedding representations corresponding to the training texts are output.

[0013] The image embedding representation and the text embedding representation are respectively input into the feature extraction network in the preset multimodal retrieval network, and the training image features corresponding to the training image and the training text features corresponding to the training text are output.

[0014] The multimodal retrieval model training method provided in this invention inputs training images and training text corresponding to multiple training products into the embedding representation layer of a preset multimodal retrieval network, outputting image embedding representations for the training images and text embedding representations for the training text, ensuring the accuracy of the output image embedding representations and text embedding representations. Then, the image embedding representations and text embedding representations are input into the feature extraction network of the preset multimodal retrieval network, outputting training image features for the training images and training text features for the training text, ensuring the accuracy of the output training image features for the training images and training text features for the training text.

[0015] In conjunction with the first embodiment of the first aspect, in the second embodiment of the first aspect, training images and training texts corresponding to multiple training products are respectively input into the embedding representation layer of a preset multimodal retrieval network, and the image embedding representations corresponding to the training images and the text embedding representations corresponding to the training texts are output, including:

[0016] For the training images, divide the training images into a preset number of sub-image blocks;

[0017] Each sub-image patch is input into the embedding representation layer in sequence. The linear projection layer in the embedding representation layer extracts features from each sub-image patch and generates the image feature code corresponding to the training image.

[0018] The positional encoding layer in the embedding representation layer performs positional encoding on the image feature encoding based on the position of each sub-image patch in the training image, thus obtaining the image positional encoding corresponding to the image feature encoding;

[0019] The block coding layer in the embedding representation layer performs block coding on the image feature code based on the block coding results of each sub-image block in the training image, and obtains the image block code corresponding to the image feature code;

[0020] The image feature codes and image position codes corresponding to the training images are concatenated using image block coding to generate the image embedding representation corresponding to the training images;

[0021] For the training text, the training text is input into the embedding representation layer in the form of a sequence. The linear fully connected layer in the embedding representation layer extracts features from the training text and generates the text feature code corresponding to the training text.

[0022] The positional encoding layer in the embedding representation layer performs positional encoding on the text feature encoding based on the position of each text character in the training text, thus obtaining the text positional encoding corresponding to the training text;

[0023] The block coding layer in the embedding representation layer performs segmented coding on the text feature encoding based on the segmentation results of each text character in the training text, thereby obtaining the text segment block encoding corresponding to the training text;

[0024] The text feature encoding, text position encoding, and text segmentation encoding corresponding to the training text are concatenated to generate the text embedding representation corresponding to the training text.

[0025] The multimodal retrieval model training method provided in this invention outputs image feature codes, image position codes, and image block codes corresponding to training images. Then, it concatenates the image feature codes and image position codes corresponding to the training images using image block codes to generate image embedding representations corresponding to the training images, ensuring the accuracy of the generated image embedding representations. Furthermore, it outputs text feature codes, text position codes, and text segment codes corresponding to training text, and then concatenates these codes to generate text embedding representations corresponding to the training text, ensuring the accuracy of the output text embedding representations. This, in turn, ensures the accuracy of the output training image features and training text features.

[0026] In conjunction with the first aspect, in the third embodiment of the first aspect, the parameters of the preset multimodal retrieval network are updated according to the relationship between the training image features and the training text features to obtain the target preset multimodal retrieval model with updated parameters, including:

[0027] Calculate the first similarity between the training image features and the training text features, and the second similarity between the training text features and the training image features;

[0028] Based on the relationship between the first similarity and the second similarity, a preset loss function is calculated;

[0029] Based on the preset loss function, the parameters of the preset multimodal retrieval network are updated to obtain the target preset multimodal retrieval model with updated parameters.

[0030] The multimodal retrieval model training method provided in this invention calculates the first similarity between training image features and training text features, and the second similarity between training text features and training image features, ensuring the accuracy of the calculated first and second similarities. Then, based on the relationship between the first and second similarities, a preset loss function is calculated, ensuring the accuracy of the calculated preset loss function. According to the preset loss function, the parameters of the preset multimodal retrieval network are updated to obtain the target preset multimodal retrieval model with updated parameters, ensuring the accuracy of the target preset multimodal retrieval model with updated parameters. This method enables simultaneous training of both training images and training text, effectively utilizing the complementary information between the two modalities, fusing training image features and training text features, and extracting more discriminative image-text retrieval features. This makes the final trained target preset multimodal retrieval model suitable for both image and text retrieval, compensating for the incomplete information of a single modality.

[0031] In conjunction with the third embodiment of the first aspect, in the fourth embodiment of the first aspect, the parameters of the preset multimodal retrieval network are updated according to the preset loss function to obtain the target preset multimodal retrieval model with updated parameters, including:

[0032] The training image features are input into the prompting network in the preset multimodal retrieval network, and the first task-independent candidate parameters and the first task-related candidate parameters corresponding to the training image features are output.

[0033] The training text features are input into the prompting network in the preset multimodal retrieval network, and the second task-independent candidate parameters and the second task-related candidate parameters corresponding to the training text features are output.

[0034] Based on the relationship between the first task-independent candidate parameters, the first task-related candidate parameters, the second task-independent candidate parameters, and the second task-related candidate parameters, as well as the preset loss function, the parameters of the preset multimodal retrieval network are updated to obtain the target preset multimodal retrieval model after parameter update.

[0035] The multimodal retrieval model training method provided in this invention inputs training image features into a prompting network within a preset multimodal retrieval network, outputting first task-independent candidate parameters and first task-related candidate parameters corresponding to the training image features, ensuring the accuracy of the output first task-independent candidate parameters and first task-related candidate parameters. Training text features are input into the prompting network within the preset multimodal retrieval network, outputting second task-independent candidate parameters and second task-related candidate parameters corresponding to the training text features, ensuring the accuracy of the output second task-independent candidate parameters and second task-related candidate parameters. Based on the relationship between the first task-independent candidate parameters, the first task-related candidate parameters, the second task-independent candidate parameters, and the second task-related candidate parameters, and a preset loss function, the preset multimodal retrieval network is updated with parameters to obtain a target preset multimodal retrieval model with updated parameters, ensuring the accuracy of the obtained target preset multimodal retrieval model. The above method enables simultaneous training on both training images and training text, effectively utilizing the complementary information between the two modalities. It fuses training image and text features to extract more discriminative image-text retrieval features, making the final trained target multimodal retrieval model suitable for both image and text retrieval, thus overcoming the incompleteness of information from a single modality. Furthermore, the cueing network in this method helps the multimodal retrieval network memorize task-independent and task-related knowledge from the visual-language modality, exhibiting better performance in downstream tasks such as product retrieval compared to other continuous learning methods. This module is also plug-and-play, meaning it can be easily integrated into other visual-language multimodal models to enhance their performance.

[0036] In conjunction with the fourth embodiment of the first aspect, in the fifth embodiment of the first aspect, training image features are input into a prompting network in a preset multimodal retrieval network, and first task-independent candidate parameters and first task-related candidate parameters corresponding to the training image features are output, including:

[0037] The training image features are input into the prompting network in the preset multimodal retrieval network to generate multiple task-independent initial parameters and multiple task-related initial parameters corresponding to the training image features;

[0038] The first degree of matching between the training parameters corresponding to the training image features and the task-independent initial parameters is calculated using a matching function.

[0039] Based on the calculated first matching degrees, determine the first task-independent candidate parameters from the task-independent initial parameters;

[0040] The second degree of matching between the training parameters and the initial parameters of each task-related type is calculated using a matching function.

[0041] Based on the calculated second matching degrees, the first task-related candidate parameters are determined from the task-related initial parameters.

[0042] The multimodal retrieval model training method provided in this invention inputs training image features into a prompting network within a preset multimodal retrieval network, generating multiple task-independent initial parameters and multiple task-related initial parameters corresponding to the training image features. Then, a matching function is used to calculate a first matching degree between the training parameters corresponding to the training image features and each task-independent initial parameter, ensuring the accuracy of the calculated first matching degree. Based on the calculated first matching degrees, first task-independent candidate parameters are determined from the task-independent initial parameters, ensuring the accuracy of the determined first task-independent candidate parameters. Next, a matching function is used to calculate a second matching degree between the training parameters and each task-related initial parameter, ensuring the accuracy of the calculated second matching degree. Based on the calculated second matching degrees, first task-related candidate parameters are determined from the task-related initial parameters, ensuring the accuracy of the determined first task-related candidate parameters.

[0043] In conjunction with the fourth embodiment of the first aspect, in the sixth embodiment of the first aspect, based on the relationship between the first task-independent candidate parameters, the first task-related candidate parameters, the second task-independent candidate parameters, and the second task-related candidate parameters, and a preset loss function, the parameters of the preset multimodal retrieval network are updated to obtain the target preset multimodal retrieval model with updated parameters, including:

[0044] Calculate the first mean squared error loss function corresponding to the first task-independent candidate parameters and the first task-related candidate parameters;

[0045] Calculate the second mean squared error loss function corresponding to the second task-independent candidate parameters and the two task-related candidate parameters;

[0046] Calculate the third mean squared error loss function between the first task-independent candidate parameters and the second task-independent candidate parameters;

[0047] Calculate the fourth mean squared error loss function between the first task-related candidate parameters and the second task-related candidate parameters;

[0048] Based on the first mean squared error loss function, the second mean squared error loss function, the third mean squared error loss function, the fourth mean squared error loss function, and the preset loss function, the parameters of the preset multimodal retrieval network are updated to obtain the target preset multimodal retrieval model with updated parameters.

[0049] The multimodal retrieval model training method provided in this embodiment of the invention ensures the accuracy of the calculated first, second, third, and fourth mean squared error loss functions. Then, based on the first, second, third, and fourth mean squared error loss functions and the preset loss function, the parameters of the preset multimodal retrieval network are updated to obtain the target preset multimodal retrieval model with updated parameters, thus ensuring the accuracy of the obtained target preset multimodal retrieval model with updated parameters.

[0050] According to a second aspect, embodiments of the present invention also provide a multimodal retrieval method, the method comprising:

[0051] Obtain the search text and / or search image corresponding to the product to be searched;

[0052] The search text and / or search image are input into a target preset multimodal retrieval model. The target preset multimodal retrieval model performs feature extraction on the search text, extracting the search text features corresponding to the search text, and / or performs feature extraction on the search image, extracting the search image features corresponding to the search image; wherein, the target preset multimodal retrieval model is trained according to the multimodal retrieval model training method in the first aspect or any embodiment of the first aspect;

[0053] Calculate the first retrieval similarity between the retrieved text features and the features of each training image in the database, and / or calculate the second retrieval similarity between the retrieved image features and the features of each training image in the database;

[0054] The training image corresponding to the maximum similarity between the first search similarity and / or the second search similarity is determined as the target training image corresponding to the product to be retrieved.

[0055] The multimodal retrieval method provided in this invention obtains the retrieval text and / or retrieval image corresponding to the product to be retrieved. Then, the retrieval text and / or retrieval image are input into a target preset multimodal retrieval model. The target preset multimodal retrieval model performs feature extraction on the retrieval text, extracting the retrieval text features corresponding to the retrieval text, and / or performs feature extraction on the retrieval image, extracting the retrieval image features corresponding to the retrieval image, ensuring the accuracy of the extracted retrieval text features and / or retrieval image features. Then, the training image corresponding to the maximum similarity between the first retrieval similarity and / or the second retrieval similarity is determined as the target training image corresponding to the product to be retrieved, ensuring the accuracy of the determined target training image. The above method enables retrieval of both text and images, compensating for the problem of incomplete information in a single modality.

[0056] According to a third aspect, embodiments of the present invention provide an electronic device, including a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform a multimodal retrieval model training method according to the first aspect or any embodiment of the first aspect.

[0057] According to a fourth aspect, embodiments of the present invention provide a computer-readable storage medium storing computer instructions for causing a computer to execute a multimodal retrieval model training method according to the first aspect or any embodiment of the first aspect. Attached Figure Description

[0058] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0059] Figure 1 This is a flowchart of the multimodal retrieval model training method provided in the embodiments of the present invention;

[0060] Figure 2 This is a flowchart of a multimodal retrieval model training method provided by another embodiment of the present invention;

[0061] Figure 3 This is a flowchart of a multimodal retrieval model training method provided by another embodiment of the present invention;

[0062] Figure 4 This is a flowchart of a multimodal retrieval model training method provided by another embodiment of the present invention;

[0063] Figure 5 This is a flowchart of a multimodal retrieval method provided by another embodiment of the present invention;

[0064] Figure 6 This is a functional block diagram of the multimodal retrieval model training device provided in the embodiments of the present invention;

[0065] Figure 7 This is a functional block diagram of the multimodal retrieval device provided in the embodiments of the present invention;

[0066] Figure 8 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of the present invention. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0068] It should be noted that the multimodal retrieval model training method provided in this application embodiment can be executed by a multimodal retrieval model training device. This device can be implemented as part or all of a computer device through software, hardware, or a combination of both. The computer device can be a server or a terminal. In this application embodiment, the server can be a single server or a server cluster composed of multiple servers. The terminal in this application embodiment can be a smartphone, personal computer, tablet computer, wearable device, or other intelligent hardware device such as an intelligent robot. In the following method embodiments, the execution subject is always described using an electronic device as an example.

[0069] In one embodiment of this application, such as Figure 1 As shown, a multimodal retrieval model training method is provided. Taking the application of this method to electronic devices as an example, the method includes the following steps:

[0070] S11. Obtain the training images and training texts corresponding to multiple training items in the training data.

[0071] Among them, the training images and training texts corresponding to the same training product have the same product identification information.

[0072] Optionally, the electronic device can acquire training images and training text corresponding to multiple training items in the training data input by the user; optionally, the electronic device can also receive training images and training text corresponding to multiple training items in the training data sent by other devices; the electronic device can also query the training images and training text corresponding to multiple training items in the training data in the database.

[0073] This application does not specifically limit the method by which electronic devices acquire training images and training texts corresponding to multiple training products in the training data.

[0074] S12. Input the training images and training texts corresponding to multiple training products into the preset multimodal retrieval network, and output the training image features corresponding to the training images and the training text features corresponding to the training texts.

[0075] Specifically, the electronic device can input training images and training texts corresponding to multiple training products into a preset multimodal retrieval network. The preset multimodal retrieval network recognizes and extracts features from the input training images and training texts, and outputs the training image features corresponding to the training images and the training text features corresponding to the training texts.

[0076] This step will be explained in detail below.

[0077] S13. Based on the relationship between the training image features and the training text features, update the parameters of the preset multimodal retrieval network to obtain the target preset multimodal retrieval model after parameter update.

[0078] In one optional embodiment of this application, the electronic device can input training image features into a prompting network in a preset multimodal retrieval network to generate multiple task-independent initial parameters and multiple task-related initial parameters corresponding to the training image features; calculate a first matching degree between the training parameters corresponding to the training image features and each task-independent initial parameter using a matching function; determine a first task-independent candidate parameter from the task-independent initial parameters based on the calculated first matching degrees; calculate a second matching degree between the training parameters and each task-related initial parameter using a matching function; and determine a first task-related candidate parameter from the task-related initial parameters based on the calculated second matching degrees.

[0079] Furthermore, the electronic device can input training text features into a prompting network within a pre-defined multimodal retrieval network to generate multiple task-independent initial parameters and multiple task-related initial parameters corresponding to the training text features; calculate the third matching degree between the training parameters corresponding to the training text features and each task-independent initial parameter corresponding to the training text features using a matching function; determine second task-independent candidate parameters from the task-independent initial parameters corresponding to the training text features based on the calculated third matching degrees; calculate the fourth matching degree between the training parameters and each task-related initial parameter corresponding to the training text features using a matching function; and determine second task-related candidate parameters from the task-related initial parameters corresponding to the training text features based on the calculated fourth matching degrees.

[0080] Then, the electronic device calculates the first mean squared error loss function corresponding to the first task-independent candidate parameter and the first task-related candidate parameter, the second mean squared error loss function corresponding to the second task-independent candidate parameter and the second task-related candidate parameter, the third mean squared error loss function between the first task-independent candidate parameter and the second task-independent candidate parameter, and the fourth mean squared error loss function between the first task-related candidate parameter and the second task-related candidate parameter.

[0081] Optionally, the electronic device can update the parameters of the preset multimodal retrieval network according to the first mean squared error loss function, the second mean squared error loss function, the third mean squared error loss function, and the fourth mean squared error loss function, respectively, to obtain the target preset multimodal retrieval model with updated parameters.

[0082] Optionally, the electronic device can also calculate the sum of the third and fourth mean squared error loss functions to obtain the fifth mean squared error loss function. Then, the parameters of the preset multimodal retrieval network are updated according to the first, second, and fifth mean squared error loss functions respectively to obtain the target preset multimodal retrieval model with updated parameters.

[0083] This step will be explained in detail below.

[0084] The multimodal retrieval model training method provided in this invention acquires training images and training text corresponding to multiple training items from the training data. These training images and text are then input into a preset multimodal retrieval network, outputting training image features corresponding to the training images and training text features corresponding to the training text, ensuring the accuracy of the output training image features and training text features. Then, based on the relationship between the training image features and training text features, the parameters of the preset multimodal retrieval network are updated to obtain the target preset multimodal retrieval model with updated parameters, ensuring the accuracy of the obtained target preset multimodal retrieval model with updated parameters. The above method inputs training images and training text corresponding to multiple training products into a preset multimodal retrieval network, and outputs training image features corresponding to the training images and training text features corresponding to the training text. This enables simultaneous training of training images and training text, effectively utilizing the complementary information between the two modalities, fusing training image features and training text features, and extracting more discriminative image-text retrieval features. As a result, the final trained target preset multimodal retrieval model is applicable to both image retrieval and text retrieval, making up for the problem of incomplete information in a single modality.

[0085] In one embodiment of this application, such as Figure 2 As shown, a multimodal retrieval model training method is provided. Taking the application of this method to electronic devices as an example, the method includes the following steps:

[0086] S21. Obtain the training images and training texts corresponding to multiple training items in the training data.

[0087] Among them, the training images and training texts corresponding to the same training product have the same product identification information.

[0088] For details on this step, please refer to [link / reference]. Figure 1 The details of S11 will not be elaborated here.

[0089] S22. Input the training images and training texts corresponding to multiple training products into a preset multimodal retrieval network, and output the training image features corresponding to the training images and the training text features corresponding to the training texts.

[0090] In an optional embodiment of this application, step S22, "inputting the training images and training texts corresponding to multiple training items into a preset multimodal retrieval network, and outputting the training image features corresponding to the training images and the training text features corresponding to the training texts," may include the following steps:

[0091] S221. Input the training images and training texts corresponding to multiple training products into the embedding representation layer of the preset multimodal retrieval network, and output the image embedding representations corresponding to the training images and the text embedding representations corresponding to the training texts.

[0092] In an optional embodiment of this application, the above step S221, "inputting the training images and training texts corresponding to multiple training items into the embedding representation layer of a preset multimodal retrieval network, and outputting the image embedding representations corresponding to the training images and the text embedding representations corresponding to the training texts," may include the following steps:

[0093] (1) For the training image, the training image is divided into a preset number of sub-image blocks.

[0094] (2) Input each sub-image block into the embedding representation layer in sequence. The linear projection layer in the embedding representation layer extracts features from each sub-image block and generates the image feature code corresponding to the training image.

[0095] (3) The position coding layer in the embedded representation layer performs position coding on the image feature coding according to the position of each sub-image block in the training image, so as to obtain the image position coding corresponding to the image feature coding;

[0096] (4) The block coding layer in the embedded representation layer performs block coding on the image feature coding according to the block coding results of each sub-image block in the training image, and obtains the image block coding corresponding to the image feature coding;

[0097] (5) The image feature codes and image position codes corresponding to the training images are concatenated using image block coding to generate the image embedding representation corresponding to the training images;

[0098] (6) For the training text, the training text is input into the embedding representation layer in the form of a sequence. The linear fully connected layer in the embedding representation layer extracts features from the training text and generates the text feature code corresponding to the training text.

[0099] (7) The position encoding layer in the embedding representation layer performs position encoding on the text feature encoding according to the position of each text character in the training text, so as to obtain the text position encoding corresponding to the training text;

[0100] (8) The block coding layer in the embedded representation layer performs segmented coding on the text feature coding according to the segmentation results of each text character in the training text, and obtains the text segment block coding corresponding to the training text.

[0101] (9) The text feature encoding, text position encoding and text segmentation encoding corresponding to the training text are concatenated to generate the text embedding representation corresponding to the training text.

[0102] Specifically, for training images, electronic devices can divide training images into a preset number of sub-image blocks. The preset number can be 4, 9, or other values. This application embodiment does not specifically limit the preset number.

[0103] Then, the electronic device can arrange the sub-image blocks in a sequence. For example, if the training image is divided into 9 sub-image blocks, they can be arranged in the order of sub-image block 1, sub-image block 2, ..., sub-image block 9. The sub-image blocks are then input into the embedding representation layer in sequence. The linear projection layer in the embedding representation layer extracts features from each sub-image block to generate the image feature code corresponding to the training image.

[0104] Then, the positional encoding layer in the embedding representation layer performs positional encoding on the image feature encoding based on the position of each sub-image block in the training image, obtaining the image positional encoding corresponding to the image feature encoding. The block encoding layer in the embedding representation layer performs block encoding on the image feature encoding based on the block division result of each sub-image block in the training image, obtaining the image block encoding corresponding to the image feature encoding.

[0105] After obtaining the image feature code, image position code, and image block code corresponding to the training image, the electronic device concatenates the image feature code, image position code, and image block code corresponding to the training image to generate the image embedding representation corresponding to the training image.

[0106] For training text, the electronic device can input the training text in sequence to the embedding representation layer. The fully connected linear layer in the embedding representation layer extracts features from the training text, generating text feature codes corresponding to the training text. Then, the positional encoding layer in the embedding representation layer performs positional encoding on the text feature codes based on the position of each text character in the training text, obtaining the text positional codes corresponding to the training text. Finally, the block encoding layer in the embedding representation layer performs segmented encoding on the text feature codes based on the segmentation results of each text character in the training text, obtaining the text segment block codes corresponding to the training text.

[0107] After obtaining the text feature code, text position code, and text segment block code corresponding to the training text, the electronic device concatenates the text feature code, text position code, and text segment code corresponding to the training text to generate the text embedding representation corresponding to the training text.

[0108] S222. Input the image embedding representation and the text embedding representation into the feature extraction network in the preset multimodal retrieval network respectively, and output the training image features corresponding to the training image and the training text features corresponding to the training text.

[0109] Specifically, after generating the image embedding representation corresponding to the training image and the text embedding representation corresponding to the training text, the electronic device can input the image embedding representation and the text embedding representation into the feature extraction network in the preset multimodal retrieval network, respectively. The feature extraction network extracts features from the image embedding representation and the text embedding representation and outputs the training image features corresponding to the training image and the training text features corresponding to the training text.

[0110] The feature extraction network can be a Vision Transformer (ViT) network or other feature extraction networks. This application does not specifically limit the feature extraction network.

[0111] When the feature extraction network is a Vision Transformer (ViT) network, attention weights can be calculated using Q and K, and then multiplied by V to obtain the training image features and training text features.

[0112] S23. Based on the relationship between the features of the training images and the features of the training text, update the parameters of the preset multimodal retrieval network to obtain the target preset multimodal retrieval model with updated parameters.

[0113] For details on this step, please refer to [link / reference]. Figure 1 The details of S13 will not be elaborated here.

[0114] For example, for a training image, the training image I of the electronic device is divided into N sub-image blocks of size P×P. Each sub-image block is input into the embedding representation layer in sequence. The linear projection layer in the embedding representation layer extracts features from each sub-image block to obtain the image feature code. The calculation formula is as follows:

[0115]

[0116] Where F represents each sub-image patch, w1 and b1 are the parameters of the fully connected layer, and σ is the activation function.

[0117] The positional encoding layer in the embedding representation layer performs positional encoding on the image feature encoding based on the position of each sub-image patch in the training image, thus obtaining the image positional encoding corresponding to the image feature encoding. The calculation formula is as follows:

[0118]

[0119] Here, B represents the position of each sub-image patch, where w2 and b2 are the parameters of the fully connected layer, and σ is the activation function.

[0120] Use the integer 0 as the block information S of the training image. img Image block coding is obtained after passing through the block coding layer (i.e., the fully connected layer) in the embedded representation layer. The calculation formula is as follows:

[0121]

[0122] Among them, S img To train image block information, w3 and b3 are the parameters of the fully connected layer, and σ is the activation function.

[0123] Then, the electronic device sums the image feature encoding, image location encoding, and image block encoding to obtain the image embedding representation:

[0124]

[0125] For the training text, the electronic device will use the training text T = (t0, t1, t2, ..., t...) L The training text is input in sequence to the embedding representation layer. The fully connected layer in the embedding representation layer extracts features from the training text and generates text feature codes corresponding to the training text. The calculation formula is as follows:

[0126]

[0127] Here, w4 and b4 are the parameters of the linear fully connected layer, and σ is the activation function.

[0128] Then, the positional encoding layer (i.e., the fully connected layer) in the embedding representation layer performs positional encoding on the text feature encoding based on the position of each text character in the training text, thus obtaining the text positional encoding corresponding to the training text. The calculation formula is as follows:

[0129]

[0130] Here, w5 and b5 are the parameters of the fully connected layer, and σ is the activation function.

[0131] The block coding layer (i.e., the fully connected layer) in the embedding representation layer uses the integer 1 as the segmentation information S of the training text T. txt The text segmentation block encoding corresponding to the training text is obtained after passing through a fully connected layer. The calculation formula is as follows:

[0132]

[0133] Where w6 and b6 are the parameters of the fully connected layer, and σ is the activation function.

[0134] Electronic devices add text feature encoding, text position encoding, and text segmentation encoding to obtain the text embedding representation E. txt ,

[0135] Electronic devices can input image embedding representations and text embedding representations separately into a feature extraction network within a pre-defined multimodal retrieval network. This feature extraction network can be a Transformer encoder. The Transformer encoder processes the image embedding representation E... img and text embedding representation E txt Perform feature extraction and encoding, and output the training image features corresponding to the training images. Training text features corresponding to the training text

[0136] The multimodal retrieval model training method provided in this invention inputs training images and training text corresponding to multiple training products into the embedding representation layer of a preset multimodal retrieval network. It outputs image feature codes, image position codes, and image block codes corresponding to the training images. Then, the image feature codes and image position codes corresponding to the training images are concatenated with the image block codes to generate image embedding representations corresponding to the training images, ensuring the accuracy of the generated image embedding representations. Similarly, it outputs text feature codes, text position codes, and text segment codes corresponding to the training text. These are then concatenated to generate text embedding representations corresponding to the training text, ensuring the accuracy of the output text embedding representations, and consequently, the accuracy of the output training image features and training text features. Finally, the image embedding representations and text embedding representations are input into the feature extraction network of the preset multimodal retrieval network, outputting training image features corresponding to the training images and training text features corresponding to the training text, ensuring the accuracy of the output training image features and training text features corresponding to the training text.

[0137] In one embodiment of this application, such as Figure 3 As shown, a multimodal retrieval model training method is provided. Taking the application of this method to electronic devices as an example, the method includes the following steps:

[0138] S31. Obtain the training images and training texts corresponding to multiple training items in the training data.

[0139] Among them, the training images and training texts corresponding to the same training product have the same product identification information.

[0140] For details on this step, please refer to [link / reference]. Figure 2 The details of S21 will not be elaborated here.

[0141] S32. Input the training images and training texts corresponding to multiple training products into the preset multimodal retrieval network respectively, and output the training image features corresponding to the training images and the training text features corresponding to the training texts.

[0142] For details on this step, please refer to [link / reference]. Figure 2 The details of S22 will not be elaborated here.

[0143] S33. Based on the relationship between the training image features and the training text features, update the parameters of the preset multimodal retrieval network to obtain the target preset multimodal retrieval model after parameter update.

[0144] In an optional embodiment of this application, step S33, "updating the parameters of the preset multimodal retrieval network based on the relationship between training image features and training text features to obtain the target preset multimodal retrieval model with updated parameters," may include the following steps:

[0145] S331. Calculate the first similarity between the training image features and the training text features, and the second similarity between the training text features and the training image features.

[0146] Specifically, the electronic device can use a preset similarity algorithm to calculate the first similarity between the training image features and the training text features, and calculate the second similarity between the training text features and the training image features.

[0147] The preset similarity algorithm can be any one of the following: Euclidean distance similarity algorithm, Manhattan distance similarity algorithm, cosine similarity algorithm, Jaccard Similarity similarity algorithm, Pearson correlation coefficient algorithm, and Minkowski distance similarity algorithm. This application does not specifically limit the preset similarity algorithm.

[0148] S332. Calculate the preset loss function based on the relationship between the first similarity and the second similarity.

[0149] In one alternative implementation, the electronic device can calculate a first loss function of training image features relative to training text features based on the relationship between a first similarity of training image features relative to training text features and a first true similarity of training image features relative to training text features, and calculate a second loss function of training text features equivalent to training image features based on the relationship between a second similarity of training text features equivalent to training image features and a second true similarity of training text features equivalent to training image features.

[0150] In one optional embodiment of this application, the first loss function and the second loss function may be the cross-entropy loss function, the infoNCE loss function, or other loss functions. This application does not specifically limit the first loss function and the second loss function.

[0151] After obtaining the first loss function and the second loss function, the electronic device can add the first loss function and the second loss function together to calculate the preset loss function.

[0152] For example, the pre-defined multimodal retrieval network needs to ensure consistency between different input modalities in order to learn the correspondence between images and text. Therefore, a cross-modal contrastive learning task is used to align image modalities and text modalities. For N image-text pairs in a training batch, there are a total of 2N data points. For each sample, the corresponding data from the other modality is considered a positive sample pair, and the remaining samples are considered negative sample pairs. For the input image-text pair ((I i ,T i The training image Transformer editor can output the first similarity between the training image features and the training text features, and the training text Transformer editor can output the second similarity between the training text features and the training image features.

[0153] Then, the electronic device calculates a preset loss function based on the relationship between the first similarity and the second similarity.

[0154] S333. Based on the preset loss function, update the parameters of the preset multimodal retrieval network to obtain the target preset multimodal retrieval model after parameter update.

[0155] In one optional embodiment of this application, the electronic device can update the parameters of a preset multimodal retrieval network according to a preset loss function to obtain a target preset multimodal retrieval model with updated parameters. This maximizes the similarity of matched image-text pairs and minimizes the similarity of unmatched image-text pairs, thereby learning discriminative image-text features and achieving semantic alignment of image-text modalities.

[0156] In an optional embodiment of this application, step S333, "updating the parameters of the preset multimodal retrieval network according to the preset loss function to obtain the target preset multimodal retrieval model with updated parameters," may further include the following steps:

[0157] (1) Input the training image features into the prompting network in the preset multimodal retrieval network, and output the first task-independent candidate parameters and the first task-related candidate parameters corresponding to the training image features.

[0158] In an optional embodiment of this application, step (1) "inputting the training image features into the prompting network in the preset multimodal retrieval network, and outputting the first task-independent candidate parameters and the first task-related candidate parameters corresponding to the training image features" may include the following steps:

[0159] (11) Input the training image features into the prompting network in the preset multimodal retrieval network to generate multiple task-independent initial parameters and multiple task-related initial parameters corresponding to the training image features;

[0160] (12) Calculate the first matching degree between the training parameters corresponding to the training image features and the task-independent initial parameters using the matching function;

[0161] (13) Based on the calculated first matching degree, determine the first task-independent candidate parameter from the task-independent initial parameters;

[0162] (14) Calculate the second degree of matching between the training parameters and the initial parameters of each task-related type using the matching function;

[0163] (15) Based on the calculated second matching degree, determine the first task-related candidate parameter from the task-related initial parameters.

[0164] Among them, task-independent initial parameters refer to the initial parameters corresponding to this task and other tasks, while task-related initial parameters refer to the initial parameters corresponding to this task.

[0165] Specifically, the electronic device can input the training image features into the prompting network in the preset multimodal retrieval network, and the prompting network generates multiple task-independent initial parameters and multiple task-related initial parameters corresponding to the training image features.

[0166] Then, the electronic device can use the matching function to calculate the first degree of matching between the training parameters corresponding to the training image features and the task-independent initial parameters.

[0167] The matching function can be any one of the following: Euclidean distance function, Manhattan distance function, cosine similarity function, Jaccard Similarity function, Pearson correlation coefficient function, and Minkowski distance function. This application does not specifically limit the matching function.

[0168] After calculating the first matching degree between the training parameters corresponding to the training image features and each task-independent initial parameter, the electronic device can sort the first matching degrees from largest to smallest, and then select the task-independent initial parameters corresponding to the top N first matches as the first task-independent candidate parameters.

[0169] Then, the electronic device can use the matching function to calculate the second degree of matching between the training parameters corresponding to the training image features and the initial parameters of each task-related type.

[0170] The matching function can be any one of the following: Euclidean distance function, Manhattan distance function, cosine similarity function, Jaccard Similarity function, Pearson correlation coefficient function, and Minkowski distance function. This application does not specifically limit the matching function.

[0171] After calculating the second matching degree between the training parameters corresponding to the training image features and each task-related initial parameter, the electronic device can sort the second matching degrees from largest to smallest, and then select the task-related initial parameters corresponding to the Top N second matching degrees as the first task-related candidate parameters.

[0172] (2) Input the training text features into the prompting network in the preset multimodal retrieval network, and output the second task-independent candidate parameters and the second task-related candidate parameters corresponding to the training text features.

[0173] Similarly, the electronic device inputs the training text features into the prompting network of a pre-defined multimodal retrieval network, generating multiple task-independent initial parameters and multiple task-related initial parameters corresponding to the training text features. Then, a matching function is used to calculate the third matching degree between the training parameters corresponding to the training text features and each of the task-independent initial parameters. Based on the calculated third matching degrees, second task-independent candidate parameters are determined from the task-independent initial parameters corresponding to the training text features. A fourth matching degree is then calculated between the training parameters and each of the task-related initial parameters corresponding to the training text features. Based on the calculated fourth matching degrees, second task-related candidate parameters are determined from the task-related initial parameters corresponding to the training text features.

[0174] In one alternative embodiment of this application, the electronic device can constrain the cueing network by utilizing the frequency at which each task-related initial parameter is selected, so that the task-related initial parameters selected for each domain are as different as possible, thereby obtaining a second task-related candidate parameter.

[0175] (3) Based on the relationship between the first task-independent candidate parameters, the first task-related candidate parameters, the second task-independent candidate parameters, and the second task-related candidate parameters, as well as the preset loss function, the parameters of the preset multimodal retrieval network are updated to obtain the target preset multimodal retrieval model after parameter update.

[0176] In one optional embodiment of this application, the electronic device can calculate a first mean squared error loss function corresponding to a first task-independent candidate parameter and a first task-related candidate parameter. Then, the electronic device can use the first mean squared error loss function to update the parameters of the feature extraction network and the prompting network in the preset multimodal retrieval network to obtain the target preset multimodal retrieval model with updated parameters.

[0177] In another optional embodiment of this application, the electronic device can calculate a second mean squared error loss function corresponding to the second task-independent candidate parameters and the two-task-related candidate parameters. Then, the electronic device can use the second mean squared error loss function to update the parameters of the feature extraction network and the prompting network in the preset multimodal retrieval network to obtain the target preset multimodal retrieval model with updated parameters.

[0178] In another optional embodiment of this application, the electronic device may further calculate the sum of a first mean squared error loss function and a second mean squared error loss function. Then, based on the sum of the first and second mean squared error loss functions, the parameters of the feature extraction network and the prompting network in the preset multimodal retrieval network are updated to obtain the parameter-updated target preset multimodal retrieval model. Then, the electronic device again updates the parameters of the feature extraction network and the prompting network in the preset multimodal retrieval network according to the preset loss function to obtain the parameter-updated target preset multimodal retrieval model. This process is repeated to update the parameters of the feature extraction network and the prompting network in the preset multimodal retrieval network to obtain the parameter-updated target preset multimodal retrieval model.

[0179] In an optional embodiment of this application, step (3) above, "updating the parameters of the preset multimodal retrieval network based on the relationship between the first task-independent candidate parameters, the first task-related candidate parameters, the second task-independent candidate parameters, and the second task-related candidate parameters, and a preset loss function, to obtain the target preset multimodal retrieval model with updated parameters," may include the following steps:

[0180] (31) Calculate the first mean squared error loss function corresponding to the first task-independent candidate parameters and the first task-related candidate parameters;

[0181] (32) Calculate the second mean squared error loss function corresponding to the second task-independent candidate parameters and the two task-related candidate parameters;

[0182] (33) Calculate the third mean squared error loss function between the first task-independent candidate parameter and the second task-independent candidate parameter;

[0183] (34) Calculate the fourth mean squared error loss function between the first task-related candidate parameters and the second task-related candidate parameters;

[0184] (35) Based on the first mean squared error loss function, the second mean squared error loss function, the third mean squared error loss function, the fourth mean squared error loss function and the preset loss function, the parameters of the preset multimodal retrieval network are updated to obtain the target preset multimodal retrieval model after parameter update.

[0185] Specifically, the electronic device calculates the first mean squared error loss function corresponding to the first task-independent candidate parameter and the first task-related candidate parameter, and calculates the second mean squared error loss function corresponding to the second task-independent candidate parameter and the second task-related candidate parameter.

[0186] Furthermore, the electronic device can project the first task-independent candidate parameters and the second task-independent candidate parameters into a shared latent space, then maintain consistency in the learned semantic projections, and then calculate a third mean squared error loss function between the first task-independent candidate parameters and the second task-independent candidate parameters.

[0187] Then, the electronic device can project the first task-related candidate parameters and the second task-related candidate parameters into a shared latent space, and then keep the learned semantic projection consistent. Then, a fourth mean squared error loss function is calculated between the first task-related candidate parameters and the second task-related candidate parameters.

[0188] Optionally, the electronic device can calculate the sum of the first mean squared error loss function, the second mean squared error loss function, and the preset loss function. Then, using the sum of the first mean squared error loss function, the second mean squared error loss function, and the preset loss function, the device can update the parameters of the feature extraction network and the prompting network in the preset multimodal retrieval network to obtain the target preset multimodal retrieval model with updated parameters.

[0189] Optionally, the electronic device can also calculate the sum of the first mean squared error loss function, the second mean squared error loss function, the third mean squared error loss function, the fourth mean squared error loss function, and the preset loss function to obtain the total loss function. Then, using the total loss function, the feature extraction network and the prompting network in the preset multimodal retrieval network are updated with parameters to obtain the target preset multimodal retrieval model with updated parameters.

[0190] The multimodal retrieval model training method provided in this invention calculates a first similarity between training image features and training text features, and a second similarity between training text features and training image features, ensuring the accuracy of the calculated first and second similarities. Then, based on the relationship between the first and second similarities, a preset loss function is calculated, ensuring the accuracy of the calculated preset loss function.

[0191] Then, the training image features are input into the prompting network of a pre-defined multimodal retrieval network to generate multiple task-independent initial parameters and multiple task-related initial parameters corresponding to the training image features. Next, a matching function is used to calculate the first matching degree between the training parameters corresponding to the training image features and each task-independent initial parameter, ensuring the accuracy of the calculated first matching degree. Based on the calculated first matching degrees, first task-independent candidate parameters are determined from the task-independent initial parameters, ensuring the accuracy of the determined first task-independent candidate parameters. Then, a matching function is used to calculate the second matching degree between the training parameters and each task-related initial parameter, ensuring the accuracy of the calculated second matching degree. Based on the calculated second matching degrees, first task-related candidate parameters are determined from the task-related initial parameters, ensuring the accuracy of the determined first task-related candidate parameters.

[0192] Based on the relationship between the first task-independent candidate parameters, the first task-related candidate parameters, the second task-independent candidate parameters, and the second task-related candidate parameters, as well as the preset loss function, the parameters of the preset multimodal retrieval network are updated to obtain the target preset multimodal retrieval model after parameter update, thus ensuring the accuracy of the obtained target preset multimodal retrieval model.

[0193] The first mean squared error loss function is calculated for the first task-independent candidate parameters and the first task-related candidate parameters; the second mean squared error loss function is calculated for the second task-independent candidate parameters and the second task-related candidate parameters; the third mean squared error loss function is calculated between the first and second task-independent candidate parameters; and the fourth mean squared error loss function is calculated between the first and second task-related candidate parameters. The accuracy of the calculated first, second, third, and fourth mean squared error loss functions is guaranteed. Then, based on the first, second, third, and fourth mean squared error loss functions and the preset loss function, the parameters of the preset multimodal retrieval network are updated to obtain the target preset multimodal retrieval model with updated parameters, ensuring the accuracy of the obtained target preset multimodal retrieval model with updated parameters.

[0194] The above method enables simultaneous training on both training images and training text, effectively utilizing the complementary information between the two modalities. It fuses training image and text features to extract more discriminative image-text retrieval features, making the final trained target multimodal retrieval model suitable for both image and text retrieval, thus overcoming the incompleteness of information from a single modality. Furthermore, the cueing network in this method helps the multimodal retrieval network memorize task-independent and task-related knowledge from the visual-language modality, exhibiting better performance in downstream tasks such as product retrieval compared to other continuous learning methods. This module is also plug-and-play, meaning it can be easily integrated into other visual-language multimodal models to enhance their performance.

[0195] In a specific embodiment, please refer to Figure 4 This invention provides a visual-language multimodal continuous learning method based on a cross-modal cue library, comprising the following steps and details:

[0196] Step 1: The electronic device can transmit the training image I = (i0, i1, i0, ..., i... K The image is divided into a predetermined number of sub-image blocks. The image feature code, image position code, and image block code corresponding to the training image are extracted through an embedding representation layer. The electronic device then adds the image feature code, image position code, and image block code to obtain the image embedding representation.

[0197] The training text T = (t0, t1, t2, ..., t) L The text embedding representation is obtained by extracting text feature codes, text position codes, and text segment codes through the embedding representation layer, and then adding these three codes together.

[0198] Specifically, for the training image, the training image I of the electronic device is divided into N sub-image blocks of size P×P. Each sub-image block is input into the embedding representation layer in sequence. The linear projection layer in the embedding representation layer extracts features from each sub-image block to obtain the image feature code. The calculation formula is as follows:

[0199]

[0200] Where F represents each sub-image patch, w1 and b1 are the parameters of the fully connected layer, and σ is the activation function.

[0201] The positional encoding layer in the embedding representation layer performs positional encoding on the image feature encoding based on the position of each sub-image patch in the training image, thus obtaining the image positional encoding corresponding to the image feature encoding. The calculation formula is as follows:

[0202]

[0203] Here, B represents the position of each sub-image patch, where w2 and b2 are the parameters of the fully connected layer, and σ is the activation function.

[0204] Use the integer 0 as the block information S of the training image. img Image block coding is obtained after passing through the block coding layer (i.e., the fully connected layer) in the embedded representation layer. The calculation formula is as follows:

[0205]

[0206] Among them, S img To train image block information, w3 and b3 are the parameters of the fully connected layer, and σ is the activation function.

[0207] Then, the electronic device sums the image feature encoding, image location encoding, and image block encoding to obtain the image embedding representation:

[0208]

[0209] For the training text, the electronic device will use the training text T = (t0, t1, t2, ..., t...) L The training text is input in sequence to the embedding representation layer. The fully connected layer in the embedding representation layer extracts features from the training text and generates text feature codes corresponding to the training text. The calculation formula is as follows:

[0210]

[0211] Here, w4 and b4 are the parameters of the linear fully connected layer, and σ is the activation function.

[0212] Then, the positional encoding layer (i.e., the fully connected layer) in the embedding representation layer performs positional encoding on the text feature encoding based on the position of each text character in the training text, thus obtaining the text positional encoding corresponding to the training text. The calculation formula is as follows:

[0213]

[0214] Here, w5 and b5 are the parameters of the fully connected layer, and σ is the activation function.

[0215] The block coding layer (i.e., the fully connected layer) in the embedding representation layer uses the integer 1 as the segmentation information S of the training text T. txt The text segmentation block encoding corresponding to the training text is obtained after passing through a fully connected layer. The calculation formula is as follows:

[0216]

[0217] Where w6 and b6 are the parameters of the fully connected layer, and σ is the activation function.

[0218] Electronic devices add text feature encoding, text position encoding, and text segmentation encoding to obtain the text embedding representation E. txt ,

[0219] Step 2: The electronic device can input the image embedding representation and the text embedding representation into the feature extraction network of a preset multimodal retrieval network, respectively. The feature extraction network can be a Transformer encoder. The Transformer encoder processes the image embedding representation E... img and text embedding representation E txt Perform feature extraction and encoding, and output the training image features corresponding to the training images. Training text features corresponding to the training text The Transformer editor corresponding to the training image and the Transformer editor corresponding to the training text each have L layers, and the calculation formula for each layer is:

[0220]

[0221]

[0222]

[0223]

[0224] Where t-1 and t are Transformer layer numbers, LN is a LayerNorm layer for feature normalization, MLP is a fully connected layer, and MSA is a multi-head attention layer, the calculation formula of which is as follows:

[0225]

[0226] Head i =Attention(HW) i Q HW i K HW i V )

[0227] MSA(H)=Concat(Head1,...,Head h W O

[0228] Where Q, K, and V represent different learnable linear parameters, d k W represents the dimension of the linear parameter. i This represents the projection parameters of the i-th head, and Concat means stacking different heads along the feature dimension.

[0229] Step 3: For the extracted image and text features, use a contrastive learning self-supervised task to train the network.

[0230] Specifically, such as Figure 5 As shown, the pre-defined multimodal retrieval network needs to ensure consistency between different input modalities in order to learn the correspondence between images and text. Therefore, a cross-modal contrastive learning task is used to align image modalities and text modalities. For N image-text pairs in a training batch, there are a total of 2N data points. For each sample, the corresponding data from the other modality is considered a positive sample pair, and the remaining samples are considered negative sample pairs. For the input image-text pair ((I i ,T i The text and image features output by the model's text Transformer and image Transformer are used to determine each pair of text and image features. Its loss function is:

[0231]

[0232] in, Used to calculate image-text pairs and The similarity between them, where τ is the temperature regulation parameter. It is a binary index that returns 1 if and only if i != j. This contrastive loss function brings paired text-image vectors closer together and pulls unpaired text-image vectors further apart.

[0233] Step 4: For the extracted image and text features, use a cross-modal cue library for continuous learning tasks to train the network.

[0234] Specifically, such as Figure 5 As shown, taking the image modality as an example (the text modality is similar), two learnable keys K and K are constructed. Use a matching function to select N hints from the hint library:

[0235]

[0236] in, This indicates the index of the first task-independent candidate parameter selected. The first task-independent candidate parameter selected is... γ is the matching function. Training parameters corresponding to training image features. The initial parameters are task-independent. The prompts selected in this process are task-independent, and the corresponding prompt library is a task-independent prompt library.

[0237] Specifically, such as Figure 5 As shown, to enable the pre-defined multimodal retrieval network to simultaneously learn common knowledge within the same domain, priority is added to help the model achieve this goal. Specifically, a prompt selection frequency is added. To constrain the model so that the suggestions for each domain are as different as possible:

[0238]

[0239] in, This indicates the index of the first task-related candidate parameter selected. The first task-related candidate parameter selected is... γ is the matching function. Training parameters corresponding to training image features. These are task-related initial parameters. The prompts selected in this process are task-related, and the corresponding prompt library is a task-specific prompt library.

[0240] Similarly, electronic devices can obtain second task-independent candidate parameters corresponding to text features. Second task-related candidate parameters

[0241] To make the learnable keys and their selected prompts closely approximate each other, the electronic device can calculate a first mean squared error loss function corresponding to a first task-independent candidate parameter and a first task-related candidate parameter, and a second mean squared error loss function corresponding to a second task-independent candidate parameter and a second task-related candidate parameter. Then, the first and second mean squared error loss functions are added together to obtain the target mean squared error loss.

[0242]

[0243] Wherein, when M=I, it represents the first mean squared error loss function obtained by calculation, and when M=T, it represents the second mean squared error loss function obtained by calculation.

[0244] Additionally, for task-independent cues, a linear layer (Linear(·)) is used to project them into a shared latent space, while preserving the learned semantic projection. and Consistent, the loss function is:

[0245]

[0246] The total loss function during the entire model training process is:

[0247]

[0248] Then, the electronic device updates the parameters of the preset multimodal retrieval network according to the total loss function to obtain the target preset multimodal retrieval model with updated parameters.

[0249] To better illustrate the multimodal retrieval model training method provided in this application, this application provides a multimodal retrieval method. It should be noted that the multimodal retrieval method provided in this application can be executed by a multimodal retrieval device. This device can be implemented as part or all of a computer device through software, hardware, or a combination of both. The computer device can be a server or a terminal. In this application embodiment, the server can be a single server or a server cluster composed of multiple servers. The terminal in this application embodiment can be a smartphone, personal computer, tablet computer, wearable device, or other intelligent hardware device such as an intelligent robot. In the following method embodiments, the execution subject is always described using an electronic device as an example.

[0250] In one embodiment of this application, such as Figure 5 As shown, a multimodal retrieval method is provided. Taking the application of this method to an electronic device as an example, the method includes the following steps:

[0251] S41. Obtain the search text and / or search image corresponding to the product to be searched.

[0252] Optionally, the electronic device can acquire the search text and / or search image corresponding to the product to be searched input by the user; optionally, the electronic device can also receive the search text and / or search image corresponding to the product to be searched sent by other devices; the electronic device can also query the search text and / or search image corresponding to the product to be searched in the database.

[0253] This application does not specifically limit the method by which electronic devices acquire the search text and / or search image corresponding to the product to be searched.

[0254] S42. Input the search text and / or search image into the target preset multimodal search model. The target preset multimodal search model performs feature extraction on the search text, extracting the search text features corresponding to the search text, and / or performs feature extraction on the search image, extracting the search image features corresponding to the search image.

[0255] The target-preset multimodal retrieval model is trained according to any of the above-mentioned multimodal retrieval model training methods.

[0256] Specifically, after acquiring the search text and / or search image, the electronic device can input the search text and / or search image into a target preset multimodal retrieval model. The embedding representation layer in the target preset multimodal retrieval model can output target text embedding representations and / or target image embedding representations. Then, the electronic device inputs the target text embedding representations and / or target image embedding representations into the feature extraction layer in the target preset multimodal retrieval model to extract features from the target text embedding representations and / or target image embedding representations, and outputs search text features and / or search image features.

[0257] S43. Calculate the first retrieval similarity between the retrieved text features and the features of each training image in the database, and / or calculate the second retrieval similarity between the retrieved image features and the features of each training image in the database.

[0258] Specifically, after obtaining the search text features and / or search image features, the electronic device can use a preset similarity algorithm to calculate the first search similarity between the search text features and each training image feature in the database, and / or calculate the second search similarity between the search image features and each training image feature in the database.

[0259] The preset similarity algorithm can be any one of the following: Euclidean distance similarity algorithm, Manhattan distance similarity algorithm, cosine similarity algorithm, Jaccard Similarity similarity algorithm, Pearson correlation coefficient algorithm, and Minkowski distance similarity algorithm. This application does not specifically limit the preset similarity algorithm.

[0260] S44. Determine the training image corresponding to the maximum similarity in the first search similarity and / or the second search similarity as the target training image corresponding to the product to be searched.

[0261] Specifically, after calculating the first search similarity and / or the second search similarity, the electronic device can determine that the training image corresponding to the maximum similarity among the first search similarity and / or the second search similarity is the target training image corresponding to the product to be searched.

[0262] The multimodal retrieval method provided in this invention obtains the retrieval text and / or retrieval image corresponding to the product to be retrieved. Then, the retrieval text and / or retrieval image are input into a target preset multimodal retrieval model. The target preset multimodal retrieval model performs feature extraction on the retrieval text, extracting the retrieval text features corresponding to the retrieval text, and / or performs feature extraction on the retrieval image, extracting the retrieval image features corresponding to the retrieval image, ensuring the accuracy of the extracted retrieval text features and / or retrieval image features. Then, the training image corresponding to the maximum similarity between the first retrieval similarity and / or the second retrieval similarity is determined as the target training image corresponding to the product to be retrieved, ensuring the accuracy of the determined target training image. The above method enables retrieval of both text and images, compensating for the problem of incomplete information in a single modality.

[0263] It should be understood that, although Figures 1-3 as well as Figure 5 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figures 1-3 as well as Figure 5 At least some of the steps in the process may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but may be executed at different times. The execution order of these steps or stages is not necessarily sequential, but may be executed in turn or alternately with other steps or at least some of the steps or stages in other steps.

[0264] like Figure 6 As shown, this embodiment provides a multimodal retrieval model training device, including:

[0265] The first acquisition module 51 is used to acquire training images and training texts corresponding to multiple training products in the training data; wherein, the product identification information corresponding to the training image and training text of the same training product is the same.

[0266] The first input module 52 is used to input training images and training texts corresponding to multiple training products into a preset multimodal retrieval network, and output the training image features corresponding to the training images and the training text features corresponding to the training texts.

[0267] The update module 53 is used to update the parameters of the preset multimodal retrieval network according to the relationship between the training image features and the training text features, so as to obtain the target preset multimodal retrieval model with updated parameters.

[0268] In one embodiment of this application, the first input module 52 is specifically used to input training images and training texts corresponding to multiple training products into the embedding representation layer of a preset multimodal retrieval network, and output the image embedding representations corresponding to the training images and the text embedding representations corresponding to the training texts; and to input the image embedding representations and text embedding representations into the feature extraction network of the preset multimodal retrieval network, and output the training image features corresponding to the training images and the training text features corresponding to the training texts.

[0269] In one embodiment of this application, the first input module 52 is specifically used to divide the training image into a preset number of sub-image blocks; input each sub-image block into the embedding representation layer in sequence; the linear projection layer in the embedding representation layer extracts features from each sub-image block to generate an image feature code corresponding to the training image; the position encoding layer in the embedding representation layer performs position encoding on the image feature code according to the position of each sub-image block in the training image to obtain the image position code corresponding to the image feature code; the block encoding layer in the embedding representation layer performs block encoding on the image feature code according to the block division result of each sub-image block in the training image to obtain the image block code corresponding to the image feature code; and the image feature code and image position code corresponding to the training image are encoded using image block encoding. The code is concatenated to generate the image embedding representation corresponding to the training image. For the training text, the training text is input into the embedding representation layer in sequence. The linear fully connected layer in the embedding representation layer extracts features from the training text to generate the text feature code corresponding to the training text. The positional encoding layer in the embedding representation layer performs positional encoding on the text feature code according to the position of each text character in the training text to obtain the text positional encoding corresponding to the training text. The block encoding layer in the embedding representation layer performs segmented encoding on the text feature code according to the segmentation result of each text character in the training text to obtain the text segment block encoding corresponding to the training text. The text feature code, text position code, and text segment code corresponding to the training text are concatenated to generate the text embedding representation corresponding to the training text.

[0270] In one embodiment of this application, the update module 53 is specifically used to calculate the first similarity between the training image features and the training text features, and the second similarity between the training text features and the training image features; calculate a preset loss function based on the relationship between the first similarity and the second similarity; and update the parameters of the preset multimodal retrieval network according to the preset loss function to obtain the target preset multimodal retrieval model after parameter update.

[0271] In one embodiment of this application, the update module 53 is specifically used to input training image features into the prompting network in a preset multimodal retrieval network, and output the first task-independent candidate parameters and the first task-related candidate parameters corresponding to the training image features; input training text features into the prompting network in the preset multimodal retrieval network, and output the second task-independent candidate parameters and the second task-related candidate parameters corresponding to the training text features; update the parameters of the preset multimodal retrieval network according to the relationship between the first task-independent candidate parameters, the first task-related candidate parameters, the second task-independent candidate parameters, and the second task-related candidate parameters, and a preset loss function, to obtain the target preset multimodal retrieval model with updated parameters.

[0272] In one embodiment of this application, the update module 53 is specifically configured to input training image features into a prompting network in a preset multimodal retrieval network, generate multiple task-independent initial parameters and multiple task-related initial parameters corresponding to the training image features; calculate a first matching degree between the training parameters corresponding to the training image features and each task-independent initial parameter using a matching function; determine a first task-independent candidate parameter from the task-independent initial parameters based on the calculated first matching degrees; calculate a second matching degree between the training parameters and each task-related initial parameter using a matching function; and determine a first task-related candidate parameter from the task-related initial parameters based on the calculated second matching degrees.

[0273] In one embodiment of this application, the update module 53 is specifically used to calculate the first mean squared error loss function corresponding to the first task-independent candidate parameters and the first task-related candidate parameters; calculate the second mean squared error loss function corresponding to the second task-independent candidate parameters and the second task-related candidate parameters; calculate the third mean squared error loss function between the first task-independent candidate parameters and the second task-independent candidate parameters; calculate the fourth mean squared error loss function between the first task-related candidate parameters and the second task-related candidate parameters; and update the parameters of the preset multimodal retrieval network according to the first mean squared error loss function, the second mean squared error loss function, the third mean squared error loss function, the fourth mean squared error loss function, and the preset loss function to obtain the target preset multimodal retrieval model after parameter update.

[0274] like Figure 7 As shown, this embodiment provides a multimodal retrieval device, including:

[0275] The second acquisition module 61 is used to acquire the search text and / or search image corresponding to the product to be searched;

[0276] The second input module 62 is used to input the search text and / or search image into the target preset multimodal retrieval model. The target preset multimodal retrieval model performs feature extraction on the search text, extracting the search text features corresponding to the search text, and / or performs feature extraction on the search image, extracting the search image features corresponding to the search image. The target preset multimodal retrieval model is trained according to the multimodal retrieval model training method of any of the above embodiments.

[0277] The calculation module 63 is used to calculate the first retrieval similarity between the retrieved text features and the features of each training image in the database, and / or to calculate the second retrieval similarity between the retrieved image features and the features of each training image in the database;

[0278] The determination module 64 is used to determine the training image corresponding to the maximum similarity in the first search similarity and / or the second search similarity as the target training image corresponding to the product to be retrieved.

[0279] For specific limitations and beneficial effects regarding the multimodal retrieval model training device and the multimodal retrieval device, please refer to the limitations of the multimodal retrieval model training method and the multimodal retrieval method mentioned above, which will not be repeated here. Each module in the aforementioned multimodal retrieval model training device and multimodal retrieval device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independently of the processor in the electronic device, or stored in software in the memory of the electronic device, so that the processor can call and execute the operations corresponding to each module.

[0280] This invention also provides an electronic device having the above-described features. Figure 6 The multimodal retrieval model training device and shown Figure 7 The multimodal retrieval device shown.

[0281] like Figure 8 As shown, Figure 8 This is a schematic diagram of the structure of an electronic device provided in an optional embodiment of the present invention, such as... Figure 8As shown, the electronic device may include: at least one processor 71, such as a CPU (Central Processing Unit), at least one communication interface 73, memory 74, and at least one communication bus 72. The communication bus 72 is used to enable communication between these components. The communication interface 73 may include a display screen or a keyboard; optionally, the communication interface 73 may also include a standard wired interface or a wireless interface. The memory 74 may be high-speed RAM (Random Access Memory) or non-volatile memory, such as at least one disk storage device. Optionally, the memory 74 may also be at least one storage device located remotely from the aforementioned processor 71. The processor 71 may be combined with... Figure 7 as well as Figure 8 The described apparatus has an application program stored in memory 74, and the processor 71 calls the program code stored in memory 74 to perform any of the above method steps.

[0282] The communication bus 72 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The communication bus 72 can be divided into an address bus, a data bus, and a control bus, etc. For ease of representation, Figure 8 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0283] The memory 74 may include volatile memory, such as random-access memory (RAM); the memory may also include non-volatile memory, such as flash memory, hard disk drive (HDD) or solid-state drive (SSD); the memory 74 may also include a combination of the above types of memory.

[0284] The processor 71 can be a central processing unit (CPU), a network processor (NP), or a combination of CPU and NP.

[0285] The processor 71 may further include a hardware chip. This hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0286] Optionally, memory 74 is also used to store program instructions. Processor 71 can call the program instructions to implement the functions described in this application. Figures 1 to 3 The multimodal retrieval model training method shown in the embodiments and Figure 5 The multimodal retrieval method shown in the embodiments.

[0287] This invention also provides a non-transitory computer storage medium storing computer-executable instructions that can execute the multimodal retrieval model training method and the multimodal retrieval method in any of the above method embodiments. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk drive (HDD), or solid-state drive (SSD), etc.; the storage medium may also include combinations of the above types of memory.

[0288] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A method for training a multimodal retrieval model, characterized in that, include: Obtain training images and training texts corresponding to multiple training products in the training data; wherein, the product identification information corresponding to the training image and training text of the same training product is the same; The training images and training texts corresponding to multiple training products are respectively input into a preset multimodal retrieval network, and the training image features corresponding to the training images and the training text features corresponding to the training texts are output. Based on the relationship between the training image features and the training text features, the parameters of the preset multimodal retrieval network are updated to obtain the target preset multimodal retrieval model with updated parameters. The step of updating the parameters of the preset multimodal retrieval network based on the relationship between the training image features and the training text features to obtain the target preset multimodal retrieval model with updated parameters includes: Calculate the first similarity between the training image features and the training text features, and the second similarity between the training text features and the training image features; Based on the relationship between the first similarity and the second similarity, a preset loss function is calculated; Based on the preset loss function, the parameters of the preset multimodal retrieval network are updated to obtain the target preset multimodal retrieval model with updated parameters; The step of updating the parameters of the preset multimodal retrieval network according to the preset loss function to obtain the target preset multimodal retrieval model with updated parameters includes: The training image features are input into the prompting network in the preset multimodal retrieval network, and the first task-independent candidate parameters and the first task-related candidate parameters corresponding to the training image features are output. The training text features are input into the prompting network in the preset multimodal retrieval network, and the second task-independent candidate parameters and the second task-related candidate parameters corresponding to the training text features are output. Based on the relationship between the first task-independent candidate parameters, the first task-related candidate parameters, the second task-independent candidate parameters, and the second task-related candidate parameters, as well as the preset loss function, the parameters of the preset multimodal retrieval network are updated to obtain the target preset multimodal retrieval model with updated parameters.

2. The method according to claim 1, characterized in that, The step of inputting the training images and training texts corresponding to multiple training items into a preset multimodal retrieval network, and outputting the training image features corresponding to the training images and the training text features corresponding to the training texts, includes: The training images and training texts corresponding to the multiple training products are respectively input into the embedding representation layer of the preset multimodal retrieval network, and the image embedding representations corresponding to the training images and the text embedding representations corresponding to the training texts are output. The image embedding representation and the text embedding representation are respectively input into the feature extraction network in the preset multimodal retrieval network, and the training image features corresponding to the training image and the training text features corresponding to the training text are output.

3. The method according to claim 2, characterized in that, The step of inputting the training images and training texts corresponding to multiple training items into the embedding representation layer of the preset multimodal retrieval network, and outputting the image embedding representations corresponding to the training images and the text embedding representations corresponding to the training texts, includes: For the training image, the training image is divided into a preset number of sub-image blocks; Each sub-image patch is input into the embedding representation layer in sequence, and the linear projection layer in the embedding representation layer extracts features from each sub-image patch to generate the image feature code corresponding to the training image. The position encoding layer in the embedding representation layer performs position encoding on the image feature encoding according to the position of each of the sub-image blocks in the training image, so as to obtain the image position encoding corresponding to the image feature encoding; The block coding layer in the embedding representation layer performs block coding on the image feature code according to the block coding results of each of the sub-image blocks in the training image, so as to obtain the image block code corresponding to the image feature code; The image feature encoding and the image position encoding corresponding to the training image are concatenated with the image block encoding to generate the image embedding representation corresponding to the training image; For the training text, the training text is input into the embedding representation layer in sequence. The linear fully connected layer in the embedding representation layer extracts features from the training text and generates the text feature code corresponding to the training text. The position encoding layer in the embedding representation layer performs position encoding on the text feature encoding according to the position of each text character in the training text, so as to obtain the text position encoding corresponding to the training text; The block coding layer in the embedded representation layer performs segmented coding on the text feature encoding according to the segmentation results of each text character in the training text, so as to obtain the text segmentation code corresponding to the training text; The text feature encoding, text position encoding, and text segmentation encoding corresponding to the training text are concatenated to generate the text embedding representation corresponding to the training text.

4. The method according to claim 1, characterized in that, The step of inputting the training image features into the prompting network in the preset multimodal retrieval network and outputting the first task-independent candidate parameters and the first task-related candidate parameters corresponding to the training image features includes: The training image features are input into the prompting network in the preset multimodal retrieval network to generate multiple task-independent initial parameters and multiple task-related initial parameters corresponding to the training image features; The first matching degree between the training parameters corresponding to the training image features and each of the task-independent initial parameters is calculated using a matching function. Based on the calculated first matching degrees, first task-independent candidate parameters are determined from the task-independent initial parameters; The second matching degree between the training parameters and each of the task-related initial parameters is calculated using a matching function. Based on the calculated second matching degrees, first task-related candidate parameters are determined from the task-related initial parameters.

5. The method according to claim 1, characterized in that, The step of updating the parameters of the preset multimodal retrieval network based on the relationship between the first task-independent candidate parameters, the first task-related candidate parameters, the second task-independent candidate parameters, and the second task-related candidate parameters, and the preset loss function, to obtain the parameter-updated target preset multimodal retrieval model, includes: Calculate the first mean squared error loss function corresponding to the first task-independent candidate parameter and the first task-related candidate parameter; Calculate the second mean squared error loss function corresponding to the second task-independent candidate parameter and the second task-related candidate parameter; Calculate the third mean squared error loss function between the first task-independent candidate parameters and the second task-independent candidate parameters; Calculate the fourth mean squared error loss function between the first task-related candidate parameters and the second task-related candidate parameters; Based on the first mean squared error loss function, the second mean squared error loss function, the third mean squared error loss function, the fourth mean squared error loss function, and the preset loss function, the parameters of the preset multimodal retrieval network are updated to obtain the target preset multimodal retrieval model with updated parameters.

6. A multimodal retrieval method, characterized in that, The method includes: Obtain the search text and / or search image corresponding to the product to be searched; The search text and / or the search image are input into a target preset multimodal retrieval model. The target preset multimodal retrieval model performs feature extraction on the search text to extract the search text features corresponding to the search text, and / or performs feature extraction on the search image to extract the search image features corresponding to the search image; wherein, the target preset multimodal retrieval model is trained according to the multimodal retrieval model training method according to any one of claims 1-5; Calculate the first retrieval similarity between the retrieved text features and each training image feature in the database, and / or calculate the second retrieval similarity between the retrieved image features and each training image feature in the database; The training image corresponding to the maximum similarity between the first retrieval similarity and / or the second retrieval similarity is determined as the target training image corresponding to the product to be retrieved.

7. An electronic device, characterized in that, The system includes a memory and a processor, wherein the memory stores computer instructions, and the processor executes the computer instructions to perform the multimodal retrieval model training method of any one of claims 1-5 and the multimodal retrieval method of claim 6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing a computer to execute the multimodal retrieval model training method of any one of claims 1-5 and the multimodal retrieval method of claim 6.

Images