A personalized multi-modal text-image retrieval method and system based on large language model enhancement

Abstract

This invention discloses a personalized multimodal image and text retrieval method and system based on large language model enhancement. The invention first performs query enhancement processing; then performs image enhancement processing; next, it maps the enhanced query and image information into a unified feature vector; then, it stacks the feature vectors through dimension stacking to form a unified sequence, which is then encoded by Transformer and obtained by mean pooling and MLP projection to obtain the CF (Catalytic Retrieval). E Subsequently, the outputs are concatenated through multi-directional attention calculations and passed through an MLP layer to obtain the CF. A ; Again, through the gating mechanism, CF E and CF A Output dynamic weighting to generate CF EA Next, the user preference information is integrated into CF through the DeepFM module. EA In the middle, we get CF EA * Then CF EA * The model is then concatenated with the cosine similarity score before being input to the output layer; finally, the model is trained. This invention can more accurately capture the user's true search intent and solves the problem of ambiguous query intent.

Description

Technical Field

[0001] This invention relates to the field of multimodal image and text processing technology, and more specifically, to a personalized multimodal image and text retrieval method and system based on large language model enhancement. Background Technology

[0002] Consumer electronics refers to electronic devices designed for personal daily use, such as smartphones, smart cameras, and smart home devices. These devices generate massive amounts of image data, providing crucial support for diverse applications such as recommendation systems, voice-interactive interfaces, and multimedia services. Text-image retrieval, by processing text queries, retrieves the most relevant matching images from massive databases for consumers, and is a key foundation for consumer electronics applications.

[0003] Improving image retrieval accuracy directly enhances user experience. For example, in smart TVs and smartphone photo album management, users expect to quickly retrieve relevant photos locally or across devices using keywords. Similarly, when taking or editing images with a smart camera, users need to accurately filter target content from massive media libraries (usually images) and editing templates based on keywords. Although some text-based image retrieval methods have image localization capabilities, these methods still cannot solve the significant limitations and application-specific challenges of image retrieval systems in consumer electronics scenarios.

[0004] Specifically, in consumer electronics scenarios, current image retrieval systems face three main challenges. First, in consumer electronics applications, text-image retrieval queries are often very concise, frequently leading to ambiguity in intent. This results in different results for the same query. Statistical analysis of user query lengths shows that over 91% of queries are between 1 and 4 words long, indicating the prevalence of short queries and their significant impact on the relevance of search results. Second, a large number of images lack text descriptions and metadata tags, and some annotation information is incorrect or unreliable. This semantic deficiency severely affects retrieval accuracy. Independent images taken by consumer electronic devices such as smartphones and smart cameras often lack supplementary descriptions. Third, users with different identities have different needs. Even with the same search content, different users may require different images. For example, when searching using "car" as a keyword, users with different roles, such as ride-hailing marketers, digital engineers, and game designers, have significantly different expectations for the desired image results.

[0005] In summary, current research on text-image retrieval suffers from problems such as short queries, lack of accompanying text descriptions for images, and difficulty in achieving personalized retrieval. Therefore, there is an urgent need to find a solution that can address these issues and improve the accuracy of text-image retrieval. Summary of the Invention

[0006] This invention addresses the shortcomings of existing technologies by proposing a personalized multimodal image and text retrieval method and system based on large language model enhancement. This invention introduces bidirectional semantic enhancement and personalized injection mechanisms, addressing the problems of ambiguous user query intent and sparse image semantic information, thereby providing highly personalized search results for different users and achieving significant improvements in model performance and user experience.

[0007] This invention aims to provide a re-ranking model that includes two core stages: data augmentation and image retrieval. For a user-submitted retrieval query and any candidate image, this invention aims to accurately evaluate the multi-dimensional similarity between the two and use this as the basis for optimizing the ranking of all candidate images.

[0008] The present invention adopts the following technical solution:

[0009] In a first aspect, this invention proposes a personalized multimodal image and text retrieval method based on large language model enhancement, comprising the following steps:

[0010] Query enhancement: Utilize user identity information and preset prompt templates to generate enhanced queries through a large language model;

[0011] Image enhancement: Performing enhancement processing on an image to obtain enhanced image information;

[0012] Multimodal feature encoding: The text information obtained from the query enhancement and the image enhancement is mapped to the image information through an encoding layer into a unified d-dimensional feature vector;

[0013] Cross-encoder fusion: The feature vectors are stacked dimensionally to form a unified sequence. This sequence is then encoded by Transformer and subjected to mean pooling and MLP projection to obtain the multimodal fusion representation (CF). E ;

[0014] Bidirectional cross-attention fusion: The outputs of multi-directional attention calculations are concatenated and passed through an MLP layer to obtain the bidirectional cross-attention fusion representation (CF). A ;

[0015] Multimodal gating fusion: This involves using a gating mechanism to control the CF... E and the CF A Output dynamic weighting to generate optimized fusion representation CF EA ;

[0016] Personalization injection: User preference information is integrated into the fused representation CF via the DeepFM module. EA In the process, a personalized fusion representation CF is obtained. EA * ;

[0017] Generate final output: The CFEA * The result is concatenated with the calculated cosine similarity score and then input into the output layer to obtain a probability value as the final matching score.

[0018] Model training: Iteratively train the model using the business dataset until it converges or reaches the preset training rounds.

[0019] Secondly, this invention proposes a personalized multimodal image and text retrieval system based on large language model enhancement, comprising:

[0020] Query enhancement module: Used to generate enhanced queries through a large language model by utilizing user identity information and preset prompt templates;

[0021] Image enhancement module: Used for enhancing images;

[0022] Multimodal feature encoding module: used to map the text information obtained from the query enhancement and the image enhancement, along with the image information, into a unified d-dimensional feature vector through an encoding layer;

[0023] Cross-encoder fusion module: used to stack the feature vectors into a unified sequence through dimensionality stacking. After the sequence is encoded by Transformer, it is then subjected to mean pooling and MLP projection to obtain the multimodal fusion representation (CF). E ;

[0024] Bidirectional cross-attention fusion module: used to concatenate the outputs of multi-directional attention calculations, and obtain the bidirectional cross-attention fusion representation (CF) through an MLP layer. A ;

[0025] Multimodal gating fusion module: used to control the CF through a gating mechanism E and the CF A Output dynamic weighting to generate optimized fusion representation CF EA ;

[0026] Personalization injection module: used to integrate user preference information into the fusion representation CF via the DeepFM module. EA In the process, a personalized fusion representation CF is obtained. EA * ;

[0027] Output module: used to output the CF EA * The result is concatenated with the calculated cosine similarity score and then input into the output layer to obtain a probability value as the final matching score.

[0028] Training module: Used to iteratively train the model using the business dataset until convergence or a preset number of training rounds are reached.

[0029] Beneficial technical effects of the present invention:

[0030] First, by introducing a large language model to personalize short and vague user queries with user identity, this invention can more accurately capture the user's true search intent, effectively solving the problem of vague search intent and improving the accuracy of retrieval from the source.

[0031] Second, this invention creatively combines multimodal large models and multiple visual models to generate rich and reliable text information for images from both macroscopic description and microscopic labeling levels. This greatly alleviates the semantic gap caused by the lack of image annotations and significantly makes up for the lack of semantic information in images.

[0032] Third, this invention not only considers user identity in the query enhancement stage, but also introduces a preference model based on user ID and image ID in the final fusion stage, which enables the search results to be dynamically adjusted according to the preferences of different users, providing personalized services for each user and achieving a highly personalized search experience.

[0033] Fourth, this invention, through a dual-stream fusion architecture (cross-encoder and bidirectional cross-attention) and gating mechanism, can robustly fuse multi-source heterogeneous information, resulting in excellent model performance and strong generalization ability. Experiments demonstrate that, on both real-world commercial datasets and public datasets, the method of this invention significantly outperforms existing baseline models in multiple key metrics such as accuracy, NDCG, and MRR. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0035] Figure 1 This is a schematic diagram of the framework of a personalized multimodal image and text retrieval method based on large language model enhancement;

[0036] Figure 2 This is a schematic diagram of the bidirectional cross-attention fusion module in a personalized multimodal image and text retrieval method based on a large language model enhancement.

[0037] Figure 3 This is a flowchart illustrating a personalized multimodal image and text retrieval method based on a large language model. Detailed Implementation

[0038] The specific embodiments of the present invention will be described below with reference to the accompanying drawings and examples. However, the following examples are only used to illustrate the present invention in detail and do not limit the scope of the present invention in any way.

[0039] like Figure 1 and Figure 3 As shown, this invention discloses a personalized multimodal image and text retrieval method based on large language model enhancement. The invention will be described in detail below with reference to embodiments, specifically including the following steps:

[0040] Step S1: Query Enhancement

[0041] Specifically, when a user inputs a query Q, the system combines the user's recorded personal identification information U (such as the user's occupation, historical search preferences, etc.) with a preset query prompt template P. Q The input is then concatenated. The combined input is then fed into a Large Language Model (LLM) for data augmentation, generating an enhanced query Q that contains richer context and clearer intent. en This process can be formally represented as:

[0042]

[0043] In this context, || represents the concatenation operation.

[0044] Step S2: Image Enhancement

[0045] Specifically, image enhancement is performed through image normalization, image description generation, and image label generation to obtain enhanced semantic information of the image. A multimodal large model is used to generate macroscopic text descriptions for candidate images; simultaneously, multiple visual models are used in parallel to extract microscopic labels from the images, and these labels are filtered and fused to obtain the final image label set.

[0046] Specifically, step S2 includes the following method steps:

[0047] Step S21, image normalization, specifically, normalizing images of various sizes and returning images of uniform format and size;

[0048] Step 22, Image Description Generation: First, convert image I to a Base64 string format I′. Then, combine this string with a preset image description prompt template P. I The images are concatenated and input into a multimodal large model (GPT-4 Vision) to generate a high-level descriptive text C that captures the overall content and style of the image. I This process can be formally represented as:

[0049]

[0050] Step S23, image label generation, specifically, extracts micro-level semantic information from the image to generate fine-grained labels, which includes three main components:

[0051] (1) SD-Tagger: In this application, the Stable Diffusion tag plugin is used to perform reverse reasoning on the image, generate multiple tags that match the image content, and encode the tags using the CLIP model. The cosine similarity between each tag and the image is calculated, and the five tags most relevant to the original image are selected to obtain the tag set T. SD .

[0052] (2) OD-Tagger: In this application, the object detection model YOLO is used, and the confidence threshold is set to 0.9 to extract the most prominent entity objects in the image as labels, thus obtaining the label set T. OD .

[0053] (3) IC-Tagger: In this application, image captioning generation models such as BLIP2 and mPLUG are used to generate short descriptions for images and extract key phrases as tags to obtain a tag set T. IC .

[0054] In this embodiment of the application, all tags (T) generated by the above three sub-modules are... SD T OD T IC The results are then summarized, and the CLIP model is used again to calculate the cosine similarity between all candidate labels and the original image. The five labels with the highest similarity are selected as the final image label set T. I .

[0055] Step S3: Multimodal feature encoding

[0056] Specifically, multimodal feature encoding is performed on the enhanced query from step S1 and the enhanced image semantic information from step S2 to obtain user-related feature representations and image-related feature representations.

[0057] Specifically, in this embodiment of the application, the text information (U, Q) obtained in steps S1 and S2 is... en C I T I The image information (I) is input into a unified feature encoding layer CLIP, which maps data from different modalities to the same high-dimensional feature space, resulting in a d-dimensional vector representation of the corresponding dimensions (batch size, sequence length): V U V Q V I V C V T Each modality is equally important, allowing for fully symmetrical interactions across modalities.

[0058] Step S4: Cross encoder fusion

[0059] In this embodiment of the application, the multimodal feature encoding in step S3 is processed by the cross-encoder fusion module to generate a unified multimodal feature representation.

[0060] Specifically, all feature vectors v1, v2, ..., v obtained in step S3 are... L Stack the sequences along their length to form a unified sequence V=[v1;v2;...;v...]. L ] R B×L×d Where B is the batch size and L is the sequence length. This sequence is input into a multi-layer Transformer encoder for fully symmetric interaction to obtain a context-rich feature vector E. V， The output E of the Transformer V A fixed-length feature vector E is obtained by performing mean pooling along the sequence length dimension. V It is then projected through an MLP layer to obtain a unified multimodal fusion representation CF. E The calculation process is as follows:

[0061]

[0062]

[0063]

[0064] Step S5: Bidirectional Cross-Attention Fusion

[0065] The multimodal feature encoding in step S3 is processed by a bidirectional cross-modal attention fusion module to achieve adaptive multimodal feature fusion. Auxiliary semantic information can supplement missing details in the image and query, while avoiding excessive noise caused by covariate features (semantic information). By fusing them through a mutual attention mechanism, the image can focus on the query, and the query can also focus on the image.

[0066] Specifically, such as Figure 2 As shown, in this application, bidirectional cross-attention fusion includes attention calculations in the following two directions:

[0067] (1) Query the features of the image of interest

[0068] Specifically, the query focuses on image features and introduces a gating mechanism based on the query vector V. Q and image vector V I The joint representation is dynamically applied to the image label vector V through an MLP layer. T and image description vector V C Assign weights g q1， gq2 Then V T， V C by g q1， g q2 The weighted vector V T’， V C’ The original image vector is passed to the Transformer encoder for feature fusion, resulting in a semantically enhanced image representation V. TI Then, through the multi-head cross-attention mechanism (MHA), the query vector V is made... Q As the query, the enhanced image representation serves as the key and value, and the attention output V on the query side is computed. MHA Q The calculation process is as follows:

[0069]

[0070] ,

[0071]

[0072]

[0073] (2) Image attention query features

[0074] Specifically, the image attention query feature will include the user identity information vector V U As query vector V Q The auxiliary information is fused through a Transformer encoder to obtain an enhanced query representation V. QU Then, through a multi-head cross-attention mechanism, the image vector V is made... I As a query, the enhanced query representation V QU Using ′ as the key and value, the attention output V on the image side is calculated. MHA I The calculation process is as follows:

[0075]

[0076]

[0077]

[0078] Finally, the attention outputs in both directions, V, are... MHA Q and V MHA I The data is concatenated and passed through an MLP layer to obtain the final bidirectional cross-attention fusion representation (CF).A This process can be formally represented as:

[0079]

[0080] Step S6: Multimodal Gated Fusion

[0081] In this application, multimodal gated fusion introduces an adaptive gating mechanism to balance fair learning across all feature types and the interaction between principal features and covariate semantic features, thereby reducing noise during the fusion process. The gating mechanism is applied to CF... E and CF A The outputs are dynamically weighted and combined to obtain a better fused representation CF. EA The calculation formula is as follows:

[0082]

[0083]

[0084] Where g f Here, σ represents the gate weights, σ ​​is the Sigmoid function, ⊙ represents element-wise multiplication, and W... g and b g These are learnable parameters.

[0085] Step S7: Personalized Injection

[0086] In this embodiment, a DeepFM-based module is introduced into the personalized injection process to learn the user ID (U id ) and Image ID (I id The interaction relationships between these elements yield a vector V representing user preferences. id The vector V id V is obtained after projection through an MLP layer. id Then, through residual connections and gating mechanisms, this preference vector is injected into the fusion representation CF in step 6. EA In this process, the personalized bias-corrected representation CF is obtained. EA * The calculation process is as follows:

[0087]

[0088]

[0089]

[0090]

[0091] Where V id'g' represents the result of projecting the user preference vector output by DeepFM onto an MLP. r W is the residual gating weight. p b is the weight of the MLP p This is the bias value for the MLP.

[0092] Step S8: Generate final output

[0093] In this application, the final fusion feature CF obtained in step S7 is used. EA * The query-image cosine similarity score S is directly calculated using the query-related feature representation and the image-related feature representation generated in step S3. QI And concatenate them to obtain the final vector CF EAS * Then, the concatenated final vector CF... EAS * The input is fed into an output layer (fully connected layer + sigmoid activation function), resulting in a probability value between 0 and 1. * This value is the final matching score between query Q and image I, used to rank all candidate images. Its calculation process is as follows:

[0094]

[0095]

[0096]

[0097] Among them, W o These are the weights of the fully connected layer, b o It is the bias value of the fully connected layer.

[0098] Step S9: Model Training

[0099] Specifically, the model is trained using data from the business dataset. After updating the model, the process returns to step S1 until the model converges or reaches the specified number of training iterations.

[0100] Furthermore, during the training process of this application, binary cross-entropy is used as the loss function to perform end-to-end optimization of the entire model. The calculation formula is as follows:

[0101]

[0102] Where N is the number of samples, y i Let i be the true label of the i-th sample (1 for a positive sample, 0 for a negative sample), and Output * Let be the predicted matching probability of the model for the i-th sample.

[0103] The present invention will be further described in detail below with reference to verification examples.

[0104] The dataset used in this validation example is a business dataset, which is an internal dataset built from employee search logs.

[0105] Specifically, in this validation example, the images that are clicked or downloaded are considered positive samples, while the non-positive samples in the top k search results are considered hard negative samples. The dataset is further divided into training set, validation set and test set in a ratio of 8:1:1.

[0106] Specifically, this validation example uses two commonly used retrieval algorithm evaluation metrics. MRR@K measures the position of the first relevant result in the sorted list, calculating the inverse mean of the highest ranking among the top K results, with a value range of (0,1], where a larger value is better. NDCG@K evaluates the relevance quality of the top K results in the sorted list, considering position weights and normalization, with a value range of [0,1], where a value closer to 1 indicates a better ranking.

[0107] In this validation example, Table 1 shows the experimental results of different retrieval models on the business dataset.

[0108] Table 1. Experimental results of different models on the business dataset.

[0109]

[0110] Specifically, the following conclusions can be drawn from Table 1:

[0111] This invention outperforms all benchmark methods across all metrics. Notably, on the particularly important metrics MRR@1 and MRR@5, this invention outperforms the suboptimal method by nearly 6% and 4%, respectively, demonstrating its superior performance in ranking and recommendation tasks. Furthermore, in the application experiments, MRR@1 and NDCG@1 values ​​were identical because, at an evaluation level of K=1, these two metrics primarily measure the binary correlation of top-ranked items.

[0112] In this verification example, Table 2 shows the ablation experiment results of the present invention on a business dataset.

[0113] Table 2 Ablation experiment results on the business dataset

[0114]

[0115] Specifically, the following conclusions can be drawn from Table 2:

[0116] This invention achieved best or near-best performance on most metrics, with the fusion component significantly improving the MRR@1 score by 5%, demonstrating the effectiveness of fusing semantic information and the fusion module. Furthermore, the impact on ID and user characteristics was less pronounced, indicating that this invention can easily adapt to public datasets, many of which lack such user-specific information; other components also contributed to the gradual improvement of the MRR@1 score. In conclusion, these results validate the effectiveness of the components designed in this invention.

[0117] Based on the same concept as the above method embodiments, this application also proposes a personalized multimodal image and text retrieval system based on large language model enhancement, including:

[0118] Query enhancement module: Used to generate enhanced queries through a large language model by utilizing user identity information and preset prompt templates;

[0119] Image enhancement module: Used for enhancing images;

[0120] Multimodal feature encoding module: used to map the text information obtained from the query enhancement and the image enhancement, along with the image information, into a unified d-dimensional feature vector through an encoding layer;

[0121] Cross-encoder fusion module: used to stack the feature vectors into a unified sequence through dimensionality stacking. After the sequence is encoded by Transformer, it is then subjected to mean pooling and MLP projection to obtain the multimodal fusion representation (CF). E ;

[0122] Bidirectional cross-attention fusion module: used to concatenate the outputs of multi-directional attention calculations, and obtain the bidirectional cross-attention fusion representation (CF) through an MLP layer. A ;

[0123] Multimodal gating fusion module: used to control the CF through a gating mechanism E and the CF A Output dynamic weighting to generate optimized fusion representation CF EA ;

[0124] Personalization injection module: used to integrate user preference information into the fusion representation CF via the DeepFM module. EA In the process, a personalized fusion representation CF is obtained. EA * ;

[0125] Output module: used to output the CF EA * The result is concatenated with the calculated cosine similarity score and then input into the output layer to obtain a probability value as the final matching score.

[0126] Training module: Used to iteratively train the model using the business dataset until convergence or a preset number of training rounds are reached.

[0127] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions, and variations can be made to these embodiments, including components, without departing from the principles and spirit of the present invention, and these variations still fall within the protection scope of the present invention.

Claims

1. A personalized multimodal image and text retrieval method based on large language model enhancement, characterized in that, Includes the following steps: Query enhancement: Utilize user identity information and preset prompt templates to generate enhanced queries through a large language model; Image enhancement: Performing enhancement processing on an image to obtain the enhanced semantic information of the image; Multimodal feature encoding: The text and image information obtained from the query enhancement and image enhancement are mapped into a unified d-dimensional feature vector through an encoding layer; Cross-encoder fusion: The feature vectors are stacked dimensionally to form a unified sequence. This sequence is then encoded by Transformer and subjected to mean pooling and MLP projection to obtain the multimodal fusion representation (CF). E ; Bidirectional cross-attention fusion: The outputs of multi-directional attention calculations are concatenated and passed through an MLP layer to obtain the bidirectional cross-attention fusion representation (CF). A ; Multimodal gating fusion: This involves using a gating mechanism to control the CF... E and the CF A Output dynamic weighting to generate optimized fusion representation CF EA ; Personalization injection: User preference information is integrated into the fused representation CF via the DeepFM module. EA In the process, a personalized fusion representation CF is obtained. EA * ; Generate final output: The CF EA * The enhanced query-image cosine similarity score is concatenated with the calculated enhanced query-image cosine similarity score and then input to the output layer to obtain the probability value as the final matching score. Model training: Iteratively train the model using the business dataset until it converges or reaches the preset training rounds.

2. The personalized multimodal image and text retrieval method based on large language model enhancement according to claim 1, characterized in that, The image enhancement includes: Image normalization: standardizing images of different sizes into a uniform format; Image description generation: Convert the image into a string format, concatenate the string with a preset prompt template, input the string into a multimodal model, and output advanced descriptive text; Image tag generation: Extracting micro-level semantic information from images to generate fine-grained tags.

3. The personalized multimodal image and text retrieval method based on large language model enhancement according to claim 2, characterized in that, The image tag generation includes the following components: SD-Tagger: Generates a tag set T through Stable Diffusion reverse reasoning and CLIP cosine similarity calculation. SD ; OD-Tagger: Utilizes YOLO high-confidence object detection to extract salient entities from images and generate a label set T. OD ; IC-Tagger: Generates short descriptions and extracts key phrases using an image captioning model to form a tag set T. IC。 4. The personalized multimodal image and text retrieval method based on large language model enhancement according to claim 3, characterized in that, T SD T OD和 T IC After summarizing the three types of labels, the CLIP model is used to calculate the cosine similarity between the candidate labels and the original images, and the top 5 highest-scoring labels are used to form the final image label set.

5. The personalized multimodal image and text retrieval method based on large language model enhancement according to claim 1, characterized in that, The attention calculation for the bidirectional cross-attention fusion includes: The query focuses on image features: Image label vectors and image description vectors are dynamically weighted using a gating mechanism, fused with the original image features, and then the query-side attention output V is generated through a Transformer encoder and a multi-head cross-attention mechanism. MHA Q ; Image-based attention query features: After augmenting the query vector with user identity information, an image-side attention output V is generated through a multi-head cross-attention mechanism. MHA I .

6. The personalized multimodal image and text retrieval method based on large language model enhancement according to claim 5, characterized in that, Bidirectional Cross-Attention Fusion Representation (CF) A It is through the V mentioned above MHA Q and the V MHA I It is obtained by splicing and then processing with MLP.

7. The personalized multimodal image and text retrieval method based on large language model enhancement according to claim 5, characterized in that, The gating mechanism enables fair learning and semantic interaction among features, reducing fusion noise.

8. The personalized multimodal image and text retrieval method based on large language model enhancement according to claim 1, characterized in that, During the model training process, end-to-end binary cross-entropy loss optimization is used.

9. The personalized multimodal image and text retrieval method based on large language model enhancement according to claim 1, characterized in that, The output layer consists of a fully connected layer and a Sigmoid activation function.

10. A personalized multimodal image and text retrieval system based on large language model enhancement, characterized in that, include: Query enhancement module: Used to generate enhanced queries through a large language model by utilizing user identity information and preset prompt templates; Image enhancement module: Used for enhancing images; Multimodal feature encoding module: used to map the text information and image information obtained from the query enhancement and image enhancement into a unified d-dimensional feature vector through the encoding layer; Cross-encoder fusion module: used to stack the feature vectors into a unified sequence through dimensionality stacking. After the sequence is encoded by Transformer, it is then subjected to mean pooling and MLP projection to obtain the multimodal fusion representation (CF). E ; Bidirectional cross-attention fusion module: used to concatenate the outputs of multi-directional attention calculations, and obtain the bidirectional cross-attention fusion representation (CF) through an MLP layer. A ; Multimodal gating fusion module: used to perform adaptive gating on the CF E and the CF A Output dynamic weighting to generate optimized fusion representation CF EA ; Personalization injection module: used to integrate user preference information into the fusion representation CF via the DeepFM module. EA In the process, a personalized fusion representation CF is obtained. EA * ; Output module: used to output the CF EA * The enhanced query-image cosine similarity score is concatenated with the input-output layer to obtain a probability value as the final matching score. Training module: Used to iteratively train the model using the business dataset until convergence or a preset number of training rounds are reached.

Images