An intelligent recall method based on a multi-modal large model

Abstract

This invention discloses an intelligent recall method based on a multimodal large model, comprising: acquiring a set of candidate materials, pre-generating a static base vector for each candidate material; modifying the static base vector to generate a dynamic decision vector by fusing dynamic contextual information and prediction results of historical semantic changes of the materials; sorting the candidate material set based on the similarity between the content intent vector and the dynamic decision vector and outputting the recall results; separating content and contextual features through a dual-tower decoupled encoder architecture, using a cross-attention mechanism and dynamic weight fusion to generate contextualized vector representations, and introducing a time-series prediction model to capture the semantic evolution trend of materials, thereby achieving accurate, personalized and forward-looking recall effects, improving the fit between the recall results and the user's real-time scenario and the long-term satisfaction of the recommended content.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to an intelligent recall method based on a multimodal large model. Background Technology

[0002] Currently, the core task of information retrieval and recommendation systems based on large models is to accurately match and recall content highly relevant to the user's immediate intent from massive amounts of data. To achieve this goal, existing technologies have evolved from traditional sparse representations based on keyword matching to a vectorized recall stage based on deep learning. In this stage, the model maps user queries and materials to the same high-dimensional semantic space and uses the distance or similarity between vectors to measure relevance. To further improve the accuracy of model recall, the industry generally optimizes the user intent representation on the query side and constructs more refined user profiles by deeply mining and fusing users' historical behavior data. Among them, CN114329199A discloses a method and apparatus for material recall. This scheme mainly sets user profiles with long, medium, and short time spans and performs gating fusion according to time sequence to generate more accurate search intent vectors that reflect users' preferences at different time scales. This scheme's design of refined modeling on the query side improves the understanding of users' historical preferences to a certain extent.

[0003] However, the aforementioned technical approach, which focuses on optimizing query-side representation, still faces several technical bottlenecks when dealing with increasingly complex and dynamic user scenarios. First, existing methods essentially place all the pressure of personalization on the query side. Once the semantic vectors of the materials are generated offline, their representation is static and context-independent. This means that the vector representation of the same material remains rigid and unchanging when facing different query scenarios from different users, unable to dynamically "respond" and "adapt" to the immediate context of the query, easily leading to discrepancies between the retrieved results and the user's actual scenario. Second, existing methods are essentially retrospective summaries of users' historical behavior, lacking the ability to proactively model the semantic evolution trends of the materials themselves. The connotation and popularity of a material (such as an emerging concept or a trending event) evolve over time. Existing methods cannot capture and predict this "semantic drift," making it difficult to proactively discover and recommend content with future potential for users.

[0004] Therefore, there is an urgent need for a design that can break through the static constraints of material representation and incorporate context awareness and future trend prediction capabilities during the model recall stage, so as to achieve a personalized and forward-looking recall method. Summary of the Invention

[0005] The purpose of this section is to outline some aspects of embodiments of the present invention and to briefly describe some preferred embodiments. Simplifications or omissions may be made in this section, as well as in the abstract and title of this application, to avoid obscuring the purpose of these documents; however, such simplifications or omissions should not be construed as limiting the scope of the invention.

[0006] In view of the aforementioned existing problems, this invention is proposed. Therefore, this invention provides an intelligent recall method based on a multimodal large model to solve the problems mentioned in the background art.

[0007] To address the aforementioned technical problems, this invention provides the following technical solution: an intelligent recall method based on a multimodal large model, comprising:

[0008] Acquire the user's multimodal query input, encode the multimodal query input, and generate a content intent vector;

[0009] Obtain a set of candidate materials, in which each candidate material corresponds to a pre-generated static basic vector;

[0010] For each candidate material in the candidate material set, a dynamic decision vector is generated by modifying the static basic vector corresponding to the candidate material.

[0011] Based on the similarity between the content intent vector and the dynamic decision vector, the candidate material set is sorted and the recall result is output.

[0012] As a preferred embodiment of the intelligent recall method based on a multimodal large model described in this invention, the step of obtaining the user's multimodal query input includes dynamic context information and prediction results of historical semantic changes of candidate materials.

[0013] As a preferred embodiment of the intelligent recall method based on a multimodal large model described in this invention, the dynamic context information includes at least one of the following: time information of the query, geographical location information, device information, network environment information, weather information, and the user's historical behavior sequence in the current session.

[0014] As a preferred embodiment of the intelligent recall method based on a multimodal large model described in this invention, the step of modifying the static foundation vector includes:

[0015] Based on the dynamic context information and the static base vector, a context adjustment vector is calculated and generated;

[0016] Based on the prediction results of the historical semantic changes of the candidate materials, a predictive vector is obtained;

[0017] The static base vector, the context adjustment vector, and the predictive vector are fused to obtain the dynamic decision vector.

[0018] As a preferred embodiment of the intelligent recall method based on a multimodal large model described in this invention, the step of obtaining a predictive vector specifically comprises:

[0019] When offline, a semantic vector time series is constructed for candidate materials, and a time series prediction model is trained using the time series.

[0020] During online recall, the time-series prediction model is used to calculate the future semantic vector of the candidate material based on the latest historical semantic vector sequence.

[0021] The predictive vector is obtained by performing a difference operation between the future semantic vector and the current static basis vector of the candidate material.

[0022] As a preferred embodiment of the intelligent recall method based on a multimodal large model described in this invention, the method further includes, before performing the static basis vector modification step:

[0023] Obtain dynamic context information associated with the multimodal query input;

[0024] The dynamic context information associated with the multimodal query input is encoded to generate a dynamic context vector;

[0025] The context adjustment vector is calculated based on the dynamic context vector and the static base vector.

[0026] As a preferred embodiment of the intelligent recall method based on a multimodal large model described in this invention, it further includes:

[0027] By constructing a dual-tower decoupled encoder architecture, multimodal query input and dynamic context information are encoded;

[0028] In the dual-tower decoupled encoder architecture, the content tower is used to generate the content intent vector, and the context tower is used to generate the dynamic context vector.

[0029] As a preferred embodiment of the intelligent recall method based on a multimodal large model described in this invention, the calculation steps of the context adjustment vector include:

[0030] A cross-attention mechanism is used to apply the dynamic context vector to the static base vector to generate a context transformation vector.

[0031] A gating value is calculated and generated based on the dynamic context vector;

[0032] The context transformation vector is adjusted using the gating value to obtain the context adjustment vector.

[0033] As a preferred embodiment of the intelligent recall method based on a multimodal large model according to the present invention, the step of fusing the static basic vector, the context adjustment vector, and the predictive vector includes:

[0034] Based on the content intent vector and dynamic context vector, a weight generation network is used to calculate and generate a first fusion weight and a second fusion weight.

[0035] The context adjustment vector is weighted by the first fusion weight, and the predictive vector is weighted by the second fusion weight;

[0036] The dynamic decision vector is obtained by summing the weighted context adjustment vector and the predictive vector with the static basis vector.

[0037] As a preferred embodiment of the intelligent recall method based on a multimodal large model described in this invention, the candidate material set is sorted by calculating the cosine similarity or dot product between the content intent vector and the dynamic decision vector as the sorting basis.

[0038] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0039] 1. By generating context adjustment vectors in real time based on the dynamic context information of user queries during the recall phase, and using them to modify the pre-generated static base vectors of materials, the vector representation of materials is transformed from static context-independent to dynamic context-aware. This allows the same material to present differentiated semantic expressions when facing different users and different scenarios, increasing the fit between the model recall results and the user's real-time and real-world scenarios, while improving the accuracy of the recall.

[0040] 2. At the same time, a predictive mechanism for the historical semantic changes of materials is introduced. That is, a forward-looking predictive vector is generated through a time-series prediction model and integrated into the final dynamic decision vector. This mechanism can effectively capture and respond to the "semantic drift" phenomenon of materials caused by factors such as hot spots and seasonality. It can not only recall currently relevant content, but also proactively discover and recommend materials with future potential for users, thereby improving the novelty of model recommendations and long-term user satisfaction.

[0041] 3. In addition, by generating dynamic decision vectors in real time on the material side, a dual-tower decoupled encoder architecture was constructed to dynamically match the query end and the material end in collaboration. This enabled more refined personalized matching, making the entire recall process more adaptable and robust when facing complex and ever-changing user intentions and scenarios. Attached Figure Description

[0042] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0043] Figure 1 This is a flowchart illustrating the overall process of an intelligent recall method based on a multimodal large model, as described in one embodiment of the present invention. Detailed Implementation

[0044] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0045] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0046] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0047] This invention is described in detail with reference to the schematic diagrams. When detailing the embodiments of this invention, for ease of explanation, the cross-sectional views illustrating the device structure may be partially enlarged, not adhering to the usual scale. Furthermore, the schematic diagrams are merely examples and should not be construed as limiting the scope of protection of this invention. In actual fabrication, the three-dimensional spatial dimensions of length, width, and depth should be included.

[0048] Furthermore, in the description of this invention, it should be noted that the terms "upper," "lower," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. These terms are used solely for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. In addition, the terms "first," "second," or "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0049] Unless otherwise explicitly specified and limited, the terms "installation," "connection," and "joining" in this invention should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; similarly, they can refer to mechanical connections, electrical connections, or direct connections, or indirect connections through an intermediate medium, or internal connections between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0050] Example 1

[0051] Reference Figure 1 This is the first embodiment of the present invention, which provides an intelligent recall method based on a multimodal large model, including:

[0052] S1. Obtain the user's multimodal query input, encode the multimodal query input, and generate a content intent vector;

[0053] It should be noted that this step is the starting point of the recall process. Its core task is to parse and transform the various query requests initiated by users in specific scenarios into two independent (decoupled) and semantically rich high-dimensional vectors through a neural network model. The high-dimensional vector consists of two parts: one is a content intent vector representing the user's explicit content needs, and the other is a dynamic context vector representing the user's implicit query scenario.

[0054] Furthermore, all relevant data streams generated when users trigger recall actions are collected in real time through the model front-end interface, and then normalized into structured data.

[0055] Specifically, the relevant data stream includes two types of information: multimodal query input and dynamic context information;

[0056] Specifically, multimodal query input aims to provide users with content that proactively expresses their core needs. This content can be a string entered by the user in the search box during text input, an image file or URL uploaded by the user during image input, or a short audio (such as a humming melody recognition) or short video clip received during audio / video input. In addition to the above, it also includes query content that combines text and images (such as a user uploading a photo of their living room with the comment "Recommend a floor lamp that matches my style").

[0057] Specifically, the model encapsulates the content from different sources obtained from the above multimodal query input in a unified manner and adds modal tags;

[0058] Specifically, dynamic context information aims to reflect the environmental information of the user's real-time scenario and potential state. It includes time information (recording server timestamps accurate to the second and deriving periodic features such as "hour" (0~24), "week" (Monday to Sunday), "whether it is a holiday", etc.), geographical location information (obtaining the user's latitude and longitude through GPS or IP address resolution and further mapping it to city ID, business district ID or scenario type (such as "home", "company", "transportation hub" etc.)), device and network information (device model, operating system, screen resolution, network type (such as 5G / Wi-Fi / 4G)), and in-session behavior sequence (the user's interaction behavior before query input in the current application session, such as the sequence of the 10 most recently clicked material IDs or the sequence of the 3 most recently searched keywords).

[0059] In addition, before the aforementioned data streams are fed into the model, they need to be preprocessed to convert them into numerical tensors that the model can accept.

[0060] Specifically, for data with modal labels, text data is segmented, converted into token ID sequences using a pre-trained model's vocabulary, and marked with special tags; image data is decoded, scaled to a fixed size (e.g., 224×224 pixels), and normalized, mapping pixel values ​​from the range [0, 255] to the range [-1, 1] or [0, 1]; for combined text and image data, text data is processed first, followed by image data.

[0061] Specifically, for dynamic context information, for categorical features such as city ID and equipment model, they are converted into integer indices through the constructed ID dictionary; for numerical features such as hour and latitude / longitude, normalization or binning is performed to eliminate differences in units; for sequential features, behavioral sequences (such as material ID sequences) are processed to a fixed length by truncating or padding to make their length consistent.

[0062] Furthermore, in order to accurately distinguish between what the user "wants" (content intent) and "in what scenario" (context state), the relevant data stream after the above processing, i.e. structured data, is sent to a constructed dual-tower decoupled encoder architecture for encoding operations.

[0063] It should be noted that the "decoupling" idea of ​​this decoder is to isolate the volatile contextual information from the relatively stable content intent at the model structure level, so as to learn pure and generalizable data features respectively.

[0064] Specifically, the dual-tower decoupled encoder architecture includes a content tower and a context tower, which are structurally independent and do not share parameters.

[0065] It should be noted that the content tower is responsible for processing the user's multimodal query input, with the goal of generating a content intent vector that does not change with the context and reflects the user's core needs; the context tower is responsible for encoding all dynamic context information in parallel, with the goal of generating a dynamic context vector that is unrelated to the query content but can characterize the user's current "state" and "scenario".

[0066] Specifically, the content tower's architecture consists of a single-modal encoder and a multi-modal fusion layer;

[0067] Specifically, in one feasible implementation of the present invention, in the unimodal encoder, the text modality branch within the content tower adopts a 12-layer Transformer encoder (such as BERT-base), and the image modality branch within the content tower adopts a visual Transformer (such as ViT-B / 16); in the multimodal fusion layer, when the input is multimodal, each unimodal encoder first outputs the respective features of the text modality branch and the image modality branch, and then, through a multimodal fusion module (such as using a Co-attention network based on a cross-attention mechanism), the combined features of text and image are interacted, allowing the semantics of the text features to guide the extraction of visual features, and vice versa. Finally, the combined features are passed through a linear projection layer to output a content intent vector with dimension d (such as d=768);

[0068] Specifically, the architecture of the context tower consists of a feature embedding layer, a sequence modeling module, and a feature interaction and aggregation layer;

[0069] Specifically, in one feasible process of the present invention, in the feature embedding layer, all preprocessed categorical and sequential feature IDs are queried through their respective embedding matrices and mapped into low-dimensional dense embedding vectors. Then, through the sequence modeling module, for the user's session behavior sequence (such as the embedded sequence of clicked material IDs), a gated recurrent unit network is used to capture the temporal dependencies and interest evolution patterns in the sequence, and output a sequence vector aggregating short-term interests. Through the feature interaction and aggregation layer, the embedding vectors of all context features (time, geography, device, network, sequence aggregation vector, etc.) are concatenated and input into a deep feedforward network. This feedforward network learns the complex interaction relationships between different context features through multi-layer nonlinear transformation characteristics, and finally outputs a dynamic context vector of the same d-dimensionality.

[0070] It should be noted that the dual-tower decoupled encoder architecture described above can separate the encoding process of content from the encoding process of context, thereby enabling the model to independently learn two different data features, enhancing the model's generalization ability and interpretability.

[0071] S2. Obtain the candidate material set. In the set, each candidate material corresponds to a pre-generated static basic vector.

[0072] It should be noted that, considering that online models usually need to have low latency requirements, it is impossible to perform dynamic vector generation operations for every material in the full material library. Therefore, the goal of this step is to quickly recall a set of candidate materials that are initially related to the user's content intent and have a controllable scale by establishing a pre-screening mechanism, and to obtain the semantic representation of these materials, i.e., static basic vectors.

[0073] Specifically, the pre-screening mechanism can be divided into an offline generation and indexing stage of static basic vectors, and an online triggering and acquisition stage of candidate material sets;

[0074] Specifically, in the offline generation and indexing stage of static base vectors, we use a large-scale multimodal material encoder to encode each material in the material library. The material encoder is structurally similar to the unimodal encoder in the content tower of step S1, but its training task is specifically designed to understand the multimodal content of the material itself. For a material, the input of the material encoder is all its available meta-information, such as: title, description text, category, key attribute tags, main image, detail image, video introduction, etc. By fusing and learning this meta-information, a fixed-dimensional vector (e.g., the same d-dimensional dimension as the content intent vector) is output for each material. This vector is the static base vector.

[0075] It needs to be explained that the so-called "static" means that once the vector is generated, it remains fixed until the next offline update. The static base vector aims to capture the most essential and unchanging intrinsic semantics of the material itself. For example, the static base vector of a "red wool turtleneck sweater" should stably encode the semantics of objective attributes such as "red", "wool", "turtle", and "sweater", regardless of the current season, temperature, or the location of the user querying the vector.

[0076] Furthermore, to achieve online millisecond-level retrieval, the static base vectors of all materials are imported into an Approximate Nearest Neighbor (ANN) index system after generation. This index system can be a vector retrieval engine built on the Facebook Faiss library or the Google ScaNN library. In this index system, the unique ID of the material forms a key-value pair with its static base vector. This index system can support fast similarity search of massive vectors through techniques such as quantization and clustering.

[0077] Specifically, in the online triggering and acquisition phase of the candidate material set, the content intent vector generated by the content tower in step S1 is used as the query vector. Subsequently, a query request is sent to the aforementioned near nearest neighbor index system. The retrieval engine of this index system will calculate and return the Top-K materials that are closest to the query vector in the vector space (i.e., the highest cosine similarity or the smallest Euclidean distance) from tens of thousands of material vectors with extremely high efficiency. Here, K is a preset hyperparameter, such as K=1000. The set of IDs composed of the Top-K materials is the candidate material set. After obtaining the set of IDs of the Top-K candidate materials, the model will retrieve the static base vectors corresponding to these K materials, which were generated in the offline phase, from a key-value storage system (such as Redis, which pre-stores the mapping from the unique ID of the material to its static base vector) through a batch query.

[0078] It should be noted that after the above pre-screening mechanism is completed, the output is a candidate material set of size K, and for each candidate material in the set, its unique ID and the corresponding static basic vector representing its inherent semantics have been obtained.

[0079] S3. For each candidate material in the candidate material set, generate a dynamic decision vector by modifying the static basic vector corresponding to the candidate material.

[0080] It should be noted that the goal of this step is to no longer directly use its static base vector for matching each candidate material obtained in step S2. Instead, based on the user's current query context and the material's own potential semantic evolution trend, the static base vector is "corrected" in real time and in a personalized manner, thereby generating a brand-new, contextualized dynamic decision vector, so that the original candidate material can adapt to each unique query request.

[0081] Specifically, the process of generating dynamic decision vectors is a fusion process, which involves fusing the static base vector of each candidate material, a context-adjusted vector "tailored" to that static base vector, and a forward-looking predictive vector.

[0082] Furthermore, the calculation of the context adjustment vector relies on the dynamic context vector and the static base vector of the current material to be processed, aiming to enable the generated dynamic decision vector to "sense" and "respond" to the user's current query scenario.

[0083] Specifically, the generation of the context adjustment vector employs a gated cross-attention mechanism, which controls the direction and intensity of the adjustment of the material's basic semantics by contextual information. The steps are as follows:

[0084] Calculating which semantic dimensions the material's static foundation vector should focus on and adjust from the current context's "perspective," by using the dynamic context vector as the query and the static foundation vector as both key and value, and inputting it into a standard cross-attention module, can be mathematically described as follows: through three independent linear transformation matrices... The input static basis vectors are projected separately, and then the attention score is calculated through the Softmax operation. The value vectors are then weighted and summed using the attention score to obtain the context transformation vector.

[0085] It should be explained that the context transformation vector is essentially a reweighted representation of the static basis vector, and its weight distribution is determined by the dynamic context vector. Therefore, the context transformation vector is equivalent to implying the information of "which semantic components in the static basis vector are more important in the current scenario".

[0086] The projection process can be expressed mathematically as follows:

[0087]

[0088]

[0089]

[0090] Where Q represents the query vector, K represents the key vector, and V represents the value vector. For querying the matrix, The key matrix, It is a value matrix; Represented as a dynamic context vector, Represented as static fundamental vectors;

[0091] The formula for calculating the attention score is expressed as follows:

[0092]

[0093] in, Let T represent the attention score, and let T represent the transpose matrix. It is the dimension of the key vector;

[0094] The context transformation vector Expressed as a formula:

[0095]

[0096] A gating value is generated based on the dynamic context vector to control the overall strength of context adjustment. If the current context is not strongly related to the material, the adjustment should be weaker, and vice versa. The dynamic context vector is input into a small neural network (which can be a single-layer or two-layer MLP) and its output is compressed to the (0, 1) interval using a sigmoid activation function. Its mathematical expression can be described as follows:

[0097]

[0098] Where g represents the gating value, which can be understood as the "permission" or "confidence" of the context in adjusting the semantics of the material;

[0099] The context adjustment vector is obtained by adjusting the context transformation vector using the gating value. The mathematical expression for this is achieved by multiplying the generated gate value element-wise with the generated context transformation vector:

[0100]

[0101] It should be noted that through the above steps, the final generated context adjustment vector not only reflects the focus of the context in terms of direction (determined by the context transformation vector), but also reflects the appropriate intensity of the context adjustment in terms of magnitude (controlled by the gating value), thus realizing fine and robust scenario-based modification of the material vector.

[0102] Furthermore, the acquisition of predictive vectors aims to enable the generated dynamic decision vectors to have "foresight" capabilities in order to capture the potential semantic evolution trends of materials. This process is divided into two stages: offline training and online prediction.

[0103] Specifically, for the offline training phase, the semantic vector time series of materials that meet the preset conditions in the model are first constructed offline. The preset conditions aim to select materials with obvious semantic evolution potential or commercial value. Static base vectors of these key materials are regenerated periodically (e.g., daily or weekly) using the latest material encoder. The static base vectors are arranged in chronological order to form a time series. Using the formed time series data, a time series prediction model is trained. The time series prediction model adopts a Long Short-Term Memory Network (LSTM) or a Transformer model with a time attention mechanism. Regardless of which time series prediction model is used, the training objective of the model is to accurately predict the static base vector of a material at the next time point, given the vector sequence of the material in the past period.

[0104] Specifically, in one implementation of the present invention, the preset condition can be specifically defined as a combination of one or more of the following: (1) materials that rank in the top 10% of cumulative exposure or clicks in the past 30 days; (2) materials newly listed in the past 7 days; (3) materials belonging to the core categories designated by the operations department; in this way, computing resources can be concentrated to serve the material group that most needs trend prediction.

[0105] Specifically, for the online prediction stage (i.e., online recall), for each material in the candidate material set in step S2, its latest historical semantic vector sequence (e.g., daily vectors for the past 7 days) is first quickly retrieved from storage, and then the vector sequence is input into the offline-trained time series prediction model to calculate the future semantic vector of the material in the next time step in real time.

[0106] Furthermore, to obtain a vector representing a "trend of change" rather than an absolute position, we performed a difference operation between the future semantic vector and the current static base vector of the material, the mathematical expression of which is:

[0107]

[0108] in, denoted as a predictive vector, which captures the most likely semantic drift direction and magnitude of the material from the present to the future; c represents the material's current static base vector;

[0109] Furthermore, after calculating the context adjustment vector and the predictive vector, they are fused with the original static basis vector. For this fusion, the present invention employs a dynamic weight fusion mechanism.

[0110] Specifically, a lightweight weight generation network is constructed (if this generation network type is the same as the small neural network mentioned above, it can be called to simplify the processing). The input of this generation network is the concatenation of the user's content intent vector and dynamic context vector. The output of this generation network is two scalar values, which are processed by a Softmax operation or a Sigmoid function to obtain the first fused weights. Second fusion weight :

[0111]

[0112] in, Represented as a weight generation network, This is represented as a process processing function (such as the Softmax operation or Sigmoid function mentioned above). Represented as a content intent vector;

[0113] It should be noted that, due to the importance of context adjustment and trend prediction, the approach actually differs for different user intentions and scenarios. For example, when the user intention is clear and context-specific (such as "restaurants nearby that are open tonight"), then it is necessary to amplify the approach. When the user's intent is vague and the context is weak (such as "Let's see what's trending lately"), then it is necessary to amplify the message. ;

[0114] Furthermore, by using a weighted summation method, the static basic vector, context-adjusted vector, and predictive vector are merged into a dynamic decision vector. Its mathematical expression is:

[0115]

[0116] S4. Based on the similarity between the content intent vector and the dynamic decision vector, sort the candidate material set and output the recall results;

[0117] It should be noted that this step is the end point of the recall process. Its purpose is to quantify and rank the relevance scores of the vector representations constructed in the previous steps, namely the content intent vector representing the user's core needs and the dynamic decision vector generated for each candidate material.

[0118] Furthermore, in order to measure the similarity between the content intent vector and each dynamic decision vector, computationally efficient and widely validated metrics in the field of vector recall can be used, such as cosine similarity or dot product.

[0119] Specifically, in one feasible implementation of the present invention, cosine similarity is used to calculate the similarity score for the i-th material (i from 1 to K) in the candidate material set. The calculation formula is:

[0120]

[0121] in, Let be the dynamic decision vector corresponding to the i-th material, and "·" represents the vector dot product. " represents the L2 norm of a vector (i.e., the magnitude of the vector);

[0122] It should be noted that the range of cosine similarity is [-1, 1]. The closer the value is to 1, the more consistent the two vectors are in direction. That is, the more the user's intention matches the dynamic semantic representation of the material in the current scenario. Since the vectors are usually normalized during the training phase to make their magnitude 1, the cosine similarity can be simplified to directly calculating the dot product when calculating online, thereby greatly improving the computational efficiency.

[0123] Specifically, in one feasible implementation of the present invention, the dot product is used, and the similarity score of the i-th material (i from 1 to K) in the candidate material set is calculated using the following formula:

[0124]

[0125] It should be noted that after vectors are normalized, their dot product is equivalent to cosine similarity.

[0126] Furthermore, after calculating the similarity scores of all K candidate materials, a score list is obtained. The model then sorts all materials in the candidate material set (identified by their material IDs) in descending order based on this score list, with the material with the highest score at the top, indicating that it best matches the user's immediate and dynamic intent. At the same time, based on the current business needs, the model extracts the Top-N materials (e.g., N=100) from the sorted list to form the final recall result list. This list contains the IDs of N materials and may optionally include their corresponding similarity scores. This recall result list will be passed to other downstream modules of the model for further processing or presented directly to the user.

[0127] It should be noted that, since the ranking is based on dynamic decision vectors rather than static base vectors, the final recall results output in this step fully consider the immediate context of the user query (reflected by context adjustment vectors) and the semantic evolution trend of the material itself (reflected by predictive vectors). This makes the recall results not only meet the user's explicitly expressed needs, but also intelligently adapt to the context in which they are located, thereby significantly improving the accuracy and context fit of the recall.

[0128] Furthermore, in order to enable all modules containing trainable parameters, such as the dual-tower decoupled encoder architecture, gated cross-attention mechanism, and weight generation network in the present invention, to work together and achieve the expected recall effect, the present invention adopts an end-to-end joint training process to learn these parameters. This training process is based on building training samples from large-scale user historical behavior logs and drives model optimization through a loss function.

[0129] Specifically, the training samples are derived from the system's historical user interaction logs. Each training sample is constructed as a tuple, which can be represented as: <(multimodal query input, dynamic context information), positive sample material ID, {set of negative sample material IDs}>;

[0130] Specifically, the query-side information (multimodal query input, dynamic context information) is directly taken from the real record of a user's successful query behavior with positive feedback (such as clicks, favorites, purchases);

[0131] Specifically, for a positive sample material ID, it is the material for which the user generated a positive response after the query;

[0132] Specifically, for the set of negative sample material IDs, that is, the model learns to distinguish between relevant and irrelevant materials, the sampling strategy for negative samples is crucial to the model's performance.

[0133] Preferably, the sampling strategy for the negative sample adopts the In-batch Negatives strategy, that is, in a training batch, for a given query input, its corresponding positive sample material is unique, while the positive sample materials corresponding to all other query inputs in the batch are regarded as negative samples of the given query input. The advantage of adopting this strategy is that it is computationally efficient and can provide high-quality difficult negative samples while forcing the model to learn a discriminative vector representation.

[0134] It should be noted that the end-to-end training of this invention uses a contrastive loss function with a temperature coefficient. Its optimization objective is to maximize the similarity between the "content intent vector" and the "dynamic decision vector of positive sample material" for each training sample, while minimizing the similarity between the "content intent vector" and the "dynamic decision vector of all negative sample materials".

[0135] Specifically, for the m-th training sample in a batch, its loss is calculated through the complete process of steps S1 to S3 of the present invention to obtain the content intent vector on the query side, as well as the dynamic decision vector of the corresponding positive sample material and the set of dynamic decision vectors of all M negative sample materials in the batch.

[0136] Specifically, the mathematical representation of this loss is as follows:

[0137]

[0138] in, This represents the loss value calculated for the m-th training sample (i.e., the m-th query) in the training batch; This vector represents the positive sample material matched by the m-th query. Here, "positive sample" refers to the material that the user actually clicked, purchased, or favorited after initiating the query, based on real user behavior logs. It's important to note that this vector is not the static base vector of that material, but rather a dynamically generated decision vector in real time. Similarly, The vector represents the j-th negative sample material that does not match the m-th query. Under the In-batch Negatives strategy described above, this negative sample refers to the positive sample material corresponding to other queries within the batch. sim represents the similarity calculation function, i.e., the cosine similarity or dot product in the aforementioned scheme. exp is the exponential function, which maps the similarity score to the positive domain and amplifies the difference between the scores. This is the standard step for calculating the probability distribution under the Softmax operation. This is represented as a temperature hyperparameter, which scales the similarity score before calculating the Softmax operation. Its main function is to control the model's focus on difficult negative samples (i.e., negative samples that are similar to the query vector). When the value is small (0.01~0.1), it amplifies the differences in similarity scores, making the distribution of the Softmax output "sharp." The model will focus on distinguishing the most difficult-to-distinguish negative samples. When the value is large (close to 1.0), the distribution of the Softmax output becomes "smooth," and the model's penalty for all negative samples tends to be consistent; in the scheme of this invention, It is set to a small value (such as 0.05) to improve the model's learning performance;

[0139] Specifically, during end-to-end training, the model randomly draws batches of data from the constructed training sample set. For each batch, a complete forward propagation is performed to compute all the required vectors and calculate the loss of the training samples in the batch. Then, the gradient of the loss function with respect to all trainable parameters of the model is calculated through the backpropagation algorithm, and the gradient of these trainable parameters is updated using the optimizer (Adam optimizer, with a learning rate of 1e-4). This process is iterated repeatedly until the performance of the model on the validation set converges.

[0140] It should be noted that, through this end-to-end training method, all trainable parameter modules in this invention can be jointly optimized to generate dynamic decision vectors that can accurately distinguish between relevant and irrelevant materials.

[0141] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of this application can be implemented using various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0142] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0143] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0144] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0145] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0146] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. An intelligent recall method based on a multimodal large model, characterized in that, include: Acquire the user's multimodal query input, encode the multimodal query input, and generate a content intent vector; Obtain a set of candidate materials, in which each candidate material corresponds to a pre-generated static basic vector; For each candidate material in the candidate material set, a dynamic decision vector is generated by modifying the static basic vector corresponding to the candidate material. The steps for modifying the static fundamental vectors include: Based on the dynamic context information and the static base vector, a context adjustment vector is calculated and generated; Obtain dynamic context information associated with the multimodal query input. Based on the prediction results of historical semantic changes of candidate materials, a predictive vector is obtained; The static base vector, the context adjustment vector, and the predictive vector are fused to obtain the dynamic decision vector; The specific steps for obtaining a predictive vector are as follows: When offline, a semantic vector time series is constructed for candidate materials, and a time series prediction model is trained using the time series. During online recall, the time-series prediction model is used to calculate the future semantic vector of the candidate material based on the latest historical semantic vector sequence. The predictive vector is obtained by performing a difference operation between the future semantic vector and the current static basis vector of the candidate material; Before performing the modification steps of the static fundamental vectors, the following are also included: Obtain dynamic context information associated with the multimodal query input; The dynamic context information associated with the multimodal query input is encoded to generate a dynamic context vector; The calculation of the context adjustment vector is based on the dynamic context vector and the static base vector. The calculation steps for the context adjustment vector include: A cross-attention mechanism is used to apply the dynamic context vector to the static base vector to generate a context transformation vector. A gating value is calculated and generated based on the dynamic context vector; The context transformation vector is adjusted using the gating value to obtain the context adjustment vector; The step of fusing the static base vector, the context-adjusted vector, and the predictive vector includes: Based on the content intent vector and dynamic context vector, a weight generation network is used to calculate and generate a first fusion weight and a second fusion weight. The context adjustment vector is weighted by the first fusion weight, and the predictive vector is weighted by the second fusion weight; The dynamic decision vector is obtained by summing the weighted context adjustment vector and the predictive vector with the static base vector. Based on the similarity between the content intent vector and the dynamic decision vector, the candidate material set is sorted and the recall result is output.

2. The intelligent recall method based on a multimodal large model as described in claim 1, characterized in that, The multimodal query input obtained from the user includes dynamic context information and prediction results of historical semantic changes of candidate materials.

3. The intelligent recall method based on a multimodal large model as described in claim 2, characterized in that, The dynamic context information includes at least one of the following: time information of the query, geographical location information, device information, network environment information, weather information, and the user's historical behavior sequence in the current session.

4. The intelligent recall method based on a multimodal large model as described in claim 1, characterized in that, Also includes: By constructing a dual-tower decoupled encoder architecture, multimodal query input and dynamic context information are encoded; In the dual-tower decoupled encoder architecture, the content tower is used to generate the content intent vector, and the context tower is used to generate the dynamic context vector.

5. The intelligent recall method based on a multimodal large model as described in claim 1, characterized in that, The candidate material set is sorted by calculating the cosine similarity or dot product between the content intent vector and the dynamic decision vector as the sorting criterion.

Images