Content query method and content query model training method

Abstract

Embodiments of the present specification provide a content query method and a content query model training method, wherein the content query method comprises: obtaining target query data and to-be-queried content; inputting the to-be-queried content into a content encoding unit in a content query model to obtain a content vector, and inputting the target query data into a query encoding unit in the content query model to obtain a query vector, wherein the content query model is trained based on a sample query vector of sample query data and a sample content vector of sample query content, the sample query content comprises a plurality of sample sub-contents, the sample content vector is determined based on the sample sub-contents and weights corresponding to the sample sub-contents, and the weights are used to reflect the importance of the corresponding sample sub-contents in the sample query content; and generating a content query result according to the content vector and the query vector. The content query model is trained based on the weighted sample content vector, thereby improving the representation capability of the content query model and the accuracy of the query result.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of artificial intelligence technology, and in particular to content query methods and content query model training methods. Background Technology

[0002] With the explosive growth of internet content, efficient semantic-based content retrieval has become a core component of applications such as search engines, recommendation systems, and intelligent question answering. Traditional keyword matching methods struggle to capture the deep semantic relationships between user queries and content, prompting researchers to widely adopt the dual-tower model based on dense vectors for semantic retrieval.

[0003] Currently, the dual-tower model, when representing the query content, fails to highlight the core semantics of the obtained content vector, contains local noise, and limits the discrimination accuracy of the representation, resulting in poor accuracy of content query results. Therefore, a more accurate content query solution is urgently needed. Summary of the Invention

[0004] In view of this, embodiments of this specification provide a content query method. One or more embodiments of this specification also relate to a content query model training method, a content query device, a content query model training device, a computing device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.

[0005] According to a first aspect of the embodiments of this specification, a content query method is provided, including: Retrieve the target query data and the content to be queried; The content to be queried is input into the content encoding unit in the content query model to obtain the content vector, and the target query data is input into the query encoding unit in the content query model to obtain the query vector. The content query model is trained based on the sample query vector of the sample query data and the sample content vector of the sample query content. The sample query content includes multiple sample sub-contents. The sample content vector is determined based on the sample sub-contents and their corresponding weights. The weights are used to reflect the importance of the corresponding sample sub-contents in the sample query content. Generate content query results based on the content vector and query vector.

[0006] According to a second aspect of the embodiments of this specification, a content query model training method is provided, comprising: Obtain sample data, which includes sample query data and sample query content, and the sample query content includes multiple sample sub-contents; The sample query content is input into the content encoding unit in the initial query model to obtain the sample content vector, and the sample query data is input into the query encoding unit in the initial query model to obtain the sample query vector. The sample content vector is determined based on the sample sub-content and the weight corresponding to the sample sub-content. The weight is used to reflect the importance of the corresponding sample sub-content in the sample query content. Based on the sample content vector and sample query vector, the parameters of the content encoding unit and query encoding unit are adjusted to obtain the content query model.

[0007] According to a third aspect of the embodiments of this specification, a content query device is provided, comprising: The first acquisition module is configured to acquire the target query data and the content to be queried. The first input module is configured to input the content to be queried into the content encoding unit in the content query model to obtain a content vector, and to input the target query data into the query encoding unit in the content query model to obtain a query vector. The content query model is trained based on the sample query vector of the sample query data and the sample content vector of the sample query content. The sample query content includes multiple sample sub-contents. The sample content vector is determined based on the sample sub-contents and their corresponding weights. The weights are used to reflect the importance of the corresponding sample sub-contents in the sample query content. The generation module is configured to generate content query results based on the content vector and the query vector.

[0008] According to a fourth aspect of the embodiments of this specification, a content query model training apparatus is provided, comprising: The second acquisition module is configured to acquire sample data, wherein the sample data includes sample query data and sample query content, wherein the sample query content includes multiple sample sub-contents; The second input module is configured to input the sample query content into the content encoding unit in the initial query model to obtain the sample content vector, and input the sample query data into the query encoding unit in the initial query model to obtain the sample query vector. The sample content vector is determined based on the sample sub-content and the weight corresponding to the sample sub-content. The weight is used to reflect the importance of the corresponding sample sub-content in the sample query content. The first adjustment module is configured to adjust the parameters of the content encoding unit and the query encoding unit based on the sample content vector and the sample query vector to obtain the content query model.

[0009] According to a fifth aspect of the embodiments of this specification, a computing device is provided, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the methods provided in the first or second aspect above.

[0010] According to a sixth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores a computer program / instructions that, when executed by a processor, implement the steps of the method provided in the first or second aspect described above.

[0011] According to a seventh aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the method provided in the first or second aspect described above.

[0012] This specification provides a content query method in one embodiment, comprising: acquiring target query data and content to be queried; inputting the content to be queried into the content encoding unit of a content query model to obtain a content vector, and inputting the target query data into the query encoding unit of the content query model to obtain a query vector. The content query model is trained based on the sample query vector of the sample query data and the sample content vector of the sample query content. The sample query content includes multiple sample sub-contents, and the sample content vector is determined based on the sample sub-contents and their corresponding weights. The weights reflect the importance of the corresponding sample sub-contents within the sample query content. Based on the content vector and the query vector, a content query result is generated. Since the model is trained based on the weighted sample content vector, its content encoding unit can learn to identify and focus on key sub-components within the query content. When processing long texts or multimodal content, the model can automatically assign higher weights to semantics strongly related to the query intent, thereby highlighting core information and suppressing irrelevant descriptions in the generated content vector, solving the problem of key semantic dilution caused by coarse aggregation. Furthermore, by dividing the sample query content into sub-contents and assigning differential weights, the model acquires the ability to identify and suppress local noise. This ensures that the final generated content vector is a fine-grained semantic summary, rather than a fuzzy average of all information, greatly enhancing the discriminative power of the vector representation and laying the foundation for subsequent matching. In summary, the content query model can generate more discriminative vector representations for unimodal or multimodal query content. Its query results are not only more accurate, but also more robust to noise and uneven information distribution within the content. Attached Figure Description

[0013] Figure 1 This is a flowchart illustrating a content query method provided in one embodiment of this specification; Figure 2This is a schematic diagram of the structure of an initial query model provided in one embodiment of this specification; Figure 3 This is an architecture diagram of a content query system provided in one embodiment of this specification; Figure 4 This is a flowchart illustrating a content query model training method provided in one embodiment of this specification; Figure 5 This is a flowchart of a content query method provided in one embodiment of this specification; Figure 6 This is a schematic diagram of the structure of a content query device provided in one embodiment of this specification; Figure 7 This is a schematic diagram of the structure of a content query model training device provided in one embodiment of this specification; Figure 8 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0014] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0015] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of one or more embodiments of this specification. The singular forms “a,” “said,” and “the” used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items. The term “at least one” in one or more embodiments of this application means “one or more,” and “a plurality of” means “two or more.” The term “comprising” is an open-ended description and should be understood as “including but not limiting,” and may include other content in addition to what has been described.

[0016] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0017] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0018] In one or more embodiments of this specification, a large model refers to a deep learning model with a large number of model parameters, typically containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even tens of trillions of model parameters. A large model can also be called a foundation model. It is pre-trained using large-scale unlabeled corpora to produce a pre-trained model with hundreds of millions of parameters. Such models can adapt to a wide range of downstream tasks and have good generalization ability. Examples include Large Language Models (LLMs) and multi-modal pre-training models.

[0019] In practical applications, large models only require a small number of samples to fine-tune the pre-trained model before they can be applied to different tasks. Large models can be widely used in fields such as Natural Language Processing (NLP) and Computer Vision. Specifically, they can be applied to computer vision tasks such as Visual Question Answering (VQA), Image Captioning (IC), and Image Generation, as well as natural language processing tasks such as text-based sentiment classification, text summarization, and machine translation. The main application scenarios of large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.

[0020] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0021] The Dual-Tower Model is a deep learning architecture widely used in recommendation systems, information retrieval, and semantic search. Its core idea is to encode two objects to be matched (such as users and items, queries and content) into vectors using two independent neural networks ("towers"), and then perform matching or recall based on similarity calculations within a shared semantic space.

[0022] Notes are text, images, or other forms of content created by individuals or teams in their studies, work, or daily lives for purposes such as recording information, organizing thoughts, and providing reminders. Notes can range from a few simple lines of text to complex electronic content containing charts, links, multimedia files, and other elements. Notes are widely used in various life and work scenarios, such as knowledge point notes and lesson plan notes in learning and education, and dietary records, exercise records, and shopping sharing notes in health and lifestyle scenarios.

[0023] Multimodal Large Language Model (MLLM) refers to a large-scale language model that can simultaneously process and understand multiple modal inputs (such as text, images, audio, and video). These models perform exceptionally well in tasks such as visual question answering, image-text generation, and cross-modal retrieval.

[0024] End-of-Sequence (EOS) token: This is a special, manually added marker, much like a period in punctuation. It explicitly informs the model of the boundary position of the current input or output sequence: in text understanding, it helps the model determine the range of text to be processed; in text generation (such as translation and summarization), the model outputting [EOS] means "the content has been generated and can stop." Furthermore, because [EOS] integrates contextual information from all preceding words during encoding, it is sometimes directly used as a vector representation of the entire text.

[0025] Average pooling: Average pooling is an operation that performs equal-weighted summation and averaging on a set of vectors, compressing the information from multiple vectors into a single composite vector. This operation treats all word vectors equally, essentially assuming that each word contributes exactly the same to the overall semantics of the text. Its output is a single vector with the same dimension as the input vectors, which is mathematically located at the "center point" of all input vectors and can roughly represent the overall semantic trend of the entire text.

[0026] BERT: A deep bidirectional pre-trained language representation model based on the Transformer architecture. At its core is a Transformer encoder consisting of multiple stacked identical layers, each containing a self-attention mechanism and a feedforward neural network, capable of simultaneously considering the contextual information of each word in the text in all directions (left and right).

[0027] ViT: A full-image encoder that successfully applies the Transformer architecture to computer vision tasks. Its core innovation lies in abandoning the traditional convolutional inductive bias, segmenting the input image into fixed-size image patches, linearly projecting these image patches into sequential "visual lexes," and then processing them through a standard Transformer encoder to achieve global context modeling.

[0028] The Transformer encoder is a sequence-to-sequence transformation module based entirely on self-attention mechanisms and stacked feedforward neural networks. It does not rely on recurrent or convolutional structures, processes the entire input sequence through parallel computation, and can efficiently capture long-range dependencies between arbitrary positions in the sequence.

[0029] Tokenizer processing: Cuts a continuous text string into a discrete sequence of semantic units according to preset rules.

[0030] Information Noise Contrastive Estimation (InfoNCE) loss is a loss function based on contrastive learning, derived from an information-theoretic improvement of noise contrastive estimation. Its core idea is to learn a representation space by maximizing the lower bound of the mutual information of positive sample pairs, thus bringing the representations of similar samples closer together and distancing the representations of dissimilar samples further apart.

[0031] In query systems (such as query engines, recommendation systems, or question-answering systems), to efficiently find results relevant to a user's query from massive amounts of content (Documents), a "dual-tower model" architecture is typically used. This architecture vectorizes both the query and the documents, and then uses vector similarity for fast retrieval. This process is an indispensable and crucial step in online services (i.e., systems that respond in real time after a user initiates a request).

[0032] The content users search for (Documents) is often multimodal (corresponding to the content to be queried). For example, a user might want to find a specific note that contains both text and images. Therefore, the dual-tower model needs to vectorize the multimodal content to obtain the content vectors corresponding to the multimodal content.

[0033] Existing multimodal representation training methods, even those that focus on multimodal consistency at the document level, still face the following fine-grained challenges: 1) Keyword overload problem: In long texts (corresponding to one modality of query content), information of keywords strongly related to images (corresponding to another modality of query content) (such as "starry sky keyboard") may be overwhelmed by a large amount of other text (such as product specifications, warranty information). The text representation (corresponding to the sub-content vector) finally obtained by the content query model is a coarse aggregation of all word vectors (corresponding to the element vector), which cannot highlight the core visual semantics. 2) Local inconsistency problem: The overall text and images of multimodal content may be related, but there is local noise. For example, a complete picture of a living room, the caption after describing the living room mentions "the free potted plant needs to be purchased separately". This text is unrelated to the image content and is local noise. Sample-level weighting cannot handle such noisy paragraphs or words within the text. 3) Limitations of coarse-grained aggregation: Using [EOS] tokens or average pooling to generate text representations is a coarse-grained information aggregation. It lacks explicit modeling of word importance and limits the discrimination accuracy of the representation.

[0034] Therefore, the problem addressed by the embodiments of this specification is: how to achieve fine-grained alignment between the target query sub-content and the query sub-content of another modality during the training of the content encoding unit, dynamically strengthen the query elements in the target query sub-content that are related to the query sub-content of another modality, weaken irrelevant query elements or noisy query elements, thereby generating a sub-content vector that focuses on the semantics of the query sub-content of another modality.

[0035] This specification describes a dynamic weighting method for query elements in a dual-tower model of target query data and query content. Internally, this method dynamically assigns weights to each first target query element by calculating the semantic relevance between the first and second target query elements within the target query sub-content, and adjusts its contribution to the corresponding sub-content vector accordingly. This specification, from the perspective of "query elements," addresses the fine-grained noise problems of key query elements being buried in the target query sub-content and local inconsistencies between the target query sub-content and query sub-content from another modality, aiming to improve the model's ability to capture core semantics and learn more discriminative vector representations.

[0036] In search scenarios, dual-tower modeling of queries and documents and extracting vectors for retrieval is an essential part of online systems. The embodiments in this specification are an important supplement and refinement to the sample-level weighting scheme, suitable for scenarios requiring fine-grained alignment of details of the target query sub-content with query sub-content of another modality. This specification provides a content query method, and also relates to a content query model training method, a content query device, a content query model training device, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0037] See Figure 1 , Figure 1 This specification shows a flowchart of a content query method according to an embodiment, which specifically includes the following steps: Step 102: Obtain the target query data and the content to be queried.

[0038] It's important to note that target query data refers to the original query content entered by the user during retrieval, used to express their information needs. Target query data can be used as input to the query encoding unit, and its semantics will be encoded into a query vector for subsequent similarity calculations. Target query data is typically text-based, including but not limited to keywords, natural language questions, or intent descriptions. For example, target query data could be "recommendations for refreshing sunscreens suitable for oily skin." Target query data can also be other modalities, such as voice, video, and images. When encoding target query data using the query encoding unit, the target query data can be converted to text before being input into the query encoding unit.

[0039] The content to be queried refers to the raw content waiting to be matched with the target query data. There can be one or more pieces of content to be queried. This content can be used as input to content encoding units, transformed into content vectors for subsequent sorting. The content to be queried can be unimodal, such as structured or unstructured text data, or multimodal multimedia content, meaning it simultaneously contains information in at least two modalities, such as images, text, video, and audio. Furthermore, the content to be queried can also cover other formats of content from any object within the content sharing platform, such as location information, group chat records, product entries, and virtual resources. The content to be queried can originate from various application scenarios, such as product detail page text and user reviews on e-commerce platforms, recommended notes and interactive comment data under notes on content sharing platforms, as well as other user-generated or system-generated content that integrates multiple media formats.

[0040] In practical applications, there are various ways to obtain the target query data and the content to be queried, and the specific method should be selected according to the actual situation. This specification does not impose any limitations on these methods in the embodiments. In one possible implementation of this specification, the target query data and the content to be queried can be read from the database of the content query system. In another possible implementation of this specification, the target query data and the content to be queried can be received from the user through a client.

[0041] Step 104: Input the content to be queried into the content encoding unit in the content query model to obtain the content vector, and input the target query data into the query encoding unit in the content query model to obtain the query vector. The content query model is trained based on the sample query vector of the sample query data and the sample content vector of the sample query content. The sample query content includes multiple sample sub-contents. The sample content vector is determined based on the sample sub-contents and their corresponding weights. The weights are used to reflect the importance of the corresponding sample sub-contents in the sample query content.

[0042] It's important to note that a content query model refers to an end-to-end neural network architecture, including content encoding units and query encoding units. For example, a content query model can be built upon a dual-tower model or a large-scale model. Through content query models, efficient semantic retrieval can be achieved, and the representation quality can be improved by weighting the query elements within the query content.

[0043] The content encoding unit (CIU) is a module (typically a deep neural network) in a content query model responsible for converting the input query content into a fixed-dimensional vector representation. The CIU can be any encoder, such as BERT or ViT. It encodes query content of different modalities (text, images, videos, etc.) or lengths into comparable vector forms, extracts high-level semantic features of the query content, and filters out noise and redundant information from the original data.

[0044] The query encoding unit is another module in the content query model, which may have the same or different structure as the content encoding unit. It is specifically responsible for converting the "target query data" into query vectors. The query encoding unit can be an LLM encoder. It encodes diverse user query intents (which could be a keyword, a question, an example image, etc.) into standardized query vectors, understanding the deep semantics of the query, rather than just surface keyword matching. During training, the query encoding unit shares or aligns its semantic space with the content encoding unit, making similar query vectors and content vectors close together.

[0045] A content vector is a numerical vector (also called an embedding vector or feature vector) output by a content encoding unit, representing the overall semantics of the "query content". As a mathematical identifier of the query content in vector space, the content vector is used to calculate similarity with the query vector to assess the relevance of the query content to the target query data. Content vectors can be pre-computed and indexed to enable fast retrieval of large-scale content.

[0046] A query vector is a numerical vector output by the query encoding unit that represents the user's query intent. The query vector is a projection of the user's intent onto the vector space. During retrieval, the query vector is used to perform a nearest neighbor search in the content vector library to find the most relevant content. Query vectors typically need to be computed online in real time to respond to immediate user queries.

[0047] In one optional embodiment of this specification, the content to be queried includes multiple sub-contents of different modalities, and the content encoding unit includes a first fusion sub-unit and multiple first content encoding sub-units, with each first content encoding sub-unit corresponding one-to-one with a sub-content to be queried; inputting the content to be queried into the content encoding unit in the content query model to obtain a content vector may include the following steps: For the target sub-content to be queried, the target sub-content to be queried is encoded by the first content encoding sub-unit to obtain the sub-content vector of the target sub-content to be queried; The first fusion subunit generates a content vector based on the sub-content vectors of multiple query sub-contents, where the target query sub-content is any one of the multiple query sub-contents, and the target first content encoding subunit is the first content encoding subunit corresponding to the target query sub-content.

[0048] It's important to note that the sub-content to be queried is a single modality of data content that constitutes the complete "query content." Each sub-content to be queried represents a fragment of information from one modality within the overall content. Decomposing complex multimodal content into independently processable parts facilitates the use of dedicated first content encoding sub-units for different modalities. This allows the model to capture the semantic features of different modalities separately, enabling finer feature extraction and providing fine-grained input sources for subsequent fusion. For example, the sub-content to be queried could be text or image content within the query content.

[0049] The first content encoding subunit is a neural network encoding module within the content encoding unit specifically designed to process a particular type of query content. Each first content encoding subunit is optimized for a specific modality or type of input data. The most suitable encoding architecture is used for different modalities (text, image, audio, etc.) (e.g., BERT for text, ViT for images). The first content encoding subunit can extract the most discriminative features for a specific data type. Different first content encoding subunits can be computed in parallel to improve efficiency. For example, a first content encoding subunit for text is used to process text content; a first content encoding subunit for image is used to process image content.

[0050] The first fusion subunit is the aggregation module within the content encoding unit. It is responsible for fusing multiple sub-content vectors output from various first content encoding subunits into a unified content vector. The first fusion subunit can fuse sub-content vectors from different modalities and sources into a unified semantic representation. It can also adaptively weight different sub-content vectors based on their contribution to the overall semantics of the queried sub-content. Aggregating multiple sub-content vectors into a single fixed-dimensional vector facilitates subsequent similarity calculations.

[0051] The target query sub-content refers to a specific query sub-content that is being processed by a particular first content encoding sub-unit in the current processing step. It can be any one of multiple query sub-content sets.

[0052] The target first content encoding subunit is a dedicated encoding module for processing the current "target query subcontent". It corresponds one-to-one with the target query subcontent, using modal matching. For example, when the target query subcontent is "product title text", the target first content encoding subunit is a text encoding subunit; when the target query subcontent is "product main image", the target first content encoding subunit is an image encoding subunit.

[0053] A sub-content vector is a local feature vector output after a single target query sub-content has been processed by its corresponding target first content encoding sub-unit. It represents the semantic information of that target query sub-content. The sub-content vector provides a local semantic representation, encodes the semantic information of each target query sub-content separately, and provides composable basic feature units for the first fusion sub-unit.

[0054] In practical applications, there are multiple ways to encode the target query sub-content using the first target content encoding sub-unit to obtain the sub-content vector of the target query sub-content. The specific method is selected according to the actual situation, and the embodiments in this specification do not limit this. In one possible implementation of this specification, a hierarchical encoding structure based on a self-attention mechanism is adopted. Specifically, the first target content encoding sub-unit consists of a sequence encoding layer and an adaptive pooling layer: First, the sequence encoding layer (such as a Transformer encoder) encodes all the basic units of the target query sub-content (such as words in text or tiles in images) into a context-aware intermediate vector sequence; then, the adaptive pooling layer dynamically calculates the importance weight of each intermediate vector through a lightweight attention scoring network, and finally aggregates them into a unified sub-content vector through weighted summation.

[0055] In another possible implementation of this specification, the target query sub-content includes multiple target query elements, and the target first content encoding sub-unit includes a first encoding layer, a first processing layer, and a first fusion layer. Encoding the target query sub-content using the first content encoding sub-unit to obtain the sub-content vector of the target query sub-content may include the following steps: After passing through the first encoding layer, multiple target query elements are encoded respectively to obtain target element vectors corresponding to the multiple target query elements; After the first processing layer, the weights of multiple target feature vectors are determined respectively. The weights are used to reflect the importance of the corresponding target feature vector among multiple target feature vectors. After passing through the first fusion layer, a sub-content vector of the target query sub-content is generated based on multiple target element vectors and their respective weights.

[0056] It's important to note that target query elements refer to basic semantically meaningful units extracted from the "target query sub-content." Decomposing complex target query sub-content into more manageable semantic units provides a foundation for fine-grained encoding. Target query elements, as the smallest semantic units for model understanding and weighting, make importance assessment more efficient. Target query elements provide a basis for interpretability because each element can correspond to a specific text fragment or image region. For example, if the target query sub-content is "The weather is very nice today," the target query elements could be "today," "weather," and "very nice."

[0057] The first encoding layer is the basic feature extraction module in the target first content encoding subunit. It is responsible for independently encoding each target query element into a dense vector (target element vector). The first encoding layer generates an independent vector representation for each query element and adopts an appropriate encoding strategy according to the element type, providing basic features for subsequent weight learning and fusion.

[0058] The first processing layer is the importance assessment module within the target first content encoding subunit. It is responsible for analyzing each target element vector and assigning it an importance weight value. The first processing layer evaluates the contribution of each target query element to the overall semantics of the subcontent, reducing the weight of irrelevant or redundant elements. The first processing layer can analyze multiple target element vectors to determine the weight of each vector. It can also analyze the correlation between the target query element and a second target query element, determining the weight of each vector, even if the second target query element has a different modality from the first.

[0059] The first fusion layer is the aggregation module within the target first content encoding subunit. It is responsible for combining the weighted target element vectors into a single, uniform-dimensional sub-content vector. The first fusion layer performs a weighted summation of each element vector according to the weights calculated by the first processing layer, outputting a fixed-dimensional vector representation to facilitate subsequent multimodal fusion. This compresses the semantic information of multiple target query elements into a compact representation.

[0060] The target feature vector is a numerical vector output from the first encoding layer that represents the semantic information of a single target query feature. The target feature vector transforms discrete target query features into a continuous vector space representation, allowing different features to be compared for similarity in the vector space. The weights of the target element vectors are scalar values ​​output by the first processing layer, representing the relative importance of the corresponding target query element in the overall target query sub-content. They are usually normalized so that the sum of all weights is 1.

[0061] The weight of the target element vector can be determined by the matching index obtained by matching the first target element vector with the second target element vector. In this case, the weight can reflect the degree of association (or similarity) between the first target element vector and the second target element vector, that is, the degree of association between the first target query element corresponding to the first target element vector and the second target query element corresponding to the second target element vector. The second target query element has a different modality than the first target query element.

[0062] For example, there are two sub-contents to be queried: text and an image. The text corresponds to a text encoder (i.e., the first encoding layer), and the image corresponds to an image encoder (i.e., the first encoding layer). The text includes multiple text words (i.e., query elements), and the image includes multiple image patches (i.e., query elements). The text can be the target sub-content to be queried. After the text encoder, a word-granularity weight allocation module (i.e., the first processing layer) is introduced. This module calculates a dynamic weight related to the image for each word token (i.e., the target query element) in the text. For the text (after tokenization) and image (after segmentation) of the query content, during the feature extraction stage, the text is processed by a text encoder (such as LLM) to obtain... Target element vectors Images are processed by an image encoder (such as ViT) to obtain... Target element vectors Use the calculated word granularity weights The output of the text encoder is weighted and aggregated to generate the final text representation. (i.e., sub-content vector), as shown in formula (1): Formula (1), in, For the first Each weight, For the first A vector of target elements.

[0063] Images can also be target query sub-content. After the image encoder, a graph-granular weight allocation module (i.e., the first processing layer) is introduced. This module calculates a dynamic weight related to the text for each image patch (i.e., the target query element) in the image. Use the calculated graph granularity weights The output of the image encoder is weighted and aggregated to generate the final image representation. (i.e., sub-content vector), as shown in formula (2): Formula (2), in, For the first Each weight, For the first A target feature vector. This weighted text representation. and image representation Used for subsequent multimodal fusion (generating content vectors) ).

[0064] In practical applications, there are various ways to determine the weights of multiple target element vectors, and the specific method should be chosen according to the actual situation. This specification does not impose any limitations on this approach. In one possible implementation, target element vectors from different modalities (such as text element vectors and image region vectors) are first projected into a unified cross-modal semantic space. Then, a shared cross-modal attention module is used to calculate the cross-attention score between each target element vector and all other modal element vectors. Finally, the final weights of the target element vectors are generated based on a weighted combination of the self-attention score (intra-modal importance) and the cross-attention score (cross-modal relevance) of each target element vector.

[0065] In another possible implementation of this specification, determining the weights of multiple target feature vectors may include the following steps: For the first target element vector, the first target element vector is matched with the second target element vector to obtain the matching index of the first target element vector. The matching index is used to reflect the degree of matching between the first target element vector and the second target element vector. The first target element vector is the target element vector corresponding to the first target query element. The first target query element is any one of multiple target query elements. The second target element vector is the target element vector corresponding to the second target query element. The second target query element has a different modality from the first target query element. The weights of the first target element vector are determined based on multiple matching indices of the first target element vector.

[0066] It should be noted that the first target element vector is the vector representation of the target query element from a specific modality (such as text) that is currently being evaluated in the multimodal element weight calculation process. It is a dense vector obtained after the "first target query element" has undergone processing in the first encoding layer. The first target element vector is the primary evaluation target that needs to be assigned importance. As the "query" in the matching calculation, it actively compares similarity with elements from other modalities, encoding the semantic information of that element within its respective modality. For example, the first target element vector might be the word vector of the text.

[0067] The second target element vector is a vector representation of other target query elements that have a different modality from the first target element vector. In the matching calculation, it acts as the "matched party" and is used for similarity comparison with the first target element vector. The second target element vector provides the first target element vector with reference points from other modalities, serving as the object of comparison in similarity calculation and verifying the semantic saliency of the first target element from different modal perspectives. There can be one or more second target element vectors, and multiple second target element vectors can share the same modality. For example, the second target element vector can be multiple image patch vectors of the image corresponding to the text.

[0068] A matching metric is a quantitative value obtained by calculating the similarity or correlation between a first target element vector and a second target element vector. It measures the degree of semantic matching between these two elements from different modalities. The matching metric quantifies the semantic correspondence strength between elements of different modalities, providing objective evidence for assessing the importance of the first target element vector. A low matching metric may indicate that the element is unimodal noise (irrelevant to other modalities). For each second target element vector, a matching metric is calculated for the first target element vector, resulting in multiple matching metrics for the first target element vector.

[0069] The primary target query element is the raw semantic unit from a specific modality that is currently being evaluated for importance. It is a concrete instance within the set of target query elements, corresponding to a fragment of raw content that needs to be encoded and weighted.

[0070] The second target query element is a different original semantic unit from the first target query element in terms of modality, used for cross-modal matching comparison with the first target query element. It represents reference content provided from the perspective of other modalities.

[0071] In practical applications, there are multiple ways to match the first target element vector with the second target element vector to obtain the matching index of the first target element vector. The specific method chosen depends on the actual situation, and this specification does not limit this approach. One possible implementation of this specification involves performing a dot product between the first and second target element vectors to obtain the matching index of the first target element vector. Another possible implementation involves calculating the cosine similarity between the first and second target element vectors to obtain the matching index of the first target element vector.

[0072] For example, the query sub-content of the first modality (such as text) corresponds to L first target element vectors, and the query sub-content of the second modality (such as image) corresponds to M second target element vectors. For each first target element vector and each second target element vector, the dot product or cosine similarity is calculated to form an association matrix (i.e., matching index) with a dimension of L*M.

[0073] In practical applications, there are various ways to determine the weight of the first target element vector based on multiple matching indicators. The specific method chosen depends on the actual situation, and this specification does not limit this approach. In one possible implementation, statistical values ​​are determined based on multiple matching indicators of the first target element vector. These statistical values ​​are then normalized to obtain the weight of the first target element vector. The statistical values ​​reflect the overall matching situation between the first target element vector and multiple second target element vectors. For example, the statistical values ​​can be obtained by averaging multiple matching indicators.

[0074] In another possible implementation of this specification, determining the weights of the first target feature vector based on multiple matching indices may include the following steps: From multiple matching indicators of the first target element vector, a target matching indicator is determined, wherein the target matching indicator is greater than the remaining matching indicators other than the target matching indicator among the multiple matching indicators; The target matching index is normalized to obtain the weights of the first target element vector.

[0075] It should be noted that the target matching index is the largest among multiple matching indices calculated from the first target element vector and all second target element vectors. It represents the degree of similarity between the first target element vector and its most relevant and best-matching second target element vector.

[0076] By using target matching metrics, the strongest semantic correspondence evidence is selected from all possible cross-modal pairings, capturing the most explicit representation of the primary target query element across other modalities. Maximization operations ignore accidental weak or noisy matches, focusing only on the most reliable cross-modal support, thus enhancing the robustness of weight allocation. In weight calculation, elements with strong correspondences in at least one other modality are given higher importance, ensuring that key elements that provide "visual evidence" are highlighted in multimodal content.

[0077] Normalization refers to scaling the target matching indices of all first target element vectors according to specific rules, so that they fall into a uniform standard range or distribution, while preserving the relative relationships of the original data as much as possible. For example, if the target query content includes four target query elements with target matching indices of 0.4, 0.5, 0.6, and 0.5 respectively, normalizing the first target matching index of 0.4 yields a weight of 0.4 / (0.4+0.5+0.6+0.5) = 0.2. Similarly, the weight of the second target query element is 0.25, the weight of the third target query element is 0.3, and the weight of the fourth target query element is 0.25, satisfying the condition that the sum of these four weights is 1.

[0078] For example, the matching indices of all the first target element vectors constitute an association matrix. For the correlation matrix By taking the maximum or average value of the second target feature vector dimension, a target matching index is obtained for each first target feature vector. As shown in formula (3) or formula (4): Formula (3), Formula (4), in, To find the maximum value function, To take the average value function, For the correlation matrix The Row data.

[0079] During normalization, for the sequence The Softmax function is applied to obtain the final weights of the target query elements. ,satisfy First-target query elements that are highly relevant to the second-target query element (such as "red" or "sports car") will receive high weight; while first-target query elements that are irrelevant to the second-target query element (such as "of" or "and") will receive low weight.

[0080] The scheme implemented in this specification uses the maximum value among the matching indices of the first target element vector and all other modal element vectors as the target matching index. Essentially, this performs a max-pooling operation, ensuring that the weight of each element is determined solely by its "strongest cross-modal evidence." This effectively filters out accidental or weak noise matching, enhancing the robustness and anti-interference capability of weight allocation. Subsequently, by normalizing the target matching indices of all elements, these absolute values ​​are converted into a relative weight distribution. This not only ensures the stability of the total weight but also allows for a fair comparison of the importance of different elements.

[0081] By applying the scheme of the embodiments in this specification, the system can quantify the semantic support strength of a feature in different modalities by calculating matching indices between a first target feature vector and second target feature vectors in all other modalities. Determining weights based on these matching indices means that the importance of a feature is no longer determined solely by its own modality content, but rather by its overall consistency and relevance in the multimodal context. Ultimately, through a cross-modal matching verification mechanism, key features with high consistency across modalities can be automatically identified and strengthened, while effectively filtering noisy content in a single modality, thereby significantly improving the semantic focus capability and noise robustness of multimodal representation.

[0082] In practical applications, there are various ways to generate a sub-content vector of the target query sub-content based on multiple target element vectors and their respective weights. The specific method is chosen according to the actual situation, and this specification does not limit this approach. In one possible implementation of this specification, the multiple target element vectors are weighted and summed according to their respective weights to obtain the sub-content vector.

[0083] In another possible implementation of this specification, each weighted target element vector is first concatenated into a long vector in sequence, and then a lightweight multilayer perceptron is used to perform nonlinear transformation and dimensionality reduction on the concatenated vector, finally outputting a sub-content vector with uniform dimensionality.

[0084] Applying the scheme of the embodiments in this specification, the first encoding layer independently encodes each target query element, ensuring that the semantic features of each element are fully extracted and preserved; subsequently, the first processing layer dynamically evaluates and quantifies the relative importance of each element in the overall sub-content through a learnable weight allocation mechanism, achieving focus on key information and suppression of noise; finally, the first fusion layer adaptively aggregates the element vector based on weights, so that the final generated sub-content vector not only integrates the information of all elements, but also strengthens the contribution of important elements, thereby significantly improving the discriminativeness and semantic density of the representation.

[0085] In practical applications, there are multiple ways to generate a content vector based on the sub-content vectors of multiple sub-contents to be queried. The specific method should be selected according to the actual situation, and the embodiments in this specification do not impose any limitations on this. In one possible implementation of this specification, multiple sub-content vectors are directly concatenated into a long vector in sequence, and then the concatenated vector is mapped to the target dimension through a learnable linear transformation layer (fully connected layer) to output the content vector.

[0086] In another possible implementation of this specification, the content vector representing the overall information is obtained by summing multiple sub-content vectors and then averaging them.

[0087] The scheme implemented in this specification involves each sub-content to be queried being processed independently by a dedicated encoding sub-unit matching its modality. This ensures that features from different modalities, such as text and images, are fully extracted by the optimized encoder, leveraging the domain advantages of each modal encoder. Subsequently, the first fusion sub-unit performs unified fusion of all sub-content vectors, aggregating semantic features from different sources into a consistent content vector through cross-modal information integration. Through this two-stage processing of modality-specific encoding and cross-modal fusion, efficient collaborative representation of multimodal content is achieved.

[0088] In one optional embodiment of this specification, the content to be queried includes multiple elements to be queried, and the content encoding unit includes a second content encoding subunit, a processing subunit, and a second fusion subunit; inputting the content to be queried into the content encoding unit in the content query model to obtain a content vector may include the following steps: The second content encoding subunit encodes multiple query elements to obtain query element vectors corresponding to each query element. The weights of multiple query element vectors are determined by the processing sub-units. The weights are used to reflect the importance of the corresponding query element vector among multiple query element vectors. The second fusion subunit generates a content vector based on multiple query element vectors and their respective weights.

[0089] It should be noted that the query element refers to the basic semantic unit belonging to the same modality, which is divided from the query content. These query elements together constitute the complete query content, and each query element represents a semantic fragment or structural component of the overall content. Here, the query content is monomodal content. For example, if the query content is text, the query element is a text fragment; if the query content is an image, the query element is a portion of the image region.

[0090] The second content encoding subunit is an encoding module specifically designed to process multiple query elements under the same modality. It is optimized for the data characteristics of this modality and can generate an independent query element vector for each query element.

[0091] The processing subunit is responsible for evaluating the importance of each query element vector within the same modality. It calculates a weight value for each query element vector, reflecting the relative importance of the query element in the overall content.

[0092] The second fusion subunit is a module that aggregates the weighted query element vectors into a single content vector. It performs a weighted combination of each query element vector according to the weights calculated by the processing subunit. The second fusion subunit integrates the semantic information of multiple query elements into a compact content vector, performs a weighted summation operation, so that the high-weight query elements contribute more to the final representation, and outputs a fixed-dimensional content vector, which is convenient for subsequent similarity calculation or retrieval.

[0093] The query element vector is a numerical vector output by the second content encoding subunit, representing the semantic information of a single query element. Each query element vector encapsulates the semantic features of the corresponding query element.

[0094] In practical applications, there are various ways to determine the weights of multiple query element vectors for a single modality of query content. The specific method chosen depends on the actual situation, and this specification does not impose any limitations on this approach. In one possible implementation, for the target query element vector, the average of all query element vectors is first calculated. Then, the similarity between the target query element vector and the average of all query element vectors is calculated, and the similarity is normalized to obtain the weight of the target query element vector. Here, the target query element vector is any one of the multiple query element vectors. The similarity can be a dot product or cosine similarity.

[0095] In another possible implementation of this specification, for the target query element vector, the similarity between the target query element vector and each other query element vector is calculated to obtain multiple similarities. Then, the multiple similarities are averaged to obtain the average similarity. The average similarity is then normalized to obtain the weight of the target query element vector.

[0096] In another possible implementation of this specification, for the target query element vector, the Euclidean distance between the target query element vector and each other query element vector is calculated to obtain multiple Euclidean distances. Then, the average of the multiple Euclidean distances is obtained to get the average Euclidean distance. The reciprocal of the average Euclidean distance is taken (the larger the distance, the lower the importance, so the reciprocal is taken) and normalized to obtain the weight of the target query element vector.

[0097] In practical applications, there are various ways to generate a content vector based on multiple query element vectors and their respective weights. The specific method chosen depends on the actual situation, and this specification does not impose any limitations on this approach. In one possible implementation, the content vector is obtained by weighted summing of the multiple query element vectors according to their respective weights.

[0098] In another possible implementation of this specification, each weighted vector of the query element is first concatenated into a long vector in sequence, and then a lightweight multilayer perceptron is used to perform nonlinear transformation and dimensionality reduction on the concatenated vector to finally output a content vector with uniform dimensions.

[0099] Applying the method of the embodiments in this specification, the second content encoding subunit independently encodes each query element, fully preserving the local semantic details of each element. Subsequently, the processing subunit dynamically evaluates the relative importance of each element in the overall content through a learnable weight allocation mechanism, thereby strengthening core information and suppressing redundant noise. Finally, the second fusion subunit performs weighted aggregation of the element vectors according to the weights, generating a content vector that not only integrates all local information but also highlights the contribution of key elements. This achieves fine-grained and discriminative representation of single-modal content.

[0100] Step 106: Generate content query results based on the content vector and query vector.

[0101] It should be noted that content query results refer to the results returned after calculation and sorting based on the content vector and query vector. Content query results can be at least one item from multiple query items that matches the target query data, or a portion of a single query item that matches the target query data. Content query results can take different forms, such as a list of items matching the target query data, complete content, content summary fragments, or structured responses with relevance scores, etc.

[0102] In practical applications, there are multiple ways to generate content query results based on content vectors and query vectors. The specific method chosen depends on the actual situation, and this specification does not impose any limitations on this approach. One possible implementation of this specification involves calculating the cosine similarity between the query vector and the content vector, and then using the cosine similarity score to recall or rank the content query results. Another possible implementation of this specification uses the dot product of the query vector and the content vector as a relevance score, and then uses this relevance score to recall or rank the content query results. Compared to cosine similarity, the dot product retains the vector magnitude information, which can reflect "confidence" or "importance."

[0103] The scheme implemented in this specification, because the model is trained based on weighted sample content vectors, enables its content encoding units to learn the ability to identify and focus on key sub-components within the query content. When processing long texts or multimodal content, the model can automatically assign higher weights to semantics strongly related to the query intent, thereby highlighting core information and suppressing irrelevant descriptions in the generated content vector, solving the problem of key semantic dilution caused by coarse aggregation. By dividing the sample query content into sub-contents and assigning differential weights, the model gains the ability to identify and suppress local noise. The explicit, importance-based weighted aggregation method replaces traditional uniform pooling or single-label representation methods, resulting in a fine-grained semantic summary of the final generated content vector, rather than a fuzzy average of all information. This greatly enhances the discriminative power of the vector representation, laying the foundation for subsequent matching. In summary, the content query model can generate more discriminative vector representations for single-modal or multimodal query content, resulting in query results that are not only more accurate but also more robust to noise and uneven information distribution within the content.

[0104] In one optional embodiment of this specification, the training method of the content query model may include the following steps: Obtain sample data, which includes sample query data and sample query content; Input the sample query content into the content encoding unit in the initial query model to obtain the sample content vector, and input the sample query data into the query encoding unit in the initial query model to obtain the sample query vector; Based on the sample content vector and sample query vector, the parameters of the content encoding unit and query encoding unit are adjusted to obtain the content query model.

[0105] It should be noted that sample data refers to the data used to train the initial query model, and at least includes sample query content and sample query data. There are usually multiple sets of sample data. Within the same set of sample data, sample query data and sample query content are related, while sample query data and sample query content in different sets of sample data may or may not be related.

[0106] Sample query content refers to the content samples used during the initial training phase of the query model. It serves as input to the content encoding unit for learning semantic representations. Sample query content can be unimodal or multimodal multimedia content. Furthermore, it can encompass other formats of content from any object within the content sharing platform. Sample query content can originate from various application scenarios, such as product detail page text and user reviews on e-commerce platforms, recommended notes and interactive comment data under notes on content sharing platforms, as well as other user-generated or system-generated content that integrates multiple media formats. Sample query data refers to user query text semantically related to the sample query content, such as keywords in search logs, natural language questions, or queries corresponding to click behaviors.

[0107] The initial query model is the raw model architecture at the start of the training process, before its parameters have been tuned using sample data. It contains the basic structure of content encoding units and query encoding units, but its internal parameters are typically randomly initialized or pre-trained and have not yet been optimized for a specific query task.

[0108] The sample content vector is the vector representation output by the content encoding unit in the initial query model after processing the sample query content. It is dynamically generated during training and represents the current model's (at a certain training stage) understanding and encoding ability of the sample content. It will be continuously optimized as the model parameters are adjusted.

[0109] A sample query vector is a vector representation output by the query encoding unit in the initial query model after processing the sample query data. It represents the current model's encoding of the query intent and is continuously optimized during the training process. The ultimate goal is to make similar sample query data and sample query content closer together in the vector space.

[0110] In one optional embodiment of this specification, the sample query content includes multiple sample query sub-contents of different modalities. The content encoding unit includes a first fusion sub-unit and multiple first content encoding sub-units, with each first content encoding sub-unit corresponding one-to-one with the sub-content to be queried. Inputting the sample query content into the content encoding unit in the initial query model to obtain a sample content vector includes: encoding the target sample query sub-content using a target first content encoding sub-unit to obtain a sample sub-content vector of the target sample query sub-content; and generating a sample content vector using the first fusion sub-unit based on the sample sub-content vectors of the multiple sample query sub-contents, wherein the target sample query sub-content is any one of the multiple sample query sub-contents, and the target first content encoding sub-unit is the first content encoding sub-unit corresponding to the target sample query sub-content.

[0111] In one optional embodiment of this specification, the target sample query sub-content includes multiple target sample query elements, and the target first content encoding sub-unit includes a first encoding layer, a first processing layer, and a first fusion layer. Encoding the target sample query sub-content through the target first content encoding sub-unit to obtain a sample sub-content vector of the target sample query sub-content includes: encoding the multiple target sample query elements respectively through the first encoding layer to obtain target sample element vectors corresponding to the multiple target sample query elements; determining the weights of the multiple target sample element vectors respectively through the first processing layer, wherein the weights are used to reflect the importance of the corresponding target sample element vector among the multiple target sample element vectors; and generating the sample sub-content vector of the target sample query sub-content based on the multiple target sample element vectors and their respective weights through the first fusion layer.

[0112] In one optional embodiment of this specification, determining the weights of multiple target sample element vectors includes: matching a first target sample element vector with a second target sample element vector to obtain a sample matching index for the first target sample element vector, wherein the sample matching index is used to reflect the degree of matching between the first target sample element vector and the second target sample element vector, the first target sample element vector is the target sample element vector corresponding to a first target sample query element, the first target sample query element is any one of multiple target sample query elements, the second target sample element vector is the target sample element vector corresponding to a second target sample query element, and the second target sample query element has a different modality from the first target sample query element; and determining the weights of the first target sample element vector based on the multiple sample matching indices of the first target sample element vector.

[0113] In one optional embodiment of this specification, determining the weight of the first target sample element vector based on multiple sample matching indices of the first target sample element vector includes: determining a target sample matching index from multiple sample matching indices of the first target sample element vector, wherein the target sample matching index is greater than the remaining sample matching indices other than the target sample matching index among the multiple sample matching indices; and normalizing the target sample matching index to obtain the weight of the first target sample element vector.

[0114] In one optional embodiment of this specification, the sample query content includes multiple sample query elements, and the content encoding unit includes a second content encoding subunit, a processing subunit, and a second fusion subunit. Inputting the sample query content into the content encoding unit in the content query model to obtain a sample content vector includes: encoding the multiple sample query elements respectively through the second content encoding subunit to obtain sample query element vectors corresponding to the multiple sample query elements; determining the weights of the multiple sample query element vectors respectively through the processing subunit, wherein the weights reflect the importance of the corresponding sample query element vector among the multiple sample query element vectors; and generating a sample content vector based on the multiple sample query element vectors and their respective weights through the second fusion subunit.

[0115] For example, the training process of the content encoding unit is as follows: For the target sample query sub-content, multiple target sample query elements are encoded to obtain target sample element vectors corresponding to each target sample query element; the weights of the multiple target sample element vectors are determined; based on the weights of the multiple target sample element vectors, a weighted sample sub-content vector of the target sample query sub-content is generated; a sample content vector is generated based on the sample sub-content vectors of the multiple sample query sub-contents; the training loss can be a contrastive loss. At this time, the gradient will be backpropagated to the first processing layer and the first encoding layer through the weighted sub-content vector, thereby learning end-to-end how to "focus" on the key information in the target sample query sub-content according to different modalities.

[0116] In practical applications, there are various ways to adjust the parameters of the content encoding unit and query encoding unit based on the sample content vector and sample query vector to obtain the content query model. The specific method is selected according to the actual situation, and the embodiments in this specification do not limit this. In one possible implementation of this specification, for each sample query vector, a positive sample content vector (related to it) and multiple negative sample content vectors (unrelated to it) are constructed. The similarity score between the query vector and the positive sample vector, and the similarity score with each negative sample vector are calculated. Then, by maximizing the score interval between positive and negative sample pairs (e.g., using triplet loss or list-based sorting loss), the parameters of the query encoding unit and the content encoding unit are updated, so that positive sample pairs are closer in the vector space, while negative sample pairs are farther apart.

[0117] In another possible implementation of this specification, the content encoding unit and query encoding unit are adjusted according to the sample content vector and sample query vector to obtain a content query model, which may include the following steps: The contrast loss is determined based on the sample query vector and the sample content vector; Based on contrast loss, the parameters of the query encoding unit and the content encoding unit are adjusted to obtain the content query model.

[0118] It should be noted that contrastive loss refers to the loss value used to measure the semantic matching degree between sample content vectors and sample query vectors. Contrastive loss can be InfoNCE loss. In a batch of sample data, contrastive loss can reduce the vector distance between sample query vectors and relevant sample query content, while increasing the vector distance between sample query vectors and irrelevant sample query content. This drives the alignment of query encoding units and content encoding units in the shared semantic space, improving query relevance and matching.

[0119] This specification proposes a method for generating a sample sub-content vector by calculating the semantic relevance (i.e., matching index) between each target sample query element in the target sample query sub-content and the sample query sub-content of another modality during the training process of a multimodal representation model, assigning a dynamic weight to each target sample query element, and performing weighted aggregation based on these weights. The semantic relevance between the target sample query element and the sample query sub-content of another modality is obtained by calculating the similarity between the target sample element vector of the target sample query element and the regional or global features of the sample query sub-content of the other modality.

[0120] This specification's embodiments introduce, for the first time, an end-to-end target element vector weighting mechanism within a multimodal contrastive learning framework. This prevents the model from treating all target query elements equally, instead enabling it to "follow the clues," actively focusing on key semantic units within the target query sub-content based on the sub-content of another modality. This surpasses sample-level weighting schemes, laying the technical foundation for achieving truly fine-grained and robust multimodal understanding.

[0121] The scheme implemented in this specification, based on the contrastive loss calculated using sample query vectors and sample content vectors, forces the model to narrow the vector distance between relevant query-content pairs while widening the distance between irrelevant pairs, thereby establishing a clear semantic clustering structure in the vector space. By backpropagating this loss to adjust the encoding unit parameters, the query encoding unit and content encoding unit are co-optimized. The resulting content query model can map semantically similar queries and content to neighboring vector positions, significantly improving the model's accuracy and robustness in open-domain retrieval and enhancing its generalization ability to unseen query-content pairs.

[0122] By applying the scheme of the embodiments in this specification, and utilizing paired samples containing sample query data and sample query content, the initial query model can simultaneously generate a sample query vector representing the query intent and a sample content vector representing the semantics of the content, providing direct supervision signals for model optimization. Subsequently, the parameters of the encoding units are adjusted based on the relationship between these two vectors (e.g., through contrastive loss), aligning the query encoding units and content encoding units in a shared semantic space. This not only improves their respective representation capabilities of the input data but, more importantly, strengthens the cross-modal interaction consistency between them. The resulting content query model thus possesses semantic matching capabilities, enabling efficient association between user queries and relevant content, significantly improving the accuracy and response efficiency of the retrieval system while reducing reliance on external annotation or complex feature engineering.

[0123] See Figure 2 , Figure 2 This specification shows a schematic diagram of the structure of an initial query model provided in one embodiment, as follows: Figure 2 As shown, the initial query model includes a query encoding unit and a content encoding unit. The content encoding unit includes a first fusion subunit and two first content encoding subunits. Each first content encoding subunit includes a first encoding layer, a first processing layer, and a first fusion layer. The sample data includes one sample query data and two sample query sub-contents with different modalities.

[0124] The query encoding unit processes the sample query data to obtain the sample query vector.

[0125] For the first sample query sub-content, L first sample element vectors are obtained through the corresponding first encoding layer; the first weights corresponding to the L first sample element vectors are determined through the corresponding first processing layer; and the first sample sub-content vector of the first sample query sub-content is determined through the corresponding first fusion layer.

[0126] For the second sample query sub-content, M second sample element vectors are obtained through the corresponding first encoding layer; the second weights corresponding to the M second sample element vectors are determined through the corresponding first processing layer; and the second sample sub-content vector of the second sample query sub-content is determined through the corresponding first fusion layer.

[0127] By matching L first sample element vectors and M second sample element vectors, an L*M dimensional matching index matrix (i.e., correlation matrix) can be obtained.

[0128] The first fusion subunit fuses the first sample sub-content vector and the second sample sub-content vector to generate a sample content vector.

[0129] Calculate the contrast loss based on the sample query vector and sample content vector, and adjust the parameters of the initial query model based on the contrast loss.

[0130] Considering the large number of parameters in the content query model and the limited computing resources on the client side, the content query method proposed in the embodiments of this specification can be applied to, for example... Figure 3 The content retrieval system shown is not limited to this. See also Figure 3 , Figure 3 This specification illustrates an architecture diagram of a content query system provided in one embodiment of the specification. The content query system may include a client 302 and a server 304. Client 302 is used to send the target query data and the content to be queried to server 304; Server 304 is used to input the content to be queried into the content encoding unit of the content query model to obtain the content vector, and input the target query data into the query encoding unit of the content query model to obtain the query vector. The content query model is trained based on the sample query vector of the sample query data and the sample content vector of the sample query content. The sample query content includes multiple sample sub-contents. The sample content vector is determined based on the sample sub-contents and their corresponding weights. The weights are used to reflect the importance of the corresponding sample sub-contents in the sample query content. Based on the content vector and the query vector, the content query result is generated and sent to client 302. Client 302 is also used to receive content query results sent by server 304.

[0131] like Figure 3 As shown, the content query model is deployed on server 304. Server 304 can connect to one or more clients 302 via LAN, WAN, Internet, or other types of data networks. Data transmitted by clients 302 may require encoding, transcoding, compression, or other processing before being published to server 304. Multiple clients 302 can establish communication connections through server 304. In the content query scenario, server 304 provides content query services between multiple clients 302. Each client 302 can act as a sender or receiver, communicating through server 304. Users can interact with server 304 through client 302 to receive data from other clients 302 or send data to other clients 302. In the content query scenario, a user can publish a data stream to server 304 through client 302. Server 304 generates content query results based on this data stream and pushes the results to other communicating clients.

[0132] Client 302 can be a browser, application (APP), or web application such as HyperText Markup Language 5 (H5) application, or a lightweight application (also known as a mini-program), or cloud application, etc. Client 302 can be obtained based on the software development kit (SDK) of the corresponding service provided by server 304, such as developing based on the Real-Time Communication (RTC) SDK. Client 302 can be deployed in electronic devices and depends on the device to run or some APP on the device to run. Electronic devices may have a display screen and support information browsing, such as personal mobile terminals such as mobile phones, tablets, and personal computers (PCs). Various other types of applications can also be configured in electronic devices, such as human-computer interaction applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc. The client 302 can also interact with the user through a graphical user interface to invoke the content query model, thereby implementing the content query method provided in the embodiments of this specification.

[0133] Server-side 304 can include servers providing various services, such as servers providing communication services to multiple clients, servers supporting backend training of models used on clients, and servers processing data sent by clients. It should be noted that server-side 304 can be implemented as a distributed server cluster composed of multiple servers, or as a single server. The server can also be a server in a distributed system, or a server integrated with blockchain. The server can also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0134] It is worth noting that the content query method provided in the embodiments of this specification is generally executed by the server. However, in other embodiments of this specification, if the client's runtime resources can meet the deployment and runtime conditions of the content query model, the client can also have similar functions to the server, thereby executing the content query method provided in the embodiments of this specification. In other embodiments, the content query method provided in the embodiments of this specification can also be executed jointly by the client and the server.

[0135] See Figure 4 , Figure 4 This specification illustrates a flowchart of a content query model training method according to an embodiment, which specifically includes the following steps: Step 402: Obtain sample data, which includes sample query data and sample query content, and the sample query content includes multiple sample sub-contents.

[0136] Step 404: Input the sample query content into the content encoding unit in the initial query model to obtain the sample content vector, and input the sample query data into the query encoding unit in the initial query model to obtain the sample query vector. The sample content vector is determined based on the sample sub-content and the weight corresponding to the sample sub-content. The weight is used to reflect the importance of the corresponding sample sub-content in the sample query content.

[0137] Step 406: Adjust the parameters of the content encoding unit and the query encoding unit based on the sample content vector and the sample query vector to obtain the content query model.

[0138] It should be noted that the implementation methods of steps 402 to 406 can refer to the training method of the above-mentioned content query model, and will not be described again in the embodiments of this specification.

[0139] The scheme implemented in this specification involves a sample content vector that is not simply encoded from the entire sample query content. Instead, it is generated based on multiple sample sub-contents and their corresponding dynamic weights. This allows the model to learn to distinguish the importance of different parts within the content during the early stages of training, automatically focusing on key sub-contents and suppressing noise. Subsequently, through learning the sample content vector and the sample query vector, the model not only optimizes the overall alignment between the query and the content but also backpropagates this fine-grained weight-awareness to the encoding units. This enables the final content query model to capture core semantics and generate more discriminative representations when understanding complex, lengthy, or multimodal content, thereby significantly improving the accuracy and robustness of cross-modal retrieval.

[0140] See Figure 5 , Figure 5 The flowchart of a content query method according to an embodiment of this specification is shown. During the content query process, the server and the client interact with each other. The client is used to send sample data to the server. The sample data includes sample query content and sample query data. The sample query content includes multiple sample sub-contents.

[0141] On the server side, sample query content is input into the content encoding unit of the initial query model to obtain a sample content vector, and sample query data is input into the query encoding unit of the initial query model to obtain a sample query vector. The sample content vector is determined based on the sample sub-content and its corresponding weight, which reflects the importance of the corresponding sample sub-content in the sample query content. Based on the sample content vector and the sample query vector, the parameters of the content encoding unit and the query encoding unit are adjusted to obtain the content query model.

[0142] The client is also used to send target query data and query content to the server.

[0143] On the server side, the content to be queried is input into the content encoding unit of the content query model to obtain a content vector, and the target query data is input into the query encoding unit of the content query model to obtain a query vector. Based on the content vector and the query vector, the content query result is generated and sent to the client.

[0144] Corresponding to the above-described embodiments of the content query method, this specification also provides embodiments of the content query device. Figure 6 A schematic diagram of the structure of a content query device according to one embodiment of this specification is shown. Figure 6 As shown, the device includes: The first acquisition module 602 is configured to acquire the target query data and the content to be queried. The first input module 604 is configured to input the content to be queried into the content encoding unit in the content query model to obtain a content vector, and to input the target query data into the query encoding unit in the content query model to obtain a query vector. The content query model is trained based on the sample query vector of the sample query data and the sample content vector of the sample query content. The sample query content includes multiple sample sub-contents. The sample content vector is determined based on the sample sub-contents and their corresponding weights. The weights are used to reflect the importance of the corresponding sample sub-contents in the sample query content. The generation module 606 is configured to generate content query results based on the content vector and the query vector.

[0145] Optionally, the content to be queried includes multiple sub-contents to be queried in different modalities. The content encoding unit includes a first fusion sub-unit and multiple first content encoding sub-units, with each first content encoding sub-unit corresponding to a sub-content to be queried. The first input module 604 is further configured to encode the target sub-content to be queried through the target first content encoding sub-unit to obtain a sub-content vector of the target sub-content to be queried. The first fusion sub-unit generates a content vector based on the sub-content vectors of the multiple sub-contents to be queried, wherein the target sub-content to be queried is any one of the multiple sub-contents to be queried, and the target first content encoding sub-unit is the first content encoding sub-unit corresponding to the target sub-content to be queried.

[0146] Optionally, the target query sub-content includes multiple target query elements, and the target first content encoding sub-unit includes a first encoding layer, a first processing layer, and a first fusion layer; the first input module 604 is further configured to encode the multiple target query elements respectively through the first encoding layer to obtain target element vectors corresponding to the multiple target query elements; determine the weights of the multiple target element vectors respectively through the first processing layer, wherein the weights are used to reflect the importance of the corresponding target element vectors in the multiple target element vectors; and generate a sub-content vector of the target query sub-content based on the multiple target element vectors and the weights corresponding to the multiple target element vectors through the first fusion layer.

[0147] Optionally, the first input module 604 is further configured to match the first target element vector with the second target element vector for the first target element vector to obtain a matching index of the first target element vector. The matching index is used to reflect the degree of matching between the first target element vector and the second target element vector. The first target element vector is the target element vector corresponding to the first target query element. The first target query element is any one of multiple target query elements. The second target element vector is the target element vector corresponding to the second target query element. The second target query element and the first target query element have different modalities. The weight of the first target element vector is determined based on the multiple matching indices of the first target element vector.

[0148] Optionally, the first input module 604 is further configured to determine a target matching index from multiple matching indices of the first target feature vector, wherein the target matching index is greater than the remaining matching indices other than the target matching index among the multiple matching indices; and to normalize the target matching index to obtain the weight of the first target feature vector.

[0149] Optionally, the content to be queried includes multiple elements to be queried, and the content encoding unit includes a second content encoding subunit, a processing subunit, and a second fusion subunit; the first input module 604 is further configured to encode the multiple elements to be queried by the second content encoding subunit to obtain the vectors to be queried corresponding to the multiple elements to be queried; to determine the weights of the multiple vectors to be queried by the processing subunit, wherein the weights are used to reflect the importance of the corresponding vector to be queried in the multiple vectors to be queried; and to generate a content vector by the second fusion subunit based on the multiple vectors to be queried and the weights corresponding to the multiple vectors to be queried.

[0150] Optionally, the device further includes: a second adjustment module configured to acquire sample data, wherein the sample data includes sample query data and sample query content; input the sample query content into the content encoding unit in the initial query model to obtain a sample content vector, and input the sample query data into the query encoding unit in the initial query model to obtain a sample query vector; and adjust the parameters of the content encoding unit and the query encoding unit according to the sample content vector and the sample query vector to obtain a content query model.

[0151] Optionally, the second adjustment module is also configured to determine the contrast loss based on the sample query vector and the sample content vector; and to adjust the parameters of the query encoding unit and the content encoding unit based on the contrast loss to obtain the content query model.

[0152] The scheme implemented in this specification, because the model is trained based on weighted sample content vectors, enables its content encoding units to learn the ability to identify and focus on key sub-components within the query content. When processing long texts or multimodal content, the model can automatically assign higher weights to semantics strongly related to the query intent, thereby highlighting core information and suppressing irrelevant descriptions in the generated content vector, solving the problem of key semantic dilution caused by coarse aggregation. By dividing the sample query content into sub-contents and assigning differential weights, the model gains the ability to identify and suppress local noise. The explicit, importance-based weighted aggregation method replaces traditional uniform pooling or single-label representation methods, resulting in a fine-grained semantic summary of the final generated content vector, rather than a fuzzy average of all information. This greatly enhances the discriminative power of the vector representation, laying the foundation for subsequent matching. In summary, the content query model can generate more discriminative vector representations for single-modal or multimodal query content, resulting in query results that are not only more accurate but also more robust to noise and uneven information distribution within the content.

[0153] The above is an illustrative scheme of a content query device according to this embodiment. It should be noted that the technical solution of this content query device and the technical solution of the above-described content query method belong to the same concept. For details not described in detail in the technical solution of the content query device, please refer to the description of the technical solution of the above-described content query method.

[0154] Corresponding to the above embodiments of the content query model training method, this specification also provides embodiments of the content query model training apparatus. Figure 7 A schematic diagram of a content query model training apparatus according to one embodiment of this specification is shown. Figure 7 As shown, the device includes: The second acquisition module 702 is configured to acquire sample data, wherein the sample data includes sample query data and sample query content, wherein the sample query content includes multiple sample sub-contents; The second input module 704 is configured to input sample query content into the content encoding unit in the initial query model to obtain a sample content vector, and input sample query data into the query encoding unit in the initial query model to obtain a sample query vector. The sample content vector is determined based on the sample sub-content and the weight corresponding to the sample sub-content. The weight is used to reflect the importance of the corresponding sample sub-content in the sample query content. The first adjustment module 706 is configured to adjust the parameters of the content encoding unit and the query encoding unit according to the sample content vector and the sample query vector to obtain the content query model.

[0155] The scheme implemented in this specification involves a sample content vector that is not simply encoded from the entire sample query content. Instead, it is generated based on multiple sample sub-contents and their corresponding dynamic weights. This allows the model to learn to distinguish the importance of different parts within the content during the early stages of training, automatically focusing on key sub-contents and suppressing noise. Subsequently, through learning the sample content vector and the sample query vector, the model not only optimizes the overall alignment between the query and the content but also backpropagates this fine-grained weight-awareness to the encoding units. This enables the final content query model to capture core semantics and generate more discriminative representations when understanding complex, lengthy, or multimodal content, thereby significantly improving the accuracy and robustness of cross-modal retrieval.

[0156] The above is an illustrative scheme of a content query model training device according to this embodiment. It should be noted that the technical solution of this content query model training device and the technical solution of the above-described content query model training method belong to the same concept. For details not described in detail in the technical solution of the content query model training device, please refer to the description of the technical solution of the above-described content query model training method.

[0157] Figure 8 This specification illustrates a structural block diagram of a computing device according to one embodiment. The components of the computing device 800 include, but are not limited to, a memory 810 and a processor 820. The processor 820 is connected to the memory 810 via a bus 830, and a database 850 is used to store data.

[0158] The computing device 800 also includes an access device 840, which enables the computing device 800 to communicate via one or more networks 860. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 840 may include one or more of any type of wired or wireless network interface (e.g., Network Interface Card (NIC)), such as an IEEE 802.11 Wireless Local Area Networks (WLAN) interface, a Wi-MAX (World Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0159] In one embodiment of this specification, the above-described components of the computing device 800 and Figure 8 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 8 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0160] The computing device 800 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers. The computing device 800 can also be a mobile or stationary server.

[0161] The processor 820 is used to execute computer programs / instructions, which, when executed by the processor, implement the steps of the above-mentioned content query method or content query model training method.

[0162] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device belongs to the same concept as the technical solutions of the content query method and the content query model training method described above. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solutions of the content query method or the content query model training method described above.

[0163] An embodiment of this specification also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the above-described content query method or content query model training method.

[0164] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium belongs to the same concept as the technical solutions of the content query method and the content query model training method described above. Details not described in detail in the technical solution of the storage medium can be found in the descriptions of the technical solutions of the content query method or the content query model training method described above.

[0165] An embodiment of this specification also provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of the above-described content query method or content query model training method.

[0166] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product belongs to the same concept as the technical solutions of the content query method and the content query model training method described above. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solutions of the content query method or the content query model training method described above.

[0167] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0168] Computer instructions include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. Computer-readable media can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in computer-readable media can be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0169] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0170] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0171] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A content query method, characterized in that, include: Retrieve the target query data and the content to be queried; The content to be queried is input into the content encoding unit in the content query model to obtain a content vector, and the target query data is input into the query encoding unit in the content query model to obtain a query vector. The content query model is trained based on the sample query vector of the sample query data and the sample content vector of the sample query content. The sample query content includes multiple sample sub-contents. The sample content vector is determined based on the sample sub-contents and the weights corresponding to the sample sub-contents. The weights are used to reflect the importance of the corresponding sample sub-contents in the sample query content. Based on the content vector and the query vector, generate content query results.

2. The method according to claim 1, characterized in that, The content to be queried includes multiple sub-contents to be queried in different modalities. The content encoding unit includes a first fusion sub-unit and multiple first content encoding sub-units, and the first content encoding sub-units correspond one-to-one with the sub-contents to be queried. The step of inputting the content to be queried into the content encoding unit in the content query model to obtain the content vector includes: For the target sub-content to be queried, the target sub-content to be queried is encoded by the first content encoding sub-unit to obtain the sub-content vector of the target sub-content to be queried; The first fusion subunit generates the content vector based on the sub-content vectors of the plurality of sub-contents to be queried, wherein the target sub-content to be queried is any one of the plurality of sub-contents to be queried, and the target first content encoding subunit is the first content encoding subunit corresponding to the target sub-content to be queried.

3. The method according to claim 2, characterized in that, The target query sub-content includes multiple target query elements, and the target first content encoding sub-unit includes a first encoding layer, a first processing layer, and a first fusion layer; The process of encoding the target query sub-content using the first content encoding sub-unit to obtain the sub-content vector of the target query sub-content includes: The first encoding layer encodes the plurality of target query elements respectively to obtain target element vectors corresponding to the plurality of target query elements; After the first processing layer, the weights of the multiple target element vectors are determined respectively, wherein the weights are used to reflect the importance of the corresponding target element vector among the multiple target element vectors; After passing through the first fusion layer, the sub-content vector of the target query sub-content is generated based on the multiple target element vectors and the weights corresponding to the multiple target element vectors respectively.

4. The method according to claim 3, characterized in that, The step of determining the weights of the multiple target feature vectors includes: For a first target element vector, the first target element vector is matched with a second target element vector to obtain a matching index for the first target element vector. The matching index is used to reflect the degree of matching between the first target element vector and the second target element vector. The first target element vector is the target element vector corresponding to a first target query element. The first target query element is any one of a plurality of target query elements. The second target element vector is the target element vector corresponding to a second target query element. The second target query element has a different modality from the first target query element. The weights of the first target element vector are determined based on a plurality of the matching indices of the first target element vector.

5. The method according to claim 4, characterized in that, Determining the weight of the first target feature vector based on multiple matching indicators of the first target feature vector includes: From the plurality of matching indicators of the first target element vector, a target matching indicator is determined, wherein the target matching indicator is greater than the remaining matching indicators other than the target matching indicator among the plurality of matching indicators; The target matching index is normalized to obtain the weight of the first target element vector.

6. The method according to claim 1, characterized in that, The content to be queried includes multiple elements to be queried, and the content encoding unit includes a second content encoding subunit, a processing subunit, and a second fusion subunit. The step of inputting the content to be queried into the content encoding unit in the content query model to obtain the content vector includes: The second content encoding subunit encodes the plurality of query elements respectively to obtain query element vectors corresponding to the plurality of query elements; The processing subunit determines the weights of the multiple query element vectors, wherein the weights are used to reflect the importance of the corresponding query element vector among the multiple query element vectors. The second fusion subunit generates the content vector based on the multiple query element vectors and the weights corresponding to the multiple query element vectors.

7. The method according to any one of claims 1 to 6, characterized in that, The training methods for the content query model include: Obtain sample data, wherein the sample data includes the sample query data and the sample query content; The sample query content is input into the content encoding unit in the initial query model to obtain the sample content vector, and the sample query data is input into the query encoding unit in the initial query model to obtain the sample query vector; Based on the sample content vector and the sample query vector, the parameters of the content encoding unit and the query encoding unit are adjusted to obtain the content query model.

8. The method according to claim 7, characterized in that, The step of adjusting the parameters of the content encoding unit and the query encoding unit based on the sample content vector and the sample query vector to obtain the content query model includes: Based on the sample query vector and the sample content vector, determine the contrast loss; Based on the contrast loss, the parameters of the query encoding unit and the content encoding unit are adjusted to obtain the content query model.

9. A method for training a content query model, characterized in that, include: Obtain sample data, wherein the sample data includes sample query data and sample query content, wherein the sample query content includes multiple sample sub-contents; The sample query content is input into the content encoding unit in the initial query model to obtain the sample content vector, and the sample query data is input into the query encoding unit in the initial query model to obtain the sample query vector. The sample content vector is determined based on the sample sub-content and the weight corresponding to the sample sub-content. The weight is used to reflect the importance of the corresponding sample sub-content in the sample query content. Based on the sample content vector and the sample query vector, the parameters of the content encoding unit and the query encoding unit are adjusted to obtain the content query model.

10. A computing device, characterized in that, include: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 9.

11. A computer-readable storage medium, characterized in that, It stores a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 9.

12. A computer program product, characterized in that, Includes a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 9.

Images