Content query method and content query model training method

Abstract

The embodiment of the invention provides a content query method and a content query model training method. The content query method comprises the steps of obtaining target query data and to-be-queried content; the content to be queried is input into a content coding unit in a content query model to obtain a content vector, the target query data is input into a query coding unit in the content query model to obtain a query vector, and the content query model comprises the content coding unit, a query coding unit and an image generation unit; the content coding unit is obtained by training based on a sample content image of the sample content and a predicted content image, and the predicted content image is obtained by processing a sample content vector of the sample content by the image generation unit; and generating a content query result according to the content vector and the query vector. And the content coding unit carries out supervised training through the sample content image and the predicted content image reconstructed by the image generation unit under the condition of the sample content vector, so that the accuracy of the content query result is improved.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of artificial intelligence technology, and in particular to content query methods and content query model training methods. Background Technology

[0002] With the explosive growth of internet content, efficient semantic-based content retrieval has become a core component of applications such as search engines, recommendation systems, and intelligent question answering. Traditional keyword matching methods struggle to capture the deep semantic relationships between user queries and content, prompting researchers to widely adopt the dual-tower model based on dense vectors for semantic retrieval.

[0003] Currently, dual-tower models typically map text to a shared embedding space using independent query encoders and content encoders, leveraging vector similarity for efficient querying. However, in this process, a significant amount of fine-grained, non-topic details are inevitably lost, leading to poor accuracy in content query results. Therefore, a more accurate content querying solution is urgently needed. Summary of the Invention

[0004] In view of this, embodiments of this specification provide a content query method. One or more embodiments of this specification also relate to a content query model training method, a content query device, a content query model training device, a computing device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.

[0005] According to a first aspect of the embodiments of this specification, a content query method is provided, including: Retrieve the target query data and the content to be queried; The content to be queried is input into the content encoding unit in the content query model to obtain the content vector, and the target query data is input into the query encoding unit in the content query model to obtain the query vector. The content query model includes a content encoding unit, a query encoding unit, and an image generation unit. The content encoding unit is trained based on the sample content image of the sample content and the predicted content image. The predicted content image is obtained by the image generation unit by processing the sample content vector of the sample content. Generate content query results based on the content vector and query vector.

[0006] According to a second aspect of the embodiments of this specification, a content query model training method is provided, comprising: Obtain sample data, which includes sample content, sample content images, and sample query data; Input the sample content into the content encoding unit in the content query model to obtain the sample content vector, and input the sample query data into the query encoding unit in the content query model to obtain the sample query vector; The image generation unit in the content query model is used to process the sample content vector to obtain the predicted content image; Based on the sample content vector, sample query vector, sample content image, and predicted content image, the parameters of the content encoding unit, query encoding unit, and image generation unit are adjusted to obtain the trained content query model.

[0007] According to a third aspect of the embodiments of this specification, a content query device is provided, comprising: The first acquisition module is configured to acquire the target query data and the content to be queried. The first input module is configured to input the content to be queried into the content encoding unit in the content query model to obtain the content vector, and input the target query data into the query encoding unit in the content query model to obtain the query vector. The content query model includes a content encoding unit, a query encoding unit, and an image generation unit. The content encoding unit is trained based on the sample content image of the sample content and the predicted content image. The predicted content image is obtained by the image generation unit by processing the sample content vector of the sample content. The generation module is configured to generate content query results based on the content vector and the query vector.

[0008] According to a fourth aspect of the embodiments of this specification, a content query model training apparatus is provided, comprising: The second acquisition module is configured to acquire sample data, wherein the sample data includes sample content, sample content images of the sample content, and sample query data; The second input module is configured to input sample content into the content encoding unit in the content query model to obtain a sample content vector, and input sample query data into the query encoding unit in the content query model to obtain a sample query vector; The first processing module is configured to use the image generation unit in the content query model to process the sample content vector and obtain the predicted content image. The first adjustment module is configured to adjust the parameters of the content encoding unit, query encoding unit, and image generation unit based on the sample content vector, sample query vector, sample content image, and predicted content image, so as to obtain the trained content query model.

[0009] According to a third aspect of the embodiments of this specification, a computing device is provided, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the methods provided in the first or second aspect above.

[0010] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores a computer program / instructions that, when executed by a processor, implement the steps of the method provided in the first or second aspect described above.

[0011] According to a fifth aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the method provided in the first or second aspect described above.

[0012] This specification provides a content query method according to one embodiment, comprising: acquiring target query data and content to be queried; inputting the content to be queried into a content encoding unit in a content query model to obtain a content vector, and inputting the target query data into a query encoding unit in the content query model to obtain a query vector. The content query model includes a content encoding unit, a query encoding unit, and an image generation unit. The content encoding unit is trained based on a sample content image and a predicted content image of the sample content. The predicted content image is obtained by the image generation unit processing the sample content vector of the sample content. Based on the content vector and the query vector, a content query result is generated. Since the content encoding unit not only learns the surface semantics of the sample content but also undergoes supervised training using the sample content image and the predicted content image reconstructed by the image generation unit with the sample content vector as a condition, it corrects the text bias problem in the sample content vector. This allows the content vector to encode sufficiently rich and structured, high-dimensional semantic information to support visual reconstruction, thereby significantly improving the semantic integrity and discriminative ability of the content vector. Therefore, when generating content query results based on the content vector and the query vector, semantically relevant and high-quality content can be identified more accurately, significantly improving the accuracy of the content query results. Attached Figure Description

[0013] Figure 1 This is a flowchart illustrating a content query method provided in one embodiment of this specification; Figure 2 This is an architecture diagram of a content query system provided in one embodiment of this specification; Figure 3 This is a flowchart illustrating a content query model training method provided in one embodiment of this specification; Figure 4 This is a schematic diagram illustrating the processing procedure of a content query model training method provided in one embodiment of this specification; Figure 5 This is a flowchart of a content query method provided in one embodiment of this specification; Figure 6 This is a schematic diagram of the structure of a content query device provided in one embodiment of this specification; Figure 7 This is a schematic diagram of the structure of a content query model training device provided in one embodiment of this specification; Figure 8 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0014] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0015] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items. The term “at least one” in one or more embodiments of this application means “one or more,” and “a plurality of” means “two or more.” The term “comprising” is an open-ended description and should be understood as “including but not limiting,” and may include other content in addition to what has been described.

[0016] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0017] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0018] In one or more embodiments of this specification, a large model refers to a deep learning model with a large number of model parameters, typically containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even tens of trillions of model parameters. A large model can also be called a foundation model. It is pre-trained using large-scale unlabeled corpora to produce a pre-trained model with hundreds of millions of parameters. Such models can adapt to a wide range of downstream tasks and have good generalization ability. Examples include Large Language Models (LLMs) and multi-modal pre-training models.

[0019] In practical applications, large models only require a small number of samples to fine-tune the pre-trained model before they can be applied to different tasks. Large models can be widely used in fields such as Natural Language Processing (NLP) and Computer Vision. Specifically, they can be applied to computer vision tasks such as Visual Question Answering (VQA), Image Captioning (IC), and Image Generation, as well as natural language processing tasks such as text-based sentiment classification, text summarization, and machine translation. The main application scenarios of large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.

[0020] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0021] The Dual-Tower Model is a deep learning architecture widely used in recommendation systems, information retrieval, and semantic search. Its core idea is to encode two objects to be matched (such as users and items, queries and content) into vectors using two independent neural networks ("towers"), and then perform matching or recall based on similarity calculations within a shared semantic space.

[0022] Representation collapse refers to a situation where, during model training, all inputs (regardless of whether they are positive or negative samples) are mapped to almost the same embedding vector (or highly clustered in a very small subspace), thus superficially satisfying the condition of minimizing contrastive loss, but actually losing semantic discriminative ability.

[0023] Contrastive Estimation (InfoNCE) only requires positive samples to be more similar to negative samples in the current batch, without requiring global discriminative power. If the model compresses all outputs to a single point, then even slightly "slightly closer" positive samples can fool the loss function.

[0024] Notes are text, images, or other forms of content created by individuals or teams in their studies, work, or daily lives for purposes such as recording information, organizing thoughts, and providing reminders. Notes can range from a few simple lines of text to complex electronic content containing charts, links, multimedia files, and other elements. Notes are widely used in various life and work scenarios, such as knowledge point notes and lesson plan notes in learning and education, and dietary records, exercise records, and shopping sharing notes in health and lifestyle scenarios.

[0025] Deep self-attention (Transformer) models are network structures based on multi-head self-attention mechanisms, primarily used for processing sequential data. A Transformer model consists of repeatedly stacked encoder and decoder units. This design allows the Transformer to efficiently learn long-term dependencies, making it suitable for various natural language processing tasks, including machine translation, text summarization, and question answering systems.

[0026] Bidirectional Encoder Representations from Transformers (BERT) is a pre-trained NLP model. By learning from large amounts of unlabeled text data, this model can capture deep semantic information from text and achieves significant performance improvements on numerous NLP tasks.

[0027] Sentence embedding model (SEM): A deep learning model based on the Transformer architecture, specifically designed to encode entire sentences or text paragraphs into fixed-dimensional vector representations (i.e., sentence vectors). These vectors capture the semantic information of sentences, making semantically similar sentences closer together in the vector space, thereby supporting downstream tasks such as semantic search, text clustering, and sentence similarity calculation.

[0028] Autoregressive refers to a process in which a model predicts the next element one by one when generating sequences (such as text, speech, and time series), and uses the already generated parts as input conditions for subsequent predictions, forming a recursive process of "regressing itself".

[0029] Convolutional Neural Network (CNN): A type of deep neural network specifically designed to process data with a grid structure (such as images). Its core idea is to automatically extract local features using convolution operations and perform high-level semantic modeling through components such as pooling, non-linear activation, and fully connected layers.

[0030] The Vision Transformer (ViT) is a model that directly applies the Transformer architecture to image recognition tasks. It segments the image into fixed-size patches, linearly embeds each patch into a vector, and then processes it using a standard Transformer encoder.

[0031] Diffusion Model: A type of generative model that generates high-quality data (such as images, audio, etc.) by simulating a forward process of gradually adding noise (diffusion process) and a backward process of learning to denoise.

[0032] Latent Diffusion Model (LDM): This is an efficient and high-quality image generation diffusion model. Its core is that the diffusion process does not take place in pixel space, but in a compressed "latent space".

[0033] In search systems (such as search engines, recommendation systems, or question-answering systems), to efficiently find results relevant to user queries from massive amounts of content (Documents), a "two-tower model" architecture is typically used. This model vectorizes both the Query and Documents separately and performs fast retrieval based on vector similarity. This process is an indispensable key component in online services (i.e., systems that respond in real-time after a user's request). In the two-tower model, the Query and Documents are encoded independently, with no cross-attention or deep interaction mechanism between the two towers. The goal of each tower is to compress high-dimensional, complex inputs (such as a piece of text or a sequence of user behavior) into a fixed-dimensional vector (embedding), and use contrastive loss functions such as InfoNCE to shorten the distance between semantically positive sample pairs (relevant Query-Documents) and widen the distance between negative sample pairs (irrelevant Query-Documents).

[0034] However, in the above schemes, a large number of fine-grained, non-thematic semantic details are inevitably discarded, resulting in limited information integrity in content representation. Especially in multimodal content understanding scenarios, it is difficult to fully model and utilize visual information such as images, resulting in insufficient alignment between text and image semantics, specifically manifested in the following aspects: First, representation bias: the content vectors obtained by Documents pyramid encoding tend to prioritize encoding explicit textual semantics, while the modeling of visual information is insufficient and not robust. Second, shallow fusion: the model often only learns shallow modal associations at the lexical level (such as associating the text of "dog" with an image of a dog), but it is difficult to capture more fine-grained visual information that is difficult to describe in text (such as the material, gloss, style, atmosphere of an object, etc., and other deep semantics). Third, cross-modal retrieval bottleneck: when the query contains strong visual intent (such as "matte red lipstick" or "vintage film filter photo"), due to the lack or weakening of visual information in the content vector, it is impossible to accurately match results that meet visual requirements, and the query accuracy drops significantly.

[0035] To address the aforementioned issues, this specification proposes a content query method, comprising: acquiring target query data and content to be queried; inputting the content to be queried into a content encoding unit in a content query model to obtain a content vector, and inputting the target query data into a query encoding unit in the content query model to obtain a query vector. The content query model includes a content encoding unit, a query encoding unit, and an image generation unit. The content encoding unit is trained based on a sample content image of the sample content and a predicted content image. The predicted content image is obtained by processing the sample content vector of the sample content by the image generation unit. Based on the content vector and the query vector, a content query result is generated.

[0036] It is worth noting that the content query method proposed in the embodiments of this specification extends the dual-tower model to the multimodal domain. By introducing an image generation unit conditioned on the sample content vector at the back end of the content encoding unit, the image reconstruction task of the image generation unit (restoring the sample content image of the sample content) can serve as a powerful regularization method, imposing a strong cross-modal semantic constraint on representation learning. This constraint requires that the final sample content vector must contain rich visual information sufficient to reconstruct the sample content image; effectively guiding the content query model in the contrastive learning process to not only focus on textual semantics, but also actively focus on fine-grained visual information that is highly related to the image content. This implicitly and structurally prompts the content encoding unit to learn a unified, universal, high-fidelity dense representation that combines language understanding and visual perception, forcing the content vector to "must" learn to encode cross-modal alignment information, fully retain the key semantics in the multimodal input, and thus learn a more informative, accurate, and generalizable unified dense representation, significantly improving the accuracy of content queries for visual intent queries.

[0037] This specification provides a content query method, and also relates to a content query model training method, a content query device, a content query model training device, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0038] See Figure 1 , Figure 1 This specification shows a flowchart of a content query method according to an embodiment, which specifically includes the following steps: Step 102: Obtain the target query data and the content to be queried.

[0039] It's important to note that target query data refers to the original query content entered by the user during retrieval, used to express their information needs. Target query data can be used as input to the query encoding unit, and its semantics will be encoded into a query vector for subsequent similarity calculations. Target query data is typically text-based, including but not limited to keywords, natural language questions, or intent descriptions. For example, target query data could be "recommendations for refreshing sunscreens suitable for oily skin." Target query data can also be other modalities, such as voice, video, and images. When encoding target query data using the query encoding unit, the target query data can be converted to text before being input into the query encoding unit.

[0040] The content to be queried refers to the raw content waiting to be matched with the target query data. There can be one or more pieces of content to be queried. This content can be used as input to content encoding units, transformed into content vectors for subsequent sorting. The content to be queried can be unimodal, such as structured or unstructured text data, or multimodal multimedia content, meaning it simultaneously contains information in at least two modalities, such as images, text, video, and audio. Furthermore, the content to be queried can also cover other formats of content from any object within the content sharing platform, such as location information, group chat records, product entries, and virtual resources. The content to be queried can originate from various application scenarios, such as product detail page text and user reviews on e-commerce platforms, recommended notes and interactive comment data under notes on content sharing platforms, as well as other user-generated or system-generated content that integrates multiple media formats.

[0041] In practical applications, there are various ways to obtain the target query data and the content to be queried, and the specific method should be selected according to the actual situation. This specification does not impose any limitations on these methods in the embodiments. In one possible implementation of this specification, the target query data and the content to be queried can be read from the database of the content query system. In another possible implementation of this specification, the target query data and the content to be queried can be received from the user through a client.

[0042] Step 104: Input the content to be queried into the content encoding unit in the content query model to obtain the content vector, and input the target query data into the query encoding unit in the content query model to obtain the query vector. The content query model includes a content encoding unit, a query encoding unit, and an image generation unit. The content encoding unit is trained based on the sample content image of the sample content and the predicted content image. The predicted content image is obtained by the image generation unit by processing the sample content vector of the sample content.

[0043] It should be noted that a content query model refers to an end-to-end neural network architecture, which includes at least a content encoding unit, a query encoding unit, and an image generation unit. Optionally, the content query model may also include a summarization unit. For example, an image reconstruction decoder can be added after the content encoding tower of a dual-tower model for image reconstruction, thereby constructing a content query model; alternatively, a content query model can be built on top of a larger model. Through the content query model, efficient semantic retrieval can be achieved, and the representation quality can be improved through the image reconstruction task. During the training phase, the content query model can jointly optimize the contrastive loss (for retrieval) and the image loss (for image generation). Optionally, a summarization loss (for summarization) can be combined with the contrastive and image losses. During the inference phase, the content query model uses only the dual-tower components (i.e., the content encoding unit and the query encoding unit) for fast querying.

[0044] A content encoding unit (CIU) is a neural network module (such as BERT or Sentence Transformer) in a content query model responsible for mapping query content into dense vectors (i.e., content vectors). The CIU generates high-quality content vectors and receives supervised training signals from the image generation unit. The key role of the CIU lies not only in semantic compression but also in joint training using the difference between the predicted content image reconstructed by the image generation unit and the real sample content image as a supervisory signal. This forces the content vectors to retain sufficient visual information (such as color, texture, shape, and layout) to support the reconstructed sample content image, thereby improving the information density and generalization ability of the representation. Optionally, the CIU can also receive supervised training signals from the summarization unit.

[0045] A query encoding unit (QU) is a neural network module in a content query model responsible for encoding target query data into query vectors. Through the QU, user intent can be accurately characterized in a semantic space aligned with the content vectors. The QU typically shares the same structure or some parameters as the content encoding unit, ensuring consistency in their embedding spaces. For example, the QU can be a BERT model isomorphic to the content encoding unit.

[0046] The image generation unit (EGU) is a generative submodule in a content query model, typically composed of a deep neural network (such as CNN, Vision Transformer, or LDM). Its function is to automatically generate corresponding predicted content images based on sample content vectors (i.e., the output of the content encoding unit) as conditional inputs. During the training phase, the EGU provides a cross-modal reconstruction supervision signal by reconstructing real images semantically related to the sample content (i.e., sample content images). This signal is used to calculate the image loss and inversely optimize the content encoding unit, ensuring that its output content vector not only contains textual semantics but also includes visual features sufficient to support visual reconstruction.

[0047] It's worth noting that the image generation unit is not used to generate arbitrary images, but rather as a validator and regularizer for representation quality: only when the content vector fully encodes the multimodal semantics of the content (including implicit information that can be mapped to the visual space) can a predicted content image be reconstructed that is semantically or visually consistent with the real sample content image. Therefore, the image generation unit can force the content encoding unit to learn a high-fidelity, unified, and dense representation that aligns with language and vision through the strong constraint of "whether the image can be reconstructed from the vector." This effectively alleviates problems such as missing visual information, representation bias, and shallow cross-modal fusion in the traditional dual-tower model, thereby improving the query accuracy and robustness of the content query model in mixed text and image scenarios.

[0048] A content vector is a low-dimensional, dense representation output by a content encoding unit. In one embodiment of this specification, the content vector is further constrained to contain sufficient information to reconstruct the sample content image. Therefore, the content vector can not only encode the textual topic of the query content but also implicitly incorporate visual information (such as material, style, and layout) of the query content. This requirement for "reconstructability" significantly enhances the discriminative power and generalization of the content vector, enabling it to accurately match relevant content even when faced with queries containing visual intent.

[0049] A query vector is a low-dimensional, dense representation output by a query encoding unit, representing the semantic embedding of the target query data. Although the target query data itself is often plain text, the content vector already incorporates visual features, and the contrastive learning process can drive the query vector to automatically align to the enhanced semantic space corresponding to the content vector. For example, when the target query data is "matte black ceramic watch," the content query model can understand the visual and material meanings behind words like "matte" and "ceramic," and match content with corresponding image features. In other words, even if the target query data does not explicitly provide an image, the query vector can still serve as a crucial bridge connecting user intent and multimodal content. The semantic relevance between the content vector and the query vector can be calculated using cosine similarity or dot product to determine the content query results (such as a recall ranking list).

[0050] In practical applications, the query encoding unit can be trained based solely on the sample content vector and the sample query vector, or it can be trained by combining the sample content vector and the predicted content vector with the sample content summary and / or the sample content image and the predicted content image.

[0051] Step 106: Generate content query results based on the content vector and query vector.

[0052] It should be noted that content query results refer to the results returned after calculation and sorting based on content vectors and query vectors. A content query result can be at least one item from multiple query items that matches the target query data, or it can be a portion of a single query item that matches the target query data. Content query results can take various forms, such as a list of items matching the target query data, complete content, summary fragments, or structured responses with relevance scores, etc. Because the semantic integrity, cross-modal alignment capability, and information density of content vectors are optimized through image reconstruction tasks, content query results are generally more accurate, more discriminative, and more responsive to queries containing visual intent than the traditional dual-tower model.

[0053] For example, the target query data is "matte finish true red lipstick", and the content query result is "Product A: [Velvet Matte Lip Gloss - True Red], Product A description: High color payoff matte formula, soft matte finish, non-drying, long-lasting color lock. Related image: Close-up of a true red lipstick, showing a delicate, non-reflective matte texture."

[0054] It is worth noting that although an image generation unit is introduced during the training of the content query model to improve the semantic integrity and cross-modal representation capability of the content vector, this image generation unit does not participate in the actual inference stage (i.e., generating content query results). The content query model only uses a dual-tower structure composed of lightweight content encoding units and query encoding units to complete content queries. This retains the representational advantages brought by multimodal supervision while ensuring low latency and high throughput of the query service, achieving engineering and algorithmic co-optimization that enhances training and optimizes inference. The content query method proposed in the embodiments of this specification can be applied to scenarios such as search engines, e-commerce product recommendations, intelligent customer service question-and-answer matching, and personalized news push.

[0055] In practical applications, there are multiple ways to generate content query results based on content vectors and query vectors. The specific method chosen depends on the actual situation, and this specification does not impose any limitations on this approach. One possible implementation of this specification involves calculating the cosine similarity between the query vector and the content vector, and then using the cosine similarity score to recall or rank the content query results. Another possible implementation of this specification uses the dot product of the query vector and the content vector as the relevance score, and then uses the relevance score to recall or rank the content query results. Compared to cosine similarity, the dot product retains the vector magnitude information, which can reflect "confidence" or "importance."

[0056] By applying the scheme of the embodiments of this specification, since the content encoding unit not only learns the surface semantics of the sample content, but also performs supervised training through the sample content image and the predicted content image reconstructed by the image generation unit with the sample content vector as a condition, the text bias problem in the sample content vector is corrected. This allows the content vector to encode sufficiently rich and structured, high-dimensional semantic information that is sufficient to support visual reconstruction, thereby significantly improving the semantic integrity and discriminative ability of the content vector. Therefore, when generating content query results based on the content vector and the query vector, semantically relevant and high-quality content can be identified more accurately, significantly improving the accuracy of the content query results.

[0057] In one optional embodiment of this specification, generating content query results based on content vectors and query vectors may include the following steps: By comparing the content vector and the query vector, a vector similarity index is obtained, which reflects the degree of similarity between the content vector and the query vector. Content query results are generated based on vector similarity metrics.

[0058] It should be noted that vector similarity metrics are numerical indicators used to quantify the degree of similarity between content vectors and query vectors. Methods for obtaining vector similarity metrics by comparing content vectors and query vectors include, but are not limited to, methods based on cosine similarity, dot product, or Euclidean distance.

[0059] The solution implemented in this specification generates content query results based on the vector similarity index between the content vector and the query vector, achieving efficient and semantically aware retrieval capabilities. Since the content vector is constrained by the image reconstruction task of the image generation unit during the training phase, the similarity calculation can not only capture surface keyword matching, but also identify the consistency of visual features, thereby improving the relevance of the recall results and user satisfaction.

[0060] Regarding the training process of the content query model, in one optional embodiment of this specification, an auxiliary image reconstruction task can be introduced for the content encoding unit based on the Query-Documents contrastive learning loss function. This task requires the content query model to reconstruct the sample content image from the sample content vector. That is, the training method of the above-mentioned content query model can include the following steps: Obtain sample data, which includes sample content, sample content images, and sample query data; Input the sample content into the content encoding unit to obtain the sample content vector, and input the sample query data into the query encoding unit to obtain the sample query vector; The image generation unit is used to process the sample content vector to obtain the predicted content image; Based on the sample content vector, sample query vector, sample content image, and predicted content image, the parameters of the content encoding unit, query encoding unit, and image generation unit are adjusted to obtain the trained content query model.

[0061] It should be noted that sample data refers to supervised data used to train the content query model, which includes at least sample content, sample content images, and sample query data. There are usually multiple sample data sets. Within the same set of sample data, there is a correlation between sample query data and sample content. However, sample query data and sample content may or may not be correlated in different sets of sample data.

[0062] Sample content refers to labeled content samples used during the training phase of the content query model, typically accompanied by real sample content images. The sample content serves as input to the content encoding unit for learning semantic representations. Sample content can be unimodal or multimodal multimedia content. Furthermore, sample content can also encompass other formats of content from any object within the content sharing platform. Sample content can originate from various application scenarios, such as product detail page text and user reviews on e-commerce platforms, recommended notes and interactive comment data under notes on content sharing platforms, and other user-generated or system-generated content that integrates multiple media formats. Sample query data refers to user query text semantically related to the sample content, such as keywords in search logs, natural language questions, or queries corresponding to click behaviors.

[0063] A sample content vector is a vector representation output by a content encoding unit after processing sample content. On one hand, sample content vectors can be used for contrastive learning (calculating similarity with sample query vectors); on the other hand, they can serve as input conditions for image generation units and / or summarization units. A sample query vector is a vector representation output by a query encoding unit after processing sample query data.

[0064] Sample content images refer to real-world images that are semantically associated with sample content, such as product main images, article illustrations, cover images, or information visualizations. Sample content images can provide content query models with fine-grained visual information beyond text, serving as a supervisory source for achieving cross-modal alignment. Image loss can be calculated using sample content images, constraining the content vector to contain semantic information that can be mapped to the visual space.

[0065] The predicted content image refers to the image automatically generated by the image generation unit using the sample content vector as input. The predicted content image is used to calculate the image loss against the sample content image. In the embodiments of this specification, pixel-level precise alignment between the predicted content image and the sample content image is not required, but the predicted content image must be consistent with the sample content image in terms of semantics, structure, or key visual information (such as color distribution, object category, and texture).

[0066] The trained content query model refers to the content query model that has converged after joint optimization. During the inference phase, only the content encoding units and query encoding units in the trained content query model are used for efficient querying.

[0067] In practical applications, there are multiple ways to adjust the parameters of the content encoding unit, query encoding unit, and image generation unit based on the sample content vector, sample query vector, sample content image, and predicted content image to obtain a trained content query model. The specific method chosen depends on the actual situation, and this specification does not limit this approach. In one possible implementation, the contrast loss is calculated based on the sample content vector and sample query vector; the image loss is calculated based on the sample content image and predicted content image; a first total loss is calculated based on the contrast loss and image loss; and the parameters of the query encoding unit, content encoding unit, and image generation unit are adjusted according to the first total loss to obtain the trained content query model. In another possible implementation, after obtaining the first total loss, the parameters of the query encoding unit are adjusted using the contrast loss, and the parameters of the content encoding unit and image generation unit are adjusted according to the first total loss to obtain the trained content query model.

[0068] By applying the scheme of the embodiments in this specification, through joint optimization of contrastive learning and image reconstruction tasks, the content encoding unit not only needs to work with the query encoding unit to complete semantic alignment, but also needs to output a sample content vector that can be accurately reconstructed into a sample content image by the image generation unit. This forces the sample content vector to simultaneously encode textual semantics and key visual information. Although the query encoding unit does not directly access the sample content image, it implicitly aligns to the enhanced multimodal semantic space under the drive of contrastive learning, significantly improving the semantic representation ability and content query accuracy of the content query model. After training, the content query model can achieve high-precision semantic query in the inference stage with only lightweight dual-tower computation. This not only significantly improves the relevance of the content query results, but also greatly enhances its generalization performance due to the stronger information integrity and visual representation ability of the content vector. It can be seamlessly transferred to downstream tasks such as image and text classification, cross-modal clustering, and multimodal retrieval.

[0069] In one optional embodiment of this specification, the above-described parameter adjustment of the content encoding unit, query encoding unit, and image generation unit based on the sample content vector, sample query vector, sample content image, and predicted content image to obtain the trained content query model may include the following steps: Calculate the contrastive loss based on the sample content vector and the sample query vector; Calculate image loss based on sample content image and predicted content image; Based on the contrast loss, the parameters of the query encoding unit are adjusted, and based on the contrast loss and the image loss, the parameters of the content encoding unit and the image generation unit are adjusted respectively, to obtain the trained content query model.

[0070] It's important to note that contrastive loss (L_contrastive) is a loss value used to measure the semantic matching degree between sample content vectors and sample query vectors. Methods for calculating this loss include, but are not limited to, InfoNCE and the Cross-Entropy loss function (CE). In a batch of sample data, contrastive loss can shorten the vector distance between the sample query vector and relevant sample content, while simultaneously widening the vector distance between the sample query vector and irrelevant sample content. This drives the alignment of query encoding units and content encoding units in the shared semantic space, improving query relevance and matching.

[0071] Image loss (L_recon_image), also known as image reconstruction loss, is a loss value used to measure the degree of difference between the predicted content image and the sample content image. Methods for calculating image loss include, but are not limited to, the mean squared error loss function and the perceptual loss function. Image loss can serve as a regularization signal, constraining the sample content vector to contain the visual semantics necessary to reconstruct the sample content image, thus preventing information loss.

[0072] In practical applications, when adjusting the parameters of the content encoding unit and the image generation unit based on contrast loss and image loss respectively, the contrast loss and image loss can be weighted to obtain the first total loss. Based on the first total loss (L_total_1), the parameters of the content encoding unit and the image generation unit are adjusted separately to obtain the trained content query model. Here, L_total_1 = L_contrastive + λ_image * L_recon_image, where λ_image is a hyperparameter used to balance the relative importance of contrast loss and image loss.

[0073] The scheme implemented in this specification allows the parameters of the query encoding unit to be adjusted only by the contrast loss, while the parameters of the content encoding unit are adjusted by both the contrast loss and the image loss. This ensures that the sample query vector focuses on matching, while the sample content vector is forced to contain visual information sufficient to reconstruct the real sample content image, thereby effectively alleviating the semantic loss and representation bias caused by information compression in the traditional dual-tower model.

[0074] In the traditional dual-tower model, during input compression, in order to retain "general" or "discriminative" semantic features (such as topic and intent), it actively discards fine-grained, non-topic details that it considers "unimportant." This means the model may only learn to match the coarsest-grained topic and fail to perceive finer semantic differences. To address this issue, in one optional embodiment of this specification, an auxiliary summary reconstruction task is introduced for the content encoding unit. This task requires the content query model to reconstruct a sample content summary from the sample content vector. The summary reconstruction task can be combined with the image reconstruction task as a dual reconstruction task, allowing the content encoding unit to simultaneously learn a more general and dense vector representation of both textual and visual information. That is, the sample data also includes sample content summaries, and the content query model also includes a summary generation unit. Before adjusting the parameters of the content encoding unit, query encoding unit, and image generation unit based on the sample content vector, sample query vector, sample content image, and predicted content image to obtain the trained content query model, the following steps may also be included: The sample content vector is processed using a summary generation unit to obtain a predicted content summary; Based on the sample content vector, sample query vector, sample content image, and predicted content image, the parameters of the content encoding unit, query encoding unit, and image generation unit are adjusted to obtain the trained content query model, which may include the following steps: Based on the sample content vector, sample query vector, sample content image, predicted content image, sample content summary, and predicted content summary, the parameters of the content encoding unit, query encoding unit, image generation unit, and summary generation unit are adjusted to obtain the trained content query model.

[0075] It should be noted that the sample content summary refers to the real summary text paired with the sample content. It is usually written by humans or extracted automatically with high quality. It represents the core textual semantics of the sample content, such as the summary, title, topic, and entities, and is used to measure the "semantic integrity" of the sample content vector.

[0076] Predicted content summaries are summary texts generated by the summarization unit based on autoregression of the sample content vectors. The predicted content summaries are compared with the sample content summaries to calculate the summarization loss and inversely optimize the content encoding units. Higher quality predicted content summaries indicate more complete retention of semantic information from the sample content vectors.

[0077] A summary generation unit (SHU) is a generative decoder that generates text based on sample content vectors. During training, the SHU generates predicted content summaries from sample content vectors using an autoregressive approach and calculates a summary loss with the actual sample content summaries. This loss serves as another regularization signal for the content encoding unit, inversely constraining it to learn more semantically complete representations, effectively alleviating information bottlenecks and representation collapse problems. For example, the SHU can be a 12-layer Transformer Decoder or a 1.5B LLM Decoder. It's worth noting that although the SHU can be introduced during the training of the content query model to improve the semantic completeness of the content vectors, it does not participate in the actual inference process.

[0078] In practical applications, there are multiple ways to adjust the parameters of the content encoding unit, query encoding unit, image generation unit, and summary generation unit based on the sample content vector, sample query vector, sample content image, predicted content image, sample content summary, and predicted content summary to obtain a trained content query model. The specific method chosen depends on the actual situation, and this specification does not limit this approach. In one possible implementation, the contrast loss is calculated based on the sample content vector and sample query vector; the image loss is calculated based on the sample content image and predicted content image; the summary loss is calculated based on the sample content summary and predicted content summary; a second total loss is calculated based on the contrast loss, image loss, and summary loss; and the parameters of the query encoding unit, content encoding unit, image generation unit, and summary generation unit are adjusted according to the second total loss to obtain the trained content query model. In another possible implementation, after obtaining the second total loss, the parameters of the query encoding unit are adjusted using the contrast loss, and the parameters of the content encoding unit, image generation unit, and summary generation unit are adjusted according to the second total loss to obtain the trained content query model.

[0079] It's worth noting that a content query model can also include a content encoding unit, a query encoding unit, and a summary generation unit, but exclude an image generation unit. In this case, the training method for the content query model includes: inputting sample content into the content encoding unit to obtain a sample content vector, and inputting sample query data into the query encoding unit to obtain a sample query vector; using the summary generation unit to process the sample content vector to obtain a predicted content summary; and adjusting the parameters of the content encoding unit, query encoding unit, and summary generation unit based on the sample content vector, sample query vector, sample content summary, and predicted content summary to obtain the trained content query model. This approach allows the content vector to encode sufficiently rich semantic information capable of generating high-quality content summaries, focusing more on the core theme and key information of the query content, rather than noise or redundant details.

[0080] Applying the scheme of the embodiments in this specification, a summary generation unit conditioned on the sample content vector is introduced at the back end of the content encoding unit. The summary reconstruction task of the summary generation unit can serve as a powerful regularization method, forcing the sample content vector to contain sufficient information to reconstruct the content summary. This effectively alleviates the information bottleneck problem in the traditional dual-tower structure, avoids the loss of a large amount of information during compression, and transforms the sample content vector from a "dedicated vector" that only serves matching into a more general "semantically complete vector." At the same time, the summary reconstruction task provides a continuous supervision signal that does not rely on negative samples, even if the contrastive learning gets stuck. Even if the representation collapses, the content query model still needs to accurately complete the summary reconstruction task, thus fundamentally suppressing representation degradation. Furthermore, since summary generation itself requires the content query model to focus on key and discriminative content in the sample content, this process implicitly guides the content encoding unit to learn the core features needed to distinguish difficult negative samples, achieving implicit hard sample learning. Ultimately, the sample content vectors learned by the content query model not only perform better in query or retrieval tasks but also exhibit stronger cross-task generalization ability due to their semantic richness, effectively transferring to downstream tasks such as classification and clustering. It is worth mentioning that, compared to complex dynamic negative sampling strategies, the content query model training method proposed in this specification only requires adding a lightweight summary generation unit, with fixed and controllable training overhead, achieving a good balance between performance and efficiency.

[0081] In one optional embodiment of this specification, the above-described parameter adjustment of the content encoding unit, query encoding unit, image generation unit, and summary generation unit based on the sample content vector, sample query vector, sample content image, predicted content image, sample content summary, and predicted content summary to obtain the trained content query model may include the following steps: Calculate the contrastive loss based on the sample content vector and the sample query vector; Calculate image loss based on sample content image and predicted content image; Calculate the summarization loss based on the sample content summary and the predicted content summary; Based on the contrast loss, the parameters of the query encoding unit are adjusted, and based on the contrast loss, image loss, and summary loss, the parameters of the content encoding unit, image generation unit, and summary generation unit are adjusted respectively to obtain the trained content query model.

[0082] It should be noted that the summarization loss (L_recon_text), also known as the summarization reconstruction loss, is a loss value used to measure the degree of difference between the predicted content summary and the sample content summary. Methods for calculating the summarization loss include, but are not limited to, the cross-entropy loss function and the negative log-likelihood loss function (NLL). The summarization loss can serve as another regularization signal, constraining the sample content vector to contain sufficient semantic information to support accurate summary generation and prevent information loss.

[0083] In practical applications, when adjusting the parameters of the content encoding unit, image generation unit, and summarization unit based on contrast loss, image loss, and summarization loss respectively, a weighted average of these losses can be used to obtain a second total loss. Then, based on this second total loss (L_total_2), the parameters of the content encoding unit, image generation unit, and summarization unit are adjusted to obtain the trained content query model. It's worth noting that L_total_2 = L_contrastive + λ_text * L_recon_text + λ_image * L_recon_image. In this formula, λ_text and λ_image are hyperparameters used to balance the relative importance of the contrast loss, image loss, and summarization loss.

[0084] By applying the scheme of the embodiments in this specification, the content query model is jointly trained by triple supervision signals (i.e., contrast loss, image loss and summarization loss), which realizes multi-dimensional collaborative optimization of text, vision and semantic summarization, forces the content encoding unit to perform deeper and more effective intermodal alignment and fusion, solves the problem of modal excitation imbalance, and thus produces a truly "multimodal" representation with quality far exceeding that of the traditional dual-tower model.

[0085] Contrast loss drives precise alignment between the query encoding unit and the content encoding unit in the shared embedding space, ensuring query relevance. Image loss requires the sample content vector to contain sufficient visual semantics to reconstruct the sample content image, enhancing its cross-modal perception capability. Summarization loss requires the sample content vector to accurately reproduce the linguistic content of the sample content, ensuring that it retains core textual semantics such as title, topic, and entities. These three constraints work together on the content encoding unit, making its output sample content vector close to the matching sample query vector in the contrast space, and able to be decoded into a correct summary and a correct image. Thus, it far surpasses the traditional dual-tower model in terms of information completeness, semantic richness, and generalization ability.

[0086] In one optional embodiment of this specification, relying solely on unsupervised or weakly supervised similarity matching methods can easily lead to ambiguous or biased representations. Therefore, this embodiment introduces explicit sample association labels to provide clear learning objectives and ensure that sample content vectors and sample query vectors are truly "converged in the same direction and separated in opposite directions" semantically. That is, the calculation of contrastive loss based on sample content vectors and sample query vectors can include the following steps: Obtain sample association tags between sample content and sample query data; By comparing the sample content vector and the sample query vector, a sample similarity index is obtained, which reflects the degree of similarity between the sample content and the sample query data. The contrast loss is calculated based on the sample association labels and sample similarity indicators.

[0087] It should be noted that sample association labels are binary or multivariate labels used to indicate whether there is a correlation between sample content and sample query data (e.g., 1 indicates correlation, 0 indicates no correlation). Sample association labels can provide supervisory signals, clarifying which sample query data-sample content pairs are positive samples and which are negative samples, thereby guiding the calculation of contrastive loss.

[0088] Sample similarity metrics refer to numerical similarity scores between sample content vectors and sample query vectors, calculated using methods such as cosine similarity and dot product. These metrics quantify the semantic matching degree between the sample content vectors and sample query vectors, serving as a relevance score for model predictions and aligning with the true labels.

[0089] The scheme implemented in this specification introduces sample association labels as supervision signals and calculates contrast loss by combining sample similarity indicators. This enables the content query model to clearly distinguish between relevant and irrelevant sample query data-sample content pairs, thereby effectively aligning the semantic space during training. This not only improves the accuracy of content queries but also enhances the robustness of the content query model to noise and irrelevant content.

[0090] Considering the large number of parameters in the content query model and the limited computing resources on the client side, the content query method proposed in the embodiments of this specification can be applied to, for example... Figure 2 The content retrieval system shown is not limited to this. See also Figure 2 , Figure 2 This specification illustrates an architecture diagram of a content query system provided in one embodiment of the specification. The content query system may include a client 202 and a server 204. Client 202 is used to send the target query data and the content to be queried to server 204; Server 204 is used to input the content to be queried into the content encoding unit of the content query model to obtain a content vector, and input the target query data into the query encoding unit of the content query model to obtain a query vector. The content query model includes a content encoding unit, a query encoding unit, and an image generation unit. The content encoding unit is trained based on the sample content image and the predicted content image of the sample content. The predicted content image is obtained by the image generation unit by processing the sample content vector of the sample content. Based on the content vector and the query vector, the server generates the content query result and sends the content query result to the client 202. Client 202 is also used to receive content query results sent by server 204.

[0091] like Figure 2 As shown, the content query model is deployed on server 204. Server 204 can connect to one or more clients 202 via LAN, WAN, Internet, or other types of data networks. Data transmitted by client 202 may require encoding, transcoding, compression, or other processing before being published to server 204. Multiple clients 202 can establish communication connections through server 204. In the content query scenario, server 204 provides content query services between multiple clients 202. Each client 202 can act as a sender or receiver, communicating through server 204. Users can interact with server 204 through client 202 to receive data from other clients 202 or send data to other clients 202. In the content query scenario, a user can publish a data stream to server 204 through client 202. Server 204 generates content query results based on this data stream and pushes the results to other communicating clients.

[0092] Client 202 can be a browser, application (APP), web application such as HyperText Markup Language 5 (H5) application, or a lightweight application (also known as a mini-program) or cloud application, etc. Client 202 can be developed based on the software development kit (SDK) of the corresponding service provided by server 204, such as based on the Real-Time Communication (RTC) SDK. Client 202 can be deployed in electronic devices and depends on the device to run or on certain APPs on the device. Electronic devices may have a display screen and support information browsing, such as personal mobile terminals such as mobile phones, tablets, and personal computers (PCs). Various other types of applications can also be configured in electronic devices, such as human-computer interaction applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc. The client 202 can also interact with the user through a graphical user interface to invoke the content query model, thereby implementing the content query method provided in the embodiments of this specification.

[0093] Server-side 204 can include servers providing various services, such as servers providing communication services to multiple clients, servers supporting backend training of models used on clients, and servers processing data sent by clients. It should be noted that server-side 204 can be implemented as a distributed server cluster composed of multiple servers, or as a single server. The server can also be a server in a distributed system, or a server integrated with blockchain. The server can also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0094] It is worth noting that the content query method provided in the embodiments of this specification is generally executed by the server. However, in other embodiments of this specification, if the client's runtime resources can meet the deployment and runtime conditions of the content query model, the client can also have similar functions to the server, thereby executing the content query method provided in the embodiments of this specification. In other embodiments, the content query method provided in the embodiments of this specification can also be executed jointly by the client and the server.

[0095] See Figure 3 , Figure 3 This specification illustrates a flowchart of a content query model training method according to an embodiment, which specifically includes the following steps: Step 302: Obtain sample data, which includes sample content, sample content images, and sample query data.

[0096] Step 304: Input the sample content into the content encoding unit in the content query model to obtain the sample content vector, and input the sample query data into the query encoding unit in the content query model to obtain the sample query vector.

[0097] Step 306: Use the image generation unit in the content query model to process the sample content vector and obtain the predicted content image.

[0098] Step 308: Based on the sample content vector, sample query vector, sample content image, and predicted content image, adjust the parameters of the content encoding unit, query encoding unit, and image generation unit to obtain the trained content query model.

[0099] It should be noted that the implementation methods of steps 302 to 308 can refer to the training method of the above content query model, and will not be repeated in the embodiments of this specification.

[0100] The scheme implemented in this specification, on the one hand, utilizes sample query data and sample content to construct semantic matching signals, enabling precise alignment of the query encoding unit and content encoding unit in the shared embedding space, thereby improving the model's query relevance. On the other hand, it introduces an image generation unit to reconstruct sample content images based on sample content vectors, forcing the sample content vectors to contain sufficient visual features to reconstruct the sample content images. This transforms the image reconstruction task into a regularization mechanism for content encoding quality, thereby driving the content encoding unit to learn to encode cross-modal alignment information. Ultimately, the trained content query model not only exhibits higher accuracy and robustness in content query tasks, but its learned content vectors also possess stronger information integrity and visual representation capabilities, significantly enhancing its generalization performance. This allows for seamless transfer to downstream tasks such as image and text classification, cross-modal clustering, and multimodal retrieval.

[0101] See Figure 4 , Figure 4This diagram illustrates the processing flow of a content query model training method according to an embodiment of this specification. The content query model training method can be viewed as a model representation constraint method based on multimodal reconstruction enhancement, and can be divided into the following six stages: query encoding stage, content encoding stage, contrastive learning stage, image reconstruction stage, summary reconstruction stage, and model training stage. The image reconstruction stage and the summary reconstruction stage can be performed in parallel, constituting the multimodal reconstruction stage. The following sections will describe each of these six stages.

[0102] Query encoding stage: Input the sample query data into the query encoding unit in the content query model to obtain the sample query vector; Content encoding stage: Input the sample content into the content encoding unit in the content query model to obtain the sample content vector; Contrastive learning phase: Obtain sample association labels between sample content and sample query data; compare sample content vectors and sample query vectors to obtain sample similarity indicators, which reflect the degree of similarity between sample content and sample query data; calculate contrastive loss based on sample association labels and sample similarity indicators; Image reconstruction stage: The image generation unit in the content query model generates a predicted content image based on the sample content vector; based on the predicted content image and the sample content image of the sample content, the image loss is calculated; Summary reconstruction stage: The summary generation unit in the content query model generates a predicted content summary based on the sample content vector in an autoregressive manner; the summary loss is calculated based on the predicted content summary and the sample content summary of the sample content. Model training phase: Based on contrast loss, image loss, and summarization loss, a second total loss is calculated; according to the contrast loss, the parameters of the query encoding unit are adjusted, and according to the second total loss, the parameters of the content encoding unit, the summarization generation unit, and the image generation unit are adjusted respectively to obtain the trained content query model.

[0103] See Figure 5 , Figure 5 The flowchart of a content query method according to an embodiment of this specification is shown. During the content query process, the server and the client interact with each other. The client is used to send sample data to the server. The sample data includes sample content, sample content images, and sample query data. On the server side, sample content is input into the content encoding unit of the content query model to obtain a sample content vector, and sample query data is input into the query encoding unit of the content query model to obtain a sample query vector. The image generation unit in the content query model is used to process the sample content vector to obtain a predicted content image. Based on the sample content vector, sample query vector, sample content image, and predicted content image, the parameters of the content encoding unit, query encoding unit, and image generation unit are adjusted to obtain the trained content query model. The client is also used to send target query data and query content to the server.

[0104] On the server side, the content to be queried is input into the content encoding unit of the trained content query model to obtain a content vector, and the target query data is input into the query encoding unit of the trained content query model to obtain a query vector. Based on the content vector and the query vector, the content query result is generated and sent to the client.

[0105] Corresponding to the above-described embodiments of the content query method, this specification also provides embodiments of the content query device. Figure 6 A schematic diagram of the structure of a content query device according to one embodiment of this specification is shown. Figure 6 As shown, the device includes: The first acquisition module 602 is configured to acquire the target query data and the content to be queried. The first input module 604 is configured to input the content to be queried into the content encoding unit in the content query model to obtain the content vector, and input the target query data into the query encoding unit in the content query model to obtain the query vector. The content query model includes a content encoding unit, a query encoding unit, and an image generation unit. The content encoding unit is trained based on the sample content image of the sample content and the predicted content image. The predicted content image is obtained by the image generation unit by processing the sample content vector of the sample content. The generation module 606 is configured to generate content query results based on the content vector and the query vector.

[0106] Optionally, the device further includes: a second adjustment module configured to acquire sample data, wherein the sample data includes sample content, sample content images, and sample query data; inputting the sample content into a content encoding unit to obtain a sample content vector, and inputting the sample query data into a query encoding unit to obtain a sample query vector; processing the sample content vector using an image generation unit to obtain a predicted content image; and adjusting the parameters of the content encoding unit, the query encoding unit, and the image generation unit based on the sample content vector, the sample query vector, the sample content image, and the predicted content image to obtain a trained content query model.

[0107] Optionally, the second adjustment module is further configured to calculate the contrast loss based on the sample content vector and the sample query vector; calculate the image loss based on the sample content image and the predicted content image; adjust the parameters of the query encoding unit according to the contrast loss; and adjust the parameters of the content encoding unit and the image generation unit respectively according to the contrast loss and the image loss to obtain the trained content query model.

[0108] Optionally, the sample data also includes a sample content summary, and the content query model also includes a summary generation unit; the device further includes: a second processing module configured to process the sample content vector using the summary generation unit to obtain a predicted content summary; and a second adjustment module further configured to adjust the parameters of the content encoding unit, query encoding unit, image generation unit, and summary generation unit based on the sample content vector, sample query vector, sample content image, predicted content image, sample content summary, and predicted content summary to obtain a trained content query model.

[0109] Optionally, the second adjustment module is further configured to calculate the contrast loss based on the sample content vector and the sample query vector; calculate the image loss based on the sample content image and the predicted content image; calculate the summary loss based on the sample content summary and the predicted content summary; adjust the parameters of the query encoding unit according to the contrast loss; and adjust the parameters of the content encoding unit, the image generation unit, and the summary generation unit according to the contrast loss, the image loss, and the summary loss, respectively, to obtain the trained content query model.

[0110] Optionally, the second adjustment module is further configured to obtain sample association labels between sample content and sample query data; compare the sample content vector and the sample query vector to obtain a sample similarity index, wherein the sample similarity index is used to reflect the degree of similarity between sample content and sample query data; and calculate the contrast loss based on the sample association labels and the sample similarity index.

[0111] Optionally, the generation module 606 is further configured to compare the content vector and the query vector to obtain a vector similarity index, wherein the vector similarity index is used to reflect the degree of similarity between the content vector and the query vector; and to generate content query results based on the vector similarity index.

[0112] By applying the scheme of the embodiments of this specification, since the content encoding unit not only learns the surface semantics of the sample content, but also performs supervised training through the sample content image and the predicted content image reconstructed by the image generation unit with the sample content vector as a condition, the text bias problem in the sample content vector is corrected. This allows the content vector to encode sufficiently rich and structured, high-dimensional semantic information that is sufficient to support visual reconstruction, thereby significantly improving the semantic integrity and discriminative ability of the content vector. Therefore, when generating content query results based on the content vector and the query vector, semantically relevant and high-quality content can be identified more accurately, significantly improving the accuracy of the content query results.

[0113] The above is an illustrative scheme of a content query device according to this embodiment. It should be noted that the technical solution of this content query device and the technical solution of the above-described content query method belong to the same concept. For details not described in detail in the technical solution of the content query device, please refer to the description of the technical solution of the above-described content query method.

[0114] Corresponding to the above embodiments of the content query model training method, this specification also provides embodiments of the content query model training apparatus. Figure 7 A schematic diagram of a content query model training apparatus according to one embodiment of this specification is shown. Figure 7 As shown, the device includes: The second acquisition module 702 is configured to acquire sample data, wherein the sample data includes sample content, sample content images of the sample content, and sample query data; The second input module 704 is configured to input sample content into the content encoding unit in the content query model to obtain a sample content vector, and input sample query data into the query encoding unit in the content query model to obtain a sample query vector; The first processing module 706 is configured to use the image generation unit in the content query model to process the sample content vector and obtain the predicted content image. The first adjustment module 708 is configured to adjust the parameters of the content encoding unit, the query encoding unit, and the image generation unit based on the sample content vector, the sample query vector, the sample content image, and the predicted content image, so as to obtain the trained content query model.

[0115] The scheme implemented in this specification, on the one hand, utilizes sample query data and sample content to construct semantic matching signals, enabling precise alignment of the query encoding unit and content encoding unit in the shared embedding space, thereby improving the model's query relevance. On the other hand, it introduces an image generation unit to reconstruct sample content images based on sample content vectors, forcing the sample content vectors to contain sufficient visual features to reconstruct the sample content images. This transforms the image reconstruction task into a regularization mechanism for content encoding quality, thereby driving the content encoding unit to learn to encode cross-modal alignment information. Ultimately, the trained content query model not only exhibits higher accuracy and robustness in content query tasks, but its learned content vectors also possess stronger information integrity and visual representation capabilities, significantly enhancing its generalization performance. This allows for seamless transfer to downstream tasks such as image and text classification, cross-modal clustering, and multimodal retrieval.

[0116] The above is an illustrative scheme of a content query model training device according to this embodiment. It should be noted that the technical solution of this content query model training device and the technical solution of the content query model training method described above belong to the same concept. For details not described in detail in the technical solution of the content query model training device, please refer to the description of the technical solution of the content query model training method described above.

[0117] Figure 8 This specification illustrates a structural block diagram of a computing device according to one embodiment. The components of the computing device 800 include, but are not limited to, a memory 810 and a processor 820. The processor 820 is connected to the memory 810 via a bus 830, and a database 850 is used to store data.

[0118] The computing device 800 also includes an access device 840, which enables the computing device 800 to communicate via one or more networks 860. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 840 may include one or more of any type of wired or wireless network interface (e.g., Network Interface Card (NIC)), such as an IEEE 802.11 Wireless Local Area Networks (WLAN) interface, a Wi-MAX (World Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0119] In one embodiment of this specification, the above-described components of the computing device 800 and Figure 8 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 8 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0120] The computing device 800 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers. The computing device 800 can also be a mobile or stationary server.

[0121] The processor 820 is used to execute computer programs / instructions, which, when executed by the processor, implement the steps of the above-mentioned content query method or content query model training method.

[0122] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device belongs to the same concept as the technical solutions of the content query method and the content query model training method described above. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solutions of the content query method or the content query model training method described above.

[0123] An embodiment of this specification also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the above-described content query method or content query model training method.

[0124] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium belongs to the same concept as the technical solutions of the content query method and the content query model training method described above. Details not described in detail in the technical solution of the storage medium can be found in the descriptions of the technical solutions of the content query method or the content query model training method described above.

[0125] An embodiment of this specification also provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of the above-described content query method or content query model training method.

[0126] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product belongs to the same concept as the technical solutions of the content query method and the content query model training method described above. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solutions of the content query method or the content query model training method described above.

[0127] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0128] Computer instructions include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. Computer-readable media can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in computer-readable media can be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0129] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0130] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0131] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A content query method, characterized by, The method comprises: obtaining target query data and to-be-queried content; inputting the to-be-queried content into a content encoding unit in a content query model to obtain a content vector, and inputting the target query data into a query encoding unit in the content query model to obtain a query vector, wherein the content query model comprises the content encoding unit, the query encoding unit, and an image generation unit, the content encoding unit is trained based on a sample content image and a predicted content image of sample content, and the predicted content image is obtained by processing a sample content vector of the sample content by the image generation unit; generating a content query result according to the content vector and the query vector.

2. The method of claim 1, wherein, The training method of the content query model comprises: obtaining sample data, wherein the sample data comprises the sample content, the sample content image, and sample query data; inputting the sample content into the content encoding unit to obtain the sample content vector, and inputting the sample query data into the query encoding unit to obtain a sample query vector; processing the sample content vector by the image generation unit to obtain the predicted content image; adjusting parameters of the content encoding unit, the query encoding unit, and the image generation unit based on the sample content vector, the sample query vector, the sample content image, and the predicted content image to obtain a trained content query model.

3. The method of claim 2, wherein, The method of adjusting parameters of the content encoding unit, the query encoding unit, and the image generation unit based on the sample content vector, the sample query vector, the sample content image, and the predicted content image to obtain a trained content query model comprises: calculating a contrast loss based on the sample content vector and the sample query vector; calculating an image loss based on the sample content image and the predicted content image; adjusting parameters of the query encoding unit according to the contrast loss, and adjusting parameters of the content encoding unit and the image generation unit according to the contrast loss and the image loss respectively to obtain the trained content query model.

4. The method of claim 2, wherein, The sample data further comprises sample content abstract, and the content query model further comprises an abstract generation unit. The method further comprises: processing the sample content vector by the abstract generation unit to obtain a predicted content abstract before adjusting parameters of the content encoding unit, the query encoding unit, and the image generation unit based on the sample content vector, the sample query vector, the sample content image, and the predicted content image to obtain a trained content query model. The method of adjusting parameters of the content encoding unit, the query encoding unit, and the image generation unit based on the sample content vector, the sample query vector, the sample content image, and the predicted content image to obtain a trained content query model comprises: Based on the sample content vector, the sample query vector, the sample content image, the predicted content image, the sample content summary and the predicted content summary, the content encoding unit, the query encoding unit, the image generation unit and the summary generation unit are parameter adjusted to obtain a trained content query model.

5. The method of claim 4, wherein, The parameter adjustment of the content encoding unit, the query encoding unit, the image generation unit and the summary generation unit based on the sample content vector, the sample query vector, the sample content image, the predicted content image, the sample content summary and the predicted content summary to obtain a trained content query model comprises: Based on the sample content vector and the sample query vector, a contrast loss is calculated; Based on the sample content image and the predicted content image, an image loss is calculated; Based on the sample content summary and the predicted content summary, a summary loss is calculated; According to the contrast loss, the query encoding unit is parameter adjusted, and according to the contrast loss, the image loss and the summary loss, the content encoding unit, the image generation unit and the summary generation unit are respectively parameter adjusted to obtain the trained content query model.

6. The method according to claim 3 or 5, characterized in that, The calculation of the contrast loss based on the sample content vector and the sample query vector comprises: Obtaining a sample association label between the sample content and the sample query data; Comparing the sample content vector and the sample query vector to obtain a sample similarity index, wherein the sample similarity index is used to reflect the similarity between the sample content and the sample query data; According to the sample association label and the sample similarity index, the contrast loss is calculated.

7. The method of claim 1, wherein, The generation of the content query result according to the content vector and the query vector comprises: Comparing the content vector and the query vector to obtain a vector similarity index, wherein the vector similarity index is used to reflect the similarity between the content vector and the query vector; Based on the vector similarity index, the content query result is generated.

8. A content query model training method, characterized in that, Comprise: Obtaining sample data, wherein the sample data comprises sample content, a sample content image of the sample content and sample query data; Inputting the sample content into a content encoding unit in a content query model to obtain a sample content vector, and inputting the sample query data into a query encoding unit in the content query model to obtain a sample query vector; Using an image generation unit in the content query model to process the sample content vector to obtain a predicted content image; Based on the sample content vector, the sample query vector, the sample content image and the predicted content image, the content encoding unit, the query encoding unit and the image generation unit are parameter adjusted to obtain a trained content query model.

9. A computing device, comprising: Comprise: Memory and processor; The memory is configured to store computer programs / instructions, and the processor is configured to execute the computer programs / instructions, and the computer programs / instructions, when executed by the processor, implement the steps of the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer program / instruction is stored in the memory, and when executed by the processor, implements the steps of the method according to any one of claims 1 to 8.

11. A computer program product, characterised in that, The computer program / instruction is stored in the memory, and when executed by the processor, implements the steps of the method according to any one of claims 1 to 8.

Images