Large model user query extension problem recommendation method based on reinforcement learning
By generating candidate question token sequences through reinforcement learning-based RQ-KMeans and large-model inference, this method solves the semantic gap and complex intent modeling challenges in traditional methods for user intent recognition. It achieves efficient intent recognition and accurate recommendation for user queries, improving the accuracy and diversity of recommendation systems.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-25
- Publication Date
- 2026-03-13
AI Technical Summary
In existing technologies, traditional user intent recognition methods suffer from semantic gaps and difficulties in modeling complex intents through semantic similarity calculations. This leads to recommendation results that deviate from user interests or lack logical coherence, especially when faced with synonyms, misspellings, and long contextual dependencies.
The RQ-KMeans method is used for discretization encoding, which decomposes the semantic vector into multi-level residual vectors and generates token sequences through K-Means clustering. Candidate question token sequences are generated by combining large model inference, and the recommendation results are optimized through codebook retrieval and reinforcement learning.
It achieves efficient identification and accurate recommendation of user query intent, improves the accuracy and recall of recommendation results, and supports multi-path recall to enhance system robustness and the diversity of recommendation results.
Smart Images
Figure CN121658710A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of intelligent question answering technology, and in particular to a method for recommending extended questions based on a large model of user queries using reinforcement learning. Background Technology
[0002] With the development of technology, the existing user intent recognition and recommendation processes mainly face the following technical bottlenecks:
[0003] First, traditional keyword-based matching methods have a significant semantic gap problem. They cannot accurately understand the user's true intent based solely on keywords. When the user's input contains synonyms, typos, or other errors, they cannot accurately recommend semantically coherent follow-up questions.
[0004] Secondly, semantic similarity methods typically involve encoding user queries with a simple encoder and then directly calculating similarity. This "encoding-similarity" paradigm is difficult to effectively model the multi-layered semantic structure of complex intents. Especially when faced with semantic ambiguity, long contextual dependencies, and combined intents, the recommended follow-up questions often deviate from user interests or lack logical coherence.
[0005] Therefore, it is necessary to provide a large-scale user query extension question recommendation method based on reinforcement learning, so as to achieve efficient identification of user query intent and accurate recommendation of user questions and schedules. Summary of the Invention
[0006] The purpose of this invention is to provide a large-scale user query extension question recommendation method based on reinforcement learning, so as to achieve efficient identification of user query intent and accurate recommendation of user questions and schedules.
[0007] To address the problems existing in the prior art, this invention provides a large-scale user query extension problem recommendation method based on reinforcement learning, comprising the following steps:
[0008] The RQ-KMeans method is used for discretization encoding. In the encoding process, the last hidden layer of the large model is used to vectorize the text of each candidate question. The semantic vector is decomposed into multi-level residual vectors using residual quantization technology. The codebook center is learned for each layer of residual vectors through K-Means clustering. The continuous intent vector is transformed into a token sequence based on the codebook. The token sequence is the structured semantic representation of the candidate question.
[0009] Based on user query input, the system uses large-scale model inference to generate a sequence of candidate follow-up question tokens that the user is interested in. Then, through codebook retrieval, each generated candidate follow-up question token is mapped back to its corresponding codebook center vector, and the intent semantic vector is reconstructed by stacking them hierarchically.
[0010] After obtaining the semantic vector of the follow-up question, a similarity search is performed directly in the existing candidate question library to obtain the Top-K recommended follow-up questions; or, recommendation results are directly generated by combining prompt words with candidate follow-up question token sequences and their corresponding semantic information.
[0011] Optionally, the reinforcement learning-based large-model user query extension problem recommendation method further includes the following steps:
[0012] The GRPO algorithm is used to optimize large models, with recall rate and output format requirements as reward signals. Through training, the quality and recall of recommendation results are continuously improved.
[0013] Optionally, in the reinforcement learning-based large-model user query extension problem recommendation method, GRPO stands for Group Relative Policy Optimization.
[0014] Compared with the prior art, the present invention has the following advantages:
[0015] (1) This invention provides a large-scale user query extension question recommendation method based on reinforcement learning, which achieves efficient identification of user query intent and accurate recommendation of user questions and schedules. This invention generates a discrete token representation of intent through RQ-KMeans, uses a large model to generate a sequence of semantically related candidate question tokens for user queries, and optimizes the recommendation effect through reinforcement learning with verifiable rewards (RLVR), thereby alleviating the shortcomings of traditional question recommendation methods in terms of semantic understanding depth, recommendation accuracy, diversity, and contextual coherence.
[0016] (2) Improved recommendation performance: By combining discrete semantic encoding with large model reasoning, a more accurate semantic representation of intent was achieved, resulting in improved performance in terms of accuracy and recall of recommendation results.
[0017] (3) Supports flexible multi-path recall: The recommendation results obtained by this method can be seamlessly integrated into existing recommendation systems, serving as an independent recall path and merging with other recall strategies in parallel, thereby improving the robustness of the overall system and the diversity of recommendation results.
[0018] (4) Diversified recommendation results: Thanks to the reasoning and generation capabilities of large models, the recommendation system can not only recall the content in the existing candidate question library, but also generate new questions with similar semantics based on the recommendation questions to enrich the recommendation results. Attached Figure Description
[0019] Figure 1A flowchart illustrating the user query extension question recommendation method provided in this embodiment of the invention. Detailed Implementation
[0020] The specific embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. The advantages and features of the present invention will become clearer from the following description. It should be noted that the drawings are all in a very simplified form and use non-precise proportions, and are only used to facilitate and clarify the illustration of the embodiments of the present invention.
[0021] In the description of this application, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.
[0022] In the following, if the methods described herein include a series of steps, the order of these steps presented herein is not necessarily the only order in which these steps can be performed, and some of the steps described may be omitted and / or some other steps not described herein may be added to the method.
[0023] With the development of technology, the existing user intent recognition and recommendation process mainly faces the following technical bottlenecks: First, traditional keyword matching methods have obvious semantic gap problems. Keywords alone cannot accurately understand the true intent of the user query. When the user input contains synonyms, typos, etc., it is impossible to accurately recommend semantically coherent follow-up questions. Second, semantic similarity methods usually encode the user query with a simple encoder and then directly calculate the similarity. This "encoding-similarity" paradigm is difficult to effectively model the multi-level semantic structure of complex intents. Especially when facing semantic ambiguity, long context dependencies, and combined intents, the recommended follow-up questions often deviate from the user's interests or lack logical coherence.
[0024] To address the problems existing in the prior art, this invention provides a large-scale user query extension recommendation method based on reinforcement learning, such as... Figure 1 As shown, the method includes the following steps:
[0025] S1: The RQ-KMeans method is used for discretization encoding. In the encoding process, the last hidden layer of the large model is used to vectorize the text of each candidate question. The semantic vector is decomposed into multi-level residual vectors using residual quantization technology. The codebook center is learned for each layer of residual vectors through K-Means clustering. The continuous intent vector is transformed into a token sequence based on the codebook. The token sequence is the structured semantic representation of the candidate question.
[0026] RQ-Kmeans is a hierarchical encoding algorithm based on vector quantization (VQ). It is mainly used in generative recommendation systems to convert high-dimensional continuous vectors (embeddings) of items into discrete token sequences to adapt to the input requirements of autoregressive models such as Transformer.
[0027] K-Means is a classic unsupervised machine learning algorithm that clusters data through iterative grouping. The K-Means algorithm is a distance-based clustering method that groups data through the following steps. The algorithm principle is as follows:
[0028] Set the number of clusters K, and randomly initialize K centroids.
[0029] Calculate the Euclidean distance between the sample point and each centroid.
[0030] The samples are assigned to the nearest centroid to form a cluster.
[0031] Recalculate the cluster mean and update the centroid.
[0032] Iterate until the centroid stabilizes (converges).
[0033] A token is a digital identifier used in various scenarios within the computer field. In information technology, it refers to a random string or encrypted block of data used for authentication, secure access, or authorization control, representing authorization information for a user, device, or session. Common forms include JSON Web Token (JWT) and OAuth Token. In natural language processing, a token, as the smallest unit of text, can be a character or a sequence of characters, serving as the basic unit for model input and output processing.
[0034] S2: Based on user query input, the system uses large-scale model inference to generate a sequence of candidate follow-up question tokens that the user is interested in. Then, through codebook retrieval, each generated candidate follow-up question token is mapped back to its corresponding codebook center vector, and the intent semantic vector is reconstructed by stacking them hierarchically.
[0035] S3: After obtaining the semantic vector of the follow-up question, perform a similarity search in the existing candidate question library to obtain the Top-K recommended follow-up questions; or, generate recommendation results directly by combining prompt words with the candidate follow-up question token sequence and its corresponding semantic information.
[0036] Optionally, the reinforcement learning-based large-model user query extension problem recommendation method further includes the following steps:
[0037] S4: The GRPO (Group Relative Policy Optimization) algorithm is used to optimize the large model, with recall rate and output format requirements as reward signals. Through training, the quality and recall of recommendation results are continuously improved.
[0038] In summary, compared with the prior art, the present invention has the following advantages:
[0039] (1) This invention provides a large-scale user query extension question recommendation method based on reinforcement learning, which achieves efficient identification of user query intent and accurate recommendation of user questions and schedules. This invention generates a discrete token representation of intent through RQ-KMeans, uses a large model to generate a sequence of semantically related candidate question tokens for user queries, and optimizes the recommendation effect through reinforcement learning with verifiable rewards (RLVR), thereby alleviating the shortcomings of traditional question recommendation methods in terms of semantic understanding depth, recommendation accuracy, diversity, and contextual coherence.
[0040] (2) Improved recommendation performance: By combining discrete semantic encoding with large model reasoning, a more accurate semantic representation of intent was achieved, resulting in improved performance in terms of accuracy and recall of recommendation results.
[0041] (3) Supports flexible multi-path recall: The recommendation results obtained by this method can be seamlessly integrated into existing recommendation systems, serving as an independent recall path and merging with other recall strategies in parallel, thereby improving the robustness of the overall system and the diversity of recommendation results.
[0042] (4) Diversified recommendation results: Thanks to the reasoning and generation capabilities of large models, the recommendation system can not only recall the content in the existing candidate question library, but also generate new questions with similar semantics based on the recommendation questions to enrich the recommendation results.
[0043] The above are merely preferred embodiments of the present invention and do not constitute any limitation on the present invention. Any equivalent substitutions or modifications made by those skilled in the art to the technical solutions and content disclosed in the present invention without departing from the scope of the present invention shall be deemed to have remained within the protection scope of the present invention.
Claims
1. A recommendation method for large-scale user query extension problems based on reinforcement learning, characterized in that, Includes the following steps: The RQ-KMeans method is used for discretization encoding. In the encoding process, the last hidden layer of the large model is used to vectorize the text of each candidate question. The semantic vector is decomposed into multi-level residual vectors using residual quantization technology. The codebook center is learned for each layer of residual vectors through K-Means clustering. The continuous intent vector is transformed into a token sequence based on the codebook. The token sequence is the structured semantic representation of the candidate question. Based on user query input, the system uses large-scale model inference to generate a sequence of candidate follow-up question tokens that the user is interested in. Then, through codebook retrieval, each generated candidate follow-up question token is mapped back to its corresponding codebook center vector, and the intent semantic vector is reconstructed by stacking them hierarchically. After obtaining the semantic vector of the follow-up question, a similarity search is performed directly in the existing candidate question library to obtain the Top-K recommended follow-up questions; or, recommendation results are directly generated by combining prompt words with candidate follow-up question token sequences and their corresponding semantic information.
2. The large-scale user query extension problem recommendation method based on reinforcement learning as described in claim 1, characterized in that, It also includes the following steps: The GRPO algorithm is used to optimize large models, with recall rate and output format requirements as reward signals. Through training, the quality and recall of recommendation results are continuously improved.
3. The large-scale user query extension problem recommendation method based on reinforcement learning as described in claim 2, characterized in that, GRPO stands for Group Relative Policy Optimization.