Semantic-based large model assistant optimization method and system in target range

By caching the output of large model in the shooting range and performing semantic similarity matching, the problems of slow streaming generation of smart assistants and high online call costs are solved, the response time and cache hit rate are improved, and the load and failure risk of large model are reduced.

CN119025666APending Publication Date: 2024-11-26BEIJING SAINING NETWORK SECURITY TECH CO LTD
View PDF 0 Cites 6 Cited by

Patent Information

Application Number
CN202411018331.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-29
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

In the shooting range, the smart assistant based on the big model has the problems of long streaming output time, high online call costs and large load on the big model, poor user experience and a single point of failure risk.

Method used

By cache the output of the big model and perform semantic similarity comparison, the calls to the big model are reduced, the user input and output are cached in KV form, and a new session is formed in combination with the user's historical session to match similarity. If the specified threshold is reached, the cache output is used, otherwise Request a big model and cache new output.

Benefits of technology

It improves user response time, reduces the probability of single point failure of large models, improves the cache hit rate and semantic context richness, and reduces the probability of matching errors.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119025666A_ABST
    Figure CN119025666A_ABST
Patent Text Reader

Abstract

The invention discloses a large model assistant optimization method and system based on semantics in a target range. The method comprises the steps that output of a large model called by a user is cached; performing similarity comparison on the current input of the user and the cached historical sessions, and if the cached historical sessions meeting conditions exist, acquiring cached large model output as reply information based on the cached historical sessions; otherwise, combining the user historical session with the current session to form a new session, performing similarity comparison based on the new session and the cached historical session, and judging whether the cached large model output can be obtained as reply information or not; and if the cache history session exceeding the specified threshold similarity is still not found when the cache retrieval end condition is met, requesting the large model and obtaining output according to the session currently input by the user, and outputting the cache large model. According to the method, the cache hit probability and accuracy can be improved, calling of the large model is reduced, the corresponding time of user input is shortened, and the probability of single-point failure of the large model is reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to a semantic-based large model assistant optimization method and system in a shooting range, belonging to the technical field of computer software and network security. Background Art

[0002] The cyber range is a platform that uses virtualization technology to simulate the real cyberspace attack and defense combat environment and can support various tests such as combat capability research and weapon equipment verification. When using the range, strong network and attack and defense related knowledge is required, which puts forward high requirements for users.

[0003] The big model plays a good role in understanding the user's artificial semantics and content generation. The RAG (Retrieval Enhanced Generation) knowledge base created by the big model can form a professional intelligent assistant with the operation methods of the shooting range and the necessary attack and defense knowledge, which can significantly reduce the user's usage cost. During use, users can use the intelligent assistant to search at any time and quickly obtain the required information, thereby improving the convenience and efficiency of operation.

[0004] At present, the intelligent assistants based on large models in the target range have the following problems: 1. When the large model executes output generation, it will only output one token each time. This streaming generation method often takes several seconds to output, which does not provide a good user experience; 2. When the online large model is called, it will be charged according to the token. Calling the large model to answer each question of the user will incur considerable expenses; 3. When using a local large model, calling the large model to answer each question of the user will increase the load of the large model, and the large model will become a single point bottleneck. Summary of the invention

[0005] Purpose of the invention: In view of the problems existing in the above-mentioned prior art, the purpose of the present invention is to provide a semantic-based large model assistant optimization method in a shooting range, which improves the cache hit probability and accuracy by caching the large model output and performing effective retrieval, reduces the call to the large model, improves the response time of user input, and reduces the probability of single point failure of the large model.

[0006] Technical solution: To achieve the above-mentioned invention object, the present invention adopts the following technical solution:

[0007] A semantic-based large model assistant optimization method in a shooting range comprises the following steps:

[0008] Cache the output of user calls to large models;

[0009] Compare the user's current input with the cached historical session for similarity. If there is a cached historical session with a similarity exceeding the specified threshold, obtain the cached large model output as the response information based on the cached historical session; otherwise, combine the user's historical session with the current session to form a new session, and compare the new session with the cached historical session for similarity. If there is a cached historical session with a similarity exceeding the specified threshold, obtain the cached large model output as the response information based on the cached historical session; if the user's historical sessions are traversed or the preset number of combinations is reached and no cached historical session with a similarity exceeding the specified threshold is found, request the large model and obtain the output based on the user's current input session, and cache the large model output.

[0010] Preferably, the large model output cache adopts KV format, where K is the vectorized representation of the user input question, and V is the output of the large model for the user input.

[0011] Preferably, the large model output corresponding to the historical conversation with the highest similarity exceeding the specified threshold similarity is selected as the reply information input by the user.

[0012] Preferably, a UserContext queue is generated for each user to record the conversation content of the most recent preset round; if the queue is full, the user input with the earliest timestamp is eliminated.

[0013] Preferably, when the large model is called to obtain output according to the user's current input session, the length of the user input is judged. If it is lower than a preset threshold, the most recent user input is taken from the user's historical session and combined with the current input to be used as an index for the large model output cache.

[0014] Preferably, the forming of a new session by combining the user's historical session with the current session is specifically as follows: traversing the user's historical session queue, taking the most recent session from the user's historical session queue each time, forming a new session with the user's current session, recording the current session as dialog, taking dialog1 for the first time, forming a new session dialog1+dialog, comparing dialog1+dialog with the cached historical session for similarity, if the similarity does not exceed a specified threshold, taking dialog2 for the second time, forming a new session dialog2+dialog1+dialog, and then comparing dialog2+dialog1+dialog with the cached historical session for similarity, and so on.

[0015] A semantic-based large model assistant optimization system in a shooting range, comprising:

[0016] The cache module is used to cache the output of the user calling the large model;

[0017] A semantic retrieval module is used to compare the user's current input with the cached historical session for similarity. If there is a cached historical session with a similarity exceeding a specified threshold, the cached large model output is obtained as response information based on the cached historical session; otherwise, a new session is formed by combining the user's historical session with the current session, and a similarity comparison is performed based on the new session and the cached historical session. If there is a cached historical session with a similarity exceeding a specified threshold, the cached large model output is obtained as response information based on the cached historical session; if the user's historical sessions are traversed or the preset number of combinations is reached and no cached historical session with a similarity exceeding the specified threshold is found, the large model is requested and the output is obtained based on the user's current input session, and the large model output is cached.

[0018] A computer program product includes a computer program / instruction, which, when executed by a processor, implements the steps of a semantic-based large model assistant optimization method in a shooting range.

[0019] Beneficial effects: Compared with the prior art, the present invention has the following advantages: 1. The present invention caches the output of the user calling the large model, compares the user's current input with the cached historical session for similarity, and selects the cache that meets the conditions as the reply information for the user input, which can reduce the call to the large model, improve the response time of the user input, and reduce the probability of single point failure of the large model. 2. The present invention caches the user's historical conversations, comprehensively utilizes multiple rounds of conversation information to form a context with richer semantics, and performs similarity matching based on this context, which can effectively improve the cache hit rate. 3. The present invention further effectively reduces the probability of cache matching errors by effectively identifying invalid semantics in user input when caching. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is an overall flow chart of an embodiment of the present invention.

[0021] Figure 2 It is a business flow chart of a specific session in an embodiment of the present invention. DETAILED DESCRIPTION

[0022] The technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings and specific embodiments.

[0023] like Figure 1As shown, a semantic-based big model assistant optimization method in a target range disclosed in an embodiment of the present invention includes the following steps: caching the output of the user calling the big model; comparing the user's current input with the cached historical session for similarity, if there is a cached historical session with a similarity exceeding a specified threshold, obtaining the cached big model output as reply information based on the cached historical session; otherwise, combining the user's historical session with the current session to form a new session, and comparing the similarity between the new session and the cached historical session, if there is a cached historical session with a similarity exceeding a specified threshold, obtaining the cached big model output as reply information based on the cached historical session; if the cache retrieval end condition is met, such as traversing the user's historical session or reaching a preset number of combinations, and still not finding a cached historical session that meets the condition, then requesting the big model and obtaining the output based on the user's current input session, and caching the big model output.

[0024] In the embodiment of the present invention, the large model output cache is represented in KV form. K is the vectorized representation of the user input question, and V is the output of the large model for the user input. For each user, in order to ensure the continuous availability of the cache, the conversation content of the most recent M rounds (such as 5 rounds) is recorded. That is, each user generates a historical conversation UserContext = {dialog1, dialog2, ..., dialogM}, where dialog1 represents the most recent cached session.

[0025] Combine the following Figure 2 The following is a detailed processing flow of a session service:

[0026] Step 1: The user's current round of conversation input is recorded as dialog, and the user's current round of input is vectorized. The obtained vector is recorded as D1, and the existing text vectorization tool can be used.

[0027] Step 2: Traverse the K values ​​in the model output cache and calculate the similarity between each K value and D1.

[0028] Step 3: Select the K value with the highest similarity in step 2 and determine whether its similarity exceeds a specified threshold value X (e.g., 80%). If the similarity exceeds the specified threshold value X, record the V corresponding to the K value as the output of the user's current input and jump to step 10; otherwise, jump to step 4.

[0029] Step 4: Combine the user's historical session information UserContext to build a new user session with context. Specifically, take the most recent session from UserContext each time and combine it with the user's current session to form a new session information dialog_new. The first time dialog1 is taken, a new session is formed.

[0030] dialog_new = dialog1 + dialog. The second time dialog2 is taken, a new session is formed

[0031] dialog_new = dialog2 + dialog1 + dialog. And so on; if all UserContexts have been traversed, jump to step 8.

[0032] Step 5: Vectorize the input of dialog_new and record the resulting vector as D2.

[0033] Step 6: Traverse the K values ​​in the model output cache and calculate the similarity between each K value and D2.

[0034] Step 7: Select the K value with the highest similarity in step 6 and determine whether its similarity exceeds the specified threshold X (such as 80%). If the similarity exceeds the specified threshold X, record the V corresponding to the K value as the output of the user's current input and jump to step 10; otherwise, jump to step 4.

[0035] Step 8: According to the conversation dialog currently input by the user, request the large model and obtain the output output, which is used as the output of the user's current input.

[0036] Step 9: Add a KV record to the cache. If the length of the current dialog is less than a specified threshold (such as 10 characters, in which case the user's input semantics is generally not rich enough), combine it with the previous input in the user's UserContext and use them together as the cached dialog, that is, cache K's dialog = dialog + dialog1. Where K = the vectorized representation of dialog, V = output, and jump to step 10.

[0037] Step 10: Update the UserContext queue, add the current round of user input to the queue, and eliminate the user input with the oldest timestamp.

[0038] Based on the same inventive concept, an embodiment of the present invention discloses a semantic-based big model assistant optimization system in a shooting range, including: a cache module, used to cache the output of the user calling the big model; a semantic retrieval module, used to compare the similarity between the user's current input and the cached historical session. If there is a cached historical session with a similarity exceeding a specified threshold, the cached big model output is obtained as reply information based on the cached historical session; otherwise, a new session is formed by combining the user's historical session with the current session, and a similarity comparison is performed based on the new session and the cached historical session. If there is a cached historical session with a similarity exceeding a specified threshold, the cached big model output is obtained as reply information based on the cached historical session; if the user's historical session is traversed or the preset number of combinations is reached and no cached historical session with a similarity exceeding the specified threshold is found, then according to the user's current input session, the big model is requested and the output is obtained, and the big model output is cached.

[0039] Based on the same inventive concept, a computer program product disclosed in an embodiment of the present invention includes a computer program / instruction, which, when executed by a processor, implements the steps of a semantics-based large model assistant optimization method in a target range.

Claims

1. A semantic-based large model assistant optimization method in a shooting range, It is characterized in that The steps include: Cache the output of user calls to large models; Compare the user's current input with the cached historical session for similarity. If there is a cached historical session with a similarity exceeding the specified threshold, obtain the cached large model output as the reply information based on the cached historical session; otherwise, combine the user's historical session with the current session to form a new session, and compare the new session with the cached historical session for similarity. If there is a cached historical session with a similarity exceeding the specified threshold, obtain the cached large model output as the reply information based on the cached historical session; If the cached historical session with a similarity exceeding the specified threshold is not found after traversing the user's historical sessions or reaching the preset number of combinations, the large model is requested and output is obtained based on the session currently input by the user, and the large model output is cached.

2. A semantic-based large model assistant optimization method in a shooting range according to claim 1, It is characterized in that The large model output cache uses KV format, where K is the vectorized representation of the user input question and V is the output of the large model for the user input.

3. The semantic-based large model assistant optimization method in a shooting range according to claim 1, It is characterized in that The large model output corresponding to the historical conversation with the highest similarity and exceeding the specified threshold similarity is selected as the reply information input by the user.

4. The semantic-based large model assistant optimization method in a shooting range according to claim 1, It is characterized in that A UserContext queue is generated for each user to record the conversation content of the most recent preset round; if the queue is full, the user input with the earliest timestamp is eliminated.

5. The semantic-based large model assistant optimization method in a shooting range according to claim 1, It is characterized in that When the large model is called to obtain output based on the user's current input session, the length of the user input is judged. If it is lower than the preset threshold, the most recent user input is taken from the user's historical session and combined with the current input to be used as the index of the large model output cache.

6. The semantic-based large model assistant optimization method in a shooting range according to claim 1, It is characterized in that The method of combining the user's historical session with the current session to form a new session specifically includes: traversing the user's historical session queue, taking the most recent session from the user's historical session queue each time, forming a new session with the user's current session, recording the current session as dialog, taking dialog1 for the first time to form a new session dialog1+dialog, comparing dialog1+dialog with the cached historical session for similarity, and if the similarity does not exceed a specified threshold, taking dialog2 for the second time to form a new session dialog2+dialog1+dialog, and then comparing dialog2+dialog1+dialog with the cached historical session for similarity, and so on.

7. A semantic-based large model assistant optimization system in a shooting range, It is characterized in that include: The cache module is used to cache the output of the user calling the large model; A semantic retrieval module is used to compare the user's current input with the cached historical session for similarity. If there is a cached historical session with a similarity exceeding a specified threshold, the cached large model output is obtained based on the cached historical session as the reply information; otherwise, a new session is formed by combining the user's historical session with the current session, and a similarity comparison is performed based on the new session and the cached historical session. If there is a cached historical session with a similarity exceeding a specified threshold, the cached large model output is obtained based on the cached historical session as the reply information; If the cached historical session with a similarity exceeding the specified threshold is not found after traversing the user's historical sessions or reaching the preset number of combinations, the large model is requested and output is obtained based on the session currently input by the user, and the large model output is cached.

8. A semantic-based large model assistant optimization system in a shooting range according to claim 7, It is characterized in that The large model output cache uses KV format, where K is the vectorized representation of the user input question and V is the output of the large model for the user input.

9. A semantic-based large model assistant optimization system in a shooting range according to claim 7, It is characterized in that When the large model is called to obtain output based on the user's current input session, the length of the user input is judged. If it is lower than the preset threshold, the most recent user input is taken from the user's historical session and combined with the current input to be used as the index of the large model output cache.

10. A computer program product comprising a computer program / instructions, It is characterized in that When the computer program / instructions are executed by a processor, the steps of a semantics-based large model assistant optimization method in a shooting range are implemented according to any one of claims 1 to 6.

Citation Information

Cited By

  • Large model assistant dynamic output rectification method and system in network target range

    CN119557526A

  • A large model assistant dynamic output rectification method and system in a network shooting range

    CN119557526B

  • Large language model efficient hybrid reasoning method and device based on routing technology and storage medium

    CN119961424A

  • Efficient hybrid reasoning method, device and storage medium for large language models based on routing technology

    CN119961424B

  • Retrieval method, device and equipment based on large model, medium and product

    CN121705395A