A method for aligning anchor points in an orchestration context scroll window and for prefix cache hits.
By using a step-aligned anchor point orchestration context scrolling window method, the problem of low cache hit rate in high-frequency lightweight orchestration calls is solved, achieving stable prefix caching and reduced orchestration decision costs, which is suitable for machine question answering and natural language query scenarios.
Patent Information
- Application Number
- CN202611324566.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-28
- Publication Date
- 2026-11-17
Smart Images

Figure CN123115331A_ABST
Abstract
Description
Technical Field
[0001] This invention mainly relates to the fields of artificial intelligence dialogue orchestration and natural language processing technology, specifically to an orchestration context scrolling window with aligned anchor points and a prefix cache hit method, which can be applied to scenarios such as machine question answering and intelligent customer service orchestration. Background Technology
[0002] In e-commerce multi-agent and human agent hybrid conversational systems, the orchestrator needs to frequently determine which agent should speak or perform an action next based on conversation events to support interactive tasks such as machine question answering and natural language queries. The orchestrator typically uses low-cost models for high-frequency calls and does not require streaming output. If each orchestration call sends the orchestration rules along with the entire set of historical messages or arbitrarily truncated historical messages as a whole to the model, the request prefix will constantly change, failing to reliably hit the prefix cache provided by the cloud vendor. This causes orchestration costs and latency to continuously worsen as the conversation lengthens.
[0003] In existing technologies, common context handling methods include retrieving the N most recent messages as context each time, replacing historical messages with digest compression, relying on cloud vendors' automatic caching mechanisms, and using the full context. In the method of retrieving the N most recent messages, the lower bound of the window slides as the maximum message sequence number decreases by N each time, causing the request prefix to change almost every round. Digest compression changes the semantic prefix, which is also detrimental to byte-level or token-level prefix caching. When relying on vendor-automatic caching, the cache hit rate is unstable if a stable prefix is not actively constructed. The full context method is costly and has high latency, making it unsuitable for high-frequency orchestration scenarios. In high-frequency, lightweight orchestration call scenarios, these methods struggle to maintain the stability of the request prefix while preserving sufficient context information, resulting in a low prefix cache hit rate and orchestration costs fluctuating with session growth. Summary of the Invention
[0004] 1. The technical problem that the invention aims to solve: This invention provides an orchestration context scrolling window with aligned anchor points and a prefix cache hit method. By aligning anchor points with step size, the orchestration context maintains a stable prefix within a step size, with new messages only appended to the tail, thereby improving the prefix cache hit rate and reducing the orchestration decision cost in machine question answering and natural language query scenarios.
[0005] 2. Technical Solution: To achieve the above objectives, the technical solution provided by this invention is: a method for aligning anchor points in an orchestration context scrolling window and for prefix cache hit detection, comprising the following steps: Get the ordered message log of the session and the current maximum message sequence number maxSeq; Set the window width W and step size S; Calculate the anchor start number (anchorStart) based on maxSeq, window width W, and step size S. Extract messages whose sequence numbers fall within the range of (anchorStart, maxSeq) and use them to form an orchestration context window; The stabilizing head is assembled with the orchestration context window into an orchestration request and sent to the orchestration model; During the lifetime of the same anchorStart, newly arriving messages are only appended to the end of the orchestration context window, while the stabilizing header and the existing front content in the orchestration context window remain unchanged, so that the prefix of the orchestration request hits the prefix cache; When maxSeq crosses the step size boundary, anchorStart jumps to the next alignment position, accepts a single prefix cache failure, and then enters a new stable interval.
[0006] Furthermore, the formula for calculating the anchor start number is as follows: ,in This indicates the floor function.
[0007] Furthermore, when the maxSeq is less than or equal to the window width W, the anchorStart value is 0, and the orchestration context window contains all messages from sequence number 1 to maxSeq.
[0008] Furthermore, in the step of capturing the orchestration context window, only messages that meet the orchestration visibility conditions are included. The orchestration visibility conditions include public dialogue messages and structured events, excluding private channel messages.
[0009] Furthermore, the stabilizing head includes orchestration rules and session roster or skill group information, and the stabilizing head remains fixed during the lifetime of the same anchorStart.
[0010] Furthermore, the assembly structure of the orchestration request is as follows: a stable header, the transcription content of the orchestration context window, and a short query; the short query is used to instruct the orchestration model to output an idle state or a expedited target.
[0011] Furthermore, in the step of capturing the arrangement context window, transient segments that are not yet finalized are not included in the arrangement context window.
[0012] Furthermore, the method is applied to the multi-agent orchestration scenario of e-commerce intelligent customer service, the orchestration model is a non-fluid model, the window width W is 200, and the step size S is 50.
[0013] Furthermore, when the anchorStart jump occurs, the new orchestration context window is a subset of the tail of the old orchestration context window, and recent messages within the overlapping area of the new and old windows maintain continuous orchestration decisions; the anchorStart jump is only triggered after the message is written to the database, and is atomically processed with the message writing to the database; after the jump, the prefix cache of the first orchestration request becomes invalid, and a new stable range is entered from the second orchestration request onwards.
[0014] The present invention also provides an orchestration context scrolling window and prefix cache hit system for aligning anchor points, including a message log acquisition module, an anchor point calculation module, a window capture module, a request assembly module, an orchestration model, and a prefix cache; The message log acquisition module is used to acquire the ordered message log of the session and the current maximum message sequence number maxSeq; The anchor point calculation module is used to calculate the anchor start number anchorStart based on maxSeq, window width W and step size S. The window capture module is used to capture messages whose message sequence number belongs to the range (anchorStart, maxSeq) to form an orchestration context window; The request assembly module is used to assemble the stabilizing head and the orchestration context window into an orchestration request and send it to the orchestration model. The orchestration model is used to output an orchestration decision based on the orchestration request; The prefix cache is used to cache the unchanging prefix token in the orchestration request; During the existence of the same anchorStart, newly arriving messages are only appended to the end of the orchestration context window, and the stabilization header and the existing front content in the orchestration context window remain unchanged, so that the prefix of the orchestration request hits the prefix cache; when the maxSeq crosses the step boundary, the anchorStart jumps to the next alignment position, accepts a single prefix cache failure, and enters a new stabilization interval.
[0015] 3. Beneficial effects: Compared with the prior art, the technical solution provided by this invention has the following advantages: Improve prefix cache hit rate: The anchor point remains unchanged within the step size, the request prefix is stable, and the window only adds new messages to the end. The stable header and the preceding text that has appeared in the window together constitute a cacheable prefix, which makes the prefix cache hit continuously. The hit rate is higher than that of the naive sliding window within the step size, which is suitable for high-frequency calling scenarios such as semantic processing and machine question answering. Smooth orchestration costs and controllable failures: High-frequency orchestration decisions can use low-cost models for a long time. The cost of a single orchestration session does not increase linearly with the session history. It only actively accepts a cache failure when crossing the step size boundary, and then enters a stable hit period again. The failure behavior is predictable, ensuring the response efficiency of interactions such as natural language queries. Decoupled from decision quality and supports optional noise reduction: still retains about W recent contexts to avoid misscheduling caused by focusing only on the last few messages; optional filtering of private messages and transient fragments to reduce irrelevant token occupation and erroneous decision signals; knowledge graph node references can be included in the stable header to assist in orchestration judgment. The forward jump transition is smooth and controllable. When the anchor point jumps forward, there is an overlap between the old and new windows. The orchestration decision relies on recent messages to maintain continuity and avoids abrupt changes. The forward jump and message storage are atomic, avoiding intermediate states. Single cache failures are predictable and can be further reduced in terms of impact on real-time response through background warm-up.
[0016] It should be noted that the structures not described in this invention are not related to the design points and improvement directions of this invention, and are the same as or can be implemented using existing technologies, and will not be elaborated here. Attached Figure Description
[0017] Figure 1 This is a schematic diagram comparing the prefix stability of the naive nearest N anchor points aligned with the step size in this invention; Figure 2 This is a schematic diagram illustrating the stepwise change of the anchor point of the present invention with the maximum message sequence number; Figure 3 This is a schematic diagram of the request layout and cacheable range of the present invention; Figure 4 This is a schematic diagram of the system module structure of the present invention; Figure 5 This is a schematic diagram of an embodiment of the e-commerce customer service arrangement of the present invention. Detailed Implementation
[0018] To facilitate understanding of the present invention, a more complete description of the invention will be given below with reference to the accompanying drawings, which illustrate several embodiments of the invention. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that the disclosure of the invention will be more thorough and complete.
[0019] See attached document Figure 1-5A system based on an orchestration context scrolling window and prefix cache hit method for aligning anchor points includes a message log acquisition module, an anchor point calculation module, a window truncation module, a request assembly module, an orchestration model, and a prefix cache. The message log acquisition module acquires the ordered message logs of the session and the current maximum message sequence number `maxSeq`. The anchor point calculation module calculates the anchor start number `anchorStart` based on `maxSeq`, the window width `W`, and the step size `S`. The window truncation module truncates messages whose sequence numbers fall within the range of `(anchorStart, maxSeq)`. The request assembly module assembles the stable header and the orchestration context window into an orchestration request. The orchestration model outputs orchestration decisions based on the orchestration request. The prefix cache caches the invariant prefix tokens in the orchestration request.
[0020] This embodiment takes the multi-agent orchestration scenario of e-commerce intelligent customer service as an example for detailed explanation. In e-commerce customer service conversation, there are multiple skill agents and human agents, covering machine question and answer and natural language query scenarios such as product consultation, order inquiry, and after-sales processing. The orchestrator needs to frequently determine who should answer the call next or whether it is necessary to expedite the process based on new messages. The orchestration call adopts a low-cost non-streaming model.
[0021] Step 1, Parameter Configuration Set the window width W to 200 and the step size S to 50. The window length is approximately 200 to 250 messages, and the anchor point jumps once every approximately 50 messages.
[0022] Step 2: Obtain the message log and current sequence number. The message log acquisition module retrieves ordered message logs from the session storage, with each message bearing a monotonically increasing sequence number (seq). The maximum sequence number of a message currently stored in the database is maxSeq.
[0023] Step 3: Calculate the starting number of the anchor point. The anchor point calculation module calculates anchorStart according to the formula:
[0024] When maxSeq is less than or equal to W, anchorStart takes the value of 0. At this time, the orchestration context window contains all messages from sequence number 1 to maxSeq. When maxSeq is greater than W, anchorStart is aligned to an integer multiple of step size S; For example, when maxSeq is 220, (220-200) / 50 = 0.4, rounded down to 0, anchorStart = 0; when maxSeq is 250, (250-200) / 50 = 1, anchorStart = 50; when maxSeq is 300, (300-200) / 50 = 2, anchorStart = 100; like Figure 2 As shown, anchorStart changes in a stepwise manner with maxSeq, remaining constant within a step size interval.
[0025] Step 4: Capture the layout context window The window capture module selects messages whose message sequence number falls within the range of (anchorStart, maxSeq). During the capture process, only messages that meet the orchestration visibility conditions are included, including public dialogue messages and structured events. Messages from private channels, such as internal private thoughts and tool original receipts, are excluded to avoid irrelevant content interfering with semantic processing and orchestration judgment. At the same time, transient fragments that are not yet finalized are not included in the orchestration context window, and only complete messages that have been stored in the database are retained.
[0026] Step 5, Assembly and Arrangement Request Request for assembly modules to be arranged in a three-section structure: The first section is the stable header, which includes the orchestration rules and conversation roster or skill set information, as well as optional product knowledge graph node references; The second section contains the transcription content arranged in the context window, sorted sequentially by message number (seq). The third segment is a short query used to instruct the orchestration model to output the idle state (IDLE) or the reminder target (NUDGE).
[0027] Step 6, prefix cache hit like Figure 1 and Figure 3 As shown, during the lifetime of the same anchorStart, the stable header remains unchanged, and the preceding text that has appeared in the orchestration context window also remains unchanged; only the tail grows as new messages arrive. The stable header and the preceding text that has already appeared in the window together form a cacheable prefix. This prefix is continuously hit in the prefix cache, and only new content at the end needs to be recalculated.
[0028] Step 7, Step Size Processing When maxSeq grows to the point where it crosses the next step size boundary, the anchor calculation module triggers an anchor jump. For example, anchorStart jumps from 50 to 100. The jump is only triggered after the new message is written to the database. It is atomically processed with the message writing and is not triggered during the transient sharding stage to avoid intermediate states.
[0029] After the previous hop, the old window is (50, maxSeq], and the new window is (100, maxSeq]. The content of the new window is the tail subset of the old window, and the overlapping interval is (100, maxSeq). The orchestration decision mainly depends on the recent messages within the overlapping interval. The discarded (50, 100] messages are older messages and have little impact on the current decision. Therefore, the orchestration decision remains continuous before and after the previous hop, without any abrupt changes.
[0030] After a hop-forward, the prefix of the first orchestration request changes overall, causing the prefix cache to fail once and requiring a complete recalculation. From the second request onwards, the system enters a new stable anchorStart interval, and the prefix cache recovers and consistently hits. Optionally, after a hop-forward, a low-priority warm-up call is initiated by the backend to pre-fill the new prefix cache, ensuring that the first real user request also hits the cache. Single cache failures across step sizes are predictable and controllable.
[0031] In the above process, compared with the naive method of taking the most recent N messages, the lower bound of the window in this invention is aligned by a step size S, rather than shifting for each message; compared with the summary compression method, this invention retains the original recent transcription to facilitate cache byte-level stability; compared with the full history method, this invention adopts a fixed window width, which is suitable for high-frequency orchestration scenarios.
[0032] In summary, this embodiment improves the prefix cache hit rate and smooths the orchestration cost by using step-size aligned anchor points to keep the orchestration context stable within a step size, with new messages only appended to the tail. Furthermore, the cache invalidation behavior is controllable, which can effectively support the high-frequency orchestration needs of natural language processing tasks such as machine question answering and natural language querying.
[0033] Extended solutions: The window width W and step size S can be adaptively adjusted according to the session activity level. When the session activity level is high, S is increased to reduce the frequency of anchor point forward jump. The stable header can be further divided into ultra-stable rules and semi-stable rosters, and only a partial failure occurs when the roster is changed; It can be combined with hard triggering and soft signal orchestration. Only soft triggering paths follow this scroll window, while hard triggering can skip the language model. The prefix cache hit rate and average orchestration time can be statistically analyzed and used as operational metrics to adjust parameters. The same anchor point formula can be applied to other high-frequency bypass models such as quality inspection coarse judgment and intent coarse classification, and is not limited to the speaking and naming scenario.
[0034] The above-described embodiments are merely illustrative of certain implementations of the present invention, and are described in a relatively specific and detailed manner. However, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements are all within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the appended claims.
Claims
1. A method for aligning anchor points in an arrangement context scrolling window and for prefix cache hit detection, characterized in that, Includes the following steps: Get the ordered message log of the session and the current maximum message sequence number maxSeq; Set the window width W and step size S; Calculate the anchor start number (anchorStart) based on maxSeq, window width W, and step size S. Extract messages whose sequence numbers fall within the range of (anchorStart, maxSeq) and use them to form an orchestration context window; The stabilizing head is assembled with the orchestration context window into an orchestration request and sent to the orchestration model; During the lifetime of the same anchorStart, newly arriving messages are only appended to the end of the orchestration context window, while the stabilizing header and the existing front content in the orchestration context window remain unchanged, so that the prefix of the orchestration request hits the prefix cache; When maxSeq crosses the step size boundary, anchorStart jumps to the next alignment position, accepts a single prefix cache failure, and then enters a new stable interval.
2. The method for aligning anchor points in an arrangement context scrolling window and for prefix cache hits according to claim 1, characterized in that: The formula for calculating the anchor start number is: ,in This indicates the floor function.
3. The method for aligning anchor points in an arrangement context scrolling window and for prefix cache hits according to claim 1, characterized in that: When the maxSeq is less than or equal to the window width W, the anchorStart value is 0, and the orchestration context window contains all messages from sequence number 1 to maxSeq.
4. The method for aligning anchor points in an orchestration context scrolling window and for prefix cache hits according to claim 1, characterized in that: In the step of capturing the orchestration context window, only messages that meet the orchestration visibility conditions are included. The orchestration visibility conditions include public dialogue messages and structured events, while private channel messages are excluded.
5. The method for aligning anchor points in an orchestration context scrolling window and for prefix cache hits according to claim 1, characterized in that: The stabilizing head includes orchestration rules and session roster or skill group information, and the stabilizing head remains fixed during the lifetime of the same anchorStart.
6. The method for aligning anchor points in an arrangement context scrolling window and for prefix cache hits according to claim 1, characterized in that: The assembly structure of the orchestration request is as follows: a stable header, the transcription content of the orchestration context window, and a short query; the short query is used to instruct the orchestration model to output an idle state or a expedited target.
7. The method for aligning anchor points in an arrangement context scrolling window and for prefix cache hits according to claim 1, characterized in that: In the step of capturing the arrangement context window, transient segments that are not yet finalized are not included in the arrangement context window.
8. The method for aligning anchor points, arranging context scrolling windows, and prefix cache hits according to claim 1, characterized in that: The method is applied to the multi-agent orchestration scenario of e-commerce intelligent customer service. The orchestration model is a non-streaming model, the window width W is 200, and the step size S is 50.
9. The method for aligning anchor points, arranging context scrolling windows, and prefix cache hits according to claim 1, characterized in that: When the anchorStart jump occurs, the new orchestration context window is a subset of the tail of the old orchestration context window, and recent messages within the overlapping area of the new and old windows maintain continuous orchestration decisions; the anchorStart jump is triggered only after the message is written to the database and is atomically processed with the message writing to the database; after the jump, the prefix cache of the first orchestration request becomes invalid, and a new stable range is entered from the second orchestration request onwards.
10. A system for aligning anchor points, arranging context scrolling windows, and a prefix cache hit mechanism, characterized in that: It includes a message log acquisition module, an anchor point calculation module, a window capture module, a request assembly module, an orchestration model, and a prefix cache; The message log acquisition module is used to acquire the ordered message log of the session and the current maximum message sequence number maxSeq; The anchor point calculation module is used to calculate the anchor start number anchorStart based on maxSeq, window width W and step size S. The window capture module is used to capture messages whose message sequence number belongs to the range (anchorStart, maxSeq) to form an orchestration context window; The request assembly module is used to assemble the stabilizing head and the orchestration context window into an orchestration request and send it to the orchestration model. The orchestration model is used to output an orchestration decision based on the orchestration request; The prefix cache is used to cache the unchanging prefix token in the orchestration request; During the existence of the same anchorStart, newly arriving messages are only appended to the end of the orchestration context window, and the stabilization header and the existing front content in the orchestration context window remain unchanged, so that the prefix of the orchestration request hits the prefix cache; when the maxSeq crosses the step boundary, the anchorStart jumps to the next alignment position, accepts a single prefix cache failure, and enters a new stabilization interval.
