Method and system for efficiently transmitting and indexing data based on data card

Abstract

The invention provides a data efficient transmission and indexing method and system based on a data card, and solves the technical problems of resource waste caused by point-to-point query and static list matching redundancy calling among network elements and model understanding deviation caused by data injection overrun in the long-period scene of multi-round dialogues in the prior art. The method comprises the following steps: acquiring a user task request, and decomposing to obtain sub-query problems; whether a corresponding data card exists or not is judged according to the sub-query problem, and if yes, the data card is quoted; otherwise, acquiring original data according to the sub-query problem, generating a data card, and injecting the reasoning context in a structured form; performing preliminary reasoning on the sub-query problem based on the data card; and when deep analysis is needed, indexing the original data, performing complementation reasoning, text analysis and logic judgment, and summarizing to obtain a final conclusion. The method can be widely applied to the technical field of data management.

Description

Technical Field

[0001] This application belongs to the field of data management technology, and more specifically, relates to a method and system for efficient data transfer and indexing based on data cards. Background Technology

[0002] In practical applications such as intelligent investment research, insurance customer service, and enterprise-level AI agents, systems often need to repeatedly access the same set of data in multi-round dialogues or long-cycle tasks. For example, when conducting fund analysis, it is necessary to repeatedly access the historical net asset value of the held products; when handling insurance inquiries, it is necessary to continuously provide feedback to customers regarding recurring questions about the same policy. However, existing systems generally adopt a model of re-calling external interfaces for each query to obtain data, resulting in the repeated retrieval of the same information, increasing database load, prolonging response latency, and wasting computing resources.

[0003] An existing Chinese invention patent with publication number CN119789128A provides a data reuse method, apparatus, electronic device, storage medium, and program product. It reduces redundant data collection and improves data service efficiency by reusing data service results stored on a data storage gateway. However, this patent's reuse relies on point-to-point queries and static list matching between network elements, resulting in low efficiency and flexibility for cross-scenario reuse. In high-concurrency, long-process, and multi-interaction business scenarios, frequent point-to-point queries between network elements generate significant communication overhead and nested call latency. Static list matching cannot dynamically adapt to changes in scenario load and heterogeneous resource characteristics, easily leading to response delays and excessive resource consumption. Consequently, query timeouts, invalid matching rules, and untimely data synchronization can cause data inconsistencies and mismatched reuse results. Furthermore, when injecting large amounts of raw data into the context of a large language model, this patent is prone to exceeding the context window limit, causing key information to be truncated or model misunderstandings.

[0004] Furthermore, due to the lack of an effective organization and semantic expression mechanism for queried data, the system struggles to efficiently reuse existing results in subsequent reasoning and often has to rely on cached key-value matching, failing to support data referencing based on natural language intent. Summary of the Invention

[0005] The purpose of this application is to provide a data card-based method and system for efficient data transmission and indexing, in order to solve the problems of redundant calls and wasted resources in point-to-point queries and static list matching between network elements, and model understanding deviations caused by excessive data injection in long-cycle scenarios with multi-turn dialogues in the prior art.

[0006] To achieve the above objectives, a first aspect of this application provides a method for efficient data transfer and indexing based on data cards, comprising the following steps: Obtain the user's task request and decompose it to obtain subquery questions; Determine whether there is a corresponding data card based on the subquery question. If there is, reference the data card; otherwise, obtain the original data based on the subquery question, generate a data card, and inject it into the reasoning context in a structured form. The data cards are used to perform preliminary reasoning on the subquery problem; when further analysis is needed, the original data is indexed, and after completing the reasoning, text analysis and logical judgment, the final conclusion is obtained.

[0007] Preferably, the process of generating data cards includes: extracting natural language summaries and data source information from the original data using a large language model based on preset prompt words; automatically generating data index IDs through code insertion or database insertion operations; generating timeliness tags based on the updated status according to the strategy; and obtaining data cards by combining natural language summaries, data source information, data index IDs, and timeliness tags.

[0008] Preferably, the original data is indexed by the data index ID, and after the large language model is called based on the preset prompt words to perform completion reasoning, text analysis and logical judgment, the reasoning results of all sub-query questions are summarized to obtain the final conclusion.

[0009] Preferably, the process of obtaining subquery questions includes: based on the steps of defining preset prompt words, clarifying the core requirements of the task request, decomposing the task request based on the core requirements, and extracting several executable subquery questions.

[0010] Preferably, the process of obtaining raw data includes: calling the key parameters required by the external service, and obtaining the raw data through the external service.

[0011] Preferably, the key parameters include the specific requirements of the subquery question, the name of the target service, and the calling tool.

[0012] Preferably, before referencing a data card, it is necessary to determine whether the timeliness tag of the data card has expired. If so, the original data is obtained according to the subquery question, and the data card is generated; otherwise, the data card is referenced.

[0013] The second aspect of this application provides a data card-based efficient data transmission and indexing system, including: a planner, an executor, an analyzer, a data card generation module, a data card caching layer, a data card referencing mechanism, and a data card parsing module; The planner is used to obtain user task requests and decompose them into subquery problems.

[0014] The executor is used to determine whether there is a corresponding data card based on the subquery question. If there is, the data card referencing mechanism references the data card from the data card cache layer; otherwise, it obtains the original data based on the subquery question and controls the data card generation module to generate the data card.

[0015] The analyzer is used for preliminary reasoning based on data cards; when in-depth analysis is required, the data card parsing module is used to index the original data, perform completion reasoning, text analysis, and logical judgment, and then summarize to obtain the final conclusion.

[0016] Preferably, the data card generation module is used to automatically generate data cards after the original data is retrieved for the first time based on the subquery question; The data card cache layer is used to temporarily store generated data cards; The data card referencing mechanism is used to reference existing data cards during the reasoning process; The data card parsing module is used to extract raw data based on the data index ID in the data card.

[0017] Preferably, the context manager is used to inject data cards into the context of a large language model in a structured form, avoiding the use of raw data. The Lifecycle Manager is used to automatically clean up, refresh, and archive data cards.

[0018] The beneficial effects of this application are as follows: This application proposes a method and system for efficient data transfer and indexing based on data cards. By converting the raw data returned by external services into lightweight data cards containing natural language summaries and unique index identifiers, the same data can be directly referenced in multi-round interactions and concurrent tasks, avoiding resource waste caused by repeated calls, and realizing efficient transfer and reuse of data information in the context of large language models; by injecting the generated data cards into the context of large language model inference in the form of structured text, retaining only key semantic information, it effectively alleviates context overflow caused by excessive raw data volume, and improves the understanding ability and inference stability of large language models of core data; by establishing a bidirectional mapping mechanism between data cards and raw data sources, it supports real-time backtracking of raw data through index when in-depth analysis is required, and combined with the timeliness tags of data cards, it realizes automatic refresh and lifecycle management of data cards, ensuring data consistency and system reliability. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This application provides an overall flowchart illustrating a data card-based method for efficient data transfer and indexing, as an embodiment of the present application. Figure 2 This is a schematic diagram of the overall process of a data card-based efficient data transmission and indexing system provided in an embodiment of this application. Detailed Implementation

[0021] To make the technical problems, technical solutions, and beneficial effects to be solved by this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the scope of this application.

[0022] This application introduces a novel data unit called data card, which transforms raw data into concise natural language descriptions and data index identifiers. This transformation method enables efficient information transfer within the context of a large language model, and allows for rapid backtracking to the original data through data indexes, achieving lightweight data transmission, efficient reuse, and traceable access, while reducing the waste of computing resources.

[0023] Please see Figure 1 The first embodiment of this application provides a method for efficient data transfer and indexing based on data cards, comprising: S1: Obtain the user's task request and decompose it to obtain subquery questions.

[0024] Users submit complex task requests to the system. The task requests enter the planner, which decomposes the task request based on preset prompts using a Large Language Model (LLM) and extracts several executable subquery questions from the task request.

[0025] In an optional embodiment, the user submits a task request to the system, "Please evaluate the performance of the funds I hold." After the request enters the planner, the planner, following the steps defined by the preset prompts, clarifies that the core requirement of the task request is "to comprehensively evaluate the profitability and market competitiveness of the user's held funds." Based on the core requirement of the task, the planner decomposes the task request "Please evaluate the performance of the funds I hold" and extracts several executable sub-query questions, such as "query detailed holding information of all funds held under the user's name," "obtain the performance data of each fund within a specified time dimension," "retrieve the performance benchmark of the corresponding fund and the average return level of similar funds in the same period," and "calculate the overall return and relative performance of the user's held funds."

[0026] S2: Determine if there is a corresponding data card based on the subquery question. If so, reference the data card.

[0027] The executor checks each subquery to see if a corresponding data card exists. If a data card exists and is not expired, the tool call is skipped, and the data card is referenced directly. The data card includes a natural language summary, a data index ID, data source information, and a timeliness tag. The data language summary briefly describes the core content of the data; the data index ID is a unique temporary or permanent ID that identifies the dataset; the data source information records the service, interface name, and parameters called; and the timeliness tag indicates whether the data has expired or needs to be refreshed.

[0028] In an optional embodiment, the executor determines the subquery "detailed holdings information of all funds held under a user's name" by searching the system for a matching data card. The search finds a data card corresponding to the subquery "detailed holdings information of all funds held under a user's name". The data card records the complete holding details of three funds held under the user's name: Fund A (code AAAAAA, 1000 units, cost 1.2 yuan / unit, purchase date January 10, 2025), Fund B (code BBBBBBB, 1500 units, cost 0.8 yuan / unit, purchase date March 15, 2025), and Fund C (code CCCCCC, 800 units, cost 1.5 yuan / unit, purchase date May 20, 2025). The data index ID is FC-USER-20251223-001; it calls the holdings query interface of the fund distribution platform. The input parameters are the user's unique identifier UID and the query timestamp, and the output parameters are the user's full fund holdings details. The data is valid until December 30, 2025, and the current status is not expired.

[0029] After evaluation, the executor determines that the data card perfectly matches the subquery "Query detailed holding information of all funds held under a user's name", and the timeliness label shows that it has not expired. Therefore, it skips the repeated calls to the fund distribution platform interface and directly references the holding information in the data card.

[0030] S3: If there is no data card that matches the subquery question, obtain the original data based on the subquery question, generate a data card, and inject it into the reasoning context in a structured form.

[0031] If no data card matching the subquery exists in the system, or if the data card is invalid, the executor confirms the key parameters required to call an external service (such as MCP), including the specific requirements of the subquery (query content), the target service name (server_name), and the calling tool (tools set). It obtains the raw data through the specified external service (such as MCP) and triggers the data card generation module. Based on preset prompts (such as "Please extract the natural language summary from the raw data," or "Data source information: {raw data}"), it uses a Large Language Model (LLM) to transform the obtained raw data into standardized data cards. These cards include a natural language summary, data source information, a data index ID automatically generated through code or database insertion operations, and a timeliness tag generated based on a strategy update status (such as a time threshold). The cards are then cached to lay the foundation for reuse in subsequent similar subquery problems. The newly generated data cards are injected into the inference context of the current task in a structured form for subsequent inference, avoiding repeated calls.

[0032] In an optional embodiment, for the subquery question "Get the performance data of each fund within a specified time dimension", the executor searches the system to see if there is a data card that matches the subquery question. If the search finds that there is no data card in the system corresponding to the subquery question "Get the performance data of each fund within a specified time dimension", the executor searches the system to see if there is a data card that matches the subquery question "Get the performance data of each fund within a specified time dimension".

[0033] The executor triggers the call process, confirming the key parameters required for the call: "Query the stage return, annualized return, and maximum drawdown of Fund A (code AAAAAA), Fund B (code BBBBBBB), and Fund C (code CCCCCC) over the past year," "Fund Performance Data Service Platform," and "Interface call protocol (HTTP / JSON) and parameter mapping rules (input parameter format for fund code and time dimension)." Based on these parameters, the external service MCP is called to obtain the raw performance data of the three funds: Fund A's stage return over the past year is 12.5%, annualized return is 12.2%, and maximum drawdown is 4.8%; Fund B's stage return over the past year is 8.3%, annualized return is 8.1%, and maximum drawdown is 3.2%; Fund C's stage return over the past year is -2.1%, annualized return is -2.2%, and maximum drawdown is 7.5%. After obtaining the raw data, the system converts this raw data into data cards.

[0034] This data card records the performance data of three funds, A, B, and C, over the past year, including their period return, annualized return, and maximum drawdown. Fund A showed the best performance, while Fund C had a negative return. The data index ID is FC-PERF-20251223-002. It uses the performance query interface of a fund performance data service platform, with input parameters being the fund code (AAAAAA / BBBBBB / CCCCCC) and the time dimension (past year), and output parameters being the fund's detailed performance data. The data is valid until January 23, 2026.

[0035] The executor stores the newly generated data card in the cache and synchronously injects it into the inference context of the current user request task "Please evaluate the performance of the funds I hold". When subsequent subquery tasks are executed, they can directly reference the data card from the context without repeatedly calling external services. This saves computing resources and reduces the model understanding bias caused by exceeding the limit of injecting a large amount of raw data.

[0036] S4: Perform preliminary reasoning on subquery questions based on data cards; when in-depth analysis is required, index the original data, call the large language model based on preset prompts to perform completion reasoning, text analysis and logical judgment, and summarize to obtain the final conclusion.

[0037] The analyzer first performs preliminary inference based on the brief descriptions of the natural language summaries in the data cards. When deeper analysis is required (such as anomaly analysis), it indexes the original data using the data index ID, calls the large language model based on preset prompts, and completes operations such as completion inference, text analysis, and logical judgment (e.g., filling the original data into preset prompts and using the large language model to obtain the corresponding analysis results). After all subqueries are completed, the system integrates the inference results from each stage, summarizes and generates the final conclusion, and feeds it back to the customer, completing the single task loop.

[0038] The system periodically checks the validity of data cards according to set strategies (such as time thresholds and data change notifications), and performs refresh or deletion operations on expired data cards to achieve dynamic updates, ensure the timeliness of data cards, and achieve system lightweighting.

[0039] In an optional embodiment, in response to the task request "Please evaluate the performance of the funds I hold," the system completes all subqueries and generates corresponding data cards. The analyzer then retrieves two core data cards from the context: the "Holdings Information Data Card" and the "Fund Performance Data Card." Based on the natural language summaries in the data cards, it performs preliminary inference to obtain the inference result. Specifically, it extracts the user's holdings and costs for Fund A, Fund B, and Fund C from the Holdings Information Data Card, and extracts the one-year returns and maximum drawdown rates of the three funds from the Fund Performance Data Card. The system preliminarily determines that Fund A's returns are greater than those of Funds B and C, and that Fund C has negative returns.

[0040] When in-depth analysis is required, the original data is indexed using the data index IDs of two data cards. The purchase time and share details of each fund are verified in the original holdings data. The monthly return volatility curves of the three funds and the performance benchmarks of similar funds are retrieved from the original performance data. Combining the original data, the analyzer completes the inference, text analysis, and logical judgment, obtaining the inference results (Fund A's high returns are accompanied by the largest drawdown, placing it at a medium risk level among similar funds; Fund B has stable returns but a poor benchmark performance; Fund C has negative returns and a drawdown rate higher than the average of similar funds). The system integrates the inference results of all sub-queries to generate a final conclusion including a holdings overview, performance comparison, and risk assessment.

[0041] Please see Figure 2 The second embodiment of this application provides a data card-based efficient data transfer and indexing system, which includes a planner, an executor, an analyzer, a data card generation module, a data card caching layer, a data card referencing mechanism, a data card parsing module, a context manager, and a lifecycle manager.

[0042] The planner is used to obtain user task requests and decompose them into subquery problems.

[0043] The executor is used to determine whether there is a corresponding data card based on the subquery question. If there is, the data card referencing mechanism references the data card from the data card cache layer; otherwise, it obtains the original data based on the subquery question and controls the data card generation module to generate the data card.

[0044] The data card generation module automatically generates data cards after the original data is retrieved for the first time based on a subquery. These cards include a natural language summary, a data index ID, and data source information. The natural language summary concisely describes the core content of the data; the data index ID is a unique temporary or permanent ID that identifies the dataset; the data source information records the service, interface name, and parameters used; and the timeliness tag indicates whether the data is expired or needs to be refreshed.

[0045] The data card cache layer is used to temporarily store generated data cards, supporting fast retrieval.

[0046] The data card referencing mechanism is used to reference existing data cards during the reasoning process.

[0047] The analyzer is used for preliminary reasoning based on data cards; when in-depth analysis is required, the data card parsing module is used to index the original data, perform completion reasoning, text analysis, and logical judgment, and finally obtain the final conclusion after summarizing.

[0048] The data card parsing module is used to extract raw data based on the data index ID for use by the analyzer.

[0049] The context manager is used to inject data cards into the context of a large language model in a structured form, avoiding the use of raw data.

[0050] The Lifecycle Manager is used to automatically clean up, refresh, and archive data cards.

[0051] This application introduces a data card mechanism that allows for one-time querying and multiple citations, significantly reducing redundant calls to external services. At the same time, it utilizes natural language summaries in the data cards to enhance the understanding of data by large language models, ensuring efficient transmission and indexing.

[0052] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0053] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A data card based data efficient transfer and indexing method, characterized by, Includes the following steps: Obtain the user's task request and decompose it to obtain subquery questions; Determine whether there is a corresponding data card based on the subquery question; if so, reference the data card. Otherwise, the original data is obtained based on the subquery question, the data card is generated, and injected into the reasoning context in a structured form; Based on the data card, a preliminary inference is made about the subquery problem; When further analysis is required, the original data is indexed, and after performing completion reasoning, text analysis, and logical judgment, the final conclusion is obtained.

2. The efficient data transfer and indexing method based on data cards as described in claim 1, characterized in that, The process of generating the data card includes: extracting natural language summaries and data source information from the original data using a large language model based on preset prompt words; automatically generating a data index ID through code insertion or database insertion operations; generating a timeliness tag based on the updated status according to the strategy; and obtaining the data card by combining the natural language summary, data source information, data index ID, and timeliness tag.

3. The efficient data transfer and indexing method based on data cards as described in claim 2, characterized in that, The original data is indexed by the data index ID. After the large language model is called based on the preset prompt words to perform completion reasoning, text analysis and logical judgment, the reasoning results of all the sub-query questions are summarized to obtain the final conclusion.

4. The efficient data transfer and indexing method based on data cards as described in claim 1, characterized in that, The process of obtaining the subquery question includes: based on the steps of defining preset prompt words, clarifying the core requirements of the task request, decomposing the task request based on the core requirements, and extracting several executable subquery questions.

5. The efficient data transfer and indexing method based on data cards as described in claim 1, characterized in that, The process of obtaining the raw data includes: calling the key parameters required by the external service, and obtaining the raw data through the external service.

6. The efficient data transfer and indexing method based on data cards as described in claim 5, characterized in that, The key parameters include the specific requirements of the subquery question, the name of the target service, and the calling tool.

7. The efficient data transfer and indexing method based on data cards as described in claim 1, characterized in that, Before referencing the data card, it is necessary to determine whether the timeliness tag of the data card has expired. If so, the original data is obtained according to the subquery question, and the data card is generated; otherwise, the data card is referenced.

8. A data card-based data efficient transfer and indexing system, applied to the data card-based data efficient transfer and indexing method of any one of claims 1-7, characterized in that, include: Planner, executor, analyzer, data card generation module, data card caching layer, data card referencing mechanism, and data card parsing module; The planner is used to obtain user task requests and decompose them into subquery problems. The executor is used to determine whether there is a corresponding data card based on the subquery question. If there is, the data card referencing mechanism references the data card from the data card cache layer. Otherwise, the original data is obtained based on the subquery question, and the data card generation module is controlled to generate the data card; The analyzer is used to perform preliminary reasoning based on the data card; when in-depth analysis is required, the original data is indexed by the data card parsing module, and after performing completion reasoning, text analysis, and logical judgment, the final conclusion is obtained.

9. The data card-based efficient data transmission and indexing system as described in claim 8, characterized in that, The data card generation module is used to automatically generate the data card after the original data is retrieved for the first time based on the subquery question; The data card cache layer is used to temporarily store the generated data cards; The data card referencing mechanism is used to reference existing data cards during the reasoning process; The data card parsing module is used to extract the original data based on the data index ID in the data card.

10. The data card based data efficient transfer and indexing system of claim 8, wherein, It also includes a context manager and a lifecycle manager; The context manager is used to inject the data cards into the context of the large language model in a structured form, avoiding the original data taking up space; The lifecycle manager is used to automatically clean, refresh, and archive the data cards.

Images