A method and system for optimizing structured data computation results

Abstract

This invention provides a method and system for optimizing the computational results of structured data, relating to the field of data processing technology. The method includes: using a third-party loaded LLM to parse the target problem proposed by the target end, and determining a list of solution IDs for the target problem based on the parsing results, field information of the original data table, and the original function; determining the target solution based on the score of CPU resources occupied by each solution; obtaining the computational results of the new function included in the target solution on the structured data; and sending the optimized computational results to the target end. This invention uses a large language model to solve the target problem and reduces the resources required to execute the target solution.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method and system for optimizing the results of structured data calculations. Background Technology

[0002] In the field of data processing and analysis, especially when dealing with massive amounts of structured data, efficiently and accurately responding to complex computational problems posed by users is a key challenge. Current technologies typically use Excel spreadsheets to solve these problems, often employing functions to perform calculations on the data. However, manually editing functions requires proficiency in various Excel operations, is inefficient, and prone to errors. The explosive growth of natural language data has created the foundation for the development of large-scale language models, which have demonstrated remarkable capabilities in understanding and generating human language. Using large language models to understand and answer these problems using Excel spreadsheets, while optimizing the computational process to reduce resource consumption, remains an ongoing challenge. Summary of the Invention

[0003] To address the aforementioned technical problems, the technical solution adopted by this invention is as follows:

[0004] According to a first aspect of the present invention, a method for optimizing the results of structured data computation is provided, the method comprising the following steps:

[0005] S100 uses a third-party loaded LLM to parse the target problem proposed by the target end, and determines the list of solution IDs for the target problem based on the parsing results, field information of the original data table, and the original function;

[0006] The solution ID list includes the IDs of several solutions. Each solution includes a new function that is a combination of several original functions. Each original function has several original parameters, which are the original fields in the original data table.

[0007] The third party stores the field information of the original data table of the original function and the original data terminal that communicates with the third party; the field information includes the description information of the original data table and the original fields, and the original data table stores the structured data corresponding to the original fields;

[0008] S200 determines the target solution based on the score of CPU resources occupied by each solution;

[0009] S300: Obtain the calculation results of the new functions included in the target scheme on the structured data, and send the optimized calculation results to the target end;

[0010] The score for each solution's CPU resource usage is determined based on the time complexity of the change in the scale of the computational parameters in the new function's dominant operation, the parameter complexity of each computational parameter, and the function complexity of each computational function multiplied by the preset weight of each computational function by the corresponding number of executions.

[0011] The main operations include loops; the calculation functions include basic arithmetic operations; if the calculation parameters need to be obtained by formula calculation, the parameter complexity is the preset complexity of the corresponding formula, otherwise the parameter complexity is 1.

[0012] According to a second aspect of the present invention, a structured data computation result optimization system is provided, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the aforementioned method.

[0013] This invention has at least the following beneficial effects: using a third-party loaded LLM to determine a list of solution IDs for the target problem, the field information of the original data table, and the original functions, the target solution is determined based on the score of the CPU resources occupied by each solution, the calculation results of the new functions included in the target solution on the structured data are obtained, and the optimized calculation results are sent to the target. This invention determines the solution to the target problem by using a large language model, and further determines the target solution by the score of the solution, finds the best solution to the target problem, and reduces the resources required to execute the target solution. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 A flowchart of a method for optimizing structured data calculation results provided in an embodiment of the present invention. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar tasks and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0018] This invention provides a method for optimizing the results of structured data computation, such as... Figure 1 As shown, the method includes the following steps:

[0019] S100 uses a third-party loaded LLM to parse the target problem proposed by the target end, and determines a list of solution IDs for the target problem based on the parsing results, field information of the original data table, and original functions. The solution ID list includes several solution IDs, each solution includes a new function that is a combination of several original functions, and each original function has several original parameters, which are the original fields in the original data table. The solution ID is a unique identifier for the solution.

[0020] The third party stores the field information of the original data table of the original function and the original data terminal that communicates with the third party; the field information includes the description information of the original data table and the original fields, and the original data table stores the structured data corresponding to the original fields.

[0021] Specifically, a third party allocates a fixed-size storage space as the data storage space; the data storage space is divided into several volumes, each volume corresponding to a raw data endpoint, and the volume stores the field information of the raw data table of the raw data endpoint and the storage address of the raw fields in the raw data endpoint.

[0022] Specifically, instructions are sent to a third party that carries an LLM: based on the field information and primitive functions of the original data table, provide a solution to the target problem, and the solution uses only the primitive functions and fields. In one embodiment of the invention, when the LLM executes the instructions, field data is input to the LLM as context to facilitate the LLM's understanding.

[0023] It is understood that in this application, the third party communicates with the original data terminal. The third party only stores the field information of the original data table, ensuring that the information of the original data table can be used without occupying too much of the third party's storage space. When the third party platform needs to use structured data, it obtains it through the communication connection with the original data terminal.

[0024] S200 determines the target solution based on the score of CPU resources used by each solution.

[0025] S300: Obtain the calculation results of the new functions included in the target scheme on the structured data, and send the optimized calculation results to the target end.

[0026] The score for CPU resources used by each solution is determined based on the time complexity of the change in the scale of the computational parameters in the new function's dominant operation, the parameter complexity of each computational parameter, and the function complexity of the preset weight of each computational function multiplied by the corresponding number of executions.

[0027] The main operations include loops, embeddings, and recursion; the computation functions include basic arithmetic operations, comparisons, mathematical functions (SQRT), aggregate functions (SUM), lookup functions, etc.; if the computation parameters need to be obtained through formula calculation, the parameter complexity is the preset complexity corresponding to the formula; otherwise, the parameter complexity is 1. Basic arithmetic operations include, for example, addition and subtraction. Formulas include, for example, basic arithmetic operations. In one embodiment of the present invention, the preset complexity corresponding to the formula is equal to the preset weight of the computation function with the same formula.

[0028] In one embodiment of the present invention, the higher the time complexity, the higher the parameter complexity, and the higher the function complexity, the lower the score of the solution's CPU resource usage, and the solution with the highest score is determined as the target solution.

[0029] In one embodiment of the present invention, Big O notation is used to determine the time complexity of the dominant operation based on the change in the size of the computational parameters in the dominant operation; the size of the computational parameters x in the dominant operation is determined, with a single loop being O(x), and the time complexity of a single loop as the dominant operation being x; nested loops are O(k×x), where k is the number of nested levels, and the time complexity of nested loops is k×x; the time complexity of recursive operations is determined based on the depth of recursion.

[0030] Specifically, the weight for basic arithmetic operations is 1, the weight for comparisons is 2, the weight for the mathematical function SQRT is 5, the weight for the aggregate function SUM is the number of calculation parameters for the aggregate function, and the weight for the lookup function VLOOKUP is log2(n), where n is the number of calculation parameters for the lookup function.

[0031] In one embodiment of the present invention, optimizing the calculation result includes: issuing an instruction to the LLM: I have a calculation result for the target problem raised by the target end, please reply to the target end in combination with the calculation result of the target problem.

[0032] Specifically, if the target end is the original data end, S300 further includes: storing the target problem and its optimized calculation results in the volume corresponding to the original data end. Specifically, when the original data end raises a problem that is the same as or related to the target problem, the optimized calculation results of the target problem are invoked.

[0033] In summary, this invention uses a third-party loaded LLM to determine a list of solution IDs for the target problem, the field information of the original data table, and the original functions. Based on the CPU resource score of each solution, the target solution is determined. The calculation results of the new functions included in the target solution on the structured data are obtained, and the optimized calculation results are sent to the target. This invention uses a large language model to determine the solution to the target problem, and further determines the target solution based on the score of the solution, thus finding the best solution to the target problem and reducing the resources required to execute the target solution.

[0034] Specifically, the score for CPU resources used by the solution is equal to the weighted sum of the time complexity a1, the parameter complexity bi of all computational parameters, and the function complexity cj of all computational functions, where the weights corresponding to the parameter complexity of all computational parameters are equal, and the weights corresponding to the function complexity of all computational functions are equal.

[0035] Specifically, the weight corresponding to time complexity is denoted as the first weight y1, the weight corresponding to parameter complexity as the second weight y2, and the weight corresponding to function complexity as the third weight y3. The score for the CPU resources used by the solution is equal to y1×a1+y2×∑ m i=1 bi+y3×∑ n j=1 cj. bi is the parameter complexity of the i-th computational parameter, where i ranges from 1 to m and m is the number of computational parameters. cj is the function complexity of the j-th computational function, where j ranges from 1 to n and n is the number of computational functions.

[0036] Furthermore, for each solution, the score for CPU resources consumed is determined based on the time complexity of the change in the scale of computational parameters in the new function's dominant operation, the parameter complexity of each computational parameter, and the function complexity of the preset weight of each computational function multiplied by the corresponding number of executions. This also includes:

[0037] S210 determines the first influence coefficient of the dominant operation based on the actual number of operations performed in the programming code and the preset theoretical minimum number of operations. It can be understood that for the same operation, different programming codes have different effects on the algorithm logic. Theoretically, the minimum number of comparisons for bubble sort is O(n²), but in the actual code execution, there may be redundant comparisons, leading to additional loops.

[0038] S220 calculates the actual average clock cycle per single operation for each computation function.

[0039] S230 determines a second influence coefficient for each computation function based on the actual average clock cycle per single operation and the preset baseline clock cycle per single operation for that computation function. It can be understood that for the same computation function, the average clock cycle of the optimal implementation on the reference hardware is not the same as the one measured using a performance counter in actual operation.

[0040] S240 optimizes the score of CPU resources used by the solution based on the first influence coefficient of the dominant operation and the second influence coefficient of each calculation function.

[0041] In summary, based on the actual number of operations of the programming code for the dominant operation and the preset theoretical minimum number of operations, the first influence coefficient of the dominant operation is determined. The actual average clock cycle of a single operation for each calculation function is calculated. Based on the actual average clock cycle of a single operation for each calculation function and the preset baseline clock cycle of a single operation for that calculation function, the second influence coefficient of each calculation function is determined. Based on the first influence coefficient of the dominant operation and the second influence coefficient of each calculation function, the score of CPU resources occupied by the solution is optimized. By estimating the actual impact of the programming code, the influence coefficient is further determined, thereby more accurately determining the score of CPU resources occupied by the solution.

[0042] Furthermore, in S240, the optimization of the CPU resource usage score of the solution based on the first influence coefficient of the dominant operation and the second influence coefficient of each calculation function also includes:

[0043] S241, the weight corresponding to the time complexity is denoted as the first weight, the weight corresponding to the parameter complexity is denoted as the second weight, and the weight corresponding to the function complexity is denoted as the third weight.

[0044] S242, obtain the first product of time complexity, first weight, and first influence coefficient z1.

[0045] S243, obtain the parameter complexity of each calculation parameter and the second product value of the second weight.

[0046] S244, obtain the third product of the function complexity, third weight, and second influence coefficient of each computation function.

[0047] S245, the sum of the first product value, the second product value of all calculation parameters, and the third product value of all calculation functions is used as the optimized score of the CPU resources occupied by the solution.

[0048] Specifically, the CPU resource consumption of the optimized solution is equal to y1×a1×z1+y2×∑ m i=1 bi+y3×∑ n j=1 (cj×hj). hj is the second influence coefficient of the j-th calculation function.

[0049] Specifically, between S100 and S200, a feasibility screening of solutions is also included. This means discarding solutions that cannot be implemented using LLM.

[0050] Furthermore, the feasibility screening of solutions specifically includes:

[0051] S110, extract all original functions referenced by the new function in the solution, as well as the original parameters used by the original functions.

[0052] S120. If the original parameter does not exist in the field information of the original information table, discard this solution.

[0053] S130, if the input-output relationship information between the computation functions included in the new function is inconsistent, this solution is discarded. The input-output relationship information includes at least the output data type and the input data type. Specifically, if the output of computation function A is the input of computation function B, then the data type of the output data of computation function A and the data type of the input data of computation function B must be the same. If they are not the same, this solution is discarded.

[0054] Furthermore, feasibility screening of solutions also includes:

[0055] S140, for the original fields referenced in the solution, the third party requests a fixed amount of structured data from the original data source.

[0056] S150: A third party uses a fixed amount of structured data to execute a new function in a secure environment and captures any exceptions that occur during the function's runtime.

[0057] S160. If the execution fails or the error belongs to the preset error list, this solution will be discarded.

[0058] In summary, this invention extracts all original functions referenced by the new function in the solution, as well as the original parameters used by those original functions. If the original parameters are not found in the fields of the original information table, the solution is discarded. If the input-output relationships between the calculation functions included in the new function are inconsistent, the solution is also discarded. For the original fields referenced by the solution, the third party requests a fixed amount of structured data from the original data source. The third party uses this fixed amount of structured data to execute the new function in a secure environment and captures any exceptions encountered during runtime. If the execution fails or the exception falls within a pre-defined list of exceptions, the solution is discarded. This invention uses several methods to discard solutions that cannot be implemented, thereby more accurately determining the target solution.

[0059] Embodiments of the present invention also provide a structured data computation result optimization method system, including: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method provided in the above embodiments.

[0060] Embodiments of the present invention also provide a computer program product including program code, which, when the program product is run on an electronic device, causes the electronic device to perform the steps of the methods described above in various exemplary embodiments of the present invention.

[0061] While specific embodiments of the invention have been described in detail by way of examples, those skilled in the art should understand that the examples are for illustrative purposes only and are not intended to limit the scope of the invention. Those skilled in the art should also understand that various modifications can be made to the embodiments without departing from the scope and spirit of the invention.

Claims

1. A method for optimizing the results of structured data computation, characterized in that, The method includes the following steps: S100 uses a third-party loaded LLM to parse the target problem proposed by the target end, and determines the list of solution IDs for the target problem based on the parsing results, field information of the original data table, and the original function; The solution ID list includes the IDs of several solutions. Each solution includes a new function that is a combination of several original functions. Each original function has several original parameters, which are the original fields in the original data table. The third party stores the field information of the original data table of the original function and the original data terminal that communicates with the third party; the field information includes the description information of the original data table and the original fields, and the original data table stores the structured data corresponding to the original fields; S200 determines the target solution based on the score of CPU resources occupied by each solution; S300: Obtain the calculation results of the new functions included in the target scheme on the structured data, and send the optimized calculation results to the target end; The score for each solution's CPU resource usage is determined based on the time complexity of the change in the scale of the computational parameters in the new function's dominant operation, the parameter complexity of each computational parameter, and the function complexity of each computational function multiplied by the preset weight of each computational function by the corresponding number of executions. The main operations include loops; the calculation functions include basic arithmetic operations; if the calculation parameters need to be obtained by formula calculation, the parameter complexity is the preset complexity of the corresponding formula, otherwise the parameter complexity is 1.

2. The method for optimizing structured data calculation results according to claim 1, characterized in that, The score for CPU resources used by the solution is equal to the weighted sum of the time complexity, the parameter complexity of all computational parameters, and the function complexity of all computational functions, where the parameter complexity of all computational parameters has the same weight, and the function complexity of all computational functions has the same weight.

3. The method for optimizing structured data calculation results according to claim 2, characterized in that, Each solution's CPU resource allocation score is determined based on the time complexity of the changes in the computational parameter size during the new function's dominant operation, the parameter complexity of each computational parameter, and the function complexity of each computational function multiplied by its preset weight and the corresponding number of executions. This also includes: The first influence coefficient of the dominant operation is determined based on the actual number of operations performed in the programming code of the dominant operation and the preset theoretical minimum number of operations. Calculate the actual average clock cycle per operation for each computation function; The second influence coefficient of each calculation function is determined based on the actual average clock cycle of a single operation and the preset benchmark clock cycle of the calculation function. The score for CPU resources used by the solution is optimized based on the first influence coefficient of the dominant operation and the second influence coefficient of each calculation function.

4. The method for optimizing structured data calculation results according to claim 3, characterized in that, The optimization of the CPU resource usage score of the solution based on the first influence coefficient of the dominant operation and the second influence coefficient of each calculation function also includes: The weight corresponding to time complexity is denoted as the first weight, the weight corresponding to parameter complexity is denoted as the second weight, and the weight corresponding to function complexity is denoted as the third weight. Obtain the first product of the time complexity, the first weight, and the first influence coefficient; Obtain the parameter complexity and the second product of the second weight for each calculation parameter; Obtain the third product of the function complexity, third weight, and second influence coefficient of each computation function; The sum of the first product value, the second product value of all calculated parameters, and the third product value of all calculated functions is used as the optimized score of the CPU resources used by the solution.

5. The method for optimizing structured data calculation results according to claim 1, characterized in that, Between S100 and S200, there is also: Feasibility screening of solutions.

6. The method for optimizing structured data calculation results according to claim 5, characterized in that, The feasibility screening of solutions specifically includes: Extract all original functions referenced by the new function in the solution, as well as the original parameters used by the original functions; If the original parameters do not exist in the field information of the original information table, this solution will be discarded. If the input-output relationship information between the computational functions included in the new function is inconsistent, this solution should be discarded. The input-output relationship information includes at least the output data type and the input data type.

7. The method for optimizing structured data calculation results according to claim 5, characterized in that, Feasibility screening of solutions also includes: For the original fields referenced in the solution, the third party requests a fixed amount of structured data from the original data source. A third party uses a fixed amount of structured data to execute a new function in a secure environment and captures any exceptions that occur during the function's runtime. If the execution fails or the error belongs to the preset error list, this solution will be discarded.

8. The method for optimizing structured data calculation results according to claim 1, characterized in that, A third party allocates a fixed-size storage space as data storage space; The data storage space is divided into several volumes, each corresponding to a raw data endpoint. The volume stores the field information of the raw data table of the raw data endpoint and the storage address of the raw fields in the raw data endpoint.

9. The method for optimizing structured data calculation results according to claim 8, characterized in that, If the target is the raw data source, S300 also includes: The target problem and the optimized calculation results are stored in the volume corresponding to the original data.

10. A system for optimizing structured data computation results, comprising: A processor, a memory, and a computer program stored in the memory and executable on the processor, characterized in that, when the processor executes the computer program, it implements the structured data computation result optimization method as described in any one of claims 1-9.

Images