Task planning evaluation method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a task planning evaluation method and device, electronic equipment and a storage medium, and relates to the technical field of computers. The method comprises the steps that a query statement is clarified, and the clarified query statement, a first sequence, a second sequence, a third sequence and a fourth sequence are obtained; obtaining a target similarity between the first sequence and the third sequence according to the longest common subsequence between the first sequence and the third sequence and a part lacking specific information or having multiple interpretations in the query statement; based on the resources occupied by the third sequence, evaluating the resources occupied by the first sequence to obtain a first relative resource consumption degree; based on the resource occupied by the fourth sequence, evaluating the resource occupied by the second sequence to obtain a second relative resource consumption degree; and obtaining an evaluation value based on the target similarity, the first relative resource consumption degree and the second relative resource consumption degree. According to the method and the device, the rationality of obtaining the first sequence from multiple dimensions is evaluated.

Description

Technical Field

[0001] The present application relates to the field of computer processing technology, and in particular to a method, device, electronic device and storage medium for evaluating task planning. Background Art

[0002] Currently, in natural language to structured query and multi-agent task planning systems, for the evaluation of system execution results, in related technologies, most technologies usually perform a simple string comparison between the structured query results output by the system and the manually constructed standard answers to evaluate whether the task planning is reasonable. They only consider whether the final output is accurate, and there is a problem that the task planning cannot be evaluated from multiple dimensions, resulting in poor robustness and adaptability of the evaluation results. Summary of the Invention

[0003] The embodiments of the present application provide a method, device, electronic device and storage medium for evaluating task planning, which are used to solve the technical problem that task planning cannot be evaluated from multiple dimensions.

[0004] According to a first aspect of an embodiment of the present application, a task planning evaluation method is provided, the method comprising: obtaining a query statement, clarifying the query statement, obtaining the clarified query statement, a first sequence, and a second sequence, the first sequence comprising first operations executed on the clarified query statement, and the second sequence comprising second operations executed to clarify the query statement; Obtaining a third sequence and a fourth sequence, wherein the third sequence includes respective third operations for executing the query statement, and the fourth sequence includes respective fourth operations required for obtaining the third sequence by clarifying the query statement; Obtain the target similarity between the first sequence and the third sequence based on the part of the query sentence that lacks specific information or has multiple interpretations, and the longest common subsequence between the first sequence and the third sequence; Taking the resources occupied by the third sequence as a benchmark, the resources occupied by the first sequence are evaluated to obtain a first relative resource consumption degree of the first sequence; Taking the resources occupied by the fourth sequence as a benchmark, the resources occupied by the second sequence are evaluated to obtain a second relative resource consumption level of the second sequence; An evaluation value for evaluating the rationality of obtaining the first sequence is obtained based on the target similarity, the first relative resource consumption degree, and the second relative resource consumption degree.

[0005] According to a second aspect of an embodiment of the present application, there is provided a task planning evaluation device, the device comprising: a first processing module configured to obtain a query statement, clarify the query statement, and obtain a clarified query statement, a first sequence, and a second sequence, wherein the first sequence includes first operations performed on the clarified query statement, and the second sequence includes second operations performed on the clarified query statement; a second processing module, configured to obtain a third sequence and a fourth sequence, wherein the third sequence includes respective third operations for executing the query statement, and the fourth sequence includes respective fourth operations required for obtaining the third sequence by clarifying the query statement; a third processing module, configured to obtain a target similarity between the first sequence and the third sequence based on the portion of the query sentence that lacks specific information or has multiple interpretations and the longest common subsequence between the first sequence and the third sequence; a fourth processing module, configured to evaluate the resources occupied by the first sequence based on the resources occupied by the third sequence, and obtain a first relative resource consumption level of the first sequence; a fifth processing module, configured to evaluate the resources occupied by the second sequence based on the resources occupied by the fourth sequence, and obtain a second relative resource consumption level of the second sequence; The sixth processing module is configured to obtain an evaluation value for evaluating the rationality of obtaining the first sequence based on the target similarity, the first relative resource consumption degree, and the second relative resource consumption degree.

[0006] According to a third aspect of an embodiment of the present application, an electronic device is provided, which includes a memory, a processor, and a computer program stored in the memory, and when the processor executes the program, the steps of the method provided in the first aspect are implemented.

[0007] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method provided in the first aspect are implemented.

[0008] According to the fifth aspect of the embodiment of the present application, a computer program product is provided, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. When the processor of a computer device reads the computer instructions from the computer-readable storage medium, the processor executes the computer instructions, so that the computer device performs the steps of implementing the method provided in the first aspect.

[0009] The beneficial effects of the technical solution provided by the embodiments of the present application are: The task planning evaluation method provided in the embodiment of the present application obtains a query statement, clarifies the query statement, obtains the clarified statement, obtains a first sequence including various first operations of executing the clarified query statement, and obtains various second operations executed by the clarified query statement. This provides strong data support for the subsequent rationality of the first sequence obtained by task planning based on the query statement from multiple dimensions, and can realize the evaluation of the rationality of obtaining the first sequence from both an operation process-oriented and result-oriented perspective.

[0010] By obtaining a third sequence comprising each third operation for executing a query statement, and by obtaining a fourth sequence comprising each fourth operation required to clarify the query statement to obtain the third sequence, a reference standard is provided for evaluating the first sequence. By obtaining the third and fourth sequences, not only can the operations for executing the query statement in the first and third sequences be compared to evaluate the first sequence, but also the operations for clarifying the query statement in the second and fourth sequences can be compared to evaluate the first sequence.

[0011] The target similarity between the first and third sequences is obtained based on the parts of the query sentence that lack specific information or have multiple interpretations and the longest common subsequence that can characterize the similarity between the first and third sequences. The influence of fuzzy expression on the determination of the first sequence is taken into account, and the structural similarity between the first and third sequences is evaluated through the longest common subsequence. The target similarity is determined based on the fuzziness and similarity of the fuzzy expression. The fuzziness is used to adjust the evaluation criteria of the target similarity, thereby enhancing the robustness and adaptability of the overall evaluation.

[0012] By evaluating the resources occupied by the first sequence based on the resources occupied by the third sequence, the first relative resource consumption level of the first sequence is obtained. By using the obtained resource consumption of the third sequence as a reference benchmark, the resources occupied by the second sequence are evaluated, thereby achieving a unified evaluation for specific operation processes and improving the robustness and adaptability of the overall evaluation.

[0013] By taking the resources occupied by the fourth sequence as a benchmark, the resources occupied by the second sequence are evaluated to obtain the second relative resources of the second sequence. The impact of the resources consumed when clarifying the query statement on the rationality of the solution is considered. The resources consumed by the operation corresponding to the fourth sequence of the third sequence obtained by the clarifying query statement are used as the reference benchmark for evaluation. The resources consumed by the operation corresponding to the first sequence obtained by the clarifying query statement are evaluated. This achieves a unified evaluation of the clarification operation process, realizes the quantity of the clarification process, and improves the robustness of the overall evaluation.

[0014] By obtaining an evaluation value for evaluating the rationality of obtaining the first sequence based on target similarity, the first relative resource consumption degree, and the second relative resource consumption degree, the rationality of the operation of obtaining the first sequence based on the query task is evaluated from multiple dimensions, ensuring the robustness and adaptability of the evaluation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments of the present application.

[0016] Figure 1 A schematic diagram of the system architecture for implementing a task planning evaluation method provided in an embodiment of the present application; Figure 2 A flowchart of a task planning evaluation method provided in an embodiment of the present application; Figure 3 A flowchart of a method for determining target similarity in a task planning evaluation method provided in an embodiment of the present application; Figure 4 A flowchart of a method for determining the degree of fuzziness in a task planning evaluation method provided in an embodiment of the present application; Figure 5 A schematic flow chart of a method for obtaining a third sequence in a task planning evaluation method provided in an embodiment of the present application; Figure 6 A flowchart of a method for determining initial similarity in a task planning evaluation method provided in an embodiment of the present application; Figure 7 A flowchart of a method for determining a first relative resource consumption level in a task planning evaluation method provided in an embodiment of the present application; Figure 8 A flowchart of a method for determining resource consumption of a sequence in a task planning evaluation method provided in an embodiment of the present application; Figure 9 A flowchart of a method for determining a second relative resource consumption level of a sequence in a task planning evaluation method provided in an embodiment of the present application; Figure 10 A flowchart of another task planning evaluation method provided in an embodiment of the present application; Figure 11 A flowchart of another task planning evaluation method provided in an embodiment of the present application; Figure 12 A schematic diagram of the structure of a task planning evaluation device provided in an embodiment of the present application; Figure 13A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0017] The following describes the embodiments of the present application in conjunction with the accompanying drawings. It should be understood that the embodiments described below in conjunction with the accompanying drawings are exemplary descriptions for explaining the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions of the embodiments of the present application.

[0018] Those skilled in the art will understand that, unless otherwise stated, the singular forms "a", "an" and "the" used herein may also include plural forms. It should be further understood that the terms "including" and "comprising" used in the embodiments of the present application mean that the corresponding features can be implemented as the presented features, information, data, steps, operations, elements and / or components, but do not exclude implementation as other features, information, data, steps, operations, elements, components and / or combinations thereof supported by the present technical field. It should be understood that when we say that an element is "connected" or "coupled" to another element, the element can be directly connected or coupled to the other element, or it can refer to that the element and the other element establish a connection relationship through an intermediate element. In addition, the "connection" or "coupling" used here can include wireless connection or wireless coupling. The term "and / or" used here indicates at least one of the items defined by the term, for example, "A and / or B" can be implemented as "A", or as "B", or as "A and B".

[0019] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0020] The following is an explanation of the terms and related technologies involved in this application: Current technologies typically use metrics like Exact Match and BLEU to assess complete consistency with standard sentences. While simple, this approach is highly sensitive to formatting and wording, with even the slightest discrepancy considered a failure. It cannot accommodate semantically equivalent but structurally different solutions. Most methods ignore the rationality of the process path and focus solely on the correctness of the final output. There is a basic lack of evaluation methods for issues such as whether the task has undergone reasonable reasoning, whether there are redundant steps, and whether there is excessive resource consumption. When executing complex queries (such as JOIN, multi-layer nesting, etc.), the task execution cost is not estimated from the perspective of actual resources (such as CPU time, memory usage), and only a rough approximation is made based on the number of task steps or query length, which lacks measurability. User input often contains fuzzy semantics (such as "last month", "recent period", etc.). When dealing with fuzzy expressions, traditional methods either rely on manual rules or rely entirely on model "guessing". In the evaluation, the uncertainty brought by this fuzziness is not modeled and is not reflected in the accuracy or cost score. When the model cannot understand the user's intention, some systems clarify the semantics through human-computer interaction or empirical reasoning, but such behavior is usually regarded as "error handling" rather than part of the normal evaluation process, and the related costs and efficiency are not standardized.

[0021] As a result, relevant evaluation technologies remain at the stage of result-oriented and structural alignment, lacking a process-oriented approach to task rationality assessment, particularly lacking a unified mechanism for integrating accuracy, cost, ambiguity, and interactive clarification. These shortcomings limit the interpretability and optimizability of evaluation methods in multi-path, multi-modal, and multi-agent collaborative scenarios.

[0022] In response to at least one of the above-mentioned technical problems or areas that need improvement in the related art, the present application proposes a task planning evaluation method. The solution obtains a query statement and clarifies the query statement to obtain a clarified statement, obtains a first sequence including each first operation of executing the clarified query statement, and obtains each second operation performed by the clarified query statement. This provides strong data support for the subsequent rationality of the first sequence obtained from multiple dimensions for task planning based on the query statement, and can realize the evaluation of the rationality of obtaining the first sequence from both the operation process and the result perspective.

[0023] By obtaining a third sequence comprising each third operation for executing a query statement, and by obtaining a fourth sequence comprising each fourth operation required to clarify the query statement to obtain the third sequence, a reference standard is provided for evaluating the first sequence. By obtaining the third and fourth sequences, not only can the operations for executing the query statement in the first and third sequences be compared to evaluate the first sequence, but also the operations for clarifying the query statement in the second and fourth sequences can be compared to evaluate the first sequence.

[0024] The target similarity between the first and third sequences is obtained based on the parts of the query sentence that lack specific information or have multiple interpretations and the longest common subsequence that can characterize the similarity between the first and third sequences. The influence of fuzzy expression on the determination of the first sequence is taken into account, and the structural similarity between the first and third sequences is evaluated through the longest common subsequence. The target similarity is determined based on the fuzziness and similarity of the fuzzy expression. The fuzziness is used to adjust the evaluation criteria of the target similarity, thereby enhancing the robustness and adaptability of the overall evaluation.

[0025] By evaluating the resources occupied by the first sequence based on the resources occupied by the third sequence, the first relative resource consumption level of the first sequence is obtained. By using the obtained resource consumption of the third sequence as a reference benchmark, the resources occupied by the second sequence are evaluated, thereby achieving a unified evaluation for specific operation processes and improving the robustness and adaptability of the overall evaluation.

[0026] By taking the resources occupied by the fourth sequence as a benchmark, the resources occupied by the second sequence are evaluated to obtain the second relative resources of the second sequence. The impact of the resources consumed when clarifying the query statement on the rationality of the solution is considered. The resources consumed by the operation corresponding to the fourth sequence of the third sequence obtained by the clarifying query statement are used as the reference benchmark for evaluation. The resources consumed by the operation corresponding to the first sequence obtained by the clarifying query statement are evaluated. This achieves a unified evaluation of the clarification operation process, realizes the quantity of the clarification process, and improves the robustness of the overall evaluation.

[0027] By obtaining an evaluation value for evaluating the rationality of obtaining the first sequence based on target similarity, the first relative resource consumption degree, and the second relative resource consumption degree, the rationality of the operation of obtaining the first sequence based on the query task is evaluated from multiple dimensions, ensuring the robustness and adaptability of the evaluation results.

[0028] The following describes several exemplary embodiments to illustrate the technical solutions of the embodiments of the present application and the technical effects produced by the technical solutions of the present application. It should be noted that the following embodiments can refer to, draw on, or combine with each other, and the same terms, similar features, and similar implementation steps in different embodiments will not be repeated.

[0029] Figure 1 A schematic diagram of the system architecture for implementing a task planning evaluation method provided in an embodiment of the present application, wherein the system architecture includes: a terminal 120 and a server 140.

[0030] The terminal 120 installs and runs an application program having a task planning evaluation method, and the terminal 120 is used to evaluate the rationality of obtaining the first sequence of query execution tasks.

[0031] The terminal 120 is connected to the server 140 via a wireless network or a wired network.

[0032] Server 140 includes at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. Illustratively, server 140 includes a processor 144 and a memory 142. Memory 142 includes a display module 1421, a control module 1422, and a receiving module 1423. Server 140 is configured to provide background services for the application program of the method. Optionally, server 140 performs primary computing tasks, and terminal 120 performs secondary computing tasks; alternatively, server 140 performs secondary computing tasks, and terminal 120 performs primary computing tasks; alternatively, server 140 and terminal 120 utilize a distributed computing architecture for collaborative computing.

[0033] Optionally, the device type of the terminal includes: at least one of a smart phone, a tablet computer, an e-book reader, a Moving Picture Experts Group Audio Layer III (MP3) player, a Moving Picture Experts Group Audio Layer IV (MP4) player, a laptop computer and a desktop computer.

[0034] Those skilled in the art will appreciate that the number of the above-mentioned terminals may be more or less. For example, the above-mentioned terminal may be only one, or the above-mentioned terminals may be dozens or hundreds, or a larger number. The embodiments of the present application do not limit the number of terminals and device types.

[0035] The present application provides an evaluation method for task planning, such as Figure 2 As shown, the method includes: S101: Obtain a query statement, clarify the query statement, and obtain the clarified query statement, a first sequence, and a second sequence.

[0036] In an embodiment of the present application, the query statement is composed of natural language, and the query statement is used to request a query to obtain certain information. The query statement usually includes a query target, query conditions, query parameters, etc. For example, the query statement can be "provide promotional activities of various merchants in the past month."

[0037] In the embodiments of the present application, in actual applications, the query statements provided by users often contain a certain degree of ambiguity, such as unclear time range, ambiguous query conditions, unclear field references, etc. This ambiguity not only affects the understanding of the query statement, but may also cause significant deviations in the decomposition and execution results of downstream tasks. Therefore, it is necessary to clarify the query statement so that the corresponding operations based on the obtained query statement can be more accurately executed to provide the user with the required query information.

[0038] In the embodiment of the present application, clarification refers to further clarifying the query intent of the query statement, eliminating the ambiguous parts in the query statement, and further clarifying the vague query conditions. The clarification operation can be to further infer the query statement based on existing data, or to supplement the query statement by interacting with the user.

[0039] In an embodiment of the present application, the first sequence includes various first operations for executing the clarified query statement, that is, after obtaining the clarified query statement, the task planning system obtains a first sequence capable of executing the clarified query statement based on the clarified query statement. The first sequence includes multiple first operations arranged in order. By executing each first operation in the order of arrangement of each first operation in the first sequence, the query target required by the query statement can be obtained. The first operation can be obtaining data within one month, summing specific values, classifying specific content, etc.

[0040] In an embodiment of the present application, the second sequence includes various second operations performed to clarify the query statement. The second operation in the second sequence refers to the operation performed in the process of clarifying the query statement. The second operation can be to further infer the query statement based on existing data, or to supplement the query statement by interacting with the user.

[0041] In an embodiment of the present application, after obtaining a query statement input by a user, the task planning system clarifies the query statement, determines a second sequence including a clarification operation (second operation), performs task planning on the clarified query statement, and obtains a first sequence including multiple first operations arranged in an orderly manner.

[0042] S102: Obtain a third sequence and a fourth sequence.

[0043] In an embodiment of the present application, the third sequence includes various third operations for executing a query statement. The third sequence is a predefined standard solution for the query statement, that is, each third operation in the third sequence is a standard operation for executing a query statement.

[0044] In an embodiment of the present application, by decomposing the query statement, multiple reasonable execution paths, i.e., standard solutions, can usually be obtained. When there are multiple standard methods for executing the query statement, the third sequence is the solution closest to the first sequence among the standard solutions.

[0045] In an embodiment of the present application, the fourth sequence includes the various fourth operations required to obtain the third sequence by clarifying the query statement. Since the query statement may be ambiguous, the third sequence can be obtained by determining which clarification operations are performed on the query task, thereby obtaining the various fourth operations required to obtain the third sequence by clarifying the query statement and determining the fourth sequence.

[0046] S103 , obtaining a target similarity between the first sequence and the third sequence based on the portion of the query sentence that lacks specific information or has multiple interpretations and the longest common subsequence between the first sequence and the third sequence.

[0047] In the embodiment of the present application, since the third sequence serves as a standard scheme for executing query statements, that is, a benchmark for judging whether the execution of query statements by each operation in the first sequence is reasonable, the similarity between the first sequence and the third sequence is determined based on the longest common subsequence between the first sequence and the third sequence. This can quantify the similarity between the first sequence and the third sequence in structure (order of operation execution) and semantics (specific operations).

[0048] In an embodiment of the present application, a portion of the query statement that lacks specific information or has multiple interpretations is obtained. Based on the portion, the degree of ambiguity of the query statement can be determined. The degree of ambiguity will affect the task planning system's judgment on each first operation required to execute the query statement. Therefore, when determining the target similarity between the first sequence and the third sequence, it is necessary to consider the impact of the portion of the query statement that lacks specific information or has multiple interpretations on the evaluation results.

[0049] In an embodiment of the present application, when there is a lack of specific information in the query statement or there are many parts with multiple interpretations, it means that the current query question is highly ambiguous, and it is necessary to lower the evaluation criteria of the target similarity part in the overall evaluation, so as to enhance the robustness and adaptability of the overall evaluation.

[0050] In the embodiment of the present application, the target similarity is used to represent the similarity between the first sequence and the third sequence based on the third sequence.

[0051] S104: Evaluate the resources occupied by the first sequence based on the resources occupied by the third sequence to obtain a first relative resource consumption level of the first sequence.

[0052] In an embodiment of the present application, since the third sequence is a predefined standard method for executing query statements, the resources required to execute all third operations in the third sequence are used as a benchmark to evaluate the resources required to execute all first operations in the first sequence to obtain the first relative resource consumption level of the first sequence.

[0053] S105 , using the resources occupied by the fourth sequence as a benchmark, evaluate the resources occupied by the second sequence to obtain a second relative resource consumption level of the second sequence.

[0054] In an embodiment of the present application, since the third sequence is a predefined standard solution for executing query statements, the fourth sequence of the third sequence is obtained by clarifying the query statement, which also serves as a standard solution for clarifying the operation. Therefore, based on the resources required for all fourth operations in the fourth sequence, the resources required for executing all second operations in the second sequence are evaluated to obtain the second relative resource consumption level of the second sequence.

[0055] S106 : Obtaining an evaluation value for evaluating the rationality of obtaining the first sequence based on the target similarity, the first relative resource consumption degree, and the second relative resource consumption degree.

[0056] In the embodiment of the present application, the product of the target similarity, the first relative resource consumption degree, and the second relative resource consumption degree is used as the evaluation value.

[0057] In an embodiment of the present application, an evaluation value for characterizing the rationality of the task planning system acquiring the first sequence is determined based on the dimensions of the relative similarity between the first sequence and the third sequence, the relative resource consumption between the first sequence and the third sequence, and the relative resource consumption between the second sequence and the fourth sequence. The evaluation value is determined from multiple dimensions by comprehensively considering the influence of the ambiguity of the query statement, the resources required for task execution, and the resources required for clarifying the query statement on the rationality of acquiring the first sequence.

[0058] In the embodiment of the present application, the importance of the target similarity, the first relative resource consumption degree, and the second relative resource consumption degree can be determined respectively, and corresponding weights can be assigned to the above values. For example, in some scenarios, the similarity between the first sequence and the third sequence of task re-acquisition is the primary goal. For example, some scenarios pay more attention to resource efficiency. Therefore, according to the importance of each content in the current application scenario, the corresponding weight is assigned to determine the evaluation value, so as to achieve a more adaptable and robust evaluation result.

[0059] In the embodiment of the present application, the evaluation value is determined by the following formula:

[0060] in, Indicates the evaluation value, represents the target similarity, Indicates the first relative resource consumption level, Indicates the second relative resource consumption level.

[0061] In the above scheme, by obtaining a query statement and clarifying the query statement, obtaining the clarified statement, obtaining a first sequence including various first operations of executing the clarified query statement, and obtaining various second operations executed by the clarified query statement, strong data support is provided for the subsequent rationality of obtaining the first sequence from multiple dimensions for task planning based on the query statement, and the rationality of obtaining the first sequence can be evaluated from both the operation process and the result perspectives.

[0062] By obtaining a third sequence comprising each third operation for executing a query statement, and by obtaining a fourth sequence comprising each fourth operation required to clarify the query statement to obtain the third sequence, a reference standard is provided for evaluating the first sequence. By obtaining the third and fourth sequences, not only can the operations for executing the query statement in the first and third sequences be compared to evaluate the first sequence, but also the operations for clarifying the query statement in the second and fourth sequences can be compared to evaluate the first sequence.

[0063] The target similarity between the first and third sequences is obtained based on the parts of the query sentence that lack specific information or have multiple interpretations and the longest common subsequence that can characterize the similarity between the first and third sequences. The influence of fuzzy expression on the determination of the first sequence is taken into account, and the structural similarity between the first and third sequences is evaluated through the longest common subsequence. The target similarity is determined based on the fuzziness and similarity of the fuzzy expression. The fuzziness is used to adjust the evaluation criteria of the target similarity, thereby enhancing the robustness and adaptability of the overall evaluation.

[0064] By evaluating the resources occupied by the first sequence based on the resources occupied by the third sequence, the first relative resource consumption level of the first sequence is obtained. By using the obtained resource consumption of the third sequence as a reference benchmark, the resources occupied by the second sequence are evaluated, thereby achieving a unified evaluation for specific operation processes and improving the robustness and adaptability of the overall evaluation.

[0065] By taking the resources occupied by the fourth sequence as a benchmark, the resources occupied by the second sequence are evaluated to obtain the second relative resources of the second sequence. The impact of the resources consumed when clarifying the query statement on the rationality of the solution is considered. The resources consumed by the operation corresponding to the fourth sequence of the third sequence obtained by the clarifying query statement are used as the reference benchmark for evaluation. The resources consumed by the operation corresponding to the first sequence obtained by the clarifying query statement are evaluated. This achieves a unified evaluation of the clarification operation process, realizes the quantity of the clarification process, and improves the robustness of the overall evaluation.

[0066] By obtaining an evaluation value for evaluating the rationality of obtaining the first sequence based on target similarity, the first relative resource consumption degree, and the second relative resource consumption degree, the rationality of the operation of obtaining the first sequence based on the query task is evaluated from multiple dimensions, ensuring the robustness and adaptability of the evaluation results.

[0067] Based on the above embodiments, as an optional embodiment, a method for determining target similarity is as follows: Figure 3 The specific contents are as follows: S201, treating each part of the query sentence that lacks specific information or has multiple interpretations as fuzzy information, and determining the fuzzy type of each fuzzy information; S202, performing statistics on each fuzzy information and fuzzy type to obtain the fuzziness level of the query statement; S203, obtaining an initial similarity between the first sequence and the third sequence based on the longest common subsequence between the first sequence and the third sequence; S204: Weight the initial similarity according to the fuzziness level to obtain the target similarity.

[0068] In S201 of the embodiment of the present application, parts of the query statement that lack specific information or have multiple interpretations are obtained, the above parts are used as fuzzy information, and the type of the fuzzy information is determined according to the specific text content of the query statement.

[0069] In the embodiment of the present application, the type of fuzzy information may be: unclear time expression, such as: "recent period of time"; the type of fuzzy information may also be: incomplete or ambiguous conditions, such as "products that users click on more"; the fuzzy type may also be: ambiguous field references, such as "this value", "this item", "previous"; the fuzzy type may also be: the existence of multiple layers of negation or double conditions, such as "does not exclude inactive users".

[0070] In S202 of the embodiment of the present application, different fuzzy types have different fuzziness levels. Statistics are collected on various fuzzy information and fuzzy types in the query statement to obtain the fuzziness level of the query statement.

[0071] In S203 of the embodiment of the present application, the longest common subsequence between the first sequence and the third sequence is obtained, and then the length of the longest common subsequence is mapped to [0, 1], thereby obtaining the initial similarity between the first sequence and the third sequence.

[0072] In S204 of the embodiment of the present application, in order to more objectively and reasonably evaluate the quality of the prediction scheme, when calculating the target similarity between the first sequence and the third sequence, the initial similarity is weighted according to the degree of fuzziness. When the degree of fuzziness is lower, the weight value is closer to 1, that is, the standard of similarity evaluation is not lowered; when the degree of fuzziness is higher, the weight value is closer to 0.5, that is, appropriate fault tolerance is given to the similarity according to the degree of fuzziness.

[0073] In the above scheme, the ambiguity degree of the query question is determined based on the existence of fuzzy information in the query question and the fuzziness type of each fuzzy information, the initial similarity between the first sequence and the third sequence is determined based on the longest common subsequence between the first sequence and the second sequence, and then the target similarity between the first sequence and the third sequence is determined based on the fuzziness degree and the initial similarity. This achieves the consideration of the impact of the fuzziness degree of the query question on the acquisition of the first sequence in the similarity calculation, improves the robustness and adaptability of the overall evaluation, and achieves a more objective and reasonable evaluation of the rationality of the acquired first sequence.

[0074] Based on the above embodiments, as an optional embodiment, the method for determining the degree of blur is as follows: Figure 4 The specific contents are as follows: S301, performing statistics on each piece of fuzzy information and its corresponding fuzzy type to obtain a first quantity of fuzzy information corresponding to each fuzzy type; S302: For each fuzzy type, obtain a first weight of the fuzzy type, and obtain a first fuzzy value of the fuzzy type based on the first weight and the first quantity; the first fuzzy value is used to represent the fuzziness level of fuzzy information of the corresponding fuzzy type in the query statement; S303: Sum the first fuzzy values of each fuzzy type, and map the summation result to a preset interval to obtain the fuzziness level of the query statement.

[0075] In S301 of the embodiment of the present application, each piece of fuzzy information is classified according to the fuzzy type, and the quantity of fuzzy information of each fuzzy type is determined, thereby obtaining a first quantity of fuzzy information corresponding to each fuzzy type.

[0076] In S302 of the embodiment of the present application, due to different fuzzy types, the degree of fuzziness of each fuzzy information will also be different. Different fuzzy types correspond to different first weights. For each fuzzy type, the first fuzzy value of all fuzzy information of the fuzzy type is determined based on the first weight and the first quantity, that is, based on the number of times the fuzzy information of the current fuzzy type appears in the query statement and the first weight corresponding to the fuzzy type, the first fuzzy value of the fuzzy information of the fuzzy type in the query statement is determined.

[0077] In S303 of the embodiment of the present application, since the fuzzy information in the query statement has at least one fuzzy type, the first fuzzy values of each fuzzy type are summed, and the summation result is mapped to a preset interval to obtain the fuzziness degree of the query statement, and the preset interval is [0,1].

[0078] In this embodiment of the present application, the fuzziness of the query statement is obtained by the following formula:

[0079]

[0080] in, is the number of fuzzy types contained in the query statement, For the A fuzzy type, For the The first weight of the fuzzy type, The first The first number of fuzzy information of fuzzy types, To sum the results, The degree of blur.

[0081] In the above formula set, when the query question is clear and unambiguous (i.e. When it is close to 0, is also close to 0, indicating that the similarity evaluation standard is not lowered; when the query question is extremely vague, It approaches 1, which is used to make appropriate tolerance to the similarity in the final evaluation.

[0082] In the above scheme, the fuzzy type of fuzzy information and the number of times each fuzzy type of information appears are determined by analyzing the query statement. The fuzziness level of the query question is determined based on the first weight of each fuzzy type and the number of times each fuzzy type of fuzzy information appears. This further reflects the overall fuzziness level of the query question more objectively and realistically, making the evaluation of the first sequence more objective.

[0083] Based on the above embodiments, as an optional embodiment, the method for obtaining the third sequence is as follows: Figure 5 The specific contents are as follows: S401, obtaining at least one reference sequence based on a query statement; S402, for each reference sequence, obtaining an initial similarity between the reference sequence and the first sequence based on the longest common subsequence between the reference sequence and the first sequence; S403: Select the reference sequence with the largest initial similarity as the third sequence.

[0084] In S401 of an embodiment of the present application, when there are multiple sequences of executable query statements, it can be pre-defined that the above sequences are all standard schemes for executing query statements, that is, they are all reference sequences, so that when a query statement is received, at least one reference sequence can be obtained based on the query statement, and the reference sequence includes various reference operations for executing the query statement.

[0085] In S402 of the embodiment of the present application, since there are multiple reference sequences, the longest common subsequence between each reference sequence and the first sequence is calculated to obtain the initial similarity between the reference sequence and the first sequence. That is, in the present application, a single sequence is not used as a reference benchmark. Instead, the first sequence is compared with each predefined reference sequence to determine the initial similarity between the first sequence and each reference sequence. In S403 of the embodiment of the present application, after determining the initial similarity between the first sequence and each reference sequence, the reference sequence closest to the first sequence is selected as the third sequence to be used as a reference benchmark for this evaluation. Therefore, the reference sequence with the greatest initial similarity is selected as the third sequence.

[0086] In the above scheme, by determining the reference sequence with the highest similarity to the first sequence from multiple reference sequences as the third sequence, and using it as the reference benchmark for this evaluation, the multi-path operation implementation of the query statement is taken into consideration. Instead of taking a single reference sequence as the standard, the reference sequence closest to the first sequence is selected for evaluation, thereby improving the objectivity of the evaluation results.

[0087] Based on the above embodiments, as an optional embodiment, the method for determining the initial similarity is as follows: Figure 6 The specific contents are as follows: S501, for any reference sequence, based on the longest common subsequence between the reference sequence and the first sequence, obtain a matching pair sequence, where each matching pair in the matching pair sequence includes a first index of an element in the longest common subsequence in the first sequence and a second index in the reference sequence; S502 , for each matching pair, obtain a corresponding first operation from a first sequence based on a first index in the matching pair, and obtain a corresponding fifth operation from a reference sequence based on a second index in the matching pair; S503, determining a first operation type of the first operation and a second operation type of the fifth operation, and determining a first similarity between the first operation and the fifth operation based on the semantic similarity between the first operation and the fifth operation, the first operation type, and the second operation type; S504 , obtaining a first operation number of operations in the reference sequence, summing up the first similarities, and using the ratio of the summation result to the first operation number as the initial similarity between the first sequence and the reference sequence.

[0088] In S501 of this embodiment of the present application, for any reference sequence, a matching pair sequence is obtained based on the longest common subsequence between the current reference sequence and the first sequence. Each matching pair in the matching pair sequence includes the index of any element between the longest common subsequences in the first sequence and the current reference sequence, i.e., the first index and the second index.

[0089] In S502 of the embodiment of the present application, for each matching pair, each matching pair corresponds to an element in the longest common subsequence. Through the first index contained in the matching pair, the first operation can be directly obtained from the first sequence, and through the second index contained in the matching pair, the corresponding fifth operation can be directly obtained from the current reference sequence.

[0090] In S503 of an embodiment of the present application, the first operation type of the acquired first operation and the second operation type of the fifth operation are determined, the similarity between the first operation type and the second operation type is compared, the semantic similarity between the first operation and the fifth operation is compared, and the first similarity between the first operation and the fifth operation is determined based on the above similarity and semantic similarity; wherein the value range of the first similarity is [0,1], where 1 indicates a complete match and 0 indicates a complete mismatch.

[0091] In S504 of the embodiment of the present application, the first number of operations in the currently processed reference sequence is obtained, the first similarities of the currently processed reference sequence are summed, and the ratio of the summed result to the first number of operations is used as the initial similarity between the current sequence and the first sequence. By using the ratio of the summed result to the first number of operations as the initial similarity, the problem of a high final similarity due to the length of a long sequence can be eliminated, allowing reference sequences of different lengths to be compared with each other.

[0092] In the above scheme, the similarity between the first operation and the fifth operation in semantics, operation type or other dimensions is determined, and the first similarities of all first operations and fifth operations are weightedly summed to obtain a total score of weighted matching, thereby achieving quantification of the structural similarity and semantic similarity between the first sequence and each reference sequence, and improving the evaluation accuracy and adaptability.

[0093] In this embodiment of the present application, the initial similarity between the first sequence and the reference sequence is obtained by the following formula:

[0094]

[0095] Among them, S is the initial similarity, represents the first operation number of the operation in the reference sequence, represents the sum of the first similarities corresponding to the reference sequence, represents the first similarity, Represents a set of first and fifth operations that match in order.

[0096] In this embodiment of the present application, the target similarity is obtained by the following formula:

[0097] in, represents the target similarity, Indicates the fuzziness of the query statement. Indicates the first sequence and the The initial similarity between the reference sequences, Indicates the The sum of the first similarities corresponding to the reference sequences, Indicates the The number of first operations in the reference sequence, n Indicates the number of reference sequences.

[0098] Based on the above embodiments, as an optional embodiment, the method for determining the first relative resource consumption level is as follows: Figure 7 The specific contents are as follows: S601, for any one of the first sequence and the third sequence, collect statistics based on the resources occupied by the executed sequence to obtain the resource consumption of the sequence; S602, obtaining a first number of first operations in a first sequence and a second number of third operations in a third sequence; S603-1, if the first amount is not greater than the second amount, taking the ratio of the second resource consumption to the first resource consumption as the first relative resource consumption level, the first resource consumption being the resource consumption of the first sequence, and the second resource consumption being the resource consumption of the third sequence; S603-2: If the first amount is greater than the second amount, adjusting the ratio between the second resource consumption and the first resource consumption based on the preset first parameter value and the first difference, and using the adjustment result as the first relative resource consumption level.

[0099] In S601 of the embodiment of the present application, for the first sequence, the resources required to execute all the first operations in the first sequence are counted to obtain the first resource consumption. For the third sequence, the resources required to execute all the third operations in the third sequence are counted to obtain the second resource consumption. The resources required for each operation include at least: memory and CPU execution time, etc.

[0100] In S602 of the embodiment of the present application, the number of first operations included in the first sequence is counted, that is, a first number is obtained, and the number of third operations included in the third sequence is counted, that is, a second number is obtained. In S603-1 of the embodiment of the present application, the number of operations in the first sequence and the third sequence are compared. If the first number is not greater than the second number, it means that the number of operations required to execute the first sequence is not greater than the number of operations required to execute the third sequence. In this case, there is no need to adjust the ratio of the second resource consumption to the first resource consumption, and it can be directly used as the first relative resource consumption level of the first sequence.

[0101] In S603-2 of the embodiment of the present application, the first difference is the difference between the first quantity and the second quantity. The number of operations in the first sequence and the third sequence are compared. If the first number is greater than the second number, it means that the number of operations required to execute the first sequence is greater than the number of operations required to execute the third sequence. The first parameter value is used to penalize the situation where the actual number of operations (first number) is greater than the number of operations of the best matching solution (second number). Therefore, it is necessary to adjust the ratio between the second resource consumption and the first resource consumption by the preset first parameter value and the first difference, and then use the adjustment result as the first relative resource consumption degree. Among them, the larger the first difference, the smaller the first relative resource consumption degree, which represents the lower score of the task planning system on the resource consumption of the first sequence of production.

[0102] In this embodiment of the present application, the first relative resource consumption level is obtained by the following formula:

[0103] in, is the first relative resource consumption level, is the second resource consumption, is the first resource consumption, is the first parameter value, is the first quantity, The second quantity.

[0104] In an embodiment of the present application, when the first quantity is less than or equal to the second quantity, the ratio of the second resource consumption to the first resource consumption is used as the first relative resource, that is, when the first operation of the operand of executing the query statement is less than or equal to the third operation, the smaller the first resource consumption, the greater the first relative resource degree, and the higher the score of the task planning system on the resource consumption of the first sequence of production. When the first quantity is greater than the third quantity, it is necessary to adjust the ratio of the second resource consumption to the first resource consumption by the first parameter value and the second difference. By respectively calculating the first resource consumption and the second resource consumption consumed by executing all the first operations and executing all the third operations, and the degree of difference in the number of operations between the operations in the first sequence and the third operation, the amount of resource consumption of the first sequence is evaluated based on the number of operands and the resources occupied by the specific operations, with the consumption of the third sequence as a benchmark, thereby improving the robustness and adaptability of the evaluation results.

[0105] Based on the above embodiments, as an optional embodiment, a method for determining the resource consumption of a sequence is as follows: Figure 8 The specific contents are as follows: S701, obtaining the maximum CPU execution time and maximum memory usage generated by historical query statement execution; S702: For each operation in any one of the first and third sequences, obtain the current first CPU execution time and first memory usage of the operation, normalize the first CPU execution time based on the maximum CPU execution time to obtain a first result corresponding to the operation, normalize the first memory usage based on the maximum memory usage to obtain a second result corresponding to the operation, and perform a weighted sum of the first and second results corresponding to the operation to obtain a third resource consumption of the operation. S703 : For any one of the first sequence and the third sequence, sum the third resource consumption of each operation in the sequence to obtain the resource consumption of the sequence.

[0106] In S701 of the embodiment of the present application, the longest CPU execution time and the maximum memory usage generated by the historical execution of each query statement are obtained, that is, the maximum execution time and the maximum memory usage are obtained from the query tasks executed in the past.

[0107] In S702 of the embodiment of the present application, for each first operation in the first sequence, a first CPU execution time and a first memory usage required to execute the first operation are determined, and the first CPU execution time is normalized by the maximum CPU execution time to obtain a first result corresponding to the first operation, which means that the first CPU execution time is normalized using the maximum CPU execution time as a normalization benchmark, and the first memory usage is normalized by the maximum memory usage, which means that the first memory usage is normalized using the maximum memory usage as a normalization benchmark to obtain a second result corresponding to the first operation. Since the importance of memory usage and CPU execution time varies for different operations, based on the respective importance of memory usage and CPU execution time, the first result and the second result of the first operation are weightedly summed to obtain a third resource consumption of the current first operation; In an embodiment of the present application, for each third operation in the third sequence, determining the first CPU execution time and the first memory usage required to execute the third operation, normalizing the first CPU execution time by the maximum CPU execution time to obtain a first result corresponding to the third operation, which means normalizing the first CPU execution time by the maximum CPU execution time as a normalization benchmark, and normalizing the first memory usage by the maximum memory usage, which means normalizing the first memory usage by the maximum memory usage as a normalization benchmark to obtain a second result corresponding to the third operation. Since the importance of memory usage and CPU execution time varies in different operations, based on the respective importance of memory usage and CPU execution time, a weighted sum is performed on the first result and the second result of the third operation to obtain a third resource consumption of the current third operation; In S703 of the embodiment of the present application, for the first sequence, the third resource consumption of each first operation in the first sequence is summed to obtain the first resource consumption of the first sequence; for the third sequence, the third resource consumption of each third operation in the third sequence is summed to obtain the second resource consumption of the third sequence.

[0108] In this embodiment of the present application, the first resource consumption is obtained by the following formula:

[0109] in, is the first resource consumption, is the first number of the first operation in the first sequence, The first sequence The first CPU execution time of the first operation, The first sequence The first memory usage of the first operation, The maximum execution time of the CPU, is the maximum memory usage, + , is the weight of CPU execution time, is the weight of memory usage.

[0110] In this embodiment of the present application, the second resource consumption is obtained by the following formula:

[0111] in, is the second resource consumption, is the second number of the third operation in the third sequence, The third sequence The first CPU execution time of the third operation, The third sequence The first memory footprint of the third operation, The maximum execution time of the CPU, is the maximum memory usage, + , is the weight of CPU execution time, is the weight of memory usage.

[0112] In the above scheme, the CPU execution time and memory usage of each operation are used as the basis for measuring resource consumption, and a weighted summation is performed based on the importance preference of the above resources to achieve reasonable quantification of the corresponding resource consumption.

[0113] Based on the above embodiments, as an optional embodiment, the second relative resource consumption degree determination method is as follows: Figure 9 The specific contents are as follows: S801: For any one of the second and fourth sequences, for each operation in the sequence, obtain the operation type corresponding to the operation, determine the fourth resource consumption of the operation based on a pre-established correspondence between resource consumption and operation type, and sum the fourth resource consumption of each operation in the sequence to obtain the resource consumption of the sequence; S802, obtaining a third number of second operations in a second sequence, obtaining a fourth number of fourth operations in a fourth sequence, and determining a second parameter value based on a difference between the third number and the fourth number; S803: Multiply the ratio of the fourth resource consumption to the third resource consumption by the second parameter value to obtain a second relative resource consumption level; the third resource consumption is the resource consumption of the second sequence, and the fourth resource consumption is the resource consumption of the third sequence.

[0114] In S801 of the embodiment of the present application, for each second operation in the second sequence, the operation type corresponding to the second operation is obtained, and the fourth resource consumption of the second operation is determined based on the pre-established correspondence between resource consumption and operation type. The fourth resource consumption of each second operation sequence in the second sequence is summed to obtain the resource consumption of the second sequence, that is, the third resource consumption; for each fourth operation in the fourth sequence, the operation type corresponding to the fourth operation is obtained, and the fourth resource consumption of the fourth operation is determined based on the pre-established correspondence between resource consumption and operation type. The fourth resource consumption of each fourth operation sequence in the fourth sequence is summed to obtain the resource consumption of the fourth sequence.

[0115] In S802 of an embodiment of the present application, the number of second operations in the second sequence is obtained, that is, the third number is obtained, and the number of fourth operations in the fourth sequence is obtained, that is, the fourth number is obtained. The second parameter value is determined based on the difference between the third number and the fourth number. The second parameter value is used to control the attenuation rate caused by redundant clarification operations.

[0116] In S803 of the embodiment of the present application, a ratio between the fourth resource consumption and the third resource consumption is obtained, and the ratio is multiplied by a second parameter value to obtain a second relative resource consumption degree of the first sequence.

[0117] In this embodiment of the present application, the second relative resource consumption level is determined by the following formula:

[0118] in, is the second relative resource consumption level, is the fourth resource consumption of the fourth sequence, is the third resource consumption of the second sequence, is the second parameter value.

[0119] In the embodiment of the present application, the third resource consumption and the fourth resource consumption are respectively determined by the following formulas:

[0120]

[0121]

[0122] in, represents the third resource consumption of the second sequence, is the third number of the second operation in the second sequence, is the fourth resource consumption of the fourth sequence, is the fourth quantity of the fourth operation in the fourth sequence, is the resource consumption required for the k-th operation in the sequence, The resource consumption corresponding to the interactive clarification operation type is Clarify the corresponding resource consumption for automatic inference of operation types. > .

[0123] In the above scheme, the resource consumption of the clarification process is quantified based on the operation type of each operation in the second and fourth sequences and the amount of resources required to execute each operation type. The resource consumption is adjusted based on the differences between the operations in the second and fourth sequences to measure the system's ability to understand and clarify user intentions, thereby improving the objectivity of the evaluation results.

[0124] Based on the above embodiments, as an optional embodiment, if the third number is not greater than the fourth number, the second parameter value is obtained based on the relative difference between the third number and the fourth number; if the third number is greater than the fourth number, the second parameter value is obtained based on the absolute difference between the third number and the fourth number.

[0125] In an embodiment of the present application, when the third number is not greater than the fourth number, the relative difference between the third number and the fourth number is determined according to the ratio between the third number and the fourth number, thereby determining the second parameter value based on the relative difference.

[0126] In the embodiment of the present application, when the third number is greater than the fourth number, the absolute difference between the third number and the fourth number is determined according to the difference between the third number and the fourth number, thereby determining the second parameter value based on the absolute difference.

[0127] In the embodiment of the present application, the second parameter value is determined by the following formula:

[0128] in, is the second parameter value, is the third number of the second operation in the second sequence, is the fourth quantity of the fourth operation in the fourth sequence, is the penalty coefficient, which is used to control the decay rate caused by unnecessary clarification operations.

[0129] The above formula can ensure that when (No clarification needed), the second parameter value is 1; when , is the dividing point, the second parameter value is 0.5; when , the second parameter value approaches 0, that is, when the third quantity is smaller than the fourth quantity, the higher the score of the second sequence in relative resource consumption, and the better the evaluation result.

[0130] In the above scheme, the second parameter value is determined by the relative difference and absolute difference between the third quantity and the fourth quantity, which can ensure that the ratio of the fourth resource consumption to the third resource consumption is adaptively adjusted according to the size of the difference between the third quantity and the fourth quantity, so that the larger the value of the third quantity is than the fourth quantity, the lower the score of the second sequence in relative resource consumption and the worse the evaluation result.

[0131] On the basis of the above embodiments, as an optional embodiment, the embodiment of the present application further provides a task planning evaluation method, such as Figure 10 The specific contents are as follows: S901, obtaining evaluation values corresponding to a first sequence of multiple query statements; S902: For each first sequence, obtain the query type to which the query statement corresponding to the first sequence belongs, and determine a second weight value of the first sequence based on a pre-established correspondence between the query type and the weight; S903, weighting and summing the evaluation values of each first sequence by the corresponding second weight value to obtain a first summation result, summing the second weight values to obtain a second summation result, and taking the ratio of the first summation result to the second summation result as the second evaluation value; In S901 of the embodiment of the present application, when the rationality of a task plan requiring multiple queries is evaluated, evaluation values corresponding to a first sequence of multiple query statements are obtained.

[0132] In S902 of the embodiment of the present application, for each first sequence, the query type to which the query statement corresponding to the first sequence belongs is determined. The query type of a query statement is related to the query complexity, query frequency or importance of the corresponding business of the query statement. That is, the query statement can be classified according to the query complexity of the query statement, or according to the historical query frequency of the query statement, or according to the importance of the business corresponding to the query statement. Generally, the higher the complexity of the query statement, the higher the query frequency or the higher the importance of the corresponding business, the higher the weight of the corresponding query type. Therefore, the correspondence between each query type and weight can be pre-established, and then when multiple query statements need to be evaluated, the second weight value given in response can be obtained based on the query type corresponding to each query statement.

[0133] In S903 of the embodiment of the present application, a weighted summation is performed based on the evaluation value of each first sequence and the corresponding second weight value to obtain a first summation result, and the second weight values are summed to obtain a second summation result. The second evaluation value is determined based on the ratio between the first summation result and the second summation result. The second evaluation value represents the rationality of obtaining the corresponding first sequence based on multiple query statements, that is, the rationality of obtaining the corresponding first sequence of multiple query statements is considered from an overall perspective.

[0134] In the embodiment of the present application, the second evaluation value is determined by the following formula:

[0135] in, is the second evaluation value, For the The evaluation value of the first sequence, For the The second weight value corresponding to the first sequence, N represents the number of query statements; In the embodiment of the present application, the second evaluation values of the multiple query statements may also be calculated using the following formula:

[0136] Among them, MacroAcc is the second evaluation value, Indicates the number of query statements. represents the qualified threshold, For the The evaluation value of the first sequence.

[0137] In the embodiment of the present application, different methods can be selected to obtain the second evaluation value according to different application scenarios. For general scenarios, the second average value can be obtained by averaging all evaluation values. For the same query statement set, when the importance of the business corresponding to the query statement is different, the second average value can be obtained by averaging all evaluation values. The corresponding formula is used to obtain the second evaluation value.

[0138] refer to Figure 11 As shown, it exemplarily shows a flow chart of a task planning evaluation method, the specific contents are as follows: S1001, obtaining a query statement, inputting the query statement into a task planning system, and obtaining a first sequence and a second sequence output by the task planning system; S1002: Obtain at least one reference sequence based on the query statement, determine initial similarities between each reference sequence and the first sequence based on the longest common subsequence between the reference sequence and the first sequence, select the reference sequence with the greatest similarity as a third sequence, and determine a fourth sequence comprising a plurality of fourth operations required to clarify the query statement and obtain the third sequence; S1003: Parts of the query sentence that lack specific information or have multiple interpretations are respectively treated as fuzzy information, and a fuzzy type of each fuzzy information is determined to obtain a first quantity of fuzzy information corresponding to each fuzzy type; S1004: For each fuzzy type, obtain a first weight of the fuzzy type, obtain a first fuzzy value of the fuzzy type based on the first weight and the first quantity, and sum the first fuzzy values of each fuzzy type to obtain a fuzziness level of the query statement; S1005, weighting the initial similarity between the first sequence and the third sequence according to the fuzziness level to obtain a target similarity; S1006. For any one of the first and third sequences, for each operation in the sequence, obtain the current first CPU execution time and first memory usage of the operation, normalize the first CPU execution time based on the maximum CPU execution time to obtain a first result corresponding to the operation, normalize the first memory usage based on the maximum memory usage to obtain a second result corresponding to the operation, perform a weighted sum of the first and second results corresponding to the operation to obtain a third resource consumption of the operation, and sum the third resource consumptions of the first operation in the first sequence and the third operation in the third sequence to obtain the first resource consumption and the second resource consumption; S1007: Obtain a first number of first operations in the first sequence and a second number of third operations in the third sequence; if the first number is not greater than the second number, use a ratio between the second resource consumption and the first resource consumption as a first relative resource consumption level; if the first number is greater than the second number, adjust the ratio between the second resource consumption and the first resource consumption based on a preset first parameter value and a second difference between the first number and the second number, and use the adjusted result as the first relative resource consumption level; S1008: For each operation in either the second sequence or the fourth sequence, obtain the operation type corresponding to the operation, determine the fourth resource consumption of the operation based on a pre-established correspondence between resource consumption and operation type, and sum the fourth resource consumption of each operation in the second sequence and the fourth sequence to obtain the third resource consumption of the second sequence and the fourth resource consumption of the fourth sequence. S1009, obtaining a third number of the second operation in the second sequence and a fourth number of the fourth operation in the fourth sequence; if the third number is not greater than the fourth number, obtaining a second parameter value based on a relative difference between the third number and the fourth number; if the third number is greater than the fourth number, obtaining a second parameter value based on an absolute difference between the third number and the fourth number; S1010, multiplying the ratio of the fourth resource consumption to the third resource consumption by the second parameter value to obtain a second relative resource consumption level; S1011 : Obtain an evaluation value based on the target similarity, the first relative resource consumption degree, and the second relative resource consumption degree.

[0139] In an embodiment of the present application, the structure and operation cost of the first operation in the first sequence of the fuzzy information appearing in the query statement are determined, and support is provided for adjusting various parameters based on context or dynamic feedback, thereby improving the stability and adaptability of the evaluation. By introducing multiple reasonable standard schemes as reference sequences, the previous reliance on a single reference sequence for rationality evaluation is broken through. Improvements are made based on the original LCS, considering the importance and semantic matching strength of each operation in the first sequence, and avoiding the evaluation bias caused by simple operation matching. The fuzzy information in the query statement is identified, and the degree of fuzziness is introduced to penalize the target similarity, so that the evaluation is closer to the understanding risk of real scenarios. By introducing the CPU execution time and memory usage required for the operation as a measure of resource consumption, the degree of deviation between the first and third sequences is used to achieve quantitative modeling of task execution efficiency. For fuzzy information, the resource consumption of the interaction between the system and the user is introduced to measure the system's ability to understand and clarify user intentions. The three indicators are combined to output the final evaluation value, forming a complete evaluation closed loop, and promoting a more comprehensive task planning system.

[0140] The embodiments of this application can be applied to scenarios such as natural language to structured task translation (such as NL2SQL, NL2API, etc.), intelligent question answering, intelligent planning, and multi-round interactive systems. Under the large model-driven reasoning framework, it provides support as the core evaluation mechanism for the execution effect of the Multi-Agent system.

[0141] In the embodiment of the present application, it no longer relies on the only "standard SQL" comparison method, but takes into account the operation level, multiple reference sequences and the ambiguity of the query statement, thereby improving the credibility and accuracy of the rationality evaluation, evaluating the task execution path, adapting to the multi-agent system collaboration mode, and facilitating planning optimization and task scheduling learning. It can perform reward learning or reinforcement fine-tuning on the model based on the three-dimensional evaluation results to form an evaluation-driven optimization mechanism, significantly enhancing the model training and fine-tuning feedback loop, and can be used as an operation monitoring indicator system for commercial deployment systems. It can be used in a production environment to track the output quality and cost of the system under real business data, and realize dynamic evaluation and alarm.

[0142] The query method for task planning provided in the embodiment of the present application introduces a dynamic weight optimization mechanism, comprehensively considers the impact of target similarity, first relative resource consumption, and second relative resource consumption on the rationality of the planned first sequence, and realizes a detailed evaluation of the multi-dimensional and dynamic adaptability of task planning, thereby improving the accuracy and practicality of task evaluation. It solves the problem in related technologies that cannot cover SQL semantic equivalence but different expressions, resulting in biased evaluation results, solves the problem of lack of perception of SQL semantic structure and difficulty in handling task planning with strong sequential dependencies, solves the problem of ignoring the fine-grained advantages and disadvantages of individual operations during task execution and insufficient training feedback, solves the problem of ignoring the understanding deviation caused by the semantic ambiguity of the problem and the inability to reflect the clarification effect, and solves the problem of ignoring high-resource-consuming operations, resulting in reduced execution efficiency or resource waste.

[0143] The embodiment of the present application provides a task planning evaluation device, such as Figure 12 As shown, the task planning evaluation device 120 may include: a first processing module 1201 , a second processing module 1202 , a third processing module 1203 , a fourth processing module 1204 , a fifth processing module 1205 and a sixth processing module 1206 .

[0144] Specifically, the first processing module 1201 is configured to obtain a query statement, clarify the query statement, and obtain a clarified query statement, a first sequence, and a second sequence, wherein the first sequence includes first operations performed on the clarified query statement, and the second sequence includes second operations performed on the clarified query statement. A second processing module 1202 is configured to obtain a third sequence and a fourth sequence, wherein the third sequence includes third operations for executing the query statement, and the fourth sequence includes fourth operations required for obtaining the third sequence by clarifying the query statement; The third processing module 1203 is configured to obtain a target similarity between the first sequence and the third sequence based on the portion of the query that lacks specific information or has multiple interpretations and the longest common subsequence between the first sequence and the third sequence; The fourth processing module 1204 is configured to evaluate the resources occupied by the first sequence based on the resources occupied by the third sequence, and obtain a first relative resource consumption level of the first sequence. The fifth processing module 1205 is configured to evaluate the resources occupied by the second sequence based on the resources occupied by the fourth sequence, and obtain a second relative resource consumption level of the second sequence; The sixth processing module 1206 is configured to obtain an evaluation value for evaluating the rationality of obtaining the first sequence based on the target similarity, the first relative resource consumption level, and the second relative resource consumption level.

[0145] The task planning evaluation device provided in the embodiment of the present application obtains a query statement, clarifies the query statement, obtains the clarified statement, obtains a first sequence including various first operations of executing the clarified query statement, and obtains various second operations executed by the clarified query statement. This provides strong data support for the subsequent rationality of the first sequence obtained by task planning based on the query statement from multiple dimensions, and can realize the evaluation of the rationality of obtaining the first sequence from both the operation process and the result perspective.

[0146] By obtaining a third sequence comprising each third operation for executing a query statement, and by obtaining a fourth sequence comprising each fourth operation required to clarify the query statement to obtain the third sequence, a reference standard is provided for evaluating the first sequence. By obtaining the third and fourth sequences, not only can the operations for executing the query statement in the first and third sequences be compared to evaluate the first sequence, but also the operations for clarifying the query statement in the second and fourth sequences can be compared to evaluate the first sequence.

[0147] The target similarity between the first and third sequences is obtained based on the parts of the query sentence that lack specific information or have multiple interpretations and the longest common subsequence that can characterize the similarity between the first and third sequences. The influence of fuzzy expression on the determination of the first sequence is taken into account, and the structural similarity between the first and third sequences is evaluated through the longest common subsequence. The target similarity is determined based on the fuzziness and similarity of the fuzzy expression. The fuzziness is used to adjust the evaluation criteria of the target similarity, thereby enhancing the robustness and adaptability of the overall evaluation.

[0148] By evaluating the resources occupied by the first sequence based on the resources occupied by the third sequence, the first relative resource consumption level of the first sequence is obtained. By using the obtained resource consumption of the third sequence as a reference benchmark, the resources occupied by the second sequence are evaluated, thereby achieving a unified evaluation for specific operation processes and improving the robustness and adaptability of the overall evaluation.

[0149] By taking the resources occupied by the fourth sequence as a benchmark, the resources occupied by the second sequence are evaluated to obtain the second relative resources of the second sequence. The impact of the resources consumed when clarifying the query statement on the rationality of the solution is considered. The resources consumed by the operation corresponding to the fourth sequence of the third sequence obtained by the clarifying query statement are used as the reference benchmark for evaluation. The resources consumed by the operation corresponding to the first sequence obtained by the clarifying query statement are evaluated. This achieves a unified evaluation of the clarification operation process, realizes the quantity of the clarification process, and improves the robustness of the overall evaluation.

[0150] By obtaining an evaluation value for evaluating the rationality of obtaining the first sequence based on target similarity, the first relative resource consumption degree, and the second relative resource consumption degree, the rationality of the operation of obtaining the first sequence based on the query task is evaluated from multiple dimensions, ensuring the robustness and adaptability of the evaluation results.

[0151] The device of the embodiment of the present application can execute the method provided by the embodiment of the present application, and its implementation principle is similar. The actions performed by each module in the device of each embodiment of the present application correspond to the steps in the method of each embodiment of the present application. For the detailed functional description of each module of the device, please refer to the description in the corresponding method shown in the previous text, and will not be repeated here.

[0152] An embodiment of the present application provides an electronic device (computer device / equipment / system) comprising a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to implement the steps of a task planning evaluation method. Compared to related art methods, this method provides strong data support for the subsequent rationality evaluation of a first sequence obtained from query-based task planning from multiple dimensions, enabling both process- and result-oriented rationality evaluation of the first sequence. By obtaining a third and fourth sequence, the first sequence can be evaluated by comparing not only the query execution operations in the first and third sequences, but also the query clarification operations in the second and fourth sequences. The structural similarity between the first and third sequences is evaluated using the longest common subsequence, and target similarity is determined based on the ambiguity and similarity of the fuzzy expressions. This fuzziness is used to adjust the target similarity evaluation criteria, enhancing the robustness and adaptability of the overall evaluation. By using the resource consumption of the obtained third sequence as a reference benchmark for evaluating the resources occupied by the second sequence, a unified evaluation is achieved for the specific process, improving the robustness and adaptability of the overall evaluation. Taking into account the impact of the resources consumed when clarifying the query statement on the rationality of the solution, the resources consumed by the operations corresponding to the fourth sequence of the third sequence of the clarifying query statement are used as the reference benchmark for evaluation. The resources consumed by the operations corresponding to the first sequence of the clarifying query statement are evaluated. This achieves a unified evaluation of the clarification operation process and an evaluation of the rationality of the operations of obtaining the first sequence based on the query task from multiple dimensions, ensuring the robustness and adaptability of the evaluation results.

[0153] In an alternative embodiment, an electronic device is provided, such as Figure 13 As shown, Figure 13The electronic device 4000 shown includes: a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, via a bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, which may be used for data exchange between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the number of transceivers 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present application.

[0154] Processor 4001 can be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 4001 can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0155] Bus 4002 may include a path for transmitting information between the above components. Bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. Bus 4002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 13 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0156] The memory 4003 may be a ROM (Read Only Memory) or other type of static storage device that can store static information and instructions, a RAM (Random Access Memory) or other type of dynamic storage device that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium that can be used to carry or store computer programs and can be read by a computer, without limitation herein.

[0157] The memory 4003 is used to store the computer program for executing the embodiment of the present application, and the execution is controlled by the processor 4001. The processor 4001 is used to execute the computer program stored in the memory 4003 to implement the steps shown in the above method embodiment.

[0158] Among them, the electronic equipment package may include but is not limited to mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., as well as fixed terminals such as digital TVs, desktop computers, etc. Figure 13 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0159] The present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps and corresponding contents of the aforementioned method embodiments. Compared to the prior art, this method provides strong data support for the subsequent evaluation of the rationality of the first sequence obtained from multiple dimensions of task planning based on a query statement, enabling both process- and result-oriented evaluation of the rationality of the first sequence. By obtaining the third and fourth sequences, the first sequence can be evaluated not only by comparing the query execution operations in the first and third sequences, but also by comparing the query clarification operations in the second and fourth sequences. The structural similarity between the first and third sequences is evaluated using the longest common subsequence, and target similarity is determined based on the ambiguity and similarity of the fuzzy expressions. This is used to adjust the target similarity evaluation criteria based on ambiguity, thereby enhancing the robustness and adaptability of the overall evaluation. By using the resource consumption of the obtained third sequence as a reference benchmark to evaluate the resources occupied by the second sequence, a unified evaluation is achieved for the specific process, improving the robustness and adaptability of the overall evaluation. Taking into account the impact of the resources consumed when clarifying the query statement on the rationality of the solution, the resources consumed by the operations corresponding to the fourth sequence of the third sequence of the clarifying query statement are used as the reference benchmark for evaluation. The resources consumed by the operations corresponding to the first sequence of the clarifying query statement are evaluated. This achieves a unified evaluation of the clarification operation process and an evaluation of the rationality of the operations of obtaining the first sequence based on the query task from multiple dimensions, ensuring the robustness and adaptability of the evaluation results.

[0160] It should be noted that the computer-readable medium mentioned in the present disclosure may be a computer-readable signal medium or a computer-readable medium, or any combination thereof. Computer-readable storage media may include, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or components, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such a propagated data signal may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wire, optical cable, RF (radio frequency), or any suitable combination thereof.

[0161] The present application also provides a computer program product, including a computer program, which, when executed by a processor, can implement the steps and corresponding contents of the aforementioned method embodiment. Compared with the prior art, it can achieve: This provides strong data support for the subsequent evaluation of the rationality of the first sequence obtained from task planning based on query statements from multiple dimensions, and can achieve an evaluation of the rationality of obtaining the first sequence from both an operational process and an outcome perspective. By obtaining the third and fourth sequences, not only can the first sequence be evaluated by comparing the operations of executing the query statement in the first and third sequences, but the first sequence can also be evaluated by comparing the operations of clarifying the query statement in the second and fourth sequences. The structural similarity of the first and third sequences is evaluated through the longest common subsequence, and the target similarity is determined based on the degree of ambiguity and similarity of the fuzzy expression. This is used to adjust the evaluation criteria of the target similarity through fuzziness, thereby enhancing the robustness and adaptability of the overall evaluation. By using the resource consumption of the obtained third sequence as a reference benchmark to evaluate the resources occupied by the second sequence, a unified evaluation oriented towards the specific operational process is achieved, improving the robustness and adaptability of the overall evaluation. Taking into account the impact of the resources consumed when clarifying the query statement on the rationality of the solution, the resources consumed by the operations corresponding to the fourth sequence of the third sequence of the clarifying query statement are used as the reference benchmark for evaluation. The resources consumed by the operations corresponding to the first sequence of the clarifying query statement are evaluated. This achieves a unified evaluation of the clarification operation process and an evaluation of the rationality of the operations of obtaining the first sequence based on the query task from multiple dimensions, ensuring the robustness and adaptability of the evaluation results.

[0162] In the specification and claims of this application and the accompanying drawings, the terms "first," "second," "third," "fourth," "1," "2," and so on (if any) are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that the terms used in this manner are interchangeable where appropriate, such that the embodiments of the present application described herein can be practiced in an order other than that shown or described.

[0163] It should be understood that, although each operation step is indicated by arrows in the flowchart of the embodiment of the present application, the order of implementation of these steps is not limited to the order indicated by the arrows. Unless otherwise clearly stated herein, in some implementation scenarios of the embodiment of the present application, the implementation steps in each flowchart can be performed in other orders according to demand. In addition, some or all of the steps in each flowchart can include multiple sub-steps or multiple stages based on actual implementation scenarios. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage in these sub-steps or stages can also be executed at different times respectively. Under different scenarios at the execution time, the execution order of these sub-steps or stages can be flexibly configured according to demand, and the embodiment of the present application does not limit this.

[0164] The above are only optional implementation methods for some implementation scenarios of this application. It should be pointed out that for ordinary technicians in this technical field, without departing from the technical concept of the solution of this application, the use of other similar implementation methods based on the technical ideas of this application also falls within the protection scope of the embodiments of this application.

Claims

1. A task planning evaluation method, characterized in that: include: Obtaining a query statement, clarifying the query statement, and obtaining a clarified query statement, a first sequence, and a second sequence, wherein the first sequence includes first operations performed on the clarified query statement, and the second sequence includes second operations performed to clarify the query statement; Obtaining a third sequence and a fourth sequence, wherein the third sequence includes respective third operations for executing the query statement, and the fourth sequence includes respective fourth operations required for obtaining the third sequence by clarifying the query statement; Obtaining a target similarity between the first sequence and the third sequence based on a portion of the query that lacks specific information or has multiple interpretations and a longest common subsequence between the first sequence and the third sequence; Evaluate the resources occupied by the first sequence based on the resources occupied by the third sequence to obtain a first relative resource consumption level of the first sequence; Taking the resources occupied by the fourth sequence as a benchmark, evaluating the resources occupied by the second sequence to obtain a second relative resource consumption level of the second sequence; An evaluation value for evaluating the rationality of obtaining the first sequence is obtained based on the target similarity, the first relative resource consumption level, and the second relative resource consumption level.

2. The method according to claim 1, characterized in that Obtaining a target similarity between the first sequence and the third sequence based on a portion of the query statement lacking specific information or having multiple interpretations and a longest common subsequence between the first sequence and the third sequence includes: each part of the query sentence that lacks specific information or has multiple interpretations is treated as fuzzy information, and a fuzzy type of each fuzzy information is determined; Counting each fuzzy information and fuzzy type to obtain the fuzziness level of the query statement; obtaining an initial similarity between the first sequence and the third sequence based on the longest common subsequence between the first sequence and the third sequence; The initial similarity is weighted according to the fuzziness level to obtain the target similarity.

3. The method according to claim 2, characterized in that The step of performing statistics on each fuzzy information and fuzzy type to obtain the fuzziness level of the query statement includes: Counting each piece of fuzzy information and its corresponding fuzzy type to obtain a first quantity of fuzzy information corresponding to each fuzzy type; For each fuzzy type, obtaining a first weight of the fuzzy type, and obtaining a first fuzzy value of the fuzzy type based on the first weight and the first quantity; the first fuzzy value is used to represent the fuzziness degree of the fuzzy information of the corresponding fuzzy type in the query statement; The first fuzzy values of each fuzzy type are summed, and the summation result is mapped to a preset interval to obtain the fuzziness level of the query statement.

4. The method according to claim 1, wherein The third sequence is obtained by: Obtain at least one reference sequence based on the query statement, where the reference sequence includes reference operations for executing the query statement; For each reference sequence, obtaining an initial similarity between the reference sequence and the first sequence based on the longest common subsequence between the reference sequence and the first sequence; The reference sequence with the largest initial similarity is selected as the third sequence.

5. The method according to claim 4, characterized in that The method for obtaining an initial similarity between the first sequence and any reference sequence based on the longest common subsequence between the first sequence and any reference sequence includes: For any reference sequence, obtaining a matching pair sequence based on the longest common subsequence between the reference sequence and the first sequence, wherein each matching pair in the matching pair sequence includes a first index of an element in the longest common subsequence in the first sequence and a second index in the reference sequence; For each matching pair, obtain a corresponding first operation from the first sequence based on a first index in the matching pair, and obtain a corresponding fifth operation from the reference sequence based on a second index in the matching pair; determining a first operation type of the first operation and a second operation type of the fifth operation, and determining a first similarity between the first operation and the fifth operation based on the semantic similarity between the first operation and the fifth operation, the first operation type, and the second operation type; A first operation number of operations in the reference sequence is obtained, each first similarity is summed, and a ratio between the summed result and the first operation number is used as an initial similarity between the first sequence and the reference sequence.

6. The method according to claim 1, characterized in that The step of evaluating the resources occupied by the first sequence based on the resources occupied by the third sequence to obtain a first relative resource consumption level of the first sequence includes: For any one of the first sequence and the third sequence, collect statistics based on the resources occupied by executing the sequence to obtain resource consumption of the sequence; Obtaining a first number of first operations in the first sequence and a second number of third operations in the third sequence; If the first amount is not greater than the second amount, the ratio of the second resource consumption to the first resource consumption is used as the first relative resource consumption level, the first resource consumption being the resource consumption of the first sequence, and the second resource consumption being the resource consumption of the third sequence; If the first quantity is greater than the second quantity, the ratio between the second resource consumption and the first resource consumption is adjusted based on the pre-set first parameter value and the first difference, and the adjustment result is used as the first relative resource consumption level; the first difference is the difference between the first quantity and the second quantity.

7. The method according to claim 6, characterized in that For any one of the first sequence and the third sequence, statistics are collected based on the resources occupied by executing the sequence to obtain the resource consumption of the sequence, including: Obtain the maximum CPU execution time and maximum memory usage generated by historical execution of the query statement; For each operation in any one of the first and third sequences, obtaining a current first CPU execution time and a first memory usage of the operation, normalizing the first CPU execution time based on the maximum CPU execution time to obtain a first result corresponding to the operation, normalizing the first memory usage based on the maximum memory usage to obtain a second result corresponding to the operation, and performing a weighted summation of the first and second results corresponding to the operation to obtain a third resource consumption of the operation; For any one of the first sequence and the third sequence, the third resource consumption of each operation in the sequence is summed to obtain the resource consumption of the sequence.

8. The method according to claim 1, characterized in that The resources occupied by the second sequence are evaluated based on the resources occupied by the fourth sequence to obtain a second relative resource consumption level of the second sequence, including: For any one of the second and fourth sequences, for each operation in the sequence, obtaining an operation type corresponding to the operation, determining a fourth resource consumption of the operation based on a pre-established correspondence between resource consumption and operation type, and summing the fourth resource consumption of each operation in the sequence to obtain a resource consumption of the sequence; obtaining a third number of second operations in the second sequence, obtaining a fourth number of fourth operations in the fourth sequence, and determining a second parameter value based on a difference between the third number and the fourth number; The ratio between the fourth resource consumption and the third resource consumption is multiplied by the second parameter value to obtain the second relative resource consumption level; the third resource consumption is the resource consumption of the second sequence, and the fourth resource consumption is the resource consumption of the third sequence.

9. The method according to claim 8, characterized in that The determining of a second parameter value based on a difference between the third quantity and the fourth quantity includes: If the third number is not greater than the fourth number, obtaining the second parameter value according to a relative difference between the third number and the fourth number; If the third number is greater than the fourth number, the second parameter value is obtained according to the absolute difference between the third number and the fourth number.

10. The method according to claim 1, characterized in that The step of obtaining an evaluation value for evaluating the rationality of obtaining the first sequence further includes: Obtain evaluation values corresponding to the first sequence of multiple query statements; For each first sequence, obtaining the query type to which the query statement corresponding to the first sequence belongs, and determining a second weight value of the first sequence based on a pre-established correspondence between query types and weights; weighting and summing the evaluation values of each first sequence by the corresponding second weight value to obtain a first summation result, summing the second weight values to obtain a second summation result, and using the ratio of the first summation result to the second summation result as the second evaluation value; The query type of a query statement is related to the query complexity, query frequency or importance of the corresponding business of the query statement.

11. A task planning evaluation device, characterized in that: include: a first processing module configured to obtain a query statement, clarify the query statement, and obtain a clarified query statement, a first sequence, and a second sequence, wherein the first sequence includes first operations performed on the clarified query statement, and the second sequence includes second operations performed to clarify the query statement; a second processing module, configured to obtain a third sequence and a fourth sequence, wherein the third sequence includes respective third operations for executing the query statement, and the fourth sequence includes respective fourth operations required for obtaining the third sequence by clarifying the query statement; a third processing module, configured to obtain a target similarity between the first sequence and the third sequence based on a portion of the query that lacks specific information or has multiple interpretations and a longest common subsequence between the first sequence and the third sequence; a fourth processing module, configured to evaluate the resources occupied by the first sequence based on the resources occupied by the third sequence, to obtain a first relative resource consumption level of the first sequence; a fifth processing module, configured to evaluate the resources occupied by the second sequence based on the resources occupied by the fourth sequence, to obtain a second relative resource consumption level of the second sequence; The sixth processing module is configured to obtain an evaluation value for evaluating the rationality of obtaining the first sequence based on the target similarity, the first relative resource consumption level, and the second relative resource consumption level.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory, wherein: The processor executes the computer program to implement the method according to any one of claims 1 to 10.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images