Task planning evaluation method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application provide a task planning evaluation method and device, electronic equipment and storage medium, and relate to the technical field of computer. The method comprises: clarifying a query statement to obtain a clarified query statement, a first sequence, a second sequence, a third sequence and a fourth sequence; obtaining a target similarity between the first sequence and the third sequence according to a longest common subsequence between the part of the query statement lacking specific information or having multiple interpretations, the first sequence and the third sequence; taking the resources occupied by the third sequence as a benchmark, evaluating the resources occupied by the first sequence to obtain a first relative resource consumption degree; taking the resources occupied by the fourth sequence as a benchmark, evaluating the resources occupied by the second sequence to obtain a second relative resource consumption degree; and obtaining an evaluation value based on the target similarity, the first relative resource consumption degree and the second relative resource consumption degree. The present application realizes evaluation of the rationality of the obtained first sequence from multiple dimensions.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer processing, in particular, the present application relates to a task planning evaluation method and device, electronic equipment and storage medium. BACKGROUND

[0002] At present, in natural language to structured query and multi-agent task planning system, for the evaluation of system execution result, in the related art, most of the technologies usually compare the structured query result output by the system with the standard answer constructed by artificial, to evaluate whether the task planning is reasonable, only consider whether the final output is accurate, there is a problem that the task planning cannot be evaluated from multiple dimensions, thereby leading to poor robustness and adaptability of the evaluation result. SUMMARY

[0003] The embodiments of the present application provide a task planning evaluation method, device, electronic equipment and storage medium, which are used to solve the technical problem that the task planning cannot be evaluated from multiple dimensions.

[0004] According to the first aspect of the embodiments of the present application, a task planning evaluation method is provided, which comprises: obtaining a query statement, clarifying the query statement, obtaining the clarified query statement, a first sequence and a second sequence, the first sequence comprising each first operation of executing the clarified query statement, and the second sequence comprising each second operation of executing the clarified query statement;

[0005] obtaining a third sequence and a fourth sequence, the third sequence comprising each third operation of executing the query statement, and the fourth sequence comprising each fourth operation required for obtaining the third sequence through the clarified query statement;

[0006] According to the part of the query statement lacking specific information or having multiple interpretations, and the longest common subsequence between the first sequence and the third sequence, obtaining the target similarity between the first sequence and the third sequence;

[0007] Taking the resources occupied by the third sequence as a benchmark, the resources occupied by the first sequence are evaluated to obtain the first relative resource consumption degree of the first sequence;

[0008] Taking the resources occupied by the fourth sequence as a benchmark, the resources occupied by the second sequence are evaluated to obtain the second relative resource consumption degree of the second sequence;

[0009] Based on the target similarity, the first relative resource consumption degree and the second relative resource consumption degree, an evaluation value for evaluating the rationality of obtaining the first sequence is obtained.

[0010] According to the second aspect of the embodiments of the present application, a task planning evaluation device is provided, which comprises:

[0011] The first processing module is configured to obtain a query statement, clarify the query statement, and obtain a clarified query statement, a first sequence and a second sequence, the first sequence including each first operation of executing the clarified query statement, and the second sequence including each second operation of executing the clarified query statement;

[0012] The second processing module is configured to obtain a third sequence and a fourth sequence, the third sequence including each third operation of executing the query statement, and the fourth sequence including each fourth operation required for obtaining the third sequence by clarifying the query statement;

[0013] The third processing module is configured to obtain a target similarity between the first sequence and the third sequence according to a part of the query statement that lacks specific information or has multiple interpretations and a longest common subsequence between the first sequence and the third sequence;

[0014] The fourth processing module is configured to evaluate a resource occupied by the first sequence based on a resource occupied by the third sequence, and obtain a first relative resource consumption degree of the first sequence;

[0015] The fifth processing module is configured to evaluate a resource occupied by the second sequence based on a resource occupied by the fourth sequence, and obtain a second relative resource consumption degree of the second sequence;

[0016] The sixth processing module is configured to obtain an evaluation value for evaluating rationality of obtaining the first sequence based on the target similarity, the first relative resource consumption degree and the second relative resource consumption degree.

[0017] According to a third aspect of the embodiments of the present application, an electronic device is provided, which includes a memory, a processor and a computer program stored in the memory, and the processor implements the steps of the method provided in the first aspect when executing the program.

[0018] According to a fourth aspect of the embodiments of the present application, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the steps of the method provided in the first aspect.

[0019] According to a fifth aspect of the embodiments of the present application, a computer program product is provided, which includes computer instructions stored in a computer readable storage medium, and when a processor of a computer device reads the computer instructions from the computer readable storage medium, the processor executes the computer instructions to enable the computer device to implement the steps of the method provided in the first aspect.

[0020] The technical scheme provided in the embodiments of the present application has the following beneficial effects:

[0021] The task planning evaluation method provided in this application obtains a query statement, clarifies the query statement, obtains a clarified statement, obtains a first sequence including each first operation of executing the clarified query statement, and obtains each second operation including each second operation executed by clarifying the query statement. This provides strong data support for the subsequent evaluation of the rationality of obtaining the first sequence based on the query statement from multiple dimensions, and can realize the evaluation of the rationality of obtaining the first sequence from both operation process-oriented and result-oriented perspectives.

[0022] By obtaining a third sequence containing the various third operations that execute the query statement, and by obtaining a fourth sequence containing the various fourth operations required to obtain the third sequence by clarifying the query statement, a reference standard is provided for evaluating the first sequence. By obtaining the third and fourth sequences, the first sequence can be evaluated not only by comparing the operations that execute the query statement in the first and third sequences, but also by comparing the operations that clarify the query statement in the second and fourth sequences.

[0023] The target similarity between the first and third sequences is obtained by considering the parts of the query statement that lack specific information or have multiple interpretations, as well as the longest common subsequence that can characterize the similarity between the first and third sequences. The influence of fuzzy expressions on the determination of the first sequence is taken into account. The structural similarity between the first and third sequences is evaluated by the longest common subsequence, and the target similarity is determined based on the degree of fuzziness and similarity of the fuzzy expressions. This is used to adjust the evaluation criteria of target similarity through fuzziness, thereby enhancing the robustness and adaptability of the overall evaluation.

[0024] By using the resources occupied by the third sequence as a benchmark to evaluate the resources occupied by the first sequence, the first relative resource consumption degree of the first sequence is obtained. By using the obtained resource consumption of the third sequence as a reference benchmark to evaluate the resources occupied by the second sequence, a unified evaluation oriented towards specific operational processes is achieved, improving the robustness and adaptability of the overall evaluation.

[0025] By evaluating the resources occupied by the second sequence based on the resources occupied by the fourth sequence, the second relative resources of the second sequence are obtained. The impact of the resources consumed when clarifying the query statement on the rationality of the solution is considered. The resources consumed by the operation of clarifying the query statement to obtain the fourth sequence corresponding to the third sequence are used as the reference benchmark for evaluation. The resources consumed by the operation of clarifying the query statement to obtain the first sequence corresponding to the first sequence are evaluated. This achieves a unified evaluation of the clarification operation process, realizes the quantity of the clarification process, and improves the robustness of the overall evaluation.

[0026] By obtaining an evaluation value based on target similarity, first relative resource consumption, and second relative resource consumption, the rationality of obtaining the first sequence is evaluated from multiple dimensions, ensuring the robustness and adaptability of the evaluation results. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below.

[0028] Figure 1 A schematic diagram of the system architecture for implementing the evaluation method of task planning provided in the embodiments of this application;

[0029] Figure 2 A flowchart illustrating a task planning evaluation method provided in an embodiment of this application;

[0030] Figure 3 A flowchart illustrating the method for determining target similarity in a task planning evaluation method provided in this application embodiment;

[0031] Figure 4 A flowchart illustrating a method for determining the degree of fuzziness in a task planning evaluation method provided in this application embodiment;

[0032] Figure 5 A flowchart illustrating the method for obtaining the third sequence in a task planning evaluation method provided in this application embodiment;

[0033] Figure 6 A flowchart illustrating the method for determining the initial similarity in a task planning evaluation method provided in this application embodiment;

[0034] Figure 7 A flowchart illustrating a method for determining the first relative resource consumption level in a task planning evaluation method provided in an embodiment of this application;

[0035] Figure 8 A flowchart illustrating a method for determining the resource consumption of a sequence in a task planning evaluation method provided in this application embodiment;

[0036] Figure 9 A flowchart illustrating a method for determining the second relative resource consumption level of a sequence in a task planning evaluation method provided in this application embodiment;

[0037] Figure 10 A flowchart illustrating another task planning evaluation method provided in this application embodiment;

[0038] Figure 11A flowchart illustrating another task planning evaluation method provided in an embodiment of this application;

[0039] Figure 12 A schematic diagram of the structure of a task planning evaluation device provided in an embodiment of this application;

[0040] Figure 13 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0041] The embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the embodiments described below with reference to the accompanying drawings are exemplary descriptions for explaining the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions of the embodiments of this application.

[0042] Those skilled in the art will understand that, unless otherwise stated, the singular forms “a,” “an,” and “the” used herein may also include the plural forms. It should be further understood that the terms “comprising” and “including” as used in embodiments of this application mean that the corresponding feature can be implemented as the presented feature, information, data, step, operation, element, and / or component, but do not exclude implementation as other features, information, data, step, operation, element, component, and / or combinations thereof supported by the art. It should be understood that when we say that an element is “connected” or “coupled” to another element, the one element can be directly connected or coupled to the other element, or it can mean that the one element and the other element establish a connection relationship through an intermediate element. Furthermore, “connected” or “coupled” as used herein can include wireless connection or wireless coupling. The term “and / or” as used herein indicates at least one of the items defined by the term; for example, “A and / or B” can be implemented as “A,” or as “B,” or as “A and B.”

[0043] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0044] The following explains the terminology and related technologies involved in this application:

[0045] Currently, related technologies typically use metrics such as Exact Match and BLEU to evaluate whether a statement is completely consistent with the standard statement. While this approach is simple, it is highly sensitive to format and syntax; even slight differences are considered failures, and it cannot accommodate semantically equivalent solutions with different structures. Most methods do not consider the rationality of the process path, focusing only on whether the final output is correct. The evaluation methods are largely lacking regarding whether the task has undergone reasonable reasoning, whether there are redundant steps, and whether there is excessive resource consumption. When executing complex queries (such as JOIN, multi-level nesting, etc.), the task execution cost is not estimated from the perspective of actual resources (such as CPU time and memory usage), but only roughly approximated by the number of task steps or query length, which lacks measurability. User input often contains ambiguous semantics (such as "last month", "recent period", etc.). When dealing with ambiguous expressions, traditional methods either rely on manual rules or rely entirely on the model to "guess". However, in the evaluation, the uncertainty brought about by this ambiguity is not modeled, nor is it reflected in the accuracy or cost score. When the model cannot understand the user's intent, some systems clarify the semantics through human-computer interaction or empirical reasoning, but such behavior is usually regarded as "error handling" rather than part of the normal evaluation process, and the related costs and efficiency are not standardized in measurement.

[0046] Therefore, related technologies remain at the stage of result-oriented and structure-aligned evaluation, lacking process-oriented methods for evaluating task rationality, especially lacking a unified mechanism for integrating accuracy, cost, ambiguity, and interactive clarification. These shortcomings limit the interpretability and optimizability of evaluation methods in multi-path, multi-modal, and multi-agent collaborative scenarios.

[0047] In view of at least one of the above-mentioned technical problems or areas for improvement in the related technologies, this application proposes a task planning evaluation method. This method obtains a query statement, clarifies the query statement, obtains a clarified statement, obtains a first sequence of first operations including the execution of the clarified query statement, and obtains a second sequence of second operations including the execution of the clarified query statement. This provides strong data support for the subsequent evaluation of the rationality of obtaining the first sequence based on the query statement from multiple dimensions, and can realize the evaluation of the rationality of obtaining the first sequence from both an operation process-oriented and result-oriented perspective.

[0048] By obtaining a third sequence containing the various third operations that execute the query statement, and by obtaining a fourth sequence containing the various fourth operations required to obtain the third sequence by clarifying the query statement, a reference standard is provided for evaluating the first sequence. By obtaining the third and fourth sequences, the first sequence can be evaluated not only by comparing the operations that execute the query statement in the first and third sequences, but also by comparing the operations that clarify the query statement in the second and fourth sequences.

[0049] The target similarity between the first and third sequences is obtained by considering the parts of the query statement that lack specific information or have multiple interpretations, as well as the longest common subsequence that can characterize the similarity between the first and third sequences. The influence of fuzzy expressions on the determination of the first sequence is taken into account. The structural similarity between the first and third sequences is evaluated by the longest common subsequence, and the target similarity is determined based on the degree of fuzziness and similarity of the fuzzy expressions. This is used to adjust the evaluation criteria of target similarity through fuzziness, thereby enhancing the robustness and adaptability of the overall evaluation.

[0050] By using the resources occupied by the third sequence as a benchmark to evaluate the resources occupied by the first sequence, the first relative resource consumption degree of the first sequence is obtained. By using the obtained resource consumption of the third sequence as a reference benchmark to evaluate the resources occupied by the second sequence, a unified evaluation oriented towards specific operational processes is achieved, improving the robustness and adaptability of the overall evaluation.

[0051] By evaluating the resources occupied by the second sequence based on the resources occupied by the fourth sequence, the second relative resources of the second sequence are obtained. The impact of the resources consumed when clarifying the query statement on the rationality of the solution is considered. The resources consumed by the operation of clarifying the query statement to obtain the fourth sequence corresponding to the third sequence are used as the reference benchmark for evaluation. The resources consumed by the operation of clarifying the query statement to obtain the first sequence corresponding to the first sequence are evaluated. This achieves a unified evaluation of the clarification operation process, realizes the quantity of the clarification process, and improves the robustness of the overall evaluation.

[0052] By obtaining an evaluation value based on target similarity, first relative resource consumption, and second relative resource consumption, the rationality of obtaining the first sequence is evaluated from multiple dimensions, ensuring the robustness and adaptability of the evaluation results.

[0053] The technical solutions of this application and their effects are described below through several exemplary embodiments. It should be noted that the following embodiments can be referenced, borrowed from, or combined with each other. Identical terms, similar features, and similar implementation steps in different embodiments will not be repeated.

[0054] Figure 1 This is a schematic diagram of the system architecture for implementing the evaluation method of task planning provided in an embodiment of this application, wherein the system architecture includes: a terminal 120 and a server 140.

[0055] Terminal 120 has an application program with a task planning evaluation method installed and running, which is used to evaluate the rationality of obtaining a first sequence of query tasks.

[0056] Terminal 120 is connected to server 140 via a wireless network or a wired network.

[0057] Server 140 includes at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. Illustratively, server 140 includes a processor 144 and a memory 142, the memory 142 including a display module 1421, a control module 1422, and a receiving module 1423. Server 140 is used to provide background services for the application of the method. Optionally, server 140 undertakes the primary computing work, and terminal 120 undertakes secondary computing work; or, server 140 undertakes secondary computing work, and terminal 120 undertakes primary computing work; or, server 140 and terminal 120 collaborate on computing using a distributed computing architecture.

[0058] Optionally, the device type of the terminal includes at least one of the following: smartphone, tablet computer, e-book reader, Moving Picture Experts Group Audio Layer III (MP3) player, Moving Picture Experts Group Audio Layer IV (MP4) player, laptop computer, and desktop computer.

[0059] Those skilled in the art will understand that the number of terminals described above can be more or less. For example, there may be only one terminal, or there may be dozens or hundreds of terminals, or even more. This application does not limit the number of terminals or the type of device.

[0060] This application provides a method for evaluating task planning, such as... Figure 2 As shown, the method includes:

[0061] S101, obtain the query statement, clarify the query statement, and obtain the clarified query statement, the first sequence, and the second sequence.

[0062] In this embodiment of the application, the query statement is composed of natural language. The query statement is used to request and obtain certain information. The query statement usually includes the query target, query conditions, query parameters, etc. For example, the query statement can be "provide the promotional activities of various merchants in the past month".

[0063] In this embodiment of the application, in practical applications, the query statements provided by users often have a certain degree of ambiguity. For example, the query statements may have unclear time ranges, vague expression of query conditions, or unclear field references. This ambiguity not only affects the understanding of the query statements but may also cause significant deviations in the breakdown and execution results of downstream tasks. Therefore, it is necessary to clarify the query statements in order to more accurately perform the corresponding operations based on the obtained query statements and provide the user with the information they need.

[0064] In this embodiment of the application, clarification refers to further clarifying the query intent of the query statement, eliminating any ambiguous parts in the query statement, and further clarifying any vague query conditions. The clarification operation can be to further infer the query statement based on existing data, or to supplement the query statement through interaction with the user.

[0065] In this embodiment, the first sequence includes various first operations for executing the clarified query statement. That is, after obtaining the clarified query statement, the task planning system obtains a first sequence that can execute the clarified query statement based on the clarified query statement. The first sequence includes multiple first operations arranged in an orderly manner. By executing each first operation in the order of the first operations in the first sequence, the query target required by the query statement can be obtained. The first operation can be to obtain data within one month, to sum specific values, or to classify specific content, etc.

[0066] In this embodiment of the application, the second sequence includes various second operations performed in clarifying the query statement. The second operations in the second sequence refer to the operations performed in the process of clarifying the query statement. The second operations may be to further infer the query statement based on existing data, or to supplement the query statement through interaction with the user.

[0067] In this embodiment of the application, after the task planning system obtains the query statement input by the user, it clarifies the query statement, determines a second sequence including clarification operations (second operations), performs task planning on the clarified query statement, and obtains a first sequence containing multiple first operations arranged in an orderly manner.

[0068] S102, obtain the third and fourth sequences.

[0069] In this embodiment of the application, the third sequence includes various third operations for executing the query statement. The third sequence is a predefined standard scheme for the query statement, that is, each third operation in the third sequence is a standard operation for executing the query statement.

[0070] In the embodiments of this application, by decomposing the query statement, multiple reasonable execution paths can usually be obtained, namely standard schemes. When there are multiple standard methods for executing the query statement, the third sequence is the scheme that is closest to the first sequence among the standard schemes.

[0071] In this embodiment of the application, the fourth sequence includes each of the fourth operations required to obtain the third sequence by clarifying the query statement. Since the query statement may be ambiguous, the third sequence can be obtained by determining which clarification operations to perform on the query task, thereby obtaining each of the fourth operations required to obtain the third sequence by clarifying the query statement, and thus determining the fourth sequence.

[0072] S103. Based on the parts of the query statement that lack specific information or have multiple interpretations, and the longest common subsequence between the first sequence and the third sequence, obtain the target similarity between the first sequence and the third sequence.

[0073] In this embodiment of the application, since the third sequence serves as the standard scheme for executing query statements, that is, as the benchmark for judging whether each operation in the first sequence executes the query statement reasonably, the similarity between the first sequence and the third sequence is determined based on the longest common subsequence between the first sequence and the third sequence, which can quantify the similarity between the first sequence and the third sequence in terms of structure (operation execution order) and semantics (specific operations).

[0074] In this embodiment of the application, the part of the query statement that lacks specific information or has multiple interpretations is obtained. Based on the above part, the ambiguity of the query statement can be determined. The ambiguity affects the task planning system's judgment on each first operation required to execute the query statement. Therefore, when determining the target similarity between the first sequence and the third sequence, it is necessary to consider the impact of the part of the query statement that lacks specific information or has multiple interpretations on the evaluation result.

[0075] In the embodiments of this application, when there are many parts in the query statement that lack specific information or have multiple interpretations, it indicates that the current query question is highly ambiguous. It is necessary to reduce the evaluation standard of the target similarity part in the whole, so as to enhance the robustness and adaptability of the overall evaluation.

[0076] In the embodiments of this application, target similarity is used to characterize the relative similarity between the first sequence and the third sequence, with the third sequence as a benchmark.

[0077] S104. Based on the resources occupied by the third sequence, evaluate the resources occupied by the first sequence to obtain the first relative resource consumption level of the first sequence.

[0078] In this embodiment of the application, since the third sequence is a predefined standard method for executing query statements, the resources required to execute all first operations in the first sequence are evaluated based on the resources required to execute all third operations in the third sequence, so as to obtain the first relative resource consumption level of the first sequence.

[0079] S105, using the resources occupied by the fourth sequence as a benchmark, evaluate the resources occupied by the second sequence to obtain the second relative resource consumption level of the second sequence.

[0080] In this embodiment of the application, since the third sequence is a predefined standard scheme for executing query statements, the fourth sequence obtained by clarifying the query statement is also used as a standard scheme for clarifying operations. Therefore, based on the resources required for all fourth operations in the fourth sequence, the resources required for executing all second operations in the second sequence are evaluated to obtain the second relative resource consumption level of the second sequence.

[0081] S106, Based on target similarity, first relative resource consumption degree, and second relative resource consumption degree, obtain an evaluation value for assessing the rationality of obtaining the first sequence.

[0082] In this embodiment of the application, the product of target similarity, first relative resource consumption degree, and second relative resource consumption degree is used as the evaluation value.

[0083] In this embodiment of the application, the evaluation value used to characterize the rationality of the task planning system in obtaining the first sequence is determined from the dimensions of the relative similarity between the first sequence and the third sequence, the relative resource consumption between the first sequence and the third sequence, and the relative resource consumption between the second sequence and the fourth sequence. The evaluation value is determined from multiple dimensions by comprehensively considering the impact of the ambiguity of the query statement, the resources consumed by the task execution, and the resources consumed by clarifying the query statement on the rationality of obtaining the first sequence.

[0084] In this embodiment, the importance of target similarity, first relative resource consumption, and second relative resource consumption can be determined separately, and corresponding weights can be assigned to each of the above values. For example, in some scenarios, the similarity between the first sequence and the third sequence obtained by the task is the primary objective; in other scenarios, resource efficiency is more important. Therefore, the evaluation value is determined by assigning corresponding weights according to the importance of each content in the current application scenario, so as to make the evaluation results more adaptable and robust.

[0085] In this embodiment of the application, the evaluation value is determined by the following formula:

[0086]

[0087] in, Indicates the evaluation value. Indicates target similarity. Indicates the first relative level of resource consumption. This indicates the second relative level of resource consumption.

[0088] In the above scheme, by obtaining the query statement and clarifying the query statement, a first sequence of first operations including the execution of the clarified query statement is obtained, and a second sequence of second operations including the execution of the clarified query statement is obtained. This provides strong data support for the subsequent task planning based on the query statement to obtain the rationality of the first sequence from multiple dimensions, and can realize the evaluation of the rationality of obtaining the first sequence from both operation process-oriented and result-oriented perspectives.

[0089] By obtaining a third sequence containing the various third operations that execute the query statement, and by obtaining a fourth sequence containing the various fourth operations required to obtain the third sequence by clarifying the query statement, a reference standard is provided for evaluating the first sequence. By obtaining the third and fourth sequences, the first sequence can be evaluated not only by comparing the operations that execute the query statement in the first and third sequences, but also by comparing the operations that clarify the query statement in the second and fourth sequences.

[0090] The target similarity between the first and third sequences is obtained by considering the parts of the query statement that lack specific information or have multiple interpretations, as well as the longest common subsequence that can characterize the similarity between the first and third sequences. The influence of fuzzy expressions on the determination of the first sequence is taken into account. The structural similarity between the first and third sequences is evaluated by the longest common subsequence, and the target similarity is determined based on the degree of fuzziness and similarity of the fuzzy expressions. This is used to adjust the evaluation criteria of target similarity through fuzziness, thereby enhancing the robustness and adaptability of the overall evaluation.

[0091] By using the resources occupied by the third sequence as a benchmark to evaluate the resources occupied by the first sequence, the first relative resource consumption degree of the first sequence is obtained. By using the obtained resource consumption of the third sequence as a reference benchmark to evaluate the resources occupied by the second sequence, a unified evaluation oriented towards specific operational processes is achieved, improving the robustness and adaptability of the overall evaluation.

[0092] By evaluating the resources occupied by the second sequence based on the resources occupied by the fourth sequence, the second relative resources of the second sequence are obtained. The impact of the resources consumed when clarifying the query statement on the rationality of the solution is considered. The resources consumed by the operation of clarifying the query statement to obtain the fourth sequence corresponding to the third sequence are used as the reference benchmark for evaluation. The resources consumed by the operation of clarifying the query statement to obtain the first sequence corresponding to the first sequence are evaluated. This achieves a unified evaluation of the clarification operation process, realizes the quantity of the clarification process, and improves the robustness of the overall evaluation.

[0093] By obtaining an evaluation value based on target similarity, first relative resource consumption, and second relative resource consumption, the rationality of obtaining the first sequence is evaluated from multiple dimensions, ensuring the robustness and adaptability of the evaluation results.

[0094] Based on the above embodiments, as an optional embodiment, the method for determining target similarity is as follows: Figure 3 As shown, the specific content is as follows:

[0095] S201, each part of the query statement that lacks specific information or has multiple interpretations is treated as fuzzy information, and the fuzzy type of each fuzzy information is determined;

[0096] S202, Perform statistics on each fuzzy information and fuzzy type to obtain the degree of fuzziness of the query statement;

[0097] S203, based on the longest common subsequence between the first sequence and the third sequence, obtain the initial similarity between the first sequence and the third sequence;

[0098] S204, the initial similarity is weighted according to the degree of ambiguity to obtain the target similarity.

[0099] In S201 of this application embodiment, each part of the query statement that lacks specific information or has multiple interpretations is obtained, and the above parts are regarded as fuzzy information. The type of fuzzy information is determined according to the specific text content of the query statement.

[0100] In the embodiments of this application, the type of fuzzy information can be: unclear time expression, such as "recent period"; the type of fuzzy information can also be: incomplete or ambiguous conditions, such as "products that users clicked a lot"; the fuzzy type can also be: fuzzy field reference, such as "this value", "this item", "previous"; the fuzzy type can also be: multiple layers of negation or double conditions, such as "not excluding users in an inactive state".

[0101] In S202 of this application embodiment, the degree of fuzziness varies for different fuzzy types. The degree of fuzziness of the query statement is obtained by statistically analyzing each fuzzy information and fuzzy type in the query statement.

[0102] In S203 of this application embodiment, the longest common subsequence between the first sequence and the third sequence is obtained, and then the length of the longest common subsequence is mapped to the range [0,1] to obtain the initial similarity between the first sequence and the third sequence.

[0103] In S204 of this application embodiment, in order to more objectively and reasonably evaluate the quality of the prediction scheme, when calculating the target similarity between the first sequence and the third sequence, the initial similarity is weighted according to the degree of ambiguity. The lower the degree of ambiguity, the closer the weight value is to 1, that is, the standard of similarity evaluation is not reduced; when the degree of ambiguity is higher, the weight value is closer to 0.5, that is, the similarity is appropriately tolerated by the degree of ambiguity.

[0104] In the above scheme, the degree of fuzziness of the query question is determined based on the existence of fuzzy information and the fuzziness type of each fuzzy information. The initial similarity between the first sequence and the third sequence is determined based on the longest common subsequence between the first sequence and the second sequence. Then, based on the degree of fuzziness and the initial similarity, the target similarity between the first sequence and the third sequence is determined. This realizes the consideration of the impact of the degree of fuzziness of the query question on the acquisition of the first sequence in the similarity calculation, improves the robustness and adaptability of the overall evaluation, and realizes a more objective and reasonable evaluation of the rationality of the acquired first sequence.

[0105] Based on the above embodiments, as an optional embodiment, the method for determining the degree of ambiguity is as follows: Figure 4 As shown, the specific content is as follows:

[0106] S301, Statistically analyze each fuzzy information and its corresponding fuzzy type to obtain the first quantity of fuzzy information corresponding to each fuzzy type;

[0107] S302, for each fuzzy type, obtain the first weight of the fuzzy type, and based on the first weight and the first quantity, obtain the first fuzzy value of the fuzzy type; the first fuzzy value is used to characterize the degree of fuzziness of the fuzzy information of the corresponding fuzzy type in the query statement;

[0108] S303, sum the first fuzzy values ​​of each fuzzy type, and map the summation result to a preset interval to obtain the fuzziness level of the query statement.

[0109] In S301 of this application embodiment, each piece of fuzzy information is classified according to the fuzzy type, and the number of fuzzy information under each fuzzy type is determined, thereby obtaining the first number of fuzzy information corresponding to each fuzzy type.

[0110] In S302 of this application embodiment, since the fuzziness types are different, the fuzziness degree of each fuzzy information will also be different. Different fuzziness types correspond to different first weights. For each fuzziness type, the first fuzzy value of all fuzzy information of that fuzziness type is determined based on the first weight and the first quantity. That is, based on the number of times the fuzzy information of the current fuzziness type appears in the query statement and the first weight corresponding to that fuzziness type, the first fuzzy value of the fuzzy information of that fuzziness type in the query statement is determined.

[0111] In S303 of this application embodiment, since there is at least one type of fuzziness in the fuzzy information in the query statement, the first fuzzy value of each fuzzy type is summed and the summation result is mapped to a preset interval to obtain the fuzziness level of the query statement. The preset interval is [0,1].

[0112] In this embodiment of the application, the fuzziness level of the query statement is obtained using the following formula:

[0113]

[0114]

[0115] in, The number of fuzzy types contained in the query statement. For the first A fuzzy type, For the first The first weight of a fuzzy type For the query statement to contain the first The first quantity of fuzzy information of a fuzzy type, To find the sum, The degree of ambiguity.

[0116] In the above set of formulas, when the query question is clear and explicit (i.e.) When it is close to 0, It is also close to 0, indicating that the evaluation criteria for similarity are not lowered; when the query question is extremely ambiguous, Approaching 1, it is used to provide appropriate tolerance for similarity in the final evaluation.

[0117] In the above scheme, the fuzzy type of fuzzy information and the number of times each fuzzy type of information appears are determined by analyzing the query statement. Based on the first weight of each fuzzy type and the number of times the fuzzy information of each fuzzy type appears, the degree of fuzziness of the query question is determined. This further reflects the overall degree of fuzziness of the query question more objectively and realistically, so as to make the evaluation of the first sequence more objective.

[0118] Based on the above embodiments, as an optional embodiment, the method for obtaining the third sequence is as follows:Figure 5 As shown, the specific content is as follows:

[0119] S401, Obtain at least one reference sequence based on the query statement;

[0120] S402, For each reference sequence, obtain the initial similarity between the reference sequence and the first sequence based on the longest common subsequence between the reference sequence and the first sequence;

[0121] S403, select the reference sequence with the highest initial similarity as the third sequence.

[0122] In S401 of this application embodiment, when there are multiple sequences of executable query statements, the sequences can be predefined as standard schemes for executing query statements, i.e., all are reference sequences. Thus, when a query statement is received, at least one reference sequence can be obtained based on the query statement. The reference sequence includes various reference operations for executing the query statement.

[0123] In S402 of this application embodiment, since there are multiple reference sequences, the longest common subsequence between each reference sequence and the first sequence is calculated to obtain the initial similarity between the reference sequence and the first sequence. That is, in this application, a unique sequence is not used as a reference benchmark. Instead, the first sequence is compared with each predefined reference sequence to determine the initial similarity between the first sequence and each reference sequence.

[0124] In S403 of this application embodiment, after determining the initial similarity between the first sequence and each reference sequence, the reference sequence closest to the first sequence is selected as the third sequence and used as the reference benchmark for this evaluation. Therefore, the reference sequence with the largest initial similarity is selected as the third sequence.

[0125] In the above scheme, by selecting the reference sequence with the highest similarity to the first sequence from multiple reference sequences as the third sequence and using it as the reference benchmark for this evaluation, the scheme takes into account the implementation of multi-path operations for query statements, does not rely on a single reference sequence, and selects the reference sequence closest to the first sequence for evaluation, thereby improving the objectivity of the evaluation results.

[0126] Based on the above embodiments, as an optional embodiment, the method for determining the initial similarity is as follows: Figure 6 As shown, the specific content is as follows:

[0127] S501, For any reference sequence, based on the longest common subsequence between the reference sequence and the first sequence, obtain a matching pair sequence. Each matching pair in the matching pair sequence includes the first index of the element in the longest common subsequence in the first sequence and the second index in the reference sequence.

[0128] S502, for each matching pair, obtain the corresponding first operation from the first sequence based on the first index in the matching pair, and obtain the corresponding fifth operation from the reference sequence based on the second index in the matching pair;

[0129] S503, determine the first operation type of the first operation and the second operation type of the fifth operation, and determine the first similarity between the first operation and the fifth operation based on the semantic similarity between the first operation and the fifth operation, the first operation type and the second operation type;

[0130] S504, obtain the first number of operations in the reference sequence, sum the first similarities, and use the ratio between the sum and the first number of operations as the initial similarity between the first sequence and the reference sequence.

[0131] In S501 of this application embodiment, for any reference sequence, a matching pair sequence is obtained based on the longest common subsequence between the current reference sequence and the first sequence. Each matching pair in the matching pair sequence contains the index of any element between the longest common subsequences in the first sequence and the current reference sequence, namely the first index and the second index.

[0132] In S502 of this application embodiment, for each matching pair, each matching pair corresponds to an element in the longest common subsequence. The first operation can be directly obtained from the first sequence through the first index contained in the matching pair, and the corresponding fifth operation can be directly obtained from the current reference sequence through the second index contained in the matching pair.

[0133] In S503 of this application embodiment, the first operation type of the first operation and the second operation type of the fifth operation are determined, the similarity between the first operation type and the second operation type is compared, the semantic similarity between the first operation and the fifth operation is compared, and the first similarity between the first operation and the fifth operation is determined based on the above similarity and semantic similarity; wherein, the value range of the first similarity is [0,1], where 1 represents a complete match and 0 represents a complete mismatch.

[0134] In S504 of this application embodiment, the first number of operations in the currently processed reference sequence is obtained, the first similarities of the currently processed reference sequence are summed, and the ratio of the summation result to the first number of operations is used as the initial similarity between the current sequence and the first sequence. By using the ratio of the summation result to the first number of operations as the initial similarity, the problem of high final similarity caused by the length of long sequences can be eliminated, so that reference sequences of different lengths can be compared with each other.

[0135] In the above scheme, the structural and semantic similarity between the first and fifth operations in terms of semantics, operation type or other dimensions is used to obtain the weighted matching total score by weighted summation of the first similarity of all first and fifth operations. This realizes the quantification of structural and semantic similarity between the first sequence and each reference sequence, and improves the evaluation accuracy and adaptability.

[0136] In this embodiment, the initial similarity between the first sequence and the reference sequence is obtained using the following formula:

[0137]

[0138]

[0139] Where S is the initial similarity. This indicates the first operation number in the reference sequence. This represents the sum of the first similarity scores corresponding to the reference sequence. Indicates the first similarity. This represents the first and fifth operations that are matched in sequence.

[0140] In this embodiment of the application, the target similarity is obtained using the following formula:

[0141]

[0142] in, Indicates target similarity. Indicates the degree of ambiguity in the query statement. Indicates the first sequence and the second... Initial similarity between reference sequences Indicates the first The sum of the first similarity scores corresponding to each of the reference sequences. Indicates the first The first number of operations in a reference sequence. n Indicates the number of reference sequences.

[0143] Based on the above embodiments, as an optional embodiment, the method for determining the first relative resource consumption level is as follows: Figure 7 As shown, the specific content is as follows:

[0144] S601, For any one of the first and third sequences, calculate the resource consumption of the sequence based on the resources used by the execution sequence;

[0145] S602, obtain the first number of the first operation in the first sequence and the second number of the third operation in the third sequence;

[0146] S603-1, If ​​the first quantity is not greater than the second quantity, then the ratio between the second resource consumption and the first resource consumption is taken as the first relative resource consumption level, the first resource consumption is the resource consumption of the first sequence, and the second resource consumption is the resource consumption of the third sequence.

[0147] S603-2, if the first quantity is greater than the second quantity, then based on the preset first parameter value and the first difference, the ratio between the second resource consumption and the first resource consumption is adjusted, and the adjustment result is used as the first relative resource consumption degree.

[0148] In S601 of this application embodiment, for the first sequence, the resources required to execute all the first operations in the first sequence are counted to obtain the first resource consumption. For the third sequence, the resources required to execute all the third operations in the third sequence are counted to obtain the second resource consumption. The resources required for each operation include at least memory and CPU execution time.

[0149] In S602 of this application embodiment, the number of first operations contained in the first sequence is counted to obtain the first quantity, and the number of third operations contained in the third sequence is counted to obtain the second quantity.

[0150] In S603-1 of this application embodiment, the number of operations in the first sequence and the third sequence are compared. If the first number is not greater than the second number, it means that the number of operations required to execute the first sequence is not greater than the number of operations required to execute the third sequence. Therefore, there is no need to adjust the ratio of the second resource consumption to the first resource consumption, and it can be directly used as the first relative resource consumption degree of the first sequence.

[0151] In S603-2 of this application embodiment, the first difference is the difference between the first quantity and the second quantity. The number of operations in the first sequence and the third sequence are compared. If the first quantity is greater than the second quantity, it means that the number of operations required to execute the first sequence is greater than the number of operations required to execute the third sequence. The first parameter value is used to penalize the case where the actual number of operations (the first quantity) is greater than the number of operations in the best matching scheme (the second quantity). Therefore, it is necessary to adjust the ratio between the second resource consumption and the first resource consumption by using the preset first parameter value and the first difference. Then, the adjustment result is used as the first relative resource consumption degree. The larger the first difference is, the smaller the first relative resource consumption degree is, which indicates that the task planning system scores lower in terms of resource consumption in the first sequence produced.

[0152] In this embodiment of the application, the first relative resource consumption level is obtained by the following formula:

[0153]

[0154] in, The first relative level of resource consumption, This is the second resource consumption. The primary resource consumption. The first parameter value, As the first quantity, This is the second quantity.

[0155] In this embodiment, when the first quantity is less than or equal to the second quantity, the ratio of the second resource consumption to the first resource consumption is used as the first relative resource. That is, when the first operation of executing the query statement is less than or equal to the third operation, the smaller the first resource consumption, the greater the degree of the first relative resource, which represents the higher the score of the task planning system in the resource consumption of the first sequence of production. When the first quantity is greater than the third quantity, the ratio of the second resource consumption to the first resource consumption needs to be adjusted by the first parameter value and the second difference. By calculating the first resource consumption and the second small resource consumption consumed in executing all first operations and all third operations respectively, and the degree of difference between the number of operations in the first sequence and the third operation, the resource consumption of the first sequence is evaluated based on the number of operations and the resources occupied by the specific operation, with the consumption of the third sequence as the benchmark, thereby improving the robustness and adaptability of the evaluation results.

[0156] Based on the above embodiments, as an optional embodiment, the method for determining the resource consumption of the sequence is as follows: Figure 8 As shown, the specific content is as follows:

[0157] S701, retrieve the longest CPU execution time and maximum memory usage generated by historically executed query statements;

[0158] S702, for any one of the first sequence and the third sequence, for each operation in the sequence, obtain the first CPU execution time and the first memory usage of the operation at the current time, normalize the first CPU execution time based on the longest CPU execution time to obtain the first result corresponding to the operation, normalize the first memory usage based on the maximum memory usage to obtain the second result corresponding to the operation, and perform a weighted sum of the first result and the second result corresponding to the operation to obtain the third resource consumption of the operation;

[0159] S703, for any one of the first and third sequences, sum the third resource consumption of each operation in the sequence to obtain the resource consumption of the sequence.

[0160] In S701 of this application embodiment, the longest CPU execution time and the maximum memory usage of each query statement executed in the past are obtained, that is, the longest execution time and the maximum memory usage are obtained from the query tasks executed in the past.

[0161] In S702 of this application embodiment, for each first operation in the first sequence, the first CPU execution time and the first memory usage required to execute the first operation are determined. The first CPU execution time is normalized by using the longest CPU execution time to obtain the first result corresponding to the first operation. This means normalizing the first CPU execution time by using the longest CPU execution time as the normalization benchmark. The first memory usage is normalized by using the maximum memory usage as the normalization benchmark to obtain the second result corresponding to the first operation. Since the importance of memory usage and CPU execution time varies in different operations, the first result and the second result of the first operation are weighted and summed based on the importance of memory usage and CPU execution time to obtain the third resource consumption of the current first operation.

[0162] In this embodiment of the application, for each third operation in the third sequence, the first CPU execution time and the first memory usage required to execute the third operation are determined. The first CPU execution time is normalized by using the longest CPU execution time as the normalization benchmark to obtain the first result corresponding to the third operation. Normalizing the first memory usage by using the maximum memory usage as the normalization benchmark to obtain the second result corresponding to the third operation. Since the importance of memory usage and CPU execution time varies in different operations, the first result and the second result of the third operation are weighted and summed based on the importance of memory usage and CPU execution time to obtain the third resource consumption of the current third operation.

[0163] In S703 of this application embodiment, for the first sequence, the third resource consumption of each first operation in the first sequence is summed to obtain the first resource consumption of the first sequence; for the third sequence, the third resource consumption of each third operation in the third sequence is summed to obtain the second resource consumption of the third sequence.

[0164] In this embodiment of the application, the first resource consumption is obtained by the following formula:

[0165]

[0166] in, The primary resource consumption. The first number of the first operation in the first sequence. For the first sequence The first CPU execution time of the first operation. For the first sequence The first memory usage of the first operation. The longest execution time for the CPU. This represents the maximum memory usage. + , Weights for CPU execution time. This is the weight for memory usage.

[0167] In this embodiment of the application, the second resource consumption is obtained using the following formula:

[0168]

[0169] in, This is the second resource consumption. This is the second number of the third operation in the third sequence. For the third sequence The first CPU execution time of the third operation For the third sequence The first memory usage of the third operation The longest execution time for the CPU. This represents the maximum memory usage. + , Weights for CPU execution time. This is the weight for memory usage.

[0170] In the above scheme, the CPU execution time and memory usage of each operation are used as the basis for measuring resource consumption. Based on the importance preference of the above resources, a weighted sum is performed to achieve a reasonable quantification of the corresponding resource consumption.

[0171] Based on the above embodiments, as an optional embodiment, the method for determining the second relative resource consumption level is as follows: Figure 9 As shown, the specific content is as follows:

[0172] S801, For any one of the second and fourth sequences, for each operation in the sequence, obtain the operation type corresponding to the operation, determine the fourth resource consumption of the operation based on the pre-established correspondence between resource consumption and operation type, and sum the fourth resource consumption of each operation in the sequence to obtain the resource consumption of the sequence.

[0173] S802, obtain the third quantity of the second operation in the second sequence, obtain the fourth quantity of the fourth operation in the fourth sequence, and determine the value of the second parameter based on the difference between the third quantity and the fourth quantity;

[0174] S803, multiply the ratio between the fourth resource consumption and the third resource consumption by the second parameter value to obtain the second relative resource consumption level; the third resource consumption is the resource consumption of the second sequence, and the fourth resource consumption is the resource consumption of the third sequence.

[0175] In S801 of this application embodiment, for each second operation in the second sequence, the operation type corresponding to the second operation is obtained, and the fourth resource consumption of the second operation is determined according to the pre-established correspondence between resource consumption and operation type. The fourth resource consumption of each second operation sequence in the second sequence is summed to obtain the resource consumption of the second sequence, i.e., the third resource consumption. For each fourth operation in the fourth sequence, the operation type corresponding to the fourth operation is obtained, and the fourth resource consumption of the fourth operation is determined according to the pre-established correspondence between resource consumption and operation type. The fourth resource consumption of each fourth operation sequence in the fourth sequence is summed to obtain the resource consumption of the fourth sequence.

[0176] In S802 of this application embodiment, the number of second operations in the second sequence is obtained, i.e., the third number is obtained, the number of fourth operations in the fourth sequence is obtained, i.e., the fourth number is obtained, and a second parameter value is determined based on the difference between the third number and the fourth number. The second parameter value is used to control the decay rate caused by the redundant clarification operations.

[0177] In S803 of this application embodiment, the ratio between the fourth resource consumption and the third resource consumption is obtained, and the ratio is multiplied by the second parameter value to obtain the second relative resource consumption degree of the first sequence.

[0178] In this embodiment, the second relative resource consumption level is determined by the following formula:

[0179]

[0180] in, The second relative level of resource consumption, This represents the fourth resource consumption of the fourth sequence. This represents the third resource consumption in the second sequence. This is the value of the second parameter.

[0181] In this embodiment of the application, the third resource consumption and the fourth resource consumption are determined by the following formulas:

[0182]

[0183]

[0184]

[0185] in, This indicates the third resource consumption of the second sequence. It is the third number of the second operation in the second sequence. This represents the fourth resource consumption of the fourth sequence. It is the fourth quantity of the fourth operation in the fourth sequence. This represents the resource consumption required for the k-th operation in the sequence. The resource consumption for the interactive clarification operation type. For operations of type "automatic inference and clarification", specify the corresponding resource consumption. > .

[0186] In the above scheme, the resource consumption of the clarification process is quantified based on the operation type of each operation in the second and fourth sequences and the amount of resources required to execute each operation type. The resource consumption is adjusted based on the differences between the operations in the second and fourth sequences, thereby measuring the system's ability to understand and clarify user intentions and improving the objectivity of the evaluation results.

[0187] Based on the above embodiments, as an optional embodiment, if the third quantity is not greater than the fourth quantity, the second parameter value is obtained based on the relative difference between the third quantity and the fourth quantity; if the third quantity is greater than the fourth quantity, the second parameter value is obtained based on the absolute difference between the third quantity and the fourth quantity.

[0188] In this embodiment of the application, when the third quantity is not greater than the fourth quantity, the relative difference between the third quantity and the fourth quantity is determined based on the ratio between the third quantity and the fourth quantity, and the second parameter value is determined based on the relative difference.

[0189] In this embodiment of the application, when the third quantity is greater than the fourth quantity, the absolute difference between the third quantity and the fourth quantity is determined based on the difference between the third quantity and the fourth quantity, and the second parameter value is determined based on the absolute difference.

[0190] In this embodiment of the application, the value of the second parameter is determined by the following formula:

[0191]

[0192] in, The value of the second parameter. It is the third number of the second operation in the second sequence. It is the fourth quantity of the fourth operation in the fourth sequence. This is a penalty coefficient used to control the decay rate caused by excessive clarification operations.

[0193] The above formula can guarantee that when (No clarification needed), the second parameter value is 1; when , is the dividing point, and the value of the second parameter is 0.5; when The second parameter value approaches 0, meaning that the smaller the value of the third quantity compared to the fourth quantity, the higher the score of the second sequence in terms of relative resource consumption, and the better the evaluation result.

[0194] In the above scheme, the second parameter value is determined by the relative and absolute differences between the third and fourth quantities. This ensures that the ratio between the fourth and third resource consumption is adaptively adjusted according to the magnitude of the difference between the third and fourth quantities. As a result, the larger the value of the third quantity compared to the fourth quantity, the lower the score of the second sequence in terms of relative resource consumption, and the worse the evaluation result.

[0195] Based on the above embodiments, as an optional embodiment, this application further provides a task planning evaluation method, such as... Figure 10 As shown, the specific content is as follows:

[0196] S901, obtain the evaluation value corresponding to the first sequence of multiple query statements;

[0197] S902, For each first sequence, obtain the query type of the query statement corresponding to the first sequence, and determine the second weight value of the first sequence according to the pre-built correspondence between query type and weight;

[0198] S903, the evaluation values ​​of each first sequence are summed by weighted summation with corresponding second weight values ​​to obtain a first summation result, and the second summation result is obtained by summing the second weight values. The ratio of the first summation result to the second summation result is used as the second evaluation value.

[0199] In S901 of this application embodiment, when the rationality of a task plan that requires multiple queries is evaluated, the evaluation value corresponding to the first sequence of multiple query statements is obtained.

[0200] In S902 of this application embodiment, for each first sequence, the query type of the query statement corresponding to the first sequence is determined. The query type of a query statement is related to the query complexity, query frequency, or importance of the corresponding business. That is, the query statement can be classified according to the query complexity, the historical query frequency, or the importance of the corresponding business. Generally, the higher the complexity, the higher the query frequency, or the higher the importance of the corresponding business, the higher the weight of the corresponding query type. Therefore, the correspondence between each query type and weight can be pre-constructed, and then when multiple query statements need to be evaluated, the second weight value given in response can be obtained based on the query type corresponding to each query statement.

[0201] In S903 of this application embodiment, a weighted summation is performed based on the evaluation value of each first sequence and the corresponding second weight value to obtain a first summation result. The second weight values ​​are then summed to obtain a second summation result. A second evaluation value is determined based on the ratio between the first summation result and the second summation result. The second evaluation value characterizes the rationality of obtaining the corresponding first sequence based on multiple query statements, that is, the rationality of obtaining the corresponding first sequence of multiple query statements from an overall perspective.

[0202] In this embodiment of the application, the second evaluation value is determined by the following formula:

[0203]

[0204] in, This is the second evaluation value. For the first The evaluation value of the first sequence, For the first The second weight value corresponds to each of the first sequences, where N represents the number of query statements;

[0205] In this embodiment of the application, the second evaluation value of multiple query statements can also be calculated using the following formula:

[0206]

[0207] MacroAcc is the second evaluation value. Indicates the number of query statements. The threshold representing the acceptable level. For the first The evaluation value of the first sequence.

[0208] In this application embodiment, different methods can be selected to obtain the second evaluation value according to different application scenarios. For general scenarios, the second average value can be obtained by averaging all evaluation values. When the importance of the business corresponding to the same set of query statements is different, the method can be adjusted accordingly. The corresponding formula is used to obtain the second evaluation value.

[0209] refer to Figure 11 As shown, it exemplarily illustrates a flowchart of a task planning evaluation method, the specific content of which is as follows:

[0210] S1001, Obtain the query statement, input the query statement into the task planning system, and obtain the first sequence and the second sequence output by the task planning system;

[0211] S1002, obtain at least one reference sequence according to the query statement, determine the initial similarity between each reference sequence and the first sequence based on the longest common subsequence between the reference sequence and the first sequence, select the reference sequence with the largest similarity as the third sequence, and determine the fourth sequence containing multiple fourth operations required to clarify the query statement to obtain the third sequence.

[0212] S1003, the parts of the query statement that lack specific information or have multiple interpretations are respectively regarded as fuzzy information, and the fuzzy type of each fuzzy information is determined to obtain the first number of fuzzy information corresponding to each fuzzy type;

[0213] S1004, for each fuzzy type, obtain the first weight of the fuzzy type, and based on the first weight and the first quantity, obtain the first fuzzy value of the fuzzy type. Sum the first fuzzy values ​​of each fuzzy type to obtain the fuzziness of the query statement.

[0214] S1005, The initial similarity between the first sequence and the third sequence is weighted according to the degree of ambiguity to obtain the target similarity;

[0215] S1006, for any one of the first sequence and the third sequence, for each operation in the sequence, obtain the first CPU execution time and the first memory usage of the operation at the current time, normalize the first CPU execution time based on the longest CPU execution time to obtain the first result corresponding to the operation, normalize the first memory usage based on the maximum memory usage to obtain the second result corresponding to the operation, perform a weighted sum of the first result and the second result corresponding to the operation to obtain the third resource consumption of the operation, and sum the third resource consumption of the first operation in the first sequence and the third operation in the third sequence to obtain the first resource consumption and the second resource consumption;

[0216] S1007, obtain the first quantity of the first operation in the first sequence and the second quantity of the third operation in the third sequence. If the first quantity is not greater than the second quantity, the ratio between the second resource consumption and the first resource consumption is taken as the first relative resource consumption degree. If the first quantity is greater than the second quantity, the ratio between the second resource consumption and the first resource consumption is adjusted based on the preset first parameter value and the second difference between the first quantity and the second quantity, and the adjustment result is taken as the first relative resource consumption degree.

[0217] S1008. For any one of the second and fourth sequences, for each operation in the sequence, obtain the operation type corresponding to the operation. Based on the pre-established correspondence between resource consumption and operation type, determine the fourth resource consumption of the operation. Sum the fourth resource consumption of each operation in the second and fourth sequences respectively to obtain the third resource consumption of the second sequence and the fourth resource consumption of the fourth sequence.

[0218] S1009, obtain the third quantity of the second operation in the second sequence and the fourth quantity of the fourth operation in the fourth sequence. If the third quantity is not greater than the fourth quantity, obtain the second parameter value based on the relative difference between the third quantity and the fourth quantity. If the third quantity is greater than the fourth quantity, obtain the second parameter value based on the absolute difference between the third quantity and the fourth quantity.

[0219] S1010, Multiply the ratio between the fourth resource consumption and the third resource consumption by the second parameter value to obtain the second relative resource consumption level;

[0220] S1011, based on target similarity, first relative resource consumption degree, and second relative resource consumption degree, an evaluation value is obtained.

[0221] In this embodiment, for the fuzzy type of fuzzy information appearing in the query statement, the structure and operation cost of the first operation in the first sequence of the query statement are executed. Various parameters are adjusted based on context or dynamic feedback, improving the stability and adaptability of the evaluation. By introducing multiple reasonable standard schemes as reference sequences, it breaks through the previous reliance on a single reference sequence for rationality evaluation. Based on the original LCS, it improves upon the importance and semantic matching strength of each operation in the first sequence, avoiding evaluation bias caused by simple operation matching. It identifies fuzzy information in the query statement and introduces a penalty for target similarity based on the degree of fuzziness, making the evaluation closer to the understanding risk of real-world scenarios. By introducing the CPU execution time and memory usage required for the operation as a measure of resource consumption, and the degree of deviation between the first and third sequences, it achieves quantitative modeling of task execution efficiency. For fuzzy information, by introducing the resource consumption of system-user interaction, it measures the system's ability to understand and clarify user intent. The final evaluation value is output by combining these three indicators, forming a complete evaluation loop and promoting a more comprehensive task planning system.

[0222] The embodiments of this application can be applied to scenarios such as natural language to structured task translation (such as NL2SQL, NL2API, etc.), intelligent question answering, intelligent planning, and multi-turn interaction systems. Under the large model-driven inference framework, it provides support as the core evaluation mechanism for the performance of multi-Agent systems.

[0223] In this embodiment, instead of relying on a single "standard SQL" comparison method, the ambiguity of operation levels, multiple reference sequences, and query statements is considered, improving the credibility and accuracy of rationality assessment. The task execution path is evaluated, adapting to the multi-agent system collaborative mode, which helps with planning optimization and task scheduling learning. The model can be rewarded or reinforced fine-tuned based on the three-dimensional evaluation results, forming an evaluation-driven optimization mechanism. This significantly enhances the model training and fine-tuning feedback loop and can serve as an operational monitoring indicator system for commercial deployment systems. In production environments, it can track the output quality and cost of the system under real business data, achieving dynamic evaluation and alerts.

[0224] The task planning query method provided in this application introduces a dynamic weight optimization mechanism, comprehensively considering the impact of target similarity, first relative resource consumption, and second relative resource consumption on the rationality of the planned first sequence. This achieves a multi-dimensional and dynamically adaptive fine evaluation of task planning, improving the accuracy and practicality of task evaluation. It solves the problem in related technologies that cannot cover SQL semantic equivalence but different expressions, leading to biased evaluation results; it solves the problem of lacking awareness of SQL semantic structure and difficulty in handling task planning with strong sequential dependencies; it solves the problem of ignoring the fine-grained advantages and disadvantages of individual operations during task execution and insufficient training feedback; it solves the problem of ignoring the misunderstanding caused by the semantic ambiguity of the problem and the inability to reflect the clarification effect; and it solves the problem of ignoring high resource consumption operations that lead to reduced execution efficiency or resource waste.

[0225] This application provides an evaluation device for task planning, such as... Figure 12 As shown, the task planning evaluation device 120 may include: a first processing module 1201, a second processing module 1202, a third processing module 1203, a fourth processing module 1204, a fifth processing module 1205, and a sixth processing module 1206.

[0226] Specifically, the first processing module 1201 is used to obtain a query statement, clarify the query statement, and obtain a clarified query statement, a first sequence, and a second sequence. The first sequence includes each first operation of executing the clarified query statement, and the second sequence includes each second operation of executing the clarified query statement.

[0227] The second processing module 1202 is used to obtain a third sequence and a fourth sequence. The third sequence includes various third operations for executing the query statement, and the fourth sequence includes various fourth operations required to obtain the third sequence by clarifying the query statement.

[0228] The third processing module 1203 is used to obtain the target similarity between the first sequence and the third sequence based on the parts of the query statement that lack specific information or have multiple interpretations, as well as the longest common subsequence between the first sequence and the third sequence.

[0229] The fourth processing module 1204 is used to evaluate the resources occupied by the first sequence based on the resources occupied by the third sequence, and obtain the first relative resource consumption degree of the first sequence.

[0230] The fifth processing module 1205 is used to evaluate the resources occupied by the second sequence based on the resources occupied by the fourth sequence, and obtain the second relative resource consumption degree of the second sequence.

[0231] The sixth processing module 1206 is used to obtain an evaluation value for assessing the rationality of obtaining the first sequence based on the target similarity, the first relative resource consumption degree, and the second relative resource consumption degree.

[0232] The task planning evaluation device provided in this application obtains a query statement, clarifies the query statement, obtains a clarified statement, obtains a first sequence including each first operation of executing the clarified query statement, and obtains each second operation including each operation executed by clarifying the query statement. This provides strong data support for the subsequent evaluation of the rationality of obtaining the first sequence based on the query statement from multiple dimensions, and can realize the evaluation of the rationality of obtaining the first sequence from both operation process and result perspectives.

[0233] By obtaining a third sequence containing the various third operations that execute the query statement, and by obtaining a fourth sequence containing the various fourth operations required to obtain the third sequence by clarifying the query statement, a reference standard is provided for evaluating the first sequence. By obtaining the third and fourth sequences, the first sequence can be evaluated not only by comparing the operations that execute the query statement in the first and third sequences, but also by comparing the operations that clarify the query statement in the second and fourth sequences.

[0234] The target similarity between the first and third sequences is obtained by considering the parts of the query statement that lack specific information or have multiple interpretations, as well as the longest common subsequence that can characterize the similarity between the first and third sequences. The influence of fuzzy expressions on the determination of the first sequence is taken into account. The structural similarity between the first and third sequences is evaluated by the longest common subsequence, and the target similarity is determined based on the degree of fuzziness and similarity of the fuzzy expressions. This is used to adjust the evaluation criteria of target similarity through fuzziness, thereby enhancing the robustness and adaptability of the overall evaluation.

[0235] By using the resources occupied by the third sequence as a benchmark to evaluate the resources occupied by the first sequence, the first relative resource consumption degree of the first sequence is obtained. By using the obtained resource consumption of the third sequence as a reference benchmark to evaluate the resources occupied by the second sequence, a unified evaluation oriented towards specific operational processes is achieved, improving the robustness and adaptability of the overall evaluation.

[0236] By evaluating the resources occupied by the second sequence based on the resources occupied by the fourth sequence, the second relative resources of the second sequence are obtained. The impact of the resources consumed when clarifying the query statement on the rationality of the solution is considered. The resources consumed by the operation of clarifying the query statement to obtain the fourth sequence corresponding to the third sequence are used as the reference benchmark for evaluation. The resources consumed by the operation of clarifying the query statement to obtain the first sequence corresponding to the first sequence are evaluated. This achieves a unified evaluation of the clarification operation process, realizes the quantity of the clarification process, and improves the robustness of the overall evaluation.

[0237] By obtaining an evaluation value based on target similarity, first relative resource consumption, and second relative resource consumption, the rationality of obtaining the first sequence is evaluated from multiple dimensions, ensuring the robustness and adaptability of the evaluation results.

[0238] The apparatus in this application embodiment can execute the method provided in this application embodiment, and the implementation principle is similar. The actions performed by each module in the apparatus of each embodiment of this application correspond to the steps in the method of each embodiment of this application. For detailed functional descriptions of each module of the apparatus, please refer to the descriptions in the corresponding methods shown above, which will not be repeated here.

[0239] This application provides an electronic device (computer device / equipment / system) including a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to implement the steps of a task planning evaluation method. Compared with related technologies, this method provides strong data support for subsequent evaluation of the rationality of obtaining a first sequence based on a query statement from multiple dimensions, enabling evaluation of the rationality of the obtained first sequence from both an operational process-oriented and result-oriented perspective. By obtaining a third and fourth sequence, the first sequence can be evaluated not only by comparing the operations executing the query statement in the first and third sequences, but also by comparing the operations clarifying the query statement in the second and fourth sequences. The structural similarity between the first and third sequences is evaluated using the longest common subsequence, and the target similarity is determined based on the degree of fuzziness and similarity of the fuzzy expression. This is used to adjust the evaluation criteria for target similarity through fuzziness, enhancing the robustness and adaptability of the overall evaluation. By using the resource consumption of the obtained third sequence as a reference benchmark to evaluate the resources occupied by the second sequence, a unified evaluation oriented towards specific operational processes is achieved, improving the robustness and adaptability of the overall evaluation. The impact of the resources consumed in clarifying the query statement on the rationality of the solution was considered. The resources consumed by the operation of obtaining the fourth sequence corresponding to the third sequence of the query statement were used as the reference benchmark for evaluation. The resources consumed by the operation of obtaining the first sequence corresponding to the query statement were evaluated. This achieved a unified evaluation of the clarification operation process and enabled the evaluation of the rationality of obtaining the first sequence operation based on the query task from multiple dimensions, ensuring the robustness and adaptability of the evaluation results.

[0240] In one alternative embodiment, an electronic device is provided, such as Figure 13 As shown, Figure 13 The illustrated electronic device 4000 includes a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, via a bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 4004 is not limited to one type, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of this application.

[0241] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0242] Bus 4002 may include a pathway for transmitting information between the aforementioned components. Bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 4002 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 13 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0243] The memory 4003 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium capable of carrying or storing computer programs and capable of being read by a computer, without limitation herein.

[0244] The memory 4003 stores computer programs that execute embodiments of this application, and its execution is controlled by the processor 4001. The processor 4001 executes the computer programs stored in the memory 4003 to implement the steps shown in the foregoing method embodiments.

[0245] The electronic device package may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and in-vehicle terminals (such as in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 13 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments disclosed herein.

[0246] This application provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program can implement the steps and corresponding content of the aforementioned method embodiments. Compared with the prior art, it can achieve the following: It provides strong data support for the subsequent evaluation of the rationality of obtaining the first sequence based on query statements from multiple dimensions, enabling an evaluation of the rationality of obtaining the first sequence from both an operational process-oriented and result-oriented perspective. By obtaining the third and fourth sequences, not only can the operations executing query statements in the first and third sequences be compared to evaluate the first sequence, but the operations clarifying query statements in the second and fourth sequences can also be compared to evaluate the first sequence. The structural similarity between the first and third sequences is evaluated using the longest common subsequence, and the target similarity is determined based on the degree of fuzziness and similarity of the fuzzy expression. This is used to adjust the evaluation criteria for target similarity through fuzziness, enhancing the robustness and adaptability of the overall evaluation. By using the resource consumption of the obtained third sequence as a reference benchmark to evaluate the resources occupied by the second sequence, a unified evaluation oriented towards specific operational processes is achieved, improving the robustness and adaptability of the overall evaluation. The impact of the resources consumed in clarifying the query statement on the rationality of the solution was considered. The resources consumed by the operation of obtaining the fourth sequence corresponding to the third sequence of the query statement were used as the reference benchmark for evaluation. The resources consumed by the operation of obtaining the first sequence corresponding to the query statement were evaluated. This achieved a unified evaluation of the clarification operation process and enabled the evaluation of the rationality of obtaining the first sequence operation based on the query task from multiple dimensions, ensuring the robustness and adaptability of the evaluation results.

[0247] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium, a computer-readable medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0248] This application also provides a computer program product, including a computer program that, when executed by a processor, can implement the steps and corresponding content of the aforementioned method embodiments. Compared with the prior art, it can achieve:

[0249] This provides strong data support for subsequent task planning based on query statements to determine the rationality of the first sequence from multiple dimensions, enabling evaluation of the rationality of the obtained first sequence from both operational and result-oriented perspectives. By obtaining the third and fourth sequences, the first sequence can be evaluated not only by comparing the operations executing the query statement in the first and third sequences, but also by comparing the operations clarifying the query statement in the second and fourth sequences. The structural similarity between the first and third sequences was evaluated using the longest common subsequence, and the target similarity was determined based on the degree of fuzziness and similarity of fuzzy representations. This was used to adjust the evaluation criteria for target similarity through fuzziness, enhancing the robustness and adaptability of the overall evaluation. By using the resource consumption of the obtained third sequence as a reference benchmark to evaluate the resources occupied by the second sequence, a unified evaluation oriented towards specific operational processes was achieved, improving the robustness and adaptability of the overall evaluation. The impact of the resources consumed in clarifying the query statement on the rationality of the solution was considered. The resources consumed by the operation of obtaining the fourth sequence corresponding to the third sequence of the query statement were used as the reference benchmark for evaluation. The resources consumed by the operation of obtaining the first sequence corresponding to the query statement were evaluated. This achieved a unified evaluation of the clarification operation process and enabled the evaluation of the rationality of obtaining the first sequence operation based on the query task from multiple dimensions, ensuring the robustness and adaptability of the evaluation results.

[0250] The terms "first," "second," "third," "fourth," "1," "2," etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in a sequence other than that shown in the illustrations or text descriptions.

[0251] It should be understood that although arrows indicate various operation steps in the flowcharts of this application's embodiments, the order in which these steps are implemented is not limited to the order indicated by the arrows. Unless explicitly stated herein, in some implementation scenarios of this application's embodiments, the implementation steps in each flowchart can be executed in other orders as required. Furthermore, some or all steps in each flowchart, based on the actual implementation scenario, may include multiple sub-steps or multiple stages. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage can also be executed at different times. In scenarios where execution times differ, the execution order of these sub-steps or stages can be flexibly configured according to requirements, and this application's embodiments do not limit this.

[0252] The above are only optional implementation methods for some implementation scenarios of this application. It should be noted that for those skilled in the art, other similar implementation methods based on the technical concept of this application, without departing from the technical concept of this application, also fall within the protection scope of the embodiments of this application.

Claims

1. A method for evaluating task planning, characterized in that, include: Obtain a query statement, clarify the query statement, and obtain a clarified query statement, a first sequence, and a second sequence. The first sequence includes each first operation of executing the clarified query statement, and the second sequence includes each second operation of executing the clarification of the query statement. Obtain a third sequence and a fourth sequence, the third sequence comprising various third operations for executing the query statement, and the fourth sequence comprising various fourth operations required to obtain the third sequence by clarifying the query statement; Based on the parts of the query statement that lack specific information or have multiple interpretations, and the longest common subsequence between the first sequence and the third sequence, the target similarity between the first sequence and the third sequence is obtained; Based on the resources occupied by the third sequence, the resources occupied by the first sequence are evaluated to obtain the first relative resource consumption degree of the first sequence; Based on the resources occupied by the fourth sequence, the resources occupied by the second sequence are evaluated to obtain the second relative resource consumption level of the second sequence; Based on the target similarity, the first relative resource consumption degree, and the second relative resource consumption degree, an evaluation value is obtained to assess the rationality of obtaining the first sequence; The step of obtaining the target similarity between the first sequence and the third sequence based on the parts of the query statement that lack specific information or have multiple interpretations, and the longest common subsequence between the first sequence and the third sequence, includes: Each part of the query statement that lacks specific information or has multiple interpretations is treated as fuzzy information, and the fuzziness type of each fuzzy information is determined. The degree of fuzziness of the query statement is obtained by statistically analyzing each fuzzy information and fuzzy type. Based on the longest common subsequence between the first sequence and the third sequence, the initial similarity between the first sequence and the third sequence is obtained; The initial similarity is weighted according to the degree of ambiguity to obtain the target similarity.

2. The method according to claim 1, characterized in that, The step of statistically analyzing various fuzzy information and fuzzy types to obtain the fuzziness level of the query statement includes: Statistical analysis is performed on each fuzzy information and its corresponding fuzzy type to obtain the first quantity of fuzzy information corresponding to each fuzzy type; For each fuzzy type, a first weight of the fuzzy type is obtained, and a first fuzzy value of the fuzzy type is obtained based on the first weight and the first quantity; the first fuzzy value is used to characterize the degree of fuzziness of the fuzzy information of the corresponding fuzzy type in the query statement; The first fuzzy value of each fuzzy type is summed, and the summation result is mapped to a preset interval to obtain the fuzziness level of the query statement.

3. The method according to claim 1, characterized in that, The third sequence is obtained in the following manner: At least one reference sequence is obtained based on the query statement, the reference sequence including various reference operations for executing the query statement; For each reference sequence, the initial similarity between the reference sequence and the first sequence is obtained based on the longest common subsequence between the reference sequence and the first sequence; The reference sequence with the highest initial similarity is selected as the third sequence.

4. The method according to claim 3, characterized in that, A method for obtaining the initial similarity between the first sequence and a reference sequence based on the longest common subsequence between the first sequence and any reference sequence includes: For any reference sequence, based on the longest common subsequence between the reference sequence and the first sequence, a matching pair sequence is obtained, wherein each matching pair in the matching pair sequence includes the first index of the element in the longest common subsequence in the first sequence and the second index in the reference sequence; For each matching pair, a corresponding first operation is obtained from the first sequence based on the first index in the matching pair, and a corresponding fifth operation is obtained from the reference sequence based on the second index in the matching pair; Determine the first operation type of the first operation and the second operation type of the fifth operation, and determine the first similarity between the first operation and the fifth operation based on the semantic similarity between the first operation and the fifth operation, the first operation type and the second operation type; Obtain the first number of operations in the reference sequence, sum the first similarities, and use the ratio between the sum and the first number of operations as the initial similarity between the first sequence and the reference sequence.

5. The method according to claim 1, characterized in that, The step of evaluating the resources occupied by the first sequence based on the resources occupied by the third sequence to obtain the first relative resource consumption level of the first sequence includes: For any one of the first sequence and the third sequence, the resource consumption of the sequence is obtained by statistically analyzing the resources used to execute the sequence. Obtain the first number of the first operation in the first sequence and the second number of the third operation in the third sequence; If the first quantity is not greater than the second quantity, then the ratio between the second resource consumption and the first resource consumption is taken as the first relative resource consumption level, the first resource consumption is the resource consumption of the first sequence, and the second resource consumption is the resource consumption of the third sequence. If the first quantity is greater than the second quantity, then based on the preset first parameter value and the first difference, the ratio between the second resource consumption and the first resource consumption is adjusted, and the adjustment result is taken as the first relative resource consumption degree; the first difference is the difference between the first quantity and the second quantity.

6. The method according to claim 5, characterized in that, For any one of the first and third sequences, the resource consumption of the sequence is statistically analyzed based on the resources used to execute the sequence, including: Obtain the longest CPU execution time and the maximum memory usage generated by the historical execution of the query statement; For any one of the first sequence and the third sequence, for each operation in the sequence, obtain the first CPU execution time and the first memory usage of the operation at the current time, normalize the first CPU execution time based on the longest CPU execution time to obtain the first result corresponding to the operation, normalize the first memory usage based on the maximum memory usage to obtain the second result corresponding to the operation, and perform a weighted sum of the first result and the second result corresponding to the operation to obtain the third resource consumption of the operation; For any one of the first sequence and the third sequence, the resource consumption of each operation in the sequence is summed to obtain the resource consumption of the sequence.

7. The method according to claim 1, characterized in that, Based on the resources occupied by the fourth sequence, the resources occupied by the second sequence are evaluated to obtain the second relative resource consumption level of the second sequence, including: For any one of the second sequence and the fourth sequence, for each operation in the sequence, obtain the operation type corresponding to the operation, determine the fourth resource consumption of the operation according to the pre-established correspondence between resource consumption and operation type, and sum the fourth resource consumption of each operation in the sequence to obtain the resource consumption of the sequence. Obtain the third quantity of the second operation in the second sequence, obtain the fourth quantity of the fourth operation in the fourth sequence, and determine the second parameter value based on the difference between the third quantity and the fourth quantity; The ratio between the fourth resource consumption and the third resource consumption is multiplied by the second parameter value to obtain the second relative resource consumption level; the third resource consumption is the resource consumption of the second sequence, and the fourth resource consumption is the resource consumption of the third sequence.

8. The method according to claim 7, characterized in that, Determining the second parameter value based on the difference between the third quantity and the fourth quantity includes: If the third quantity is not greater than the fourth quantity, then the second parameter value is obtained based on the relative difference between the third quantity and the fourth quantity; If the third quantity is greater than the fourth quantity, then the second parameter value is obtained based on the absolute difference between the third quantity and the fourth quantity.

9. The method according to claim 1, characterized in that, The process of obtaining an evaluation value for assessing the reasonableness of obtaining the first sequence further includes: Obtain the evaluation value corresponding to the first sequence of multiple query statements; For each first sequence, obtain the query type of the query statement corresponding to the first sequence, and determine the second weight value of the first sequence according to the pre-built correspondence between query type and weight; The evaluation values ​​of each first sequence are summed by weighted summation with corresponding second weight values ​​to obtain a first summation result. The second weight values ​​are summed to obtain a second summation result. The ratio of the first summation result to the second summation result is used as the second evaluation value. The query type of a query statement is related to the query complexity, query frequency, or the importance of the corresponding business.

10. A task planning evaluation device, characterized in that, include: A first processing module is configured to obtain a query statement, clarify the query statement, and obtain a clarified query statement, a first sequence, and a second sequence. The first sequence includes each first operation performed on the clarified query statement, and the second sequence includes each second operation performed on the clarified query statement. The second processing module is used to obtain a third sequence and a fourth sequence, wherein the third sequence includes various third operations for executing the query statement, and the fourth sequence includes various fourth operations required to obtain the third sequence by clarifying the query statement; The third processing module is used to obtain the target similarity between the first sequence and the third sequence based on the parts of the query statement that lack specific information or have multiple interpretations, and the longest common subsequence between the first sequence and the third sequence; The fourth processing module is used to evaluate the resources occupied by the first sequence based on the resources occupied by the third sequence, and obtain the first relative resource consumption degree of the first sequence. The fifth processing module is used to evaluate the resources occupied by the second sequence based on the resources occupied by the fourth sequence, and obtain the second relative resource consumption degree of the second sequence; The sixth processing module is used to obtain an evaluation value for assessing the rationality of obtaining the first sequence based on the target similarity, the first relative resource consumption degree, and the second relative resource consumption degree. The step of obtaining the target similarity between the first sequence and the third sequence based on the parts of the query statement that lack specific information or have multiple interpretations, and the longest common subsequence between the first sequence and the third sequence, includes: Each part of the query statement that lacks specific information or has multiple interpretations is treated as fuzzy information, and the fuzziness type of each fuzzy information is determined. The degree of fuzziness of the query statement is obtained by statistically analyzing each fuzzy information and fuzzy type. Based on the longest common subsequence between the first sequence and the third sequence, the initial similarity between the first sequence and the third sequence is obtained; The initial similarity is weighted according to the degree of ambiguity to obtain the target similarity.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the method according to any one of claims 1-9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-9.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-9.

Images