Delivery methods, systems, devices, apparatuses, and storage media

Abstract

The present disclosure provides a delivery method, system, device, electronic equipment and storage medium, relates to the technical field of computers, in particular to the technical field of artificial intelligence, large model, intelligent delivery, intelligent research and development, and can be applied to application scenarios such as intelligent assistants and virtual assistants. The specific implementation scheme is: for a to-be-delivered function, a target test plan is generated; based on the target test plan, a target object is guided to interact in a natural language manner to obtain an interaction result; in the case that the interaction result indicates that the target test plan is executed, the target test plan is executed to obtain a target test result of the to-be-delivered function.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer technology, in particular to the technical field of artificial intelligence, large model, intelligent delivery, intelligent research and development, and can be applied to application scenarios such as intelligent assistants and virtual assistants. BACKGROUND

[0002] The research and development delivery process of software / algorithm / program is a systematic and orderly process. In order to finally complete the upgrade iteration delivery of requirements or the delivery of software products, the delivery process involves many links. In this process, project managers, research and development, testing, operation and maintenance of multiple roles work together, each performing their own duties at different nodes to ensure the smooth progress and successful delivery of the project. SUMMARY

[0003] The present disclosure provides a delivery method, system, device, equipment and storage medium.

[0004] According to an aspect of the present disclosure, a delivery method is provided, comprising:

[0005] For a function to be delivered, a target test plan is generated;

[0006] Based on the target test plan, a target object is guided to interact in a natural language manner to obtain an interaction result;

[0007] In the case where the interaction result indicates that the target test plan is executed, the target test plan is executed to obtain a target test result of the function to be delivered.

[0008] According to another aspect of the present disclosure, a delivery system is provided, comprising an intelligent delivery engine and an intelligent delivery assistant, wherein:

[0009] The intelligent delivery engine is configured to generate a target test plan for a function to be delivered, and send the target test plan to the intelligent delivery assistant;

[0010] The intelligent delivery assistant is configured to guide a target object to interact in a natural language manner based on the target test plan to obtain an interaction result;

[0011] The intelligent delivery engine is further configured to execute the target test plan in the case where the interaction result indicates that the target test plan is executed, to obtain a target test result of the function to be delivered.

[0012] According to another aspect of the present disclosure, a delivery device is provided, comprising:

[0013] A generation module is configured to generate a target test plan for a function to be delivered;

[0014] An interaction module configured to guide the target object to interact in a natural language manner based on the target test plan, to obtain an interaction result.

[0015] An execution module configured to execute the target test plan to obtain a target test result of the to-be-delivered function, in a case where the interaction result indicates that the target test plan is executed.

[0016] According to another aspect of the present disclosure, an electronic device is provided, comprising:

[0017] at least one processor; and

[0018] a memory in communication with the at least one processor; wherein

[0019] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any of the embodiments of the present disclosure.

[0020] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to perform the method according to any of the embodiments of the present disclosure.

[0021] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the method according to any of the embodiments of the present disclosure.

[0022] In the embodiments of the present disclosure, natural language is used as the basis for interaction, so that the target object can complete the target test plan, and the target object can use the most simple way of human beings, i.e., natural language, to complete the delivery of the whole process. Based on the dialog LUI (Language User Interface) intelligent delivery engine, the delivery threshold of the RD can be reduced, and the delivery efficiency is improved.

[0023] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0024] The accompanying drawings are used to better understand the present scheme, and do not limit the present disclosure. Among them:

[0025] Figure 1 is a schematic diagram of a payment method according to an embodiment of the present disclosure;

[0026] Figure 2 is a flowchart of a payment method according to a first embodiment of the present disclosure;

[0027] Figure 3 is a flowchart of a payment method according to a second embodiment of the present disclosure;

[0028] Figure 4 is a flowchart of a payment method according to a third embodiment of the present disclosure;

[0029] Figure 5 is a framework diagram of a payment system according to an embodiment of the present disclosure;

[0030] Figure 6 is a diagram of an offline testing agent according to an embodiment of the present disclosure;

[0031] Figure 7 is a diagram of an online testing agent according to an embodiment of the present disclosure;

[0032] Figure 8 is a diagram of a payment system according to a first embodiment of the present disclosure;

[0033] Figure 9 is a diagram of a payment system according to a second embodiment of the present disclosure;

[0034] Figure 10 is a diagram of a payment method according to another embodiment of the present disclosure;

[0035] Figure 11 is a structural diagram of a payment device according to another embodiment of the present disclosure;

[0036] Figure 12 is a block diagram of an electronic device for implementing a payment method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0037] Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, which include various details of the embodiments of the present disclosure to help the understanding of the present disclosure. These should be considered in the context of the overall description and are not intended to limit the scope of the present disclosure, which should be limited only by the appended claims. Also, for the sake of brevity and clarity, descriptions of well-known functions and structures incorporated herein can be omitted.

[0038] In the related art, a conventional delivery mode is divided into three parts, such as Figure 1As shown, in the case of RD (Research and Development, research and development) personnel entering a new working environment, it is necessary to first learn the business, study the specifications, and be familiar with the tools and internal platforms. The content to be learned can specifically include business architecture, code knowledge, process specifications, usage instructions of multiple platforms, operation manuals, etc. After the preliminary learning is completed, code can be written, locally tested, and after the ontology test is completed, the code can be submitted and executed for admission. In the process of writing code, various problems may be encountered, such as how to use the tool, how to fail to compile, etc. How to use QA (Quality Assurance, quality assurance) to ensure the quality of the code, and how to proceed with the next step of testing in the case of admission. After the test is passed, it can be prepared for release and online, continuous deployment, and observation of system indicators and online effects.

[0039] Therefore, the new RD personnel has a high cost to get started, the overall delivery process depends on the platform, the operation is cumbersome, and at the same time depends on the communication response efficiency between people. The RD needs to learn and be familiar with a large amount of business, architecture, specification guide manual at the initial stage. The delivery full process crosses the cumbersome operation of multiple platforms, needs to learn the usage specifications of multiple test tools, and needs to repeatedly communicate to solve problems when problems are encountered.

[0040] Therefore, in the embodiments of the present disclosure, a delivery method is provided, which is mainly used for implementing intelligent testing in intelligent delivery. The method can reduce the operation threshold and improve the efficiency by automatically planning the test task and completing the delivery in a man-machine cooperative manner. Specifically, it can be implemented as Figure 2 As shown, it comprises:

[0041] S201, generating a target test plan for a to-be-delivered function.

[0042] In the embodiments of the present disclosure, the to-be-delivered function refers to the function that needs to be delivered. The function is developed by the R&D personnel and is used to update and iterate the business system. For example, a new function point needs to be added to the business system. After the to-be-delivered function for implementing the new function point is developed, the delivery method provided by the present disclosure can be executed to automatically generate a target test plan and automatically execute each link in the delivery process to execute the plan. The delivery process involved in the delivery method of the present disclosure can include admission -> testing -> testing -> release -> online, and the specific links involved in the delivery process can be determined based on the actual situation, which is not limited by the present disclosure.

[0043] S202, guiding the target object to interact in a natural language manner based on the target test plan, and obtaining an interaction result.

[0044] The target object can be an RD personnel.

[0045] The interaction result may exemplarily include confirming the target test plan, may continue to execute the plan, or includes modification of the target test plan, needs further improvement, etc. Thus, whether to continue to execute the target test plan can be determined based on information of the interaction result.

[0046] S203, in a case where the interaction result indicates to execute the target test plan, executing the target test plan to obtain a target test result of the to-be-delivered function.

[0047] In the embodiments of the present disclosure, the natural language is used as the interaction basis, so that the target object can complete the target test plan, and the target object can complete the delivery full process in the most simple way of human, i.e., the natural language. Based on the conversational LUI (Language User Interface) intelligent delivery engine, the delivery threshold of the RD can be reduced, and the delivery efficiency is improved.

[0048] The delivery method provided in the embodiments of the present disclosure is designed to achieve at least one of the following targets: being able to actively understand the risk that may cause a system problem in the to-be-delivered function, analyzing the risk and actively planning a target test plan, automatically scheduling execution of the target test plan and verification, and further making intelligent decisions based on facts, memory, and reasoning, resolving the risk or exposing potential risks, so as to give repair suggestions for the risk to assist R&D self-repair, and as far as possible, achieving automatic repair in some scenarios.

[0049] Under the cognitive background of the target, the embodiments of the present disclosure can perform intelligent delivery from two aspects of offline quality assurance and / or online risk control. Therefore, the target test plan in the embodiments of the present disclosure includes at least one of the following: a first test plan for offline testing of the to-be-delivered function; and a second test plan for online testing of the to-be-delivered function.

[0050] In the embodiments of the present disclosure, the first test plan for offline quality assurance and the second test plan for online risk control are selected from one of them or combined in the manner of both, which can actively understand the risk that may cause a system problem, analyze the risk and actively plan a solution, automatically schedule execution of a verification task, make intelligent decisions based on facts, memory, and reasoning, resolve the risk or expose the risk, give repair suggestions for the risk to assist R&D self-repair, achieve automatic repair in some scenarios, and further improve the delivery efficiency.

[0051] The first test plan and the second test plan will be described below respectively.

[0052] 1) a first test plan for testing the to-be-delivered function offline.

[0053] In some embodiments, generating a target test plan for a function to be delivered can be implemented as follows: Figure 3 As shown, it includes:

[0054] S301, if the target test plan includes the first test plan, identify potential risks to the functions to be delivered.

[0055] In some embodiments, determining the potential risks of a function to be delivered can be implemented by determining the potential risks of a function to be delivered based on a preset risk analysis strategy.

[0056] During implementation, the potential risks of the functions to be delivered are determined based on the preset risk analysis strategy. For example, this can be understood as analyzing code changes and identifying impact scenarios based on the background of requirements, while combining historical data to gain insights into potential risks.

[0057] In this embodiment of the disclosure, the potential risks of the functions to be delivered can be further identified by combining the preset risk analysis strategy, laying the foundation for ensuring the security of the functions to be delivered.

[0058] In some embodiments, determining the potential risks of the function to be delivered based on a preset risk analysis strategy can be implemented as steps A1 and / or A2:

[0059] Step A1: Identify potential risks from the description information of the functions to be delivered based on preset rules.

[0060] In cases where the description of the function to be delivered is in code form, preset rules can be manually specified rule strategies to identify potential risks. These rule strategies can rely on the written code implementation, thereby uncovering potential risks within the code of the function to be delivered.

[0061] In this embodiment of the disclosure, since the description information of the function to be delivered is composed of preset rules, potential risks can be identified from the description information of the function to be delivered based on the preset rules. This method is simple and easy to implement.

[0062] In some embodiments, potential risks are identified from the description information of the function to be delivered based on preset rules, which can be implemented as follows:

[0063] Step A11: Perform semantic analysis on the functional requirements text of the function to be delivered to obtain the key requirements of the function to be delivered.

[0064] The functional requirements document can be a requirements document formed from preliminary discussions of the functional points of the function to be delivered. It can also be further explanations provided by the user. This requirements document may include at least one of the following: a functional description of the function to be delivered, the scope and degree of its business impact, etc.

[0065] The semantic analysis of the function requirement text can be performed using a large model to obtain the requirement points corresponding to the to-be-delivered function.

[0066] The requirement points can exemplarily include a business category and an overview of the requirement. It can be understood that the requirement and the influence of the code are understood theoretically.

[0067] In step A12, white-box testing is performed on the code of the to-be-delivered function to obtain first-type risk points.

[0068] The white-box testing of the code of the to-be-delivered function is to understand the requirement and the influence of the code practically. Based on the actual code changes, the influence of the code change on the business scope is analyzed, and some potential risks are analyzed based on the code. The obtained first-type risk points can exemplarily include stability risks (out of Core), performance risks (memory, CPU rise, processing time increase), and the like.

[0069] It should be noted that the execution order of step A11 and step A12 is not limited. Step A11 is to analyze potential problems from the perspective of requirement text, and step A12 is to analyze potential problems in specific implementation. The two can assist each other to complete the risk analysis comprehensively.

[0070] In step A13, the requirement points, the first-type risk points, and the function features of the to-be-delivered function are analyzed based on preset rules to obtain second-type risk points.

[0071] The description information includes the requirement points, the first-type risk points, and the function features; and the potential risks include the first-type risk points and the second-type risk points.

[0072] In implementation, the preset risk features can be matched from the requirement points, the first-type risk points, and the function features of the to-be-delivered function based on preset rules, so as to analyze the potential risks. Specific rules used can be determined based on actual conditions, and the present disclosure does not limit this.

[0073] In the present disclosure, the first-type risk points are used to understand the requirement background, analyze the code change, and identify the range of influence scenarios. The second-type risk points are generated in combination with the requirement points and the function features of the to-be-delivered function. The multi-aspect combination makes the obtained potential risks more comprehensive, and the potential risks can be obtained automatically based on this manner, saving manpower and resources.

[0074] Step A1 describes the analysis of potential risks based on preset rules. These preset rules need to be listed by experts, and corresponding rule-based judgment functions need to be developed. When new problems arise, new preset rules need to be added, resulting in a certain lag in risk insight. Therefore, in this disclosed example, to improve the generalization ability of risk insight, the powerful knowledge background and logical reasoning capabilities of a large model can be leveraged to identify potential risks. Specifically, in step A2, potential risks can be identified from the description information of the functions to be delivered based on the large model.

[0075] Among these, large models can be large language models (LLMs). Large language models refer to specific types of large-scale models specifically designed for processing text data. These models are neural network-based natural language processing models that can be used to generate, understand, and process text data. Large language models can have hundreds of billions of parameters, generate high-quality text, and can be used for various natural language processing tasks, such as question answering, text generation, and dialogue systems.

[0076] Large language models possess excellent reasoning and few-shot learning capabilities. This is because large language models are comprehensible in that they can be trained on a large number of samples. Based on such large models, accurate semantic understanding can be achieved.

[0077] In this embodiment of the disclosure, the reasoning ability of the large model can be used to accurately identify potential risks from the description information of the function to be delivered. Not only can potential risks be identified, but with the prompts of the large model's capabilities, new potential risks can be automatically identified as the risks change, thereby improving the ability to gain risk insight.

[0078] Similarly, in some embodiments, identifying potential risks from the description information of the functions to be delivered based on a large model can be implemented as follows:

[0079] Step A21: Perform semantic analysis on the functional requirements text of the function to be delivered to obtain the key requirements of the function to be delivered.

[0080] Specifically, the methods for obtaining the requirements of the functions to be delivered have been described above, and will not be repeated here in this disclosure.

[0081] Step A22: Perform white-box testing on the code of the function to be delivered to identify the first type of risk.

[0082] Specifically, the method for obtaining the first type of risk point has been described above, and will not be repeated here in this disclosure.

[0083] Step A23: Based on the large model analysis of the key requirements, the first type of risk points, and the functional characteristics of the functions to be delivered, the third type of risk points are obtained.

[0084] The description information includes demand points, first-class risk points, and functional features, and the potential risks include the first-class risk points and third-class risk points.

[0085] In implementation, the prompt information obtained directly from the demand points, the first-class risk points, and the functional features is too long, which is not conducive to logical reasoning by the large model. Therefore, part of the key information can be selected from the demand points, the first-class risk points, and the functional features of the to-be-delivered function based on the large model to construct the prompt information input to the large model for risk insight.

[0086] In the embodiments of the present disclosure, the first-class risk points are used to understand the demand background, analyze the code changes, and identify the scope of the impact scene. The first-class risk points, the demand points, and the functional features of the to-be-delivered function are used to generate third-class risk points by using the large model. The combination of multiple aspects and the strong reasoning ability of the large model make the potential risks more accurate. At the same time, the potential risks can be obtained automatically based on the above-mentioned manner, which saves manpower and resources.

[0087] In implementation, either step A1 or step A2 can be used to identify the potential risks from the description information of the to-be-delivered function, or a combination of the two can be used to identify more potential risks from the description information of the to-be-delivered function.

[0088] The potential risk points obtained by using the combination of the two include the first risk points, the second risk points, and the third risk points.

[0089] S302, generating a first test plan based on the potential risks of the to-be-delivered function by using the large model.

[0090] In the embodiments of the present disclosure, the potential risks of the to-be-delivered function are determined from the perspective of offline quality assurance, and further planning can be made based on the potential risks, so that the to-be-delivered function can be quality assured at the linear test node.

[0091] In some embodiments, generating a first test plan based on the potential risks of the to-be-delivered function by using the large model can be implemented as:

[0092] Step C1, generating recommended input information based on the white-box test result of the to-be-delivered function.

[0093] In implementation, the white-box test result can include the impact of the business scope, and the recommended input information is generated based on the impact of the business scope.

[0094] The recommended input information can be understood as input information required for testing.

[0095] Step C2, based on the recommended input information and potential risks, a first test plan is generated. The first test plan includes recommended input information, test scenarios, and test tasks.

[0096] The test scenario can be understood as a scenario that needs to be tested for the to-be-delivered function. The test scenario includes, for example, system services, traffic ranges, and change types.

[0097] The test task can be understood as a test point that needs to be verified in this test. For example, the test task can be stability testing, performance 01 testing, and queue proportion testing.

[0098] In implementation, based on the recommended input information and potential risks, and in combination with historical risk examples, the large model is input together, the inference ability of the large model is used to infer the first test plan.

[0099] In the embodiments of the present disclosure, after obtaining the potential risks, the first test plan is automatically obtained based on the strong inference ability of the large model, which can realize autonomous completion of the test plan and improve delivery efficiency.

[0100] In implementation, the first test plan can be confirmed by the RD. In the case of RD confirmation, the first test plan is continued to be executed to obtain a test result. The test result can include, for example, 0 / 1 result, risk index, test coverage range, and the like. The 0 / 1 result indicates whether the test passes; the risk index indicates whether the time consumption of the function point of the to-be-delivered function in the node of the current delivery process meets the expectation; and the test coverage range indicates whether the test is comprehensive by comparing the risk range covered in the actual test process with the risk range evaluated before the test (i.e., the risk range of the potential risks described above).

[0101] In some embodiments, the test result of the first test plan analyzed by the large model based on the potential risks of the to-be-delivered function can be implemented as follows: historical test data of the to-be-delivered function is queried to obtain a historical data acquisition result; and the historical data acquisition result and the potential risks are input into the large model to obtain a decision result of the large model on the test result of the first test plan.

[0102] In implementation, in the case that the first test plan is executed for the first time or in the case that no test data related to the test is stored, the historical data acquisition result is empty, that is, the potential risks and some related content can be input into the large model to obtain a decision result of the large model on the test result of the first test plan. The related content can include, for example, influence range, historical experience, test execution result, system index performance, and the like.

[0103] In the case where the decision result indicates that retesting is required, the RD can be interacted with again to reconfirm which risk points in the potential risks need to be tested and which risk points do not need to be tested. Or the RD can repair the to-be-delivered function and then perform testing again. In this case, the historical data acquisition result can be obtained, and the historical data acquisition result and the potential risks are input into the large model to obtain a decision result of the large model on the test result of the first test plan.

[0104] In the embodiments of the present disclosure, the historical data acquisition result is combined to enable the large model to take the historical content as a reference, so that the large model understands the historical progress of the to-be-delivered function, and so that the large model can make targeted decisions on the to-be-delivered function.

[0105] In some embodiments, the large model analyzes the test result of the first test plan based on the potential risks of the to-be-delivered function, which can realize intelligent decision-making on the test result. The large model can learn all tools, expert knowledge, etc. related to testing. The knowledge background emphasized can give a decision result of intelligent decision-making. The decision result includes at least one of the following:

[0106] indication information of whether the to-be-delivered function passes the offline testing;

[0107] a coverage range of the test result for the potential risks, that is, which risk points in the potential risks complete testing and pass, and which risk points do not pass or are not tested;

[0108] indication information of a risk point that has been resolved in the potential risks, that is, which risk points in the potential risks have passed testing and the risk point has been resolved;

[0109] a risk level of a residual risk;

[0110] whether supplementary testing is required;

[0111] a potential fault;

[0112] a solution to the potential fault.

[0113] In implementation, in the case where the indication information of whether the to-be-delivered function passes the offline testing indicates that the to-be-delivered function passes the offline testing, the next process can be performed, and in the case where the indication information indicates that the to-be-delivered function does not pass the offline testing, the system performance and user demand need to be comprehensively judged. Other decision results can be determined based on actual conditions how to flow, and in addition, the decision results can be adaptively added or reduced based on actual conditions, which are not limited in the embodiments of the present disclosure.

[0114] In the embodiments of the present disclosure, multiple decision results are used, each decision result performs a corresponding subsequent operation, and can cover most cases, so that the decision result has universality.

[0115] 2) a second test plan for online testing of the to-be-delivered function.

[0116] In some embodiments, for the to-be-delivered function, the target test plan is generated, which can be implemented as: in the case that the target test plan includes a second test plan, a large model is used to analyze the test result of the first test plan based on the potential risk of the to-be-delivered function; in the case that it is determined that the test result of the first test plan is test passed, the second test plan is generated based on the test result of the first test plan.

[0117] The specific way of obtaining the test result of the first test plan has been described in the foregoing description, and the embodiments of the present disclosure will not be repeated here. It should be noted that the first test plan involved in generating the second test plan can be intelligently planned based on the foregoing method, or can be performed in a linear programming manner, and the embodiments of the present disclosure do not limit this.

[0118] The second test plan of the online test in the embodiments of the present disclosure is determined by using the combination of online testing and offline testing, so that in the case that the test result of the first test plan offline is test passed, the second test plan is generated again, which can save computing resources while ensuring the quality of the to-be-delivered function.

[0119] In some embodiments, based on the test result of the first test plan, the second test plan is generated, which can be implemented as:

[0120] Step D1, in the case that the business system where the to-be-delivered function is located is changed, obtaining a target risk with a risk level lower than a preset level and not solved in offline testing in the test result of the first test plan.

[0121] The business system change exemplarily can be that the system version is updated.

[0122] Step D2, determining the target risk as an online key monitoring risk in the second test plan.

[0123] In the embodiments of the present disclosure, offline testing can be understood as testing completed before the to-be-delivered function is submitted to the business system for online operation. Online testing can be understood as testing completed when the to-be-delivered function is submitted to the business system for online operation.

[0124] In order to improve the efficiency of offline testing, some low risks with a risk level lower than a preset level can not be tested linearly, so for low risk points, they will be focused on in online testing.

[0125] Step D3, based on the analysis of the large model on the online key monitoring risk and the demand change knowledge of the to-be-delivered function, the second test plan is generated.

[0126] In the embodiments of the present disclosure, when it is perceived that the business system has changed, a second test plan for online testing is generated based on the offline-online risk linkage mechanism and in combination with the risk indicators of offline testing, thereby providing a favorable basis for the second test plan.

[0127] In the second test plan generated in response to the change in the business system, at least one of the following is included:

[0128] a key monitoring item;

[0129] a monitoring item of a newly added business policy;

[0130] an alarm information processing rule.

[0131] The key monitoring item is a monitoring item generated for a target risk detected offline.

[0132] The monitoring item of the newly added business policy is a monitoring item generated based on a similar business policy with respect to a newly added business policy of the business system, so that the monitoring item can be automatically supplemented for the newly added business policy, thereby improving the test efficiency.

[0133] The alarm information processing rule can include an alarm information processing strategy related to the monitoring item, or can include an alarm information processing strategy related to other monitoring items. The alarm information processing rule defines an alarm condition corresponding to different monitoring items. For example, an alarm interception threshold can be provided, and an alarm is triggered when the monitoring item exceeds the alarm interception threshold.

[0134] In the embodiments of the present disclosure, when it is perceived that the business system has changed, a second test plan for online testing is generated based on the offline-online risk linkage mechanism and in combination with the risk indicators of offline testing, intelligent planning of project-level key monitoring indicators, automatic supplement of monitoring items, alarm processing rules, and the like, and key monitoring is performed at each node of the project online to further ensure the safety of the to-be-delivered function.

[0135] In some embodiments, based on the test results of the first test plan, the second test plan is generated, which can be implemented as:

[0136] Step E1, in the case where a business indicator of a business system in which the to-be-delivered function is located changes, acquiring a system state of the business system and a knowledge graph of the business system;

[0137] In the case where the business indicator changes, the to-be-delivered function modifies the business indicator, and the business indicator needs to be continuously detected. For example, changes in the environment and hotspots can cause changes in the business indicator.

[0138] Step E2, processing the system state of the business system, the knowledge graph, and the functional requirement knowledge of the to-be-delivered function based on a large model to generate the second test plan.

[0139] The knowledge graph can be composed of a plurality of historical system states, business indicators, and historical plans.

[0140] In implementation, the system state of the business system, the knowledge graph, and the functional requirement knowledge of the to-be-delivered function are constructed into prompt information, and the prompt information is input into the large model to obtain a second test plan.

[0141] In the embodiments of the present disclosure, for the case where it is found that the business indicators of the business system change, a second test plan is intelligently planned by combining the real-time system state and the knowledge graph and using the strong reasoning capability of the system, so that the process is automatically completed, and human resources are saved.

[0142] The second test plan generated under the business indicator change condition includes at least one of the following:

[0143] Key abnormal indicators;

[0144] Recommended alarm thresholds;

[0145] Key inspection objects.

[0146] The key abnormal indicators represent business indicators that need to be focused on. The recommended alarm threshold can determine the alarm threshold based on the normal range. The recommended alarm threshold can be graded using 1-10, where 1 represents no need for temporary processing, i.e., the lowest alarm, and 10 represents an urgent alarm that needs to be handled immediately. For example, if the value exceeds the normal range by 50%, and considering the overall system situation and user expectations, the alarm threshold is determined to be an urgent alarm.

[0147] The key inspection objects can include business system nodes that need to be focused on. For example, if the business system includes a plurality of service nodes, some of the service nodes need to be focused on, and these service nodes are set as key inspection objects.

[0148] In the embodiments of the present disclosure, based on the plurality of planning contents in the second test plan, the intelligence level of the second test plan can be improved, thereby improving the delivery efficiency.

[0149] In the embodiments of the present disclosure, the test result of the second test plan is processed based on the related processing parameters of the alarm information in the second test plan.

[0150] In implementation, after the second test plan is executed, the test result is obtained, and based on the related processing parameters of the alarm information, whether to alarm, whether to intercept, loss stop suggestions, and other related content can be given.

[0151] In the embodiments of the present disclosure, in the case of generating a second test plan, the test results of the second test plan can be processed based on the related processing parameters of the alarm information in the second test plan, a corresponding disposal strategy is given based on the test results, and delivery efficiency is submitted.

[0152] In the embodiments of the present disclosure, in order to further improve delivery efficiency, human-computer interaction can be performed with RD to cooperatively complete intelligent testing work. The entire delivery process (from R&D -> online) can be completed through LUI dialogue. The user's problem can be accurately understood, the planning and execution can be accurately planned, the processing capacity can be powerful, and the natural and personalized answers can be given. The design of the overall LUI dialogue function includes core capabilities and core services (Agent):

[0153] Among them, the core capabilities can include intent recognition capabilities; the core services are implemented by using multiple plug-ins, which can include test plug-ins, problem positioning plug-ins, process management plug-ins, operation guiding plug-ins, knowledge question and answer plug-ins, and specific contents will be described later.

[0154] In some embodiments, based on the target test plan, the target object is guided to interact in a natural language manner, and the interaction result can be implemented as:

[0155] Step F1, in the case where the target object needs to provide the to-be-supplemented information in the target test plan, the target object is guided to provide the to-be-supplemented information based on the natural language manner.

[0156] For example, as described above, in the case where a potential risk is detected, the target test plan can provide the direction of the supplementary content or the suggestion of the modified content, and the target object is guided to supplement the information based on the natural language manner based on the method.

[0157] In some embodiments, step F1 can be implemented as: based on the recommended input information in the target test plan, the recommended input information is obtained from a known information set; in the case where the to-be-supplemented information in the recommended input information is missing in the known information set, the target object is guided to provide the to-be-supplemented information in a natural language manner.

[0158] In the embodiments of the present disclosure, in the case where the target object needs to supplement information, the recommended input information can be obtained from a known information set, in the case where the missing information is determined, the target object is guided to provide the to-be-supplemented information in a natural language manner to make up for the missing information, so as to be able to smoothly complete the target test plan.

[0159] Step F2, in response to the to-be-supplemented information, the target test plan is updated.

[0160] In implementation, based on the to-be-supplemented information, the offline test agent and the online test agent are used for re-planning to obtain a new target test plan. The specific planning process has been described above, and thus will not be described here again.

[0161] In the embodiments of the present disclosure, in the case where the to-be-supplemented information is needed, the intelligent delivery assistant can timely inform the target object to supplement the content, and the human-computer combined manner can improve the delivery efficiency.

[0162] In some embodiments, in response to the to-be-supplemented information, updating the target test plan can be implemented as follows:

[0163] Step G1, based on the required mandatory parameters and optional parameters of the target node in the delivery process, extracting the supplement parameters from the to-be-supplemented information.

[0164] Since there are multiple nodes in the delivery process, each node has multiple parameters, and thus in the case of testing a certain node, the mandatory parameters and optional parameters for executing the node need to be selected from a large number of parameters, and the supplement parameters are extracted from the to-be-supplemented information.

[0165] Step G2, based on the supplement parameters, perfecting the target test plan.

[0166] In the embodiments of the present disclosure, the necessary parameters and optional parameters are selected, and the supplement parameters are extracted from the to-be-supplemented information to perfect the target test plan, so as to ensure that the target test plan can be smoothly performed and improve the delivery efficiency.

[0167] In some embodiments, the aforementioned core capabilities include an intent recognition capability, which can be implemented as follows: Figure 4 As shown in the figure, the core capabilities include:

[0168] S401, obtaining a to-be-processed sentence provided by a target object in a natural language manner.

[0169] S402, determining a target plug-in for processing the to-be-processed sentence.

[0170] The target plug-in can be used to assist in delivering a to-be-delivered function.

[0171] The target plug-in can be understood as any one of a test plug-in, a problem positioning plug-in, a process management plug-in, an operation guiding plug-in, and a knowledge question and answer plug-in, and the use of each plug-in will be described later, and thus will not be described here again.

[0172] S403, processing the to-be-processed sentence based on the target plug-in.

[0173] In implementation, the corresponding plug-in is called to execute the to-be-processed sentence to obtain a processing result. The processing result can be fed back to the target object to understand the execution status.

[0174] In the embodiments of the present disclosure, the delivery can be completed based on the natural language of the target object, the delivery threshold is reduced, and the delivery efficiency and capability are improved.

[0175] In some embodiments, since there are multiple plug-ins, determining the target plug-in for processing the to-be-processed sentence can be implemented as follows:

[0176] Step H1, performing intent recognition on the to-be-processed sentence to obtain a target intent.

[0177] In implementation, the BERT (Bidirectional Encoder Representation from Transformer) model can be used for intent recognition to obtain the target intent; or the TextCNN (Text Convolutional Neural Networks) model, a large model, can be used for intent recognition to obtain the target intent, which is not limited in the embodiments of the present disclosure.

[0178] In order to make the recognized intent more accurate to determine the target plug-in for completing the task, the intent recognition on the to-be-processed sentence to obtain the target intent can be implemented as follows: extracting keyword slot information from the to-be-processed sentence; rewriting the to-be-processed sentence based on the keyword slot information to obtain a rewritten sentence; performing intent recognition on the rewritten sentence to obtain the target intent.

[0179] In implementation, the keyword hard matching and the regular expression can be used to extract the keyword slot information from the to-be-processed sentence, and then rewrite the to-be-processed sentence.

[0180] In the embodiments of the present disclosure, the keyword slot extraction can be used to obtain the key information, and then rewrite based on the key information to obtain the rewritten sentence, which can clearly express the target intent of the to-be-processed sentence, so as to accurately determine the target plug-in.

[0181] Step H2, assigning at least one candidate plug-in to the target intent by using a large model.

[0182] Step H3, interacting with the at least one candidate plug-in according to the assignment priority of the at least one candidate plug-in, to determine the plug-in with the highest priority in the candidate plug-in capable of processing the to-be-processed sentence as the target plug-in.

[0183] In the embodiments of the present disclosure, multiple plug-ins are improved. For similar to-be-processed statements, multiple plug-ins can all be able to process, but due to different capabilities of different plug-ins, there are differences in processing results. Therefore, in the embodiments of the present disclosure, at least one candidate plug-in capable of processing can be screened out by a large model, and then the inquiry functions of each plug-in are called to inquire whether each plug-in can process the to-be-processed statement, thereby realizing interaction with each candidate plug-in, and finally, the most reasonable candidate plug-in can be screened out to process the to-be-processed statement, thereby improving the accuracy of human-computer interaction.

[0184] In the embodiments of the present disclosure, the inquiry detection operation is added to the planned at least one candidate plug-in. The candidate plug-in can preliminarily determine whether to process the to-be-processed statement this time, and when both reach an agreement, the system preferentially selects the plug-in to execute the task of the to-be-processed statement this time, so as to improve the execution accuracy.

[0185] In some embodiments, in the case where the target plug-in is a delivery process management plug-in, based on the target plug-in processing the to-be-processed statement, the implementation can be that the supplementary parameters are extracted from the to-be-processed statement based on the required mandatory parameters and optional parameters of the target plug-in.

[0186] The process management plug-in is used to ensure that each node is executed according to the predetermined standard process. Each process node to the next process node can customize the decision to be made and the report to be generated. For example, the typical process of a self-test type requirement is: admission-measurement-test-release-online. The decision strategy of the process management plug-in can be used to judge whether each process node flows to the next process node. Each process has an information management service belonging to the process.

[0187] In implementation, the mandatory parameters and the optional parameters can be obtained based on the regular expression and the hard rule matching, and the way of extracting the supplementary parameters from the to-be-processed statement can also be the regular expression and the hard rule matching.

[0188] In the embodiments of the present disclosure, in the case where the target plug-in is a delivery process management plug-in, the mandatory parameters, the optional parameters and the supplementary parameters are used to ensure the normal operation of the delivery process.

[0189] In some embodiments, based on the mandatory parameters and the optional parameters required by the target plug-in, the supplementary parameters are extracted from the to-be-processed statement, which can be implemented as follows: the mandatory parameters and the optional parameters are extracted from the to-be-processed statement; in the case where at least one sub-parameter in the mandatory parameters is missing in the to-be-processed statement, the target object is guided to supplement at least one sub-parameter through multiple rounds of dialogue.

[0190] In addition, the pipeline parameter can also be obtained from the statement to be processed. The pipeline can be understood as a carrier of information, and some key information required by the test can be obtained from the pipeline. The pipeline can be used to automatically fill in the parameters, some parameters in which the mandatory parameters and the optional parameters do not appear, and the parameters required to be provided by the RD can be obtained from the pipeline or the carrier to facilitate the interaction with the RD and supplement the parameters.

[0191] In the embodiments of the present disclosure, the obtained parameters based on the manner are the parameters required for the test, and the manner based on multiple interactions ensures the integrity of the parameters and improves the test efficiency.

[0192] In some embodiments, after obtaining the test result, the test result of the target test plan can also be obtained; based on the test result, it is determined whether to enter the next delivery link. This operation can be completed by the aforementioned process management plug-in. In the implementation, when the test result of the target test plan is passed, the process management plug-in confirms to enter the next link, and when the test result of the target test plan is not passed, the process management plug-in confirms to enter the next link. For example, in the case where the process includes compilation->admission->testing->release->online, when the test result of the admission process indicates that it is passed, the process management plug-in confirms to enter the testing link.

[0193] Through the automatic judgment of whether to circulate the link, the entire delivery process can be automatically executed, and the delivery efficiency is improved.

[0194] In the embodiments of the present disclosure, the execution of the test can call the test plug-in to complete. The test plug-in uses the online multi-test tool, focuses on assisting the RD in all needs in the self-test link, such as vocabulary construction, environment building, request sending, etc., and simultaneously serves as a pre-link of the delivery process to provide key data reference for the intelligent circulation of the subsequent process management plug-in.

[0195] In some embodiments, in each link in the delivery process, whether the start condition of the target test plan is met can also be determined based on the obtained parameters required for executing the target test plan.

[0196] In the implementation, the parameters required for the target test plan are the aforementioned necessary parameters. It should be noted that all the necessary parameters need to be screened to meet the start condition of the target test plan. The method can be completed by the aforementioned process management plug-in, so that the process management plug-in can comprehensively manage the start of the test and the process.

[0197] In the embodiments of the present disclosure, the parameters required for executing the target test plan are obtained to determine whether the start condition of the target test plan is met, so as to ensure the accuracy and integrity of the test and improve the delivery efficiency.

[0198] In some embodiments, the LUI-based form is also supported to provide intelligent problem positioning to the target object. When implemented, the problem description information of the function to be delivered can be determined; based on the problem description information, a problem expert model is determined; based on the problem expert model, the problem description information is analyzed to obtain the code segment of the problem and the repair suggestion.

[0199] The problem description information can be obtained by interacting with the target object, or can be obtained according to error information in the test process. This test can be a test plan actively formulated by the target object, or a target test plan formulated by the wisdom in the preceding text. The problem positioning can be completed by the problem positioning plug-in.

[0200] Due to various problems that may be encountered in the integrated delivery process, the problem positioning plug-in can combine execution logs, changed code slices, requirement background analysis and other data information, and according to different types of problems, adjust the prompt information to the corresponding problem expert model, to obtain the code segment of the problem and the repair suggestion.

[0201] In the embodiments of the present disclosure, the problem expert model is used to obtain the corresponding code segment of the problem and the repair suggestion, which facilitates the target object to perform subsequent repair of the function to be delivered and the like.

[0202] In some embodiments, not only can the target test plan be automatically planned and completed automatically, but also the offline test of the target function can be completed by interacting with the target object.

[0203] The target function can be a function to be delivered, or a function offline to the function to be delivered. The target object can use the LUI technology to test any function developed offline. The offline test can be implemented based on the test plug-in of the integrated test tool.

[0204] In the embodiments of the present disclosure, the offline test is completed by interacting with the target object, which facilitates the target object to flexibly perform the test.

[0205] In some embodiments, in order to improve the user experience, the delivery knowledge can be provided to the target object by interacting with the target object in the embodiments of the present disclosure.

[0206] For example, the knowledge Q&A plug-in reads the business knowledge, system architecture, process specification, and all-around expert, and answers questions in real time online, which can efficiently answer all problems encountered by the RD in the delivery process.

[0207] In the embodiments of the present disclosure, the knowledge Q&A plug-in can provide delivery knowledge for the target object, so that the target object can obtain answers in real time, and the user experience of the target object is improved.

[0208] In some embodiments, in order to guide the target object to quickly complete the delivery, the target object can also be interacted with to provide the target object with the operation method of the delivery system.

[0209] For example, by operating the guide plug-in, the scope of the delivery system itself, the operation method, etc. can be explicitly indicated to guide the target object to correctly and efficiently use the delivery system that integrates the delivery method of the embodiments of the present disclosure.

[0210] In the embodiments of the present disclosure, the target object can quickly learn the use knowledge to assist the target object to quickly complete the delivery task.

[0211] Based on the same technical concept, the embodiments of the present disclosure also provide a delivery system, as shown in Figure 5 The delivery system includes an intelligent delivery engine and an intelligent delivery assistant, wherein:

[0212] The intelligent delivery engine is configured to generate a target test plan for a to-be-delivered function, and send the target test plan to the intelligent delivery assistant.

[0213] The intelligent delivery assistant is configured to guide the target object to interact in a natural language manner based on the target test plan, and obtain an interaction result.

[0214] The intelligent delivery engine is further configured to execute the target test plan in a case where the interaction result indicates that the target test plan is executed, and obtain a target test result of the to-be-delivered function.

[0215] In the embodiments of the present disclosure, the intelligent delivery assistant and the intelligent delivery engine are combined, the interaction between the intelligent delivery assistant and the target object is based on the intelligent delivery engine as a whole support, the target object uses the natural language dialogue to complete the delivery task, the whole system simplifies the complex into the simple, all delivery processes are completed through the intelligent delivery assistant, the human-to-human work card communication mode is broken through, and the cumbersome information communication is avoided.

[0216] Still taking the delivery system as shown in Figure 5As shown, the intelligent delivery assistant includes an interaction portal, an interface layer, an intent layer, a multi-agent, a data layer, and an offline service. The interaction portal is used to interact with the target object. The interface layer is used to obtain a to-be-processed query of the target object and return test results, alarm information, modification suggestions, and other information to the target object. The intent layer is used to rewrite the query and determine the target intent. The multi-agent includes a process management plug-in, a problem positioning plug-in, a test plug-in, a knowledge question and answer plug-in, and an operation guide plug-in to support the corresponding functions of the intelligent interaction assistant. The data layer includes a process management database, a problem positioning database, a tool process management database, data such as a glossary, a knowledge base document, and a human setup background capability description. The data layer is used to provide reference content for the intelligent interaction assistant and can also provide the required knowledge for the large model. The offline service is used to push real-time progress to the target object and to position the problem when the task fails. In addition, the training process of the large model is also an offline process.

[0217] The intelligent delivery engine includes an offline test agent and an online test agent. In addition, the data engineering stores data required for the to-be-delivered function to flow to each process node and various underlying data. Examples include interaction history, code, and the like. The underlying data is used to support the data required for the to-be-delivered process to flow, and examples include test professional knowledge, code professional knowledge, business system knowledge, and the like.

[0218] In some embodiments, the target test plan includes at least one of:

[0219] a first test plan for offline testing of the to-be-delivered function;

[0220] a second test plan for online testing of the to-be-delivered function.

[0221] In the embodiments of the present disclosure, from the first test plan of offline quality assurance and the second test plan of online risk control, or in a combined manner of both, the risk that may cause system problems can be actively discovered, the risk can be analyzed and a solution can be actively planned, the verification task can be automatically scheduled and executed, intelligent decisions can be made according to facts, memory, and reasoning, the risk can be resolved or exposed, repair suggestions can be given for the risk to assist R&D in self-repair, automatic repair can be achieved in some scenarios, and delivery efficiency is further improved.

[0222] In some embodiments, in the case where the target test plan includes the first test plan, the intelligent delivery engine includes an offline test agent, configured to: determine a potential risk of the to-be-delivered function; and generate the first test plan based on the large model for the potential risk of the to-be-delivered function.

[0223] In the embodiments of the present disclosure, the offline test agent can identify potential risks of the to-be-delivered function based on the perspective of offline quality assurance, and then make further planning based on the potential risks, so as to ensure the quality of the to-be-delivered function at the next process node.

[0224] In some embodiments, the offline test agent is specifically configured to determine the potential risks of the to-be-delivered function based on a preset risk analysis strategy.

[0225] In the embodiments of the present disclosure, the offline test agent can provide a basis for identifying potential risks of the to-be-delivered function in combination with the preset risk analysis strategy, so as to effectively ensure the security of the to-be-delivered function.

[0226] In some embodiments, the offline test agent is specifically configured to: identify the potential risks from the description information of the to-be-delivered function based on a preset rule; and / or identify the potential risks from the description information of the to-be-delivered function based on a large model.

[0227] In the embodiments of the present disclosure, the description information of the to-be-delivered function is composed of a preset rule, so the potential risks can be identified from the description information of the to-be-delivered function based on the preset rule, which is simple and easy to implement.

[0228] In the embodiments of the present disclosure, the potential risks are obtained by using the preset rule and the large model, which ensures the accuracy of the potential risks in the to-be-delivered function.

[0229] In some embodiments, in the case of identifying the potential risks from the description information of the to-be-delivered function based on a preset rule, the offline test agent is configured to: perform semantic analysis on the function requirement text of the to-be-delivered function to obtain the requirement points of the to-be-delivered function; and perform white-box testing on the code of the to-be-delivered function to obtain first-type risk points; analyze the requirement points, the first-type risk points, and the function characteristics of the to-be-delivered function based on a preset rule to obtain second-type risk points; the description information includes the requirement points, the first-type risk points, and the function characteristics; and the potential risks include the first-type risk points and the second-type risk points.

[0230] The function requirement text can be a requirement document formed by discussing the function points of the to-be-delivered function in the early stage. It can also be a further explanation provided by the user. The requirement document can include at least one of the following contents: function description of the to-be-delivered function, business impact range and impact degree of the to-be-delivered function, and the like.

[0231] The white-box testing on the code of the to-be-delivered function is to understand the requirements and impacts of the code in practice. Based on the actual code changes, the impact of the code changes on the business scope is analyzed, and some potential risks are analyzed based on the code.

[0232] In the embodiments of the present disclosure, the first type of risk points are used to understand the demand background, analyze the code changes, identify the scope of the impact scene, and generate the second type of risk points in combination with the demand points and the functional features of the to-be-delivered function. The combination of multiple aspects makes the potential risks more accurate, and the offline test agent can automatically obtain the potential risks based on this manner, thereby saving manpower and resources.

[0233] In some embodiments, after the potential risks are identified from the description information of the to-be-delivered function based on the large model, the offline test agent is specifically configured to: perform semantic analysis on the functional requirement text of the to-be-delivered function to obtain the demand points of the to-be-delivered function; and perform white-box testing on the code of the to-be-delivered function to obtain the first type of risk points; analyze the demand points, the first type of risk points, and the functional features of the to-be-delivered function based on the large model to obtain the third type of risk points; the description information includes the demand points, the first type of risk points, and the functional features; and the potential risks include the first type of risk points and the third type of risk points.

[0234] In the embodiments of the present disclosure, the first type of risk points are used to understand the demand background, analyze the code changes, identify the scope of the impact scene, and generate the third type of risk points using the large model in combination with the first type of risk points, the demand points, and the functional features of the to-be-delivered function. The combination of multiple aspects and the strong reasoning capability of the large model make the potential risks more accurate, and the offline test agent can automatically obtain the potential risks based on this manner, thereby saving manpower and resources.

[0235] In some embodiments, the offline test agent is configured to: generate recommended input information based on the white-box testing result of the to-be-delivered function; analyze the recommended input information and the potential risks based on the large model to generate a first test plan; and the first test plan includes the recommended input information, a test scene, and a test task.

[0236] In the embodiments of the present disclosure, after the potential risks are obtained, the offline test agent can automatically obtain the first test plan based on the strong reasoning capability of the large model, and then autonomously complete the test.

[0237] In some embodiments, in the case where the target test plan includes a second test plan, the intelligent delivery engine includes an offline test agent and an online test agent, wherein:

[0238] The offline test agent is configured to analyze the test result of the first test plan based on the potential risks of the to-be-delivered function using the large model;

[0239] The online test agent is configured to, in the case where the test result of the first test plan is determined to be test passed, generate a second test plan based on the test result of the first test plan.

[0240] Since the embodiment of the present disclosure is determined by combining online testing and offline testing, the test result of the first test plan is passed in the case of online testing, and the second test plan is regenerated, which can save computing resources and improve delivery efficiency.

[0241] In some embodiments, the offline test agent is specifically configured to: query historical test data of the to-be-delivered function to obtain a historical data acquisition result; and input the historical data acquisition result and the potential risk into the large model to obtain a decision result of the large model on the test result of the first test plan.

[0242] In the embodiment of the present disclosure, the historical data acquisition result is combined to enable the large model to understand the historical progress of the to-be-delivered function based on the historical content as a reference, so that the large model can make targeted decisions on the to-be-delivered function.

[0243] In some embodiments, the large model analyzes the test result of the first test plan based on the potential risk of the to-be-delivered function, which can realize intelligent decision-making on the test result. The large model can learn all tools, expert knowledge, etc. related to testing. The decision result of intelligent decision-making can be given depending on the emphasized knowledge background. The decision result includes at least one of the following:

[0244] Indication information of whether the to-be-delivered function passes the offline test; that is, which risk points in the potential risk have passed the test, and the risk points have been resolved;

[0245] Coverage range of the test result for the potential risk;

[0246] Indication information of the risk points that have been resolved in the potential risk;

[0247] Risk level of the residual risk;

[0248] Whether a supplementary test is needed;

[0249] Potential failure;

[0250] Solution to the potential failure.

[0251] In the embodiment of the present disclosure, multiple decision results are used, and each decision result performs a corresponding subsequent operation, which can cover most cases, so that the decision result has universality.

[0252] In addition, the risk level can be directly determined by the large model based on whether the risk is resolved, whether it can be released, and whether a supplementary test is needed, or the risk level can be further determined by the large model based on the above three results and other content.

[0253] In summary, the intelligent delivery engine is used for the offline delivery intelligent agent of the offline test, such as Figure 6As shown, it can be divided into four working steps, divided into risk insight, intelligent planning, test execution and intelligent decision-making.

[0254] Risk insight is mainly used to input a large model based on the description information of the to-be-delivered function and the demand points of the to-be-delivered function obtained in the code, and the potential risks are analyzed by the preset rules combined with the large model mode. Among them, the demand points of the to-be-delivered function can be business impact, code change, and system linkage. Business impact represents the impact of the to-be-delivered function in the business scope, code change represents the change content of the to-be-delivered function in the code, and system linkage represents whether the to-be-delivered function affects the entire system or a single module in the system.

[0255] The analysis and feature extraction of the code include code, related code slices, and few-shot. The code represents the related code of the to-be-delivered function. In the case of modifying a line of code and calling the line of code, the linkage part of the line of code needs to be sliced out for modification, that is, the related code slice. Few-shot can be understood as historical risk examples. Based on the foregoing information, the demand points of the to-be-delivered function are obtained, and then the prompt information is input into the large model to obtain potential risks.

[0256] Intelligent planning is mainly used for testing the large model to obtain the first test plan. Intelligent planning mainly realizes that the white box test results of the to-be-delivered function are based on the white box test results to generate recommended input information. The recommended input information exemplarily can include service name, small flow number, stress test duration, compilation method, flow proportion, QPS (Queries Per Seconds, Queries Per Seconds), etc., and is combined with potential risks to input into the large model. Among them, the small flow number is used to distribute traffic. The compilation method represents the compilation method required to execute the code. In addition, the test cases are selected from the system service, traffic range and change type test scenarios, and the use cases for the scenarios are generated to generate the first test plan. Among them, the system service includes system name and other content. The traffic range is determined based on the distribution of the foregoing small flow number. The change type represents the type corresponding to the code change. The first test plan can include running stability test, performance 01 test, queue proportion, etc. Test tasks.

[0257] Test execution mainly relies on the test center platform capability to obtain test parameters and environment deployment, and obtains test data after test execution.

[0258] The intelligent decision is used to combine potential risks, process data, test data and the like to generate prompt information, input a large model, obtain a risk level and data analysis result, and make a preliminary conclusion and a positioning conclusion based on the risk level and the data analysis result. The preliminary conclusion refers to that the code can proceed to a next process node, and the positioning conclusion refers to that an abnormal position and a cause of a potential abnormality of the code are specified. The foregoing decision result is obtained.

[0259] In the offline test in the embodiments of the present disclosure, historical and current test behavior data are autonomously analyzed, potential risks, influence ranges, historical experiences, test execution results, system index performances and the like are combined as inputs, a large model is used for decision and judgment, a decision result is obtained, and it is known based on the decision result whether the risks are removed, whether the code can be released, whether the test needs to be supplemented and the like, so as to divide risk levels and further determine subsequent process state transitions and subsequent behaviors. For risk-free and low-risk projects, no test personnel intervention is needed in the whole process, that is, no manual intervention is needed; for high-risk projects, observable quality evaluation conclusions and reliable repair suggestions are given to assist self-repair of the R&D, and system self-repair is tried in some scenarios to improve human efficiency.

[0260] In some embodiments, the online test intelligent agent is specifically used for: in a case where a business system where the to-be-delivered function is located changes, obtaining a target risk in a test result of a first test plan, the target risk being a risk whose risk level is lower than a preset level and not solved in linear testing; determining the target risk as an online key monitoring risk in a second test plan; and generating the second test plan based on a large model analyzing the online key monitoring risk and requirement change knowledge of the to-be-delivered function.

[0261] In the embodiments of the present disclosure, the online test intelligent agent is used to perceive that the business system changes, and in a case where it is perceived that the business system changes, a second test plan for online testing is generated based on a risk linkage mechanism of offline and online testing and in combination with risk indexes of offline testing.

[0262] In some embodiments, in a case where the business system changes, the second test plan generated includes at least one of the following:

[0263] a key monitoring item;

[0264] a monitoring item of a newly added business policy;

[0265] an alarm information processing rule related to the key monitoring item.

[0266] The key monitoring item is a monitoring item generated based on the target risk detected by the offline test.

[0267] In the embodiments of the present disclosure, when it is perceived that the business system has changed, according to the offline-online risk linkage mechanism, in combination with the risk indicators of offline testing, the intelligent planning of project-level key monitoring indicator items, the automatic supplement of monitoring items, the alarm processing rules and the like, key monitoring is performed in each node of the project online to further ensure the safety of the to-be-delivered function.

[0268] In some embodiments, the baseline test agent is specifically used for: in the case that the business indicators of the business system where the to-be-delivered function is located change, acquiring the system state of the business system and the knowledge graph of the business system; and based on the large model processing the system state of the business system, the knowledge graph and the functional requirement knowledge of the to-be-delivered function, to generate a second test plan.

[0269] In the embodiments of the present disclosure, when it is perceived that the business indicators of the business system change, in combination with the real-time system state and the knowledge graph, the large model uses its powerful reasoning ability to intelligently plan a second test plan, so that the process is automatically completed, saving human resources.

[0270] In some embodiments, the second test plan generated under the condition of business indicator change includes at least one of the following:

[0271] Key abnormal indicators;

[0272] Recommended alarm threshold;

[0273] Key inspection objects.

[0274] In the embodiments of the present disclosure, based on the multiple planning contents in the second test plan when the online test agent perceives the business indicator change, the intelligence level of the second test plan can be improved, thereby improving the delivery efficiency.

[0275] In some embodiments, the online test agent is further used for: based on the related processing parameters of the alarm information in the second test plan, processing the test results of the second test plan.

[0276] In the embodiments of the present disclosure, in the case of generating the second test plan, the test results of the second test plan can be processed based on the related processing parameters of the alarm information in the second test plan, and the corresponding disposal strategy is given based on the test results to improve the delivery efficiency.

[0277] In some embodiments, the intelligent delivery assistant is used for:

[0278] In the case that the target object needs to provide to-be-supplemented information in the target test plan, guiding the target object to provide the to-be-supplemented information based on a natural language mode;

[0279] In response to the to-be-supplemented information, updating the target test plan.

[0280] In the embodiments of the present disclosure, in the case of needing supplementary information, the intelligent delivery assistant can timely inform the target object to supplement the content, and the use of human-computer combination can improve work efficiency.

[0281] In summary, in the online risk control environment, such as Figure 7 As shown in the figure, the online test of the intelligent delivery engine is divided into two cases, namely business system change and business index change, and each case can be divided into four steps of insight, planning, execution and decision:

[0282] 1) Business system change

[0283] In the insight step, the real-time system state, change requirement background knowledge and offline-online risk linkage index of the system are input into the big model, and the second test plan is generated, and the second test plan is executed. The second test plan includes at least one of the following: key monitoring items; monitoring items of new business strategies; alarm information processing rules related to key monitoring items. The execution process is from single machine->single machine room->…->full machine room, and the test is performed in small to large range and based on the order of each stage (before\middle\after), and each range and each stage is checked, and finally the test result is obtained. Finally, in the decision step, the test result is decided, which is divided into three decision modes of whether to alarm, whether to intercept and stop loss mode, and the decision is notified to the relevant personnel (actor).

[0284] 2) Business index change

[0285] The main detection range of business index change is the real-time system state, the knowledge graph of the business and the change data information, and the second test plan is obtained, which includes at least one of the following information: key abnormal index, recommended alarm threshold and key inspection object. The second test plan is systemically inspected, and the test result is obtained after the inspection is completed. According to the test result, the decision is made, which is divided into three decision modes of whether to alarm, whether to intercept and stop loss suggestion, and the decision is notified to the relevant personnel (actor).

[0286] In some embodiments, the offline test agent or the online test agent of the intelligent delivery engine is used to determine the to-be-supplemented information based on the following method: based on the recommended input information in the target test plan, the recommended input information is obtained from the known information set; in the case that the to-be-supplemented information in the recommended input information is missing in the known information set, the target object is guided to provide the to-be-supplemented information in a natural language manner.

[0287] In the embodiments of the present disclosure, in the case where the target object needs to supplement information, recommended input information can be obtained from a known information set, and in the case where missing information is determined, the intelligent delivery assistant can guide the target object to provide the to-be-supplemented information in a natural language manner to supplement the missing information, so as to smoothly complete the target test plan.

[0288] In some embodiments, the intelligent delivery assistant is specifically configured to: extract a supplement parameter from the to-be-supplemented information based on the mandatory parameter and the optional parameter required by the target node in the delivery process; and perfect the target test plan based on the supplement parameter.

[0289] In the embodiments of the present disclosure, the necessary parameter and the optional parameter are screened out, and the supplement parameter is extracted from the to-be-supplemented information to perfect the target test plan, so as to ensure that the target test plan can be smoothly carried out and improve the delivery efficiency.

[0290] In some embodiments, the intelligent delivery assistant is further configured to: obtain a to-be-processed sentence provided by the target object in a natural language manner; determine a target plug-in for processing the to-be-processed sentence; and process the to-be-processed sentence based on the target plug-in.

[0291] In the embodiments of the present disclosure, the intelligent delivery assistant can complete the delivery based on the natural language of the target object, thereby reducing the delivery threshold and improving the delivery efficiency and capability.

[0292] In some embodiments, the intelligent delivery assistant is specifically configured to: perform intent recognition on the to-be-processed sentence to obtain a target intent; assign at least one candidate plug-in to the target intent by using a large model; and interact with the at least one candidate plug-in according to an assignment priority of the at least one candidate plug-in, to determine a plug-in with the highest priority in the candidate plug-in capable of processing the to-be-processed sentence as the target plug-in.

[0293] In the embodiments of the present disclosure, the intelligent delivery assistant adds an inquiry detection operation to the at least one candidate plug-in planned, and the candidate plug-in can preliminarily determine whether to process the to-be-processed sentence this time, and when the two reach an agreement, the system preferentially selects the plug-in to execute the task of the to-be-processed sentence this time, so as to improve the execution accuracy.

[0294] In the embodiments of the present disclosure, the intelligent delivery assistant is specifically configured to: extract keyword slot information from the to-be-processed sentence; rewrite the to-be-processed sentence based on the keyword slot information to obtain a rewritten sentence; and perform intent recognition on the rewritten sentence to obtain a target intent.

[0295] In the embodiments of the present disclosure, the keyword slot extraction method can obtain key information, and then the rewritten sentence is obtained based on the key information, which can clearly express the target intent of the to-be-processed sentence, so as to accurately determine the target plug-in.

[0296] In some embodiments, when the target plug-in is a delivery process management plug-in, the delivery process management plug-in is configured to extract the supplementary parameters from the to-be-processed statement based on the required mandatory parameters and optional parameters of the target plug-in.

[0297] In the embodiments of the present disclosure, when the target plug-in is a delivery process management plug-in, the mandatory parameters, the optional parameters and the supplementary parameters are used to ensure the normal operation of the process.

[0298] In some embodiments, the delivery process management plug-in is specifically configured to extract the mandatory parameters and the optional parameters from the to-be-processed statement, and in the case that at least one of the sub-parameters in the mandatory parameters is missing in the to-be-processed statement, guide the target object to supplement the at least one sub-parameter through a multi-round dialogue.

[0299] In the embodiments of the present disclosure, the parameters obtained by the delivery process management plug-in based on the above-mentioned method are the parameters required for the test, and the multi-round interaction ensures the completeness of the parameters and improves the test efficiency.

[0300] In some embodiments, the delivery process management plug-in is further configured to obtain a test result of the target test plan, and determine whether to enter a next delivery link based on the test result.

[0301] In the embodiments of the present disclosure, the automatic judgment of whether to circulate the link can make the entire delivery process automatically executed, thereby improving the delivery efficiency.

[0302] In some embodiments, the delivery process management plug-in is further configured to determine whether the start condition of the target test plan is met based on the obtained parameters required for executing the target test plan.

[0303] In the embodiments of the present disclosure, the parameters required for executing the target test plan are obtained, and it is determined whether the start condition of the target test plan is met, so as to ensure the accuracy and completeness of the test and improve the delivery efficiency.

[0304] In some embodiments, the target plug-in further includes a problem positioning plug-in, which supports providing intelligent problem positioning to the target object in the form of LUI. In implementation, the problem description information of the to-be-delivered function can be determined, the problem expert model can be determined based on the problem description information, and the code segment with the problem and the repair suggestion can be obtained by analyzing the problem description information based on the problem expert model.

[0305] In the embodiments of the present disclosure, the problem positioning plug-in uses the problem expert model to obtain the corresponding code segment with the problem and the repair suggestion, which facilitates the target object to perform subsequent repair on the to-be-delivered function.

[0306] In some embodiments, the target plug-in further includes a test tool set, which is configured to complete offline testing of the target function by interacting with the target object.

[0307] In the embodiments of the present disclosure, based on the test plug-in required by the to-be-delivered function of the target object, the target object can flexibly perform the test.

[0308] In the embodiments of the present disclosure, the target plug-in further includes a question and answer plug-in, which is configured to interact with the target object and provide the target object with delivery knowledge.

[0309] In the embodiments of the present disclosure, the knowledge question and answer plug-in can provide the target object with delivery knowledge, so that the target object can obtain answers in real time, and the user experience of the target object is improved.

[0310] In the embodiments of the present disclosure, the target plug-in further includes an operation guide plug-in, which is configured to interact with the target object and provide the target object with an operation method of the delivery system.

[0311] In the embodiments of the present disclosure, the operation guide plug-in can make the target object quickly adapt to the operation of the intelligent delivery assistant, so as to assist the target object to quickly complete the task.

[0312] In some embodiments, in order to facilitate understanding, the entire delivery process is as shown in Figure 8 In the embodiments of the present disclosure, the target object sends a request for a to-be-processed statement, first determines whether the target object needs to clean up historical dialogues, in the case of determining not to clean up, obtains historical dialogues from a database (DB), rewrites the to-be-processed statement and the historical dialogues to obtain rewritten statements and key parameters (mandatory parameters), combines the rewritten statements and the key parameters into prompt information, inputs the prompt information into a large model for plug-in recognition, selects a target plug-in from multiple candidate plug-ins, and in the case that the large model does not determine the plug-in, uses an operation guide plug-in to interact with the target object again.

[0313] In the embodiments of the present disclosure, the specific implementation process of determining the target plug-in based on the intent is as shown in Figure 9As shown, since the target object can obtain the session information (session) in multiple interaction modes, the same user is the same session, and the history session of the session in the recent period (exemplarily n times) can be based on the session, or the intention identified in the last time of the session, in addition, the history session in the preset period and the to-do event can be combined, and at least one history information can be selected from the foregoing multiple history information as reference information to obtain the intention recognition of the current conversation combined with the to-be-processed sentence. The target of this recognition is to obtain the mandatory parameter, the optional parameter and the target plug-in. The content in the target can be recognized step by step, the necessary parameter of the plug-in needs to be determined under the condition that the target plug-in is accurately obtained, if the necessary parameter of the plug-in exists, it is necessary to determine whether the necessary parameter is complete, under the condition that all the necessary parameters are determined, it is judged whether there is a permission, if there is a calling permission, the plug-in is called to execute the task, and the processing result of the plug-in is returned to the target object. In the case where the target plug-in is not accurately obtained, the question and answer plug-in is called to ask the target object, and after a positive reply is obtained, subsequent operations are performed. In the case where the necessary parameter is detected to be missing, the target object is interacted again to obtain all the necessary parameters, and subsequent operations are performed. In the case where it is determined that there is no calling permission, the application calling permission needs to be contacted, and in the case where a preset number (exemplarily 3 times) of processes are reached, the target object is informed to contact the on-duty QA.

[0314] Based on the natural language dialogue mode for the target object in the embodiments of the present disclosure, the delivery task is completed. In the related art, the delivery task is implemented, for example, Figure 10 As shown, in this delivery mode, the project iteration whole process is based on the GUI graphical interface for interaction, and must be executed in turn according to the established standard, delivery process (coding, self-test, compilation, access, test, release, online) script operation, which is usually a fixed, rule, strict, and even some complex operation mode and execution process. The development needs to go through a complex process, such as: consulting learning materials, code development, local self-test (which may involve the use of multiple test tools), integration into the pipeline access test, exception failure troubleshooting, communication with the test on-duty, test submission, release, online observation, confirmation of abnormal interception indicators, etc. This process often needs to perform tedious operations for multiple platforms / tools, such as: demand space platform, local development platform, integration pipeline, test plan, environment deployment, test submission platform, online platform, indicator monitoring platform, etc. In the process of each demand delivery, two core problems may be faced:

[0315] (1) In the access & test process: who tests, what tests, how tests, and whether it can be online?

[0316] To address this issue, QA typically needs strong code comprehension skills, extensive business knowledge, and rich experience.

[0317] (2) During the online and deployment process: What to observe, how to observe, whether there are any abnormalities, and how to stop losses.

[0318] This problem severely tests the R&D team's self-awareness, system and business capabilities, and troubleshooting experience.

[0319] To address issue (1), the intelligent delivery engine realizes the transformation from human intervention, human formulation, and human decision-making to AI insight, AI planning, automatic execution, and AI decision-making. Based on the accumulation of extensive testing and delivery experience, it has trained a large-scale delivery expert model applicable to multiple sub-directions of information flow business, including risk insight, test planning, decision admission, monitoring item generation, decision alarm, and decision loss mitigation. Combined with the guiding principles of the four-step method for building delivery agent applications (insight, planning, execution, and design), it has built an intelligent delivery engine for information flow business to drive unattended delivery of requirements. The questions of who will test, what to test, and how to test are left to the offline testing intelligence agent. The questions of what to observe, how to observe, whether there are any anomalies, and whether to mitigate losses are left to the online testing intelligence agent.

[0320] Regarding issue (2): The intelligent delivery assistant is used to transform the entire delivery process from being oriented towards multiple platforms to being oriented towards the intelligent delivery assistant. Based on LUI, the conversational intelligent delivery assistant understands queries through a large model, integrates the core critical paths in the delivery process, connects all underlying platform capabilities in the entire process of development self-testing-delivery and launch, brings the testing tool copilot online, and makes process management intelligent. Based on the knowledge background of delivery experts, it builds a delivery problem location and Q&A intelligent agent, which is a personal assistant by the side of the RD. It is always ready to solve the problems encountered by the RD in the delivery process, provide assistance, and help the RD to use natural language to complete the entire delivery process smoothly, improve delivery efficiency, and the human-friendly interaction greatly enhances the project delivery experience.

[0321] Based on the two types of intelligent agents of the intelligent delivery engine and the intelligent delivery assistant proposed in the embodiments of the present disclosure, the traditional delivery mode is broken, and a new type of human-computer collaborative delivery mode is constructed: taking LUI natural language as the interaction basis, the knowledge background of a series of delivery experts such as business background, code engineering and testing is instilled into the large model, and the understanding, generation, logic and memory capabilities of the large model are used to realize risk identification and detection, test planning, and to enable RD to use the most simple way of human beings-natural language to complete the whole delivery process. Based on the AI native application <dialogic LUI intelligent delivery system>, the traditional script delivery mode is changed to the human-computer collaborative natural dialogue delivery mode, the delivery threshold is reduced, and the delivery efficiency is improved. The whole delivery process realizes the change from the traditional script mode to the human-computer collaborative natural dialogue mode.

[0322] Based on the same technical concept, the embodiments of the present disclosure propose a delivery device 1100, as shown in Figure 11 comprises:

[0323] The generation module 1101 is configured to generate a target test plan for a to-be-delivered function.

[0324] The interaction module 1102 is configured to guide a target object to interact in a natural language manner based on the target test plan, and obtain an interaction result.

[0325] The execution module 1103 is configured to execute the target test plan in a case where the interaction result indicates that the target test plan is executed, and obtain a target test result of the to-be-delivered function.

[0326] In some embodiments, the target test plan comprises at least one of:

[0327] A first test plan for offline testing of the to-be-delivered function;

[0328] A second test plan for online testing of the to-be-delivered function.

[0329] In some embodiments, the generation module comprises:

[0330] The determination unit is configured to determine a potential risk of the to-be-delivered function in a case where the target test plan comprises the first test plan.

[0331] The first generation unit is configured to generate the first test plan based on a large model for the potential risk of the to-be-delivered function.

[0332] In some embodiments, the determination unit is configured to:

[0333] determine the potential risk of the to-be-delivered function based on a preset risk analysis strategy.

[0334] In some embodiments, the determining unit is configured to:

[0335] identify the potential risk from description information of the to-be-delivered function based on a preset rule; and / or,

[0336] identify the potential risk from the description information of the to-be-delivered function based on a large model.

[0337] In some embodiments, the determining unit is configured to:

[0338] perform semantic analysis on function requirement text of the to-be-delivered function to obtain requirement points of the to-be-delivered function; and,

[0339] perform white-box testing on code of the to-be-delivered function to obtain first-type risk points;

[0340] analyze the requirement points, the first-type risk points, and function features of the to-be-delivered function based on the preset rule to obtain second-type risk points; the description information comprises the requirement points, the first-type risk points, and the function features;

[0341] The potential risk comprises the first-type risk points and the second-type risk points.

[0342] In some embodiments, the determining unit is configured to:

[0343] perform semantic analysis on function requirement text of the to-be-delivered function to obtain requirement points of the to-be-delivered function; and,

[0344] perform white-box testing on code of the to-be-delivered function to obtain first-type risk points;

[0345] analyze the requirement points, the first-type risk points, and function features of the to-be-delivered function based on the large model to obtain third-type risk points; the description information comprises the requirement points, the first-type risk points, and the function features;

[0346] The potential risk comprises the first-type risk points and the third-type risk points.

[0347] In some embodiments, the first generating unit is configured to:

[0348] generate recommended input information based on a white-box testing result of the to-be-delivered function;

[0349] generate the first test plan based on analysis of the recommended input information and the potential risk by the large model.

[0350] The first test plan includes the recommended input information, a test scenario, and a test task.

[0351] In some embodiments, the generation module includes:

[0352] The analysis unit is configured to, in a case where the target test plan includes the second test plan, analyze a test result of the first test plan based on the potential risk of the to-be-delivered function by using the large model.

[0353] The second generation unit is configured to, in a case where it is determined that the test result of the first test plan is test pass, generate the second test plan based on the test result of the first test plan.

[0354] In some embodiments, the analysis unit is configured to:

[0355] query historical test data of the to-be-delivered function to obtain a historical data acquisition result;

[0356] input the historical data acquisition result and the potential risk into the large model to obtain a decision result of the large model on the test result of the first test plan.

[0357] In some embodiments, the decision result includes at least one of the following:

[0358] indication information about whether the to-be-delivered function passes offline testing;

[0359] a coverage range of the test result with respect to the potential risk;

[0360] indication information about a risk point that has been resolved in the potential risk;

[0361] a risk level of a residual risk;

[0362] whether a supplementary test is needed;

[0363] a potential fault;

[0364] a solution to the potential fault.

[0365] In some embodiments, the second generation unit is configured to:

[0366] in a case where the business system in which the to-be-delivered function is located is changed, acquire a target risk in the test result of the first test plan, the target risk having a risk level lower than a preset level and not being solved in offline testing;

[0367] determine the target risk as an online key monitoring risk in the second test plan;

[0368] analyzing, based on the large model, the online key monitoring risk and the requirement change knowledge of the to-be-delivered function, to generate the second test plan.

[0369] In some embodiments, the second test plan includes at least one of:

[0370] a key monitoring item;

[0371] a monitoring item of a newly added business policy;

[0372] an alarm information processing rule.

[0373] In some embodiments, the second generation unit is configured to:

[0374] in a case where a business indicator of the business system where the to-be-delivered function is located is changed, acquire a system state of the business system, a knowledge graph of the business system, and a functional requirement knowledge of the to-be-delivered function.

[0375] based on the large model, process the system state of the business system, the knowledge graph, and the functional requirement knowledge of the to-be-delivered function, to generate the second test plan.

[0376] In some embodiments, the second test plan includes at least one of:

[0377] a key abnormal indicator;

[0378] a recommended alarm threshold;

[0379] a key inspection object.

[0380] In some embodiments, the method further includes a first processing module configured to:

[0381] based on a related processing parameter of alarm information in the second test plan, process a test result of the second test plan.

[0382] In some embodiments, the interaction module includes:

[0383] an interaction unit configured to, in a case where the target test plan needs the target object to provide to-be-supplemented information, guide the target object to provide the to-be-supplemented information based on a natural language mode;

[0384] an updating unit configured to, in response to the to-be-supplemented information, update the target test plan.

[0385] In some embodiments, the method further includes a screening unit configured to determine the to-be-supplemented information based on the following method:

[0386] acquire the recommended input information from a known information set based on the recommended input information in the target test plan;

[0387] in a case where missing information to be supplemented in the recommended input information in the known information set, guide the target object to provide the missing information in a natural language manner.

[0388] In some embodiments, the updating unit is configured to:

[0389] extract a supplementary parameter from the missing information based on a mandatory parameter and an optional parameter required by a target node in a delivery process;

[0390] perfect the target test plan based on the supplementary parameter.

[0391] In some embodiments, the second processing module is further configured to:

[0392] acquire a to-be-processed sentence provided by the target object in a natural language manner;

[0393] determine a target plug-in for processing the to-be-processed sentence;

[0394] process the to-be-processed sentence based on the target plug-in.

[0395] In some embodiments, the second processing module is specifically configured to:

[0396] perform intent recognition on the to-be-processed sentence to obtain a target intent;

[0397] assign at least one candidate plug-in to the target intent by using a large model;

[0398] interact with the at least one candidate plug-in according to an assignment priority of the at least one candidate plug-in, to determine a plug-in with the highest priority in the candidate plug-in that can process the to-be-processed sentence as the target plug-in.

[0399] In some embodiments, the second processing module is specifically configured to:

[0400] extract keyword slot information from the to-be-processed sentence;

[0401] rewrite the to-be-processed sentence based on the keyword slot information to obtain a rewritten sentence;

[0402] perform intent recognition on the rewritten sentence to obtain the target intent.

[0403] In some embodiments, in a case where the target plug-in is a delivery process management plug-in, the second processing module is specifically configured to:

[0404] extracting a supplementary parameter from the to-be-processed statement based on the required mandatory parameter and the optional parameter required by the target plug-in.

[0405] In some embodiments, the second processing module is specifically configured to:

[0406] extracting the mandatory parameter and the optional parameter from the to-be-processed statement;

[0407] in a case where at least one sub-parameter in the mandatory parameter is missing in the to-be-processed statement, guiding the target object to supplement the at least one sub-parameter through multiple rounds of dialogues.

[0408] In some embodiments, the third processing module is further configured to:

[0409] obtaining a test result of the target test plan;

[0410] determining whether to enter a next delivery link based on the test result.

[0411] In some embodiments, the fourth processing module is further configured to:

[0412] determining whether a start condition of the target test plan is met based on the obtained parameter required for executing the target test plan.

[0413] In some embodiments, the fifth processing module is further configured to:

[0414] determining problem description information of the to-be-delivered function;

[0415] determining a problem expert model based on the problem description information;

[0416] analyzing the problem description information based on the problem expert model to obtain a code fragment with a problem and a repair suggestion.

[0417] In some embodiments, the sixth processing module is further configured to:

[0418] interacting with the target object to complete offline testing of a target function.

[0419] In some embodiments, the seventh processing module is further configured to:

[0420] interacting with the target object to provide delivery knowledge for the target object.

[0421] In some embodiments, the eighth processing module is further configured to:

[0422] interacting with the target object to provide an operation method of the delivery system for the target object.

[0423] The specific functions and examples of the modules and sub-modules of the apparatuses in the embodiments of the present disclosure are described in the corresponding steps of the above method embodiments, which will not be described here.

[0424] In the technical solutions of the present disclosure, the acquisition, storage and application of user personal information comply with relevant laws and regulations and do not violate public order and good customs.

[0425] According to the embodiments of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium and a computer program product.

[0426] Figure 12 A schematic block diagram of an example electronic device 1200 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the present disclosure described and / or claimed in this document.

[0427] As shown in Figure 12 Figure 12 The device 1200 includes a computing unit 1201 that can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory (ROM) 1202 or loaded into a random access memory (RAM) 1203 from a storage unit 1208. Various programs and data required for the operation of the device 1200 can also be stored in the RAM 1203. The computing unit 1201, the ROM 1202, and the RAM 1203 are connected to each other through a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.

[0428] Various components in the device 1200 are connected to the I / O interface 1205, including an input unit 1206, such as a keyboard, a mouse, etc., an output unit 1207, such as various types of displays, speakers, etc., a storage unit 1208, such as a magnetic disk, an optical disk, etc., and a communication unit 1209, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1209 allows the device 1200 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunications networks.

[0429] The computing unit 1201 can be various general purpose and / or special purpose processing components having processing and computing capabilities. Some examples of the computing unit 1201 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1201 performs various methods and processes described above, such as the payment method. For example, in some embodiments, the payment method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 1208. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 1200 via the ROM 1202 and / or the communication unit 1209. When the computer program is loaded onto the RAM 1203 and executed by the computing unit 1201, one or more steps of the payment method described above can be performed. Alternatively, in other embodiments, the computing unit 1201 can be configured to perform the payment method by any other suitable means, such as by means of firmware.

[0430] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (PLD), a computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0431] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or the block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0432] In the context of this disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0433] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0434] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0435] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0436] It should be understood that the various forms of flow shown above can be used to reorder, add, or remove steps. For example, the steps recited in the present disclosure can be performed in parallel, in series, or in a different order, without limitation herein, so long as the desired results of the technology disclosed in the present disclosure are achieved.

[0437] The specific implementation described above does not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the principles of the present disclosure should be included in the protection scope of the present disclosure.

Claims

1. A delivery method, comprising: generating a second test plan in a target test plan for a to-be-delivered function, including: analyzing a potential risk of the to-be-delivered function based on a large model to generate a test result of a first test plan; and generating the second test plan based on the test result of the first test plan, in a case where it is determined that the test result of the first test plan is a test pass; the target test plan including: the first test plan for offline testing of the to-be-delivered function; and the second test plan for online testing of the to-be-delivered function; guiding a target object to interact in a natural language manner based on the target test plan to obtain an interaction result; in a case where the interaction result indicates that the target test plan is executed, executing the target test plan to obtain a target test result of the to-be-delivered function.

2. The method of claim 1, wherein, The generating of the target test plan for the to-be-delivered function includes: determining a potential risk of the to-be-delivered function, in a case where the target test plan includes the first test plan; generating the first test plan based on the potential risk of the to-be-delivered function by using the large model.

3. The method of claim 2, wherein, The generating of the first test plan based on the potential risk of the to-be-delivered function by using the large model includes: generating recommended input information based on a white-box test result of the to-be-delivered function; generating the first test plan based on the large model analyzing the recommended input information and the potential risk; the first test plan including the recommended input information, a test scenario, and a test task.

4. The method of claim 1, wherein, The generating of the second test plan based on the test result of the first test plan includes: in a case where a business system in which the to-be-delivered function is located is changed, obtaining a target risk with a risk level lower than a preset level and not solved in offline testing in the test result of the first test plan; determining the target risk as an online key monitoring risk in the second test plan; generating the second test plan based on the large model analyzing the online key monitoring risk and demand change knowledge of the to-be-delivered function.

5. The method of claim 1, wherein, The generating of the second test plan based on the test result of the first test plan includes: in a case where a business index of the business system in which the to-be-delivered function is located is changed, obtaining a system state of the business system and a knowledge graph of the business system; generating the second test plan based on the large model processing the system state of the business system, the knowledge graph, and function demand knowledge of the to-be-delivered function.

6. The method of claim 1, further comprising: obtaining a to-be-processed sentence provided by a target object in a natural language manner; determining a target plug-in for processing the to-be-processed sentence; processing the to-be-processed sentence based on the target plug-in.

7. The method of claim 6, wherein, The determining of the target plug-in for processing the to-be-processed sentence includes: performing intent recognition on the to-be-processed sentence to obtain a target intent; assigning at least one candidate plug-in to the target intent by using a large model; According to an allocation priority of the at least one candidate plug-in, interact with the at least one candidate plug-in to determine a plug-in with the highest priority in candidate plug-ins capable of processing the to-be-processed sentence as the target plug-in.

8. The method of claim 7, wherein, The intent recognition on the to-be-processed sentence comprises: extracting keyword slot information from the to-be-processed sentence; rewriting the to-be-processed sentence based on the keyword slot information to obtain a rewritten sentence; performing intent recognition on the rewritten sentence to obtain the target intent.

9. A delivery system comprising an intelligent delivery engine and an intelligent delivery assistant, wherein: the intelligent delivery engine is configured to generate a second test plan in a target test plan for a to-be-delivered function, and send the target test plan to the intelligent delivery assistant; the intelligent delivery engine comprises an offline test intelligent agent and an online test intelligent agent, wherein: the offline test intelligent agent is configured to analyze a test result of a first test plan based on a potential risk of the to-be-delivered function using a large model; the online test intelligent agent is configured to generate the second test plan based on the test result of the first test plan if it is determined that the test result of the first test plan is a test pass; the target test plan comprises: a first test plan for offline testing of the to-be-delivered function; and a second test plan for online testing of the to-be-delivered function; the intelligent delivery assistant is configured to guide a target object to interact in a natural language manner based on the target test plan, and obtain an interaction result; the intelligent delivery engine is further configured to execute the target test plan if the interaction result indicates that the target test plan is executed, and obtain a target test result of the to-be-delivered function.

10. The system of claim 9, wherein, the offline test intelligent agent is configured to: generate recommended input information based on a white-box test result of the to-be-delivered function; generate the first test plan based on the large model analyzing the recommended input information and the potential risk; the first test plan comprises the recommended input information, a test scenario, and a test task.

11. The system of claim 9, wherein, the online test intelligent agent is specifically configured to: obtain a target risk with a risk level lower than a preset level and not solved in linear testing in a test result of the first test plan if a business system where the to-be-delivered function is located is changed; determine the target risk as an online key monitoring risk in the second test plan; generate the second test plan based on the large model analyzing the online key monitoring risk and demand change knowledge of the to-be-delivered function.

12. The system of claim 9, wherein, the online test intelligent agent is specifically configured to: obtain a system state of the business system and a knowledge graph of the business system if a business index of the business system where the to-be-delivered function is located is changed; process the system state of the business system, the knowledge graph, and function demand knowledge of the to-be-delivered function based on the large model to generate the second test plan.

13. The system of claim 9, wherein the intelligent delivery assistant is further configured to: The target object obtains a to-be-processed sentence provided in a natural language manner; determining a target plug-in for processing the to-be-processed sentence; processing the to-be-processed sentence based on the target plug-in.

14. The system of claim 13, wherein, The intelligent delivery assistant is specifically configured to: performing intent recognition on the to-be-processed sentence to obtain a target intent; assigning at least one candidate plug-in to the target intent using a large model; interacting with the at least one candidate plug-in according to the assignment priority of the at least one candidate plug-in to determine the plug-in with the highest priority among the candidate plug-ins capable of processing the to-be-processed sentence as the target plug-in.

15. A delivery device, comprising: a generation module configured to generate a second test plan in a target test plan for a to-be-delivered function, including: analyzing a test result of a first test plan based on potential risks of the to-be-delivered function using a large model; and generating the second test plan based on the test result of the first test plan in a case where it is determined that the test result of the first test plan is a test pass; the target test plan including: a first test plan for offline testing of the to-be-delivered function; and a second test plan for online testing of the to-be-delivered function; an interaction module configured to guide a target object to interact in a natural language manner based on the target test plan to obtain an interaction result; an execution module configured to execute the target test plan to obtain a target test result of the to-be-delivered function in a case where the interaction result indicates that the target test plan is executed.

16. An electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.

17. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to make the computer execute the method according to any one of claims 1-8.

18. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1-8.

Images