Delivery method, system, device and equipment and storage medium

Abstract

The invention provides a delivery method, system and device, electronic equipment and a storage medium, relates to the technical field of computers, in particular to the technical fields of artificial intelligence, large models, intelligent delivery, intelligent research and development and the like, and can be applied to application scenes such as intelligent assistants and virtual assistants. According to the specific implementation scheme, a target test plan is generated for a to-be-delivered function; based on the target test plan, guiding a target object to interact in a natural language mode to obtain an interaction result; and under the condition that the interaction result indicates to execute the target test plan, executing the target test plan to obtain a target test result of the to-be-delivered function.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to technical fields such as artificial intelligence, large models, intelligent delivery, and intelligent research and development, and can be used in application scenarios such as intelligent assistants and virtual assistants. Background Art

[0002] The R&D delivery process of software / algorithms / programs is a systematic and orderly process. In order to ultimately complete the upgrade and iterative delivery of requirements or the delivery of software products, the delivery process involves many links. In this process, project managers, R&D, testing, and operation and maintenance work together to perform their respective duties at different nodes to ensure the smooth and successful delivery of the project. Summary of the invention

[0003] The present disclosure provides a delivery method, system, device, equipment and storage medium.

[0004] According to one aspect of the present disclosure, there is provided a delivery method, comprising:

[0005] Generate target test plans for the functions to be delivered;

[0006] Based on the target test plan, guiding the target object to interact in a natural language manner to obtain an interaction result;

[0007] In the case where the interaction result indicates to execute the target test plan, the target test plan is executed to obtain the target test result of the function to be delivered.

[0008] According to another aspect of the present disclosure, a delivery system is provided, including an intelligent delivery engine and an intelligent delivery assistant, wherein:

[0009] The intelligent delivery engine is used to generate a target test plan for the function to be delivered, and send the target test plan to the intelligent delivery assistant;

[0010] The intelligent delivery assistant is used to guide the target object to interact in a natural language manner based on the target test plan to obtain an interaction result;

[0011] The intelligent delivery engine is further configured to execute the target test plan when the interaction result indicates to execute the target test plan, and obtain the target test result of the function to be delivered.

[0012] According to another aspect of the present disclosure, there is provided a delivery device, comprising:

[0013] The generation module is used to generate target test plans for the functions to be delivered;

[0014] An interaction module, used to guide the target object to interact in a natural language manner based on the target test plan to obtain an interaction result;

[0015] An execution module is used to execute the target test plan when the interaction result indicates to execute the target test plan, so as to obtain the target test result of the function to be delivered.

[0016] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0017] at least one processor; and

[0018] a memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any method in the embodiments of the present disclosure.

[0020] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any method according to the embodiments of the present disclosure.

[0021] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements any method according to the embodiments of the present disclosure when executed by a processor.

[0022] In the disclosed embodiment, natural language is used as the basis for interaction, so that the target object completes the target test plan, and the target object can use the most simple human way, that is, natural language, to complete the entire delivery process. Based on the conversational LUI (Language User Interface) intelligent delivery engine, the delivery threshold of RD can be lowered, thereby improving delivery efficiency.

[0023] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0025] Figure 1 is a schematic diagram of a payment method according to an embodiment of the present disclosure;

[0026] Figure 2 is a flow chart of a payment method according to the first embodiment of the present disclosure;

[0027] Figure 3 is a flow chart of a payment method according to a second embodiment of the present disclosure;

[0028] Figure 4 is a flow chart of a payment method according to a third embodiment of the present disclosure;

[0029] Figure 5 is a schematic diagram of a payment system according to an embodiment of the present disclosure;

[0030] Figure 6 is a schematic diagram of an offline test agent according to an embodiment of the present disclosure;

[0031] Figure 7 is a schematic diagram of an online test agent according to an embodiment of the present disclosure;

[0032] Figure 8 is a schematic diagram of a payment system according to a first embodiment of the present disclosure;

[0033] Fig. 9 is a schematic diagram of a payment system according to a second embodiment of the present disclosure;

[0034] Fig.10 is a schematic diagram of a payment method according to another embodiment of the present disclosure;

[0035] Fig.11 is a schematic diagram of the structure of a payment device according to another embodiment of the present disclosure;

[0036] Fig.12 It is a block diagram of an electronic device used to implement the payment method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0037] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0038] In related technologies, the traditional delivery model is divided into three parts, such as Figure 1As shown, when RD (Research and Development) personnel enter a new work environment, they need to first learn the business, study the specifications, and become familiar with the tools and internal platforms. The specific content that needs to be learned may include business architecture, code knowledge, process specifications, instructions for the use of multiple platforms, operation manuals, etc. After the initial learning is completed, you can write code and conduct local self-tests. After the main body test is completed, you can submit the code and execute access. In the process of writing code, you may encounter various problems, such as how to use the tool, why the compilation failed, how to use QA (Quality Assurance) to ensure the quality of the code, and how to proceed to the next step of testing in the case of access. After the test passes, you can prepare for the release and launch, conduct continuous deployment, and observe system indicators and online effects.

[0039] Therefore, the cost of getting started for new RDs is very high, the overall delivery process is platform-dependent, the operations are cumbersome, and it is also very dependent on the efficiency of interpersonal communication. RDs need to learn and become familiar with a large number of business, architecture, and specification manuals in the early stages. The entire delivery process involves tedious operations across multiple platforms, and requires learning the usage specifications of multiple testing tools. When problems arise, repeated communication is required to solve them.

[0040] In view of this, a delivery method is proposed in the embodiment of the present disclosure, which is mainly used to realize intelligent testing in intelligent delivery. The method automatically plans test tasks and completes delivery in a human-machine collaborative manner, which can reduce the operation threshold and improve efficiency. It can be implemented as follows Figure 2 As shown, including:

[0041] S201, generating a target test plan for the functions to be delivered.

[0042] Among them, the function to be delivered in the embodiment of the present disclosure refers to the function that needs to be delivered. The function is developed by R&D personnel and is used to update and iterate the business system. For example, if a new function point needs to be added to the business system, after the R&D and development of the function to be delivered to implement the new function point, the delivery method provided by the embodiment of the present disclosure can be executed to automatically generate a target test plan and automatically execute each link in the delivery process to execute the plan. The delivery process involved in the delivery method of the embodiment of the present disclosure may include access->testing->testing->release->online, and the specific links involved in the delivery process can be determined based on the actual situation, and the embodiment of the present disclosure does not limit this.

[0043] S202, based on the target test plan, guiding the target object to interact in a natural language manner to obtain an interaction result.

[0044] Among them, the target object can be RD personnel.

[0045] The interaction result may exemplarily include confirmation of the target test plan, and the plan can be continued to be executed, or the target test plan has been modified and needs to be further improved. Therefore, it can be determined whether to continue to execute the target test plan based on the information of the interaction result.

[0046] S203: When the interaction result indicates to execute the target test plan, the target test plan is executed to obtain the target test result of the function to be delivered.

[0047] In the disclosed embodiment, natural language is used as the basis for interaction, so that the target object completes the target test plan, and the target object can use the most simple human way, that is, natural language, to complete the entire delivery process. Based on the conversational LUI (Language User Interface) intelligent delivery engine, the delivery threshold of RD can be lowered, thereby improving delivery efficiency.

[0048] The delivery method provided by the embodiments of the present disclosure is designed to implement at least one of the following objectives: being able to proactively understand the risks that may cause system problems in the function to be delivered, analyze the risks and proactively plan the target test plan, automatically schedule the execution of the target test plan and verify it; in addition, it is necessary to make intelligent decisions based on facts, memory, and reasoning to eliminate risks or expose potential risks, so as to provide repair suggestions for risks, assist in the development of self-service repair, and achieve automatic repair in some scenarios as much as possible.

[0049] In the context of this goal, the disclosed embodiment can perform intelligent delivery from the perspectives of offline quality assurance and / or online risk management. Therefore, the target test plan in the disclosed embodiment includes at least one of the following: a first test plan for offline testing of the function to be delivered; a second test plan for online testing of the function to be delivered.

[0050] In the disclosed embodiments, by selecting one of the first test plan for offline quality assurance and the second test plan for online risk management, or a combination of the two, it is possible to proactively gain insight into risks that may cause system problems, analyze risks and proactively plan solutions, automatically schedule execution of verification tasks, make intelligent decisions based on facts, memory, and reasoning, eliminate risks or expose risks, give repair suggestions for risks, assist in the development of self-service repair, and achieve automatic repair in some scenarios, further improving delivery efficiency.

[0051] The first test plan and the second test plan are described below respectively.

[0052] 1) The first test plan for offline testing of the functions to be delivered.

[0053] In some embodiments, generating a target test plan for the function to be delivered may be implemented as follows: Figure 3 As shown, including:

[0054] S301: When the target test plan includes a first test plan, determine potential risks of functions to be delivered.

[0055] In some embodiments, determining the potential risk of the function to be delivered may be implemented as follows: determining the potential risk of the function to be delivered based on a preset risk analysis strategy.

[0056] During implementation, determining the potential risks of the functions to be delivered based on the preset risk analysis strategy can be exemplarily understood as analyzing code changes based on the demand background, identifying impact scenarios, and combining historical data to gain insight into potential risks.

[0057] In the disclosed embodiment, the potential risks of the functions to be delivered can be further identified in combination with the preset risk analysis strategy, thus laying a foundation for ensuring the safety of the functions to be delivered.

[0058] In some embodiments, determining the potential risk of the function to be delivered based on a preset risk analysis strategy may be implemented as step A1 and / or step A2:

[0059] Step A1, identifying potential risks from the description information of the functions to be delivered based on preset rules.

[0060] Wherein, when the description information of the function to be delivered is in the form of code, the preset rule may be a manually specified rule strategy to identify potential risks. The rule strategy may be implemented by the written code, thereby discovering potential risks in the code of the function to be delivered.

[0061] In the disclosed embodiment, since the description information of the function to be delivered is composed of preset rules, potential risks can be identified from the description information of the function to be delivered based on the preset rules. This method is simple and easy to implement.

[0062] In some embodiments, identifying potential risks from the description information of the function to be delivered based on preset rules may be implemented as follows:

[0063] Step A11, semantic analysis is performed on the functional requirement text of the function to be delivered to obtain the key requirements of the function to be delivered.

[0064] The functional requirement text may be a requirement document formed by discussing the functional points of the function to be delivered in the early stage. It may also be a further explanation provided by the user. The requirement document may include at least one of the following contents: functional description of the function to be delivered, business impact scope and impact degree of the function to be delivered, etc.

[0065] Among them, the big model can be used to perform semantic analysis on the functional requirement text to obtain the key requirements corresponding to the function to be delivered.

[0066] The requirements points may include, for example, the business category and an overview of the requirements, which can be understood as a theoretical understanding of the requirements and impact of the code.

[0067] Step A12: Perform white box testing on the code of the function to be delivered to obtain the first type of risk points.

[0068] White box testing of the code to be delivered is to understand the requirements and impact of the code from a practical perspective. Based on the actual code changes, analyze the impact of code changes on the business scope, and analyze some potential risks based on the code. The first type of risk points obtained can include stability risks (Core outages), performance risks (memory, CPU increases, processing time increases), etc.

[0069] It should be noted that the execution order of step A11 and step A12 is not limited. Step A11 is to analyze the potential problems that may be involved from the perspective of the requirement text, and step A12 is to analyze the potential problems in the specific implementation. The two can assist each other to fully complete the risk analysis.

[0070] Step A13, analyzing the key points of the requirements, the first type of risk points, and the functional characteristics of the functions to be delivered based on preset rules to obtain the second type of risk points.

[0071] Among them, the description information includes key points of requirements, first-category risk points and functional characteristics; the potential risks include first-category risk points and second-category risk points.

[0072] During implementation, the preset risk features can be matched from the requirements, the first risk points, and the functional features of the functions to be delivered based on the preset rules, so as to analyze the potential risks. The specific rules to be used can be determined based on the actual situation, and the embodiments of the present disclosure do not limit this.

[0073] In the disclosed embodiment, the first type of risk points is used to understand the demand background, analyze code changes, identify the scope of impact scenarios, and generate the second type of risk points based on the key points of the demand and the functional characteristics of the functions to be delivered. The multi-faceted combination makes the potential risks more comprehensive. At the same time, based on this method, potential risks can be obtained automatically, saving manpower and material resources.

[0074] Step A1 describes the analysis of potential risks based on preset rules. The preset rules need to be listed by experts and the corresponding rule judgment function needs to be developed. When new problems arise, new preset rules need to be added, and risk insight has a certain lag. Therefore, in the disclosed example, in order to improve the generalization ability of risk insight, the powerful knowledge background and logical reasoning ability of the big model can be used to identify potential risks. Specifically, in step A2, potential risks can be identified from the description information of the functions to be delivered based on the big model.

[0075] Among them, the large model can be a large language model (LLMs). Large language models refer to a specific type of large model that is specifically used to process text data. This type of model is a natural language processing model based on a neural network, which can be used to generate, understand and process text data. Large language models can have tens of billions of parameters, can generate high-quality text, and can be used for various natural language processing tasks, such as question answering, text generation, dialogue systems, etc.

[0076] The large language model has good reasoning ability and small sample learning ability. The large language model can understand the model on a large scale or can be trained on a large number of samples. Based on the large model, accurate semantic understanding can be achieved.

[0077] In the disclosed embodiments, the reasoning capability of the big model can be used to accurately identify potential risks from the description information of the functions to be delivered. Not only can potential risks be identified, but with the prompts of the big model capabilities, new potential risks can be automatically identified as risks change, thereby improving the ability of risk insight.

[0078] Similarly, in some embodiments, identifying potential risks from the description information of the functions to be delivered based on the big model may be implemented as follows:

[0079] Step A21, semantic analysis is performed on the functional requirement text of the function to be delivered to obtain the key requirements of the function to be delivered.

[0080] Specifically, the method of obtaining the key requirements of the functions to be delivered has been described above, and the embodiments of the present disclosure will not be repeated here.

[0081] Step A22, perform white box testing on the code of the function to be delivered to obtain the first type of risk points.

[0082] Specifically, the method of obtaining the first type of risk points has been described above, and the embodiments of the present disclosure will not be described one by one here.

[0083] Step A23, based on the large model, analyze the key points of requirements, the first type of risk points, and the functional characteristics of the functions to be delivered to obtain the third type of risk points.

[0084] Among them, the description information includes key points of requirements, first-category risk points and functional characteristics; the potential risks include first-category risk points and third-category risk points.

[0085] During implementation, because the requirements, the first-class risk points, and functional features directly constitute the prompt information, the prompt information obtained is too long, which is not conducive to the logical reasoning of the big model. Therefore, based on the big model, some key information can be filtered out from the requirements, the first-class risk points, and the functional features of the functions to be delivered to construct prompt information and input it into the big model for risk insight.

[0086] In the disclosed embodiment, the first type of risk points are used to understand the demand background, analyze code changes, and identify the scope of impact scenarios. The first type of risk points, demand points, and functional characteristics of the functions to be delivered are generated into the third type of risk points using a large model. The multi-faceted combination and the strong reasoning ability of the large model make the potential risks more accurate. At the same time, based on this method, potential risks can be obtained automatically, saving manpower and material resources.

[0087] During implementation, one of step A1 and step A2 may be selected to identify potential risks from the description information of the functions to be delivered, or a combination of the two may be used to identify more potential risks from the description information of the functions to be delivered.

[0088] The potential risk points obtained by combining the two methods include the first risk point, the second risk point and the third risk point.

[0089] S302: Generate a first test plan based on the big model for potential risks of the functions to be delivered.

[0090] In the disclosed embodiment, from the perspective of offline quality assurance, the potential risks of the function to be delivered are determined, and further planning can be made based on the potential risks, so that the quality of the function to be delivered can be guaranteed at the online linear test node.

[0091] In some embodiments, generating a first test plan based on the big model for potential risks of the functions to be delivered may be implemented as follows:

[0092] Step C1, generating recommended input information based on the white box test results of the functions to be delivered.

[0093] When implemented, white box testing results can include business scope impacts, based on which recommended input information is generated.

[0094] The recommended input information may be understood as input information required for testing.

[0095] Step C2: Analyze the recommended input information and potential risks based on the large model to generate a first test plan, which includes the recommended input information, test scenarios, and test tasks.

[0096] The test scenario can be understood as the scenario that needs to be tested for the function to be delivered. Exemplary test scenarios include system services, traffic ranges, and change types.

[0097] The test task can be understood as the test point that needs to be verified in this test, and examples include stability test, performance 01 test, queue ratio test, etc.

[0098] During implementation, the recommended input information and potential risks are combined with historical risk examples and jointly input into the big model. The reasoning ability of the big model is used to infer the first test plan.

[0099] In the disclosed embodiment, after the potential risks are obtained, the first test plan is automatically obtained based on the strong reasoning ability of the large model, which can realize the autonomous completion of the test plan and improve the delivery efficiency.

[0100] During implementation, the first test plan can be confirmed by RD. If confirmed by RD, the first test plan will continue to be executed to obtain the test results. Exemplary test results may include 0 / 1 results, risk indicators, test coverage, etc. Among them, 0 / 1 results indicate whether the test has passed; risk indicators refer to whether the time consumption of the function points of the functions to be delivered in the nodes of the current delivery process meets expectations; test coverage refers to the comparison of the risk range covered in the actual test process and the risk range assessed before the test (i.e., the risk range of potential risks described above) to see whether the test is comprehensive.

[0101] In some embodiments, the test results of the first test plan are analyzed based on the potential risks of the functions to be delivered using a big model, which can be implemented as follows: querying the historical test data of the functions to be delivered to obtain historical data acquisition results; inputting the historical data acquisition results and potential risks into the big model to obtain the decision results of the big model on the test results of the first test plan.

[0102] During implementation, if the first test plan is executed for the first time or if there is no test data stored for the relevant test, the historical data acquisition result is empty, that is, the potential risks and some related content can be input into the big model to obtain the decision result of the big model on the test result of the first test plan. Some related content may include, for example, the scope of influence, historical experience, test execution results, system indicator performance, etc.

[0103] If the decision result indicates that retesting is required, the RD can be interacted with again to confirm which risk points in the potential risks need to be tested and which risk points do not need to be tested. Or the RD can repair the functions to be delivered and then perform the test again. In this case, the historical data acquisition results can be obtained, and the historical data acquisition results and potential risks can be input into the big model to obtain the big model's decision results on the test results of the first test plan.

[0104] In the disclosed embodiment, historical data acquisition results are combined so that the big model can be based on historical content as a reference, so that the big model can understand the historical process of the function to be delivered, and the big model can make targeted decisions on the function to be delivered.

[0105] In some embodiments, the test results of the first test plan are analyzed based on the potential risks of the functions to be delivered by using a large model, so that intelligent decision-making on the test results can be achieved. The large model can learn all tools, expert knowledge, etc. related to the test. The decision result of the intelligent decision can be given by relying on the emphasized knowledge background. The decision result includes at least one of the following:

[0106] Indication of whether the function to be delivered has passed offline testing;

[0107] The test results cover the potential risks, that is, which risk points in the potential risks have completed the test and passed, and which risk points have not passed or have not been tested;

[0108] Indication information of risk points that have been resolved among potential risks, that is, which risk points among potential risks have passed the test and have been resolved;

[0109] Risk level of residual risk;

[0110] whether additional testing is needed;

[0111] Potential failures;

[0112] Solutions to potential failures.

[0113] During implementation, if the indication information of whether the function to be delivered has passed the offline test indicates that it has passed the offline test, the next process can be carried out. If it indicates that it has not passed the offline test, further judgment is required based on the system performance and user needs. Other decision results can be determined based on actual conditions. In addition, the decision results can be adaptively increased or decreased based on actual conditions. This disclosure embodiment does not limit this.

[0114] In the disclosed embodiment, a variety of decision results are used, and each decision result executes a corresponding subsequent operation, which can cover most situations, thus making the decision result universal.

[0115] 2) A second test plan for online testing of the functions to be delivered.

[0116] In some embodiments, a target test plan is generated for the function to be delivered, which can be implemented as follows: when the target test plan includes a second test plan, a large model is used to analyze the test results of the first test plan based on the potential risks of the function to be delivered; when it is determined that the test result of the first test plan is a passed test, a second test plan is generated based on the test result of the first test plan.

[0117] The specific method of obtaining the test results of the first test plan has been described above, and the embodiments of the present disclosure will not be repeated here. It should be noted that the first test plan involved in generating the second test plan can be intelligently planned based on the above method, or it can be carried out in a linear programming manner, and the embodiments of the present disclosure are not limited to this.

[0118] The second test plan for online testing in the disclosed embodiment is determined by combining online testing and offline testing. Therefore, when the test result of the offline first test plan is that the test passes, the second test plan is generated, which can save computing resources while ensuring the quality of the functions to be delivered.

[0119] In some embodiments, generating a second test plan based on the test results of the first test plan may be implemented as follows:

[0120] Step D1, when a business system where the function to be delivered is located is changed, obtain the target risks in the test results of the first test plan whose risk level is lower than the preset level and which are not resolved by the offline test.

[0121] An exemplary business system change is an update of the system version.

[0122] Step D2, determining the target risk as the online key monitoring risk in the second test plan.

[0123] In the embodiments of the present disclosure, offline testing can be understood as testing completed before the function to be delivered is submitted to the business system for online operation. Online testing can be understood as testing completed when the function to be delivered is submitted to the business system for online operation.

[0124] In order to improve the efficiency of offline testing, some low-risk items with risk levels lower than the preset levels may not be included in the linear test. Therefore, low-risk points will be given special attention during online testing.

[0125] Step D3, based on the large model, analyze the online key monitoring risks and the requirement change knowledge of the functions to be delivered, and generate a second test plan.

[0126] In the disclosed embodiment, when a change in the business system is detected, a favorable basis is provided for generating a second test plan for online testing based on the offline and online risk linkage mechanism and the risk indicators of offline testing.

[0127] Wherein, in the case of a business system change, the corresponding second test plan generated includes at least one of the following:

[0128] Key monitoring items:

[0129] Added monitoring items for new business strategies;

[0130] Alarm information processing rules.

[0131] Among them, key monitoring items are monitoring items generated by target risks detected offline;

[0132] The monitoring items of the newly added business strategy refer to the monitoring items generated based on similar business strategies relative to the newly added business strategy of the business system. In this way, the monitoring items can be automatically supplemented for the newly added business strategy to improve the testing efficiency;

[0133] The alarm information processing rules may include alarm information processing strategies related to the monitoring items in the system, and may also include alarm information processing strategies related to other monitoring items. The alarm information processing rules define the alarm conditions corresponding to different monitoring items. For example, it may include providing an alarm interception threshold, and triggering an alarm when the monitoring item exceeds the alarm interception threshold.

[0134] In the disclosed embodiment, when changes are detected in the business system, based on the offline and online risk linkage mechanism, combined with the risk indicators of offline testing, intelligent planning of project-level key monitoring indicators, automatic supplementary monitoring items, alarm processing rules, etc., key monitoring is performed at each node of the project launch to further ensure the security of the functions to be delivered.

[0135] In some embodiments, generating a second test plan based on the test results of the first test plan may be implemented as follows:

[0136] Step E1, when the business indicators of the business system where the function to be delivered is located are changed, the system status of the business system and the knowledge graph of the business system are obtained;

[0137] Among them, the business indicator change means that the function to be delivered modifies the business indicator, which requires continuous detection of the business indicators of the business system. For example, changes in the environment and hot spots may cause changes in business indicators.

[0138] Step E2, based on the big model processing business system status, knowledge graph and functional requirement knowledge of the functions to be delivered, to generate a second test plan.

[0139] Among them, the knowledge graph can be composed of multiple historical system states, business indicators, and historical plans.

[0140] During implementation, the system status of the business system, the knowledge graph, and the functional requirement knowledge of the functions to be delivered are used to construct prompt information, which is then input into the big model to obtain the second test plan.

[0141] In the disclosed embodiment, when it is discovered that a business indicator of a business system has changed, the system combines the real-time system status with the knowledge graph and uses its powerful reasoning ability to intelligently plan a second test plan, so that the process is completed automatically and human resources are saved.

[0142] Wherein, when the business indicator changes, the corresponding second test plan generated includes at least one of the following:

[0143] Key abnormal indicators;

[0144] Recommended alarm thresholds;

[0145] Key inspection targets.

[0146] Among them, the key abnormal indicators represent the business indicators that need to be focused on; the recommended alarm threshold can be determined based on the normal range. The recommended alarm threshold can be divided into levels from 1 to 10, 1 means no need to deal with it temporarily, that is, the lowest alarm, increasing level by level, and 10 means an emergency alarm that needs to be dealt with immediately. For example, if it exceeds the normal range by 50%, combined with the overall situation of the system and user expectations, the alarm threshold is determined to be an emergency alarm.

[0147] The key inspection objects may include business system nodes that need key inspection. For example, if a business system includes multiple service nodes, which service nodes need key inspection, then these service nodes are set as key inspection objects.

[0148] In the disclosed embodiment, based on the corresponding multiple planning contents in the second test plan, the intelligence level of the second test plan can be improved, thereby improving the delivery efficiency.

[0149] In the disclosed embodiment, the method further includes: processing the test results of the second test plan based on relevant processing parameters of the alarm information in the second test plan.

[0150] During implementation, after the second test plan is executed and completed, the test results are obtained. Based on the relevant processing parameters of the alarm information, relevant contents such as whether to alarm, whether to intercept, stop loss suggestions, etc. can be given.

[0151] In the embodiment of the present disclosure, when the second test plan is generated, the test results of the second test plan can be processed based on the relevant processing parameters of the alarm information in the second test plan, and a corresponding disposal strategy can be given based on the test results to improve delivery efficiency.

[0152] In order to further improve the delivery efficiency in the disclosed embodiment, human-computer interaction with RD can be performed to collaborate on intelligent testing. The entire delivery process (from R&D -> online) can be completed through LUI dialogue. It can accurately understand user questions, accurately plan execution, powerful processing capabilities, and give natural and humanized answers. The design of the overall LUI dialogue function includes core capabilities and core services (Agent):

[0153] Among them, the core capabilities may include intent recognition capabilities; the core services are implemented using multiple plug-ins, which may include test plug-ins, problem location plug-ins, process management plug-ins, operation guidance plug-ins, and knowledge question and answer plug-ins. The specific content will be explained later.

[0154] In some embodiments, based on the target test plan, guiding the target object to interact in a natural language manner to obtain the interaction result can be implemented as follows:

[0155] Step F1, when the target object is required to provide the information to be supplemented in the target test plan, guide the target object to provide the information to be supplemented based on natural language.

[0156] For example, as mentioned above, when a potential risk is detected, the target test plan may provide directions for supplementing content or suggestions for modifying content, thereby guiding the target object to supplement information in a natural language manner.

[0157] In some embodiments, step F1 can be implemented as follows: based on the recommended input information in the target test plan, obtain recommended input information from a known information set; if the known information set lacks the information to be supplemented in the recommended input information, guide the target object to provide the information to be supplemented in a natural language manner.

[0158] In the disclosed embodiment, when supplementary information of the target object is required, recommended input information can be obtained from a known information set. When missing information is determined, the target object is guided to provide the information to be supplemented using natural language to supplement the missing information so that the target test plan can be successfully completed.

[0159] Step F2: updating the target test plan in response to the information to be supplemented.

[0160] During implementation, based on the information to be supplemented, offline test agents and online test agents are used to re-plan and obtain a new target test plan. The specific planning process has been described above, and the embodiments of the present disclosure will not be repeated here.

[0161] In the disclosed embodiment, when additional information is needed, the intelligent delivery assistant can promptly inform the target object to supplement the content, and the use of a human-machine combination can improve delivery efficiency.

[0162] In some embodiments, in response to the information to be supplemented, updating the target test plan may be implemented as follows:

[0163] Step G1, based on the mandatory parameters and optional parameters required by the target node in the delivery process, extract the supplementary parameters from the information to be supplemented.

[0164] Since there are multiple nodes in the delivery process and each node has multiple parameters, when testing a node, it is necessary to filter out the required parameters and optional parameters for executing the node from the numerous parameters, and extract the supplementary parameters from the information to be supplemented.

[0165] Step G2, improving the target test plan based on the supplementary parameters.

[0166] In the disclosed embodiment, necessary parameters and optional parameters are screened out, and supplementary parameters are extracted from the information to be supplemented, so as to improve the target test plan, thereby ensuring that the target test plan can proceed smoothly and improving the delivery efficiency.

[0167] In some embodiments, the aforementioned core capabilities include intent recognition capabilities, which can be implemented as follows: Figure 4 As shown, including:

[0168] S401, obtaining a sentence to be processed provided by a target object based on natural language.

[0169] S402, determining a target plug-in for processing the statement to be processed.

[0170] Among them, target plug-ins can be used to assist in delivering the features to be delivered.

[0171] Among them, the target plug-in can be understood as any one of the test plug-in, problem location plug-in, process management plug-in, operation guidance plug-in, and knowledge question and answer plug-in. The purpose of each plug-in will be explained later, and the embodiments of the present disclosure will not be repeated here.

[0172] S403, processing the pending statement based on the target plug-in.

[0173] During implementation, the corresponding plug-in is called to execute the statement to be processed to obtain the processing result. The processing result can be fed back to the target object to understand the execution status.

[0174] In the disclosed embodiments, delivery can be completed based on the natural language of the target object, which lowers the delivery threshold and further improves delivery efficiency and capabilities.

[0175] In some embodiments, since there are multiple plug-ins, determining a target plug-in for processing a statement to be processed may be implemented as follows:

[0176] Step H1, perform intent recognition on the sentence to be processed to obtain the target intent.

[0177] During implementation, the BERT (Bidirectional Encoder Representation from Transformer) model can be used for intent recognition to obtain the target intent; the TextCNN (Text Convolutional Neural Networks) model or a large model can also be used for intent recognition to obtain the target intent, which is not limited in the embodiments of the present disclosure.

[0178] In order to make the recognized intent more accurately determine the target plug-in used to complete the task, the intent recognition is performed on the statement to be processed. The target intent can be implemented as follows: extracting keyword slot information from the statement to be processed; rewriting the statement to be processed based on the keyword slot information to obtain a rewritten statement; and performing intent recognition on the rewritten statement to obtain the target intent.

[0179] During implementation, keyword hard matching and regular expressions can be used to extract keyword slot information from the statement to be processed, and then the statement to be processed can be rewritten.

[0180] In the disclosed embodiment, key information can be obtained by extracting keyword slots, and then rewritten based on the key information to obtain a rewritten statement, which can clearly express the target intent of the statement to be processed, so as to accurately determine the target plug-in.

[0181] Step H2, using the large model to assign at least one candidate plug-in to the target intent.

[0182] Step H3: interact with at least one candidate plug-in according to the assigned priority of at least one candidate plug-in to determine a plug-in with the highest priority among the candidate plug-ins that can process the statement to be processed as a target plug-in.

[0183] In the disclosed embodiment, multiple plug-ins are provided. For similar statements to be processed, multiple plug-ins may be able to process them, but due to the different capabilities of different plug-ins, the processing results are different. Therefore, in the disclosed embodiment, the large model can screen out at least one candidate plug-in that can process, and then call the query function of each plug-in to inquire about the processing capabilities of each plug-in for the statement to be processed, thereby realizing interaction with each candidate plug-in. Finally, the most reasonable gap can be screened out to process the statement to be processed, thereby improving the accuracy of human-computer interaction.

[0184] In the disclosed embodiment, a query detection operation is added to at least one planned candidate plug-in, and the candidate plug-in can preliminarily decide whether to process the statement to be processed. When the two reach an agreement, the system gives priority to selecting this plug-in to execute the task of the statement to be processed to improve the execution accuracy.

[0185] In some embodiments, when the target plug-in is a delivery process management plug-in, processing the statement to be processed based on the target plug-in can be implemented as follows: extracting supplementary parameters from the statement to be processed based on the required parameters and optional parameters required by the target plug-in.

[0186] The process management plug-in is used to ensure that each node is executed according to the predetermined standard process. Each process node to the next process node can customize the decisions to be made and the reports to be generated, such as the typical process of self-test requirements: admission-submission-testing-release-online. The decision-making strategy of the process management plug-in can be used to judge the flow from each process node to the next process node. Each process has its own information management service.

[0187] During implementation, the mandatory parameters and optional parameters can be obtained based on regular expressions and hard rule matching, and the supplementary parameters can also be extracted from the statement to be processed by regular expressions and hard rule matching.

[0188] In the embodiment of the present disclosure, when the target plug-in is a delivery process management plug-in, the normal operation of the delivery process is ensured based on the required parameters, optional parameters and supplementary parameters.

[0189] In some embodiments, based on the required parameters and optional parameters required by the target plug-in, supplementary parameters are extracted from the statement to be processed, which can be implemented as follows: extracting required parameters and optional parameters from the statement to be processed; when at least one sub-parameter of the required parameters is missing in the statement to be processed, guiding the target object to supplement at least one sub-parameter through multiple rounds of dialogue.

[0190] In addition, pipeline parameters can also be obtained from the statement to be processed. The pipeline can be understood as a carrier of information. Some key information required for testing can be obtained from this pipeline to complete the automatic filling of parameters. Some parameters that do not appear in the required parameters and optional parameters and need to be provided by RD can also be obtained from this pipeline or carrier to facilitate interaction with RD and supplement parameters.

[0191] In the disclosed embodiment, the parameters obtained in this way are the parameters required for the test, and based on the multiple interactions, the integrity of the parameters is ensured and the test efficiency is improved.

[0192] In some embodiments, after obtaining the test results, the test results of the target test plan can also be obtained; based on the test results, determine whether to enter the next delivery link. This operation can be completed by the aforementioned process management plug-in. During implementation, if the test result of the target test plan is passed, the process management plug-in confirms entering the next link, and if the test result of the target test plan is not passed, the process management plug-in confirms entering the next link. Exemplarily, in the case where the process includes compilation->access->submission of test->release->online, when the test result of the access process indicates a pass, the process management plug-in confirms entering the submission of test link.

[0193] By automatically determining whether a link has been circulated, the entire delivery process can be automatically executed, thereby improving delivery efficiency.

[0194] In the disclosed embodiment, the execution of the test can be completed by calling the test plug-in. The test plug-in uses multiple online test tools to focus on assisting RD in all the needs of the self-test link, such as vocabulary construction, environment construction, request sending, etc. At the same time, as a pre-stage of the delivery process, it provides key data reference for the intelligent flow of subsequent process management plug-ins.

[0195] In some embodiments, in each link of the delivery process, it is also possible to determine whether the start conditions of the target test plan are met based on obtaining the parameters required to execute the target test plan.

[0196] During implementation, the parameters required for the target test plan are the aforementioned necessary parameters. It should be noted that all necessary parameters need to be screened out to meet the start conditions of the target test plan. This method can be completed by the aforementioned process management plug-in, so that the process management plug-in can comprehensively manage the start and process of the test.

[0197] In the disclosed embodiment, the parameters required for executing the target test plan are obtained to determine whether the start conditions of the target test plan are met, so as to ensure the accuracy and completeness of the test and improve the delivery efficiency.

[0198] In some embodiments, intelligent problem location is provided to the target object in the form of LUI. During implementation, the problem description information of the function to be delivered can be determined; based on the problem description information, the problem expert model can be determined; based on the problem expert model, the problem description information can be analyzed to obtain the code snippet with the problem and the repair suggestion.

[0199] The problem description information may be obtained by interacting with the target object, or may be obtained based on the error information during the test. This test may be a test plan actively formulated by the target object, or may be a target test plan formulated by the intelligent preamble. The problem location may be completed by the problem location plug-in.

[0200] Due to the various problems that may be encountered during the integration and delivery process, the problem location plug-in can combine multiple data information such as execution logs, changed code slices, demand background analysis, etc., and according to different types of problems, comprehensively set and adjust prompt information to the corresponding problem expert model to obtain the problematic code snippets and repair suggestions.

[0201] In the disclosed embodiment, a problem expert model is used to obtain corresponding problematic code snippets and repair suggestions, so as to facilitate the target object to perform subsequent repairs on the functions to be delivered.

[0202] In some embodiments, not only can the automatically planned target test plan be completed automatically, but offline testing of the target function can also be completed by interacting with the target object.

[0203] The target function can be a function to be delivered or an offline function other than the function to be delivered. The target object can use LUI technology to perform offline testing on any developed function. The offline test can be implemented based on the test plug-in of the integrated test tool.

[0204] In the disclosed embodiment, offline testing is completed by interacting with the target object, so that the target object can be tested flexibly.

[0205] In some embodiments, in order to improve user experience, the embodiments of the present disclosure may also interact with the target object and provide delivery knowledge to the target object.

[0206] For example, by using the knowledge question and answer plug-in, all-round experts who are familiar with business knowledge, system architecture, and process specifications can answer questions online in real time and efficiently answer all problems encountered by RD during the delivery process.

[0207] In the disclosed embodiment, the knowledge question and answer plug-in can be used to provide delivery knowledge to the target object, so that the target object can get answers in real time, thereby improving the user experience of the target object.

[0208] In some embodiments, in order to guide the target object to quickly complete the delivery, it is also possible to interact with the target object and provide the target object with an operation method of the delivery system.

[0209] For example, by using an operation guidance plug-in to clarify the delivery system's own capabilities, operation methods, etc., the target object can be guided to correctly and efficiently use the delivery system that integrates the delivery method of the embodiment of the present disclosure.

[0210] In the disclosed embodiment, the target object is enabled to quickly learn usage knowledge to assist the target object in quickly completing the delivery task.

[0211] Based on the same technical concept, the embodiment of the present disclosure also provides a delivery system, such as Figure 5 As shown, it includes an intelligent delivery engine and an intelligent delivery assistant, wherein:

[0212] The intelligent delivery engine is used to generate a target test plan for the functions to be delivered and send the target test plan to the intelligent delivery assistant;

[0213] Intelligent delivery assistant, which is used to guide the target object to interact in natural language based on target test planning and obtain interaction results;

[0214] The intelligent delivery engine is also used to execute the target test plan when the interaction result indicates to execute the target test plan, and obtain the target test result of the function to be delivered.

[0215] In the disclosed embodiments, a combination of an intelligent delivery assistant and an intelligent delivery engine is used. Based on the interaction between the intelligent delivery assistant and the target object, the intelligent delivery engine is used as the overall support, so that the target object completes the delivery task by using natural language dialogue. The entire system is simplified as a whole, and all delivery processes are completed through the intelligent delivery assistant, which opens up the communication mode between people and eliminates tedious information communication.

[0216] Or as Figure 5As shown in the figure, the intelligent delivery assistant includes an interactive portal, an interface layer, an intention layer, a multi-role collaborative service (Multi-Agent), a data layer, and an offline service. The interactive portal is used to interact with the target object, and the interface layer is used to obtain the target object's pending statement (Query) and return other information such as test results, alarm information, and modification suggestions to the target object. The intention layer is used to rewrite the Query and determine the target intention. Multi-Agent includes a process management plug-in, a problem location plug-in, a test plug-in, a knowledge question and answer plug-in, and an operation guidance plug-in to support the corresponding functions of the intelligent interactive assistant. The data layer includes a process management database, a problem location database, a tool process management database, a vocabulary and other data, a knowledge base document, a description of the background and capabilities of the person, etc. The data layer is used to provide reference content for the intelligent interactive assistant and can also provide the required knowledge for the large model. The offline service is used to push real-time progress to the target object and the corresponding problem location when the task fails. In addition, the training process of the large model is also an offline execution process.

[0217] The intelligent delivery engine includes offline test agents and online test agents. In addition, data engineering stores the data required for the delivery functions to flow to each process node, as well as various underlying data. Examples include interaction history, code, etc. The underlying data is used to support the flow of the delivery process, and examples include testing expertise, code expertise, business system knowledge, etc.

[0218] In some embodiments, the target test plan includes at least one of the following:

[0219] The first test plan for offline testing of the functions to be delivered;

[0220] The second test plan is used for online testing of the functions to be delivered.

[0221] In the disclosed embodiments, by selecting one of the first test plan for offline quality assurance and the second test plan for online risk management, or a combination of the two, it is possible to proactively gain insight into risks that may cause system problems, analyze risks and proactively plan solutions, automatically schedule execution of verification tasks, make intelligent decisions based on facts, memory, and reasoning, eliminate risks or expose risks, give repair suggestions for risks, assist in the development of self-service repair, and achieve automatic repair in some scenarios, further improving delivery efficiency.

[0222] In some embodiments, when the target test plan includes a first test plan, the intelligent delivery engine, including an offline test agent, is used to: determine potential risks of the functions to be delivered; and generate a first test plan based on the big model for the potential risks of the functions to be delivered.

[0223] In the disclosed embodiment, the offline testing agent identifies the potential risks of the function to be delivered based on the offline quality assurance perspective, and can make further plans based on the potential risks to ensure the quality of the function to be delivered in the next process node.

[0224] In some embodiments, the offline testing agent is specifically used to determine the potential risks of the functions to be delivered based on a preset risk analysis strategy.

[0225] In the disclosed embodiment, the offline test agent combined with the preset risk analysis strategy can provide a basis for identifying the potential risks of the functions to be delivered, so that the offline test agent can effectively ensure the safety of the functions to be delivered.

[0226] In some embodiments, the offline testing agent is specifically used to: identify potential risks from the description information of the functions to be delivered based on preset rules; and / or identify potential risks from the description information of the functions to be delivered based on a large model.

[0227] In the disclosed embodiment, since the description information of the function to be delivered is composed of preset rules, potential risks can be identified from the description information of the function to be delivered based on the preset rules. This method is simple and easy to implement.

[0228] In the disclosed embodiment, potential risks are obtained by using preset rules and a large model, thereby ensuring the accuracy of potential risks in the functions to be delivered.

[0229] In some embodiments, when potential risks are identified from the description information of the function to be delivered based on preset rules, the offline testing agent is used to: perform semantic analysis on the functional requirement text of the function to be delivered to obtain the key points of the requirements of the function to be delivered; and perform white box testing on the code of the function to be delivered to obtain the first type of risk points; analyze the key points of the requirements, the first type of risk points and the functional characteristics of the function to be delivered based on preset rules to obtain the second type of risk points; the description information includes the key points of the requirements, the first type of risk points and the functional characteristics; the potential risks include the first type of risk points and the second type of risk points.

[0230] The functional requirement text may be a requirement document formed by discussing the functional points of the function to be delivered in the early stage. It may also be a further explanation provided by the user. The requirement document may include at least one of the following contents: functional description of the function to be delivered, business impact scope and impact degree of the function to be delivered, etc.

[0231] Among them, white box testing of the code to be delivered is to understand the requirements and impact of the code from a practical perspective. Based on the actual code changes, the impact of code changes on the business scope is analyzed, and some potential risks are analyzed based on the code.

[0232] In the disclosed embodiment, the first type of risk points is used to understand the demand background, analyze code changes, identify the scope of impact scenarios, and generate the second type of risk points based on the demand points and the functional characteristics of the functions to be delivered. The multi-faceted combination makes the potential risks more accurate. Based on this method, the offline test agent can automatically obtain potential risks, saving manpower and material resources.

[0233] In some embodiments, in identifying potential risks from the description information of the function to be delivered based on the big model, the offline testing agent is specifically used to: perform semantic analysis on the functional requirement text of the function to be delivered to obtain the key points of the requirements of the function to be delivered; and perform white box testing on the code of the function to be delivered to obtain the first type of risk points; analyze the key points of the requirements, the first type of risk points and the functional characteristics of the function to be delivered based on the big model to obtain the third type of risk points; the description information includes the key points of the requirements, the first type of risk points and the functional characteristics; the potential risks include the first type of risk points and the third type of risk points.

[0234] In the disclosed embodiment, the first type of risk points are used to understand the demand background, analyze code changes, and identify the scope of impact scenarios. The first type of risk points, demand points, and functional characteristics of the functions to be delivered are generated into the third type of risk points using a large model. The multi-faceted combination and the strong reasoning ability of the large model make the potential risks obtained more accurate. Based on this method, the offline test agent can automatically obtain potential risks, saving manpower and material resources.

[0235] In some embodiments, the offline test agent is used to: generate recommended input information based on the white box test results of the functions to be delivered; generate a first test plan based on the large model analysis of the recommended input information and potential risks; the first test plan includes recommended input information, test scenarios and test tasks.

[0236] In the disclosed embodiment, after obtaining the potential risk, the offline test agent can automatically obtain the first test plan based on the strong reasoning ability of the large model, and then complete the test autonomously.

[0237] In some embodiments, when the target test plan includes the second test plan, the intelligent delivery engine includes an offline test agent and an online test agent, wherein:

[0238] An offline test agent for analyzing the test results of the first test plan based on the potential risks of the functions to be delivered using a large model;

[0239] The online test agent is used to generate a second test plan based on the test result of the first test plan when it is determined that the test result of the first test plan is a test pass.

[0240] Since the embodiment of the present disclosure is determined by combining online testing and offline testing, when the test result of the offline first test plan is that the test passes, a second test plan is generated, which can save computing resources and improve delivery efficiency.

[0241] In some embodiments, the offline test agent is specifically used to: query the historical test data of the function to be delivered to obtain the historical data acquisition results; input the historical data acquisition results and potential risks into the big model to obtain the decision results of the big model on the test results of the first test plan.

[0242] In the disclosed embodiment, combined with the historical data acquisition results, the big model can understand the historical process of the function to be delivered based on the historical content as a reference, so that the big model can make targeted decisions on the function to be delivered.

[0243] In some embodiments, the test results of the first test plan are analyzed based on the potential risks of the functions to be delivered by using a large model, so that intelligent decision-making on the test results can be achieved. The large model can learn all tools, expert knowledge, etc. related to the test. The decision result of the intelligent decision can be given by relying on the emphasized knowledge background. The decision result includes at least one of the following:

[0244] Indication information on whether the functions to be delivered have passed offline testing; that is, which risk points among potential risks have passed the test and have been resolved;

[0245] The coverage of test results for potential risks;

[0246] Indication of the points in the potential risk that have been resolved;

[0247] Risk level of residual risk;

[0248] whether additional testing is needed;

[0249] Potential failures;

[0250] Solutions to potential failures.

[0251] In the disclosed embodiment, a variety of decision results are used, and each decision result executes a corresponding subsequent operation, which can cover most situations, thus making the decision result universal.

[0252] In addition, the risk level can be determined by the big model directly based on whether the risk has been eliminated, whether it can be approved, and whether additional testing is needed. The big model can also be based on the above three results, combined with other content, and then further analyzed by the big model to finally determine the risk level.

[0253] In summary, the intelligent delivery engine is used for offline delivery agents in offline testing, such as Figure 6As shown, it can be divided into four work steps, namely risk insight, intelligent planning, test execution and intelligent decision-making.

[0254] Risk insight is mainly used to input the description information of the function to be delivered and the key requirements of the function to be delivered obtained from the code into the big model, and analyze potential risks by combining preset rules with the big model. Among them, the key requirements of the function to be delivered can be business impact, code change, and system linkage. Business impact refers to the impact of the function to be delivered within the business scope, code change refers to the changes in the code of the function to be delivered, and system linkage refers to whether the function to be delivered affects the entire system or a single module in the system.

[0255] The code analysis and feature extraction include code, related code slices, and few-shots. The code represents the related code of the function to be delivered. When a line of code is modified and the line of code is called, the linkage part of the line of code needs to be sliced ​​out and modified together, which is the related code slice. Few-shots can be understood as historical risk examples. Based on the above information, the requirements of the function to be delivered are obtained, and then the prompt information is purchased and input into the large model to obtain potential risks.

[0256] Intelligent planning is mainly used to test the large model and obtain the first test plan. Intelligent planning mainly realizes the white box test results of the functions to be delivered, and generates recommended input information based on the white box test results. The recommended input information may include service name, small traffic number, stress test duration, compilation method, traffic proportion, QPS (Queries PerSeconds, number of requests per second), etc., which are combined with potential risks and input into the large model. Among them, the small traffic number is used to distribute traffic. The compilation method indicates the compilation method required to execute the code. In addition, use cases are screened from the test scenarios of system services, traffic ranges and change types, and use cases for the scenario are generated at the same time to generate the first test plan. Among them, the system service includes the system name and other contents. The traffic range is determined based on the distribution of the aforementioned small traffic number. The change type indicates the type corresponding to the code change. The first test plan may include test tasks such as operation stability test, performance 01 test, and queue proportion.

[0257] Test execution mainly relies on the capabilities of the test platform to obtain test parameters and deploy the environment, and then obtain test data after test execution.

[0258] Intelligent decision-making is used to combine potential risks, process data, test data and other related content, generate prompt information, input the big model, obtain risk levels and data analysis results, and make accurate conclusions and positioning conclusions based on risk levels and data analysis results. The accurate conclusion means that the code can proceed to the next process node, and the positioning conclusion means locating the abnormal position and cause of the potential abnormality generated by the code to obtain the aforementioned decision results.

[0259] In the disclosed embodiment, the offline test can autonomously analyze historical and current test behavior data, and use the insight into potential risks, impact scope, historical experience, test execution results, system indicator performance, etc. as inputs to make decisions using a large model to obtain decision results. Based on the decision results, it can be known whether the risk has been eliminated, whether it can be approved, whether additional testing is needed, etc., in order to divide the risk level, and then determine the subsequent process state flow and subsequent decision-making behavior. For risk-free and low-risk projects, the entire process can be achieved without the intervention of testers, that is, unattended; for high-risk projects, observable quality assessment conclusions and reliable repair suggestions are given to assist R&D in self-service repair of problems, and some scenarios attempt system self-repair to improve labor efficiency.

[0260] In some embodiments, the online testing agent is specifically used to: when a business system where the function to be delivered is located changes, obtain the target risk whose risk level is lower than a preset level and has not been resolved by the online linear test in the test results of the first test plan; determine the target risk as an online key monitoring risk in the second test plan; and generate a second test plan based on the large model analysis of the online key monitoring risks and the knowledge of the requirement changes of the function to be delivered.

[0261] In the disclosed embodiment, the online test agent is used to detect changes in the business system. When the changes in the business system are detected, based on the offline and online risk linkage mechanism and combined with the risk indicators of the offline test, a favorable foundation is provided for the second test plan generated for the online test.

[0262] In some embodiments, when the business system is changed, the corresponding second test plan generated includes at least one of the following:

[0263] Key monitoring items:

[0264] Added monitoring items for new business strategies;

[0265] Alarm information processing rules related to key monitoring items.

[0266] Among them, key monitoring items are monitoring items generated by target risks detected offline.

[0267] In the disclosed embodiment, when changes are detected in the business system, based on the offline and online risk linkage mechanism, combined with the risk indicators of offline testing, intelligent planning of project-level key monitoring indicators, automatic supplementary monitoring items, alarm processing rules, etc., key monitoring is performed at each node of the project launch to further ensure the security of the functions to be delivered.

[0268] In some embodiments, the baseline test agent is specifically used to: obtain the system status of the business system and the knowledge graph of the business system when the business indicators of the business system where the function to be delivered is located change; process the system status of the business system, the knowledge graph and the functional requirement knowledge of the function to be delivered based on the big model to generate a second test plan.

[0269] In the disclosed embodiment, when it is found that the business indicators of the business system have changed, the big model uses its powerful reasoning ability to intelligently plan a second test plan in combination with the real-time system status and knowledge graph, so that the process is completed automatically and human resources are saved.

[0270] In some embodiments, when a business indicator changes, the corresponding second test plan generated includes at least one of the following:

[0271] Key abnormal indicators;

[0272] Recommended alarm thresholds;

[0273] Key inspection targets.

[0274] In the disclosed embodiment, when the online test agent has insight into changes in business indicators, it can improve the intelligence of the second test plan based on the corresponding multiple planning contents in the second test plan, thereby improving delivery efficiency.

[0275] In some embodiments, the online test agent is further used to: process the test results of the second test plan based on relevant processing parameters of the alarm information in the second test plan.

[0276] In the embodiment of the present disclosure, when the second test plan is generated, the test results of the second test plan can be processed based on the relevant processing parameters of the alarm information in the second test plan, and a corresponding disposal strategy can be given based on the test results to improve delivery efficiency.

[0277] In some embodiments, the intelligent delivery assistant is used to:

[0278] When the target object is required to provide supplementary information in the target test plan, guide the target object to provide the supplementary information based on natural language;

[0279] In response to the information to be supplemented, the target test plan is updated.

[0280] In the disclosed embodiment, when additional information is needed, the intelligent delivery assistant can promptly inform the target object to supplement the content, and the use of a human-machine combination can improve work efficiency.

[0281] In summary, in the online risk management environment, Figure 7 As shown in the figure, the online test of the intelligent delivery engine is divided into two situations: business system changes and business indicator changes. Each situation can be divided into four steps: insight, planning, execution, and decision-making:

[0282] 1) Business system changes

[0283] The insight step of business system changes mainly involves inputting the real-time system status of the system, the background knowledge of change requirements, and the offline and online risk linkage indicators into the big model, and then generating the second test plan and executing the second test plan. Among them, the second test plan includes at least one of the following: key monitoring items; monitoring items of new business policies; alarm information processing rules related to key monitoring items. The execution process is from single machine -> single computer room ->... -> full computer room, from small to large scope and based on the order of each stage (front\middle\back), and each scope and each stage are checked at the same time, and finally the test results are obtained. Finally, in the decision step, decisions will be made based on the test results, which are divided into three decision-making methods: whether to alarm, whether to intercept, and stop loss method, and the decision will be notified to the relevant personnel (actor).

[0284] 2) Changes in business indicators

[0285] The main detection scope of business indicator changes is the real-time system status, the knowledge graph of the business, and the change data information. The second test plan is obtained. The second test plan includes at least one key abnormal indicator, recommended alarm threshold and key inspection objects in the following information. The second test plan is inspected systematically, and the test results are obtained after the inspection is completed. Decisions are made based on the test results, which are divided into three decision-making methods: whether to alarm, whether to intercept, and stop loss recommendations, and the decision is notified to the relevant personnel (actors).

[0286] In some embodiments, the offline test agent or online test agent of the intelligent delivery engine is used to determine the information to be supplemented based on the following method: based on the recommended input information in the target test plan, the recommended input information is obtained from a known information set; when the information to be supplemented in the recommended input information is missing in the known information set, the target object is guided to provide the information to be supplemented in a natural language.

[0287] In the disclosed embodiments, when supplementary information of the target object is required, recommended input information can be obtained from a known information set. When missing information is determined, the intelligent delivery assistant can guide the target object to provide the information to be supplemented using natural language to supplement the missing information, so as to successfully complete the target test plan.

[0288] In some embodiments, the intelligent delivery assistant is specifically used to: extract supplementary parameters from the information to be supplemented based on the required parameters and optional parameters required by the target node in the delivery process; and improve the target test plan based on the supplementary parameters.

[0289] In the disclosed embodiment, necessary parameters and optional parameters are screened out, and supplementary parameters are extracted from the information to be supplemented, so as to improve the target test plan, thereby ensuring that the target test plan can proceed smoothly and improving the delivery efficiency.

[0290] In some embodiments, the intelligent delivery assistant is further used to: obtain the statements to be processed provided by the target object based on natural language; determine the target plug-in for processing the statements to be processed; and process the statements to be processed based on the target plug-in.

[0291] In the disclosed embodiment, the intelligent delivery assistant can complete the delivery based on the natural language of the target object, thereby lowering the delivery threshold and improving the delivery efficiency and capabilities.

[0292] In some embodiments, the intelligent delivery assistant is specifically used to: perform intent recognition on the statement to be processed to obtain the target intent; use a large model to assign at least one candidate plug-in to the target intent; and interact with at least one candidate plug-in based on the assignment priority of at least one candidate plug-in to determine the plug-in with the highest priority among the candidate plug-ins that can process the statement to be processed as the target plug-in.

[0293] In the disclosed embodiment, the intelligent delivery assistant adds an inquiry detection operation to at least one planned candidate plug-in. The candidate plug-in can preliminarily decide whether to process the statement to be processed. When the two reach an agreement, the system gives priority to selecting this plug-in to execute the task of the statement to be processed to improve the execution accuracy.

[0294] In the disclosed embodiment, the intelligent delivery assistant is specifically used for: extracting keyword slot information from the sentence to be processed; rewriting the sentence to be processed based on the keyword slot information to obtain a rewritten sentence; and performing intent recognition on the rewritten sentence to obtain a target intent.

[0295] In the disclosed embodiment, key information can be obtained by extracting keyword slots, and then rewritten based on the key information to obtain a rewritten statement, which can clearly express the target intent of the statement to be processed, so as to accurately determine the target plug-in.

[0296] In some embodiments, when the target plug-in is a delivery process management plug-in, the delivery process management plug-in is used to: extract supplementary parameters from the statement to be processed based on the required parameters and optional parameters required by the target plug-in.

[0297] In the embodiment of the present disclosure, when the target plug-in is a delivery process management plug-in, the normal operation of the process is ensured based on the required parameters, optional parameters and supplementary parameters.

[0298] In some embodiments, the delivery process management plug-in is specifically used to: extract mandatory parameters and optional parameters from the statement to be processed; when at least one sub-parameter of the mandatory parameters is missing in the statement to be processed, guide the target object to supplement at least one sub-parameter through multiple rounds of dialogue.

[0299] In the disclosed embodiment, the parameters obtained by the delivery process management plug-in in this way are the parameters required for the test. Based on the multiple interactions, the integrity of the parameters is ensured and the test efficiency is improved.

[0300] In some embodiments, the delivery process management plug-in is further used to: obtain the test results of the target test plan; and determine whether to enter the next delivery stage based on the test results.

[0301] In the disclosed embodiment, by automatically determining whether a link has been circulated, the entire delivery process can be automatically executed, thereby improving delivery efficiency.

[0302] In some embodiments, the delivery process management plug-in is further used to: determine whether the start condition of the target test plan is met based on obtaining the parameters required to execute the target test plan.

[0303] In the disclosed embodiment, the parameters required for executing the target test plan are obtained to determine whether the start conditions of the target test plan are met, so as to ensure the accuracy and completeness of the test and improve the delivery efficiency.

[0304] In some embodiments, the target plug-in also includes a problem location plug-in that supports providing intelligent problem location to the target object in the form of LUI. During implementation, the problem description information of the function to be delivered can be determined; based on the problem description information, the problem expert model can be determined; based on the problem expert model, the problem description information can be analyzed to obtain the code snippet with the problem and the repair suggestion.

[0305] In the disclosed embodiment, the problem location plug-in uses the problem expert model to obtain the corresponding problematic code snippets and repair suggestions, so as to facilitate the target object to perform subsequent repairs on the functions to be delivered.

[0306] In some embodiments, the target plug-in also includes a test tool set for interacting with the target object to complete offline testing of the target function.

[0307] In the disclosed embodiment, the target object is convenient for flexible testing based on the test plug-in required by the to-be-delivered function of the target object.

[0308] In the disclosed embodiment, the target plug-in further includes a question-and-answer plug-in for interacting with the target object and providing delivery knowledge to the target object.

[0309] In the disclosed embodiment, the knowledge question and answer plug-in can be used to provide delivery knowledge to the target object, so that the target object can get answers in real time, thereby improving the user experience of the target object.

[0310] In the disclosed embodiment, the target plug-in further includes an operation guide plug-in for interacting with the target object and providing the target object with an operation method of the delivery system.

[0311] In the disclosed embodiment, the use of the operation guidance plug-in can enable the target object to quickly adapt to the operation of the intelligent delivery assistant to assist the target object in quickly completing the task.

[0312] In some embodiments, for ease of understanding, the entire delivery process is as follows Figure 8 As shown, the target object sends a request for a statement to be processed. First, it is determined whether the target object needs to clean up historical conversations. If it is determined not to clean up, the historical conversations are obtained from the database (data base, DB), the statement to be processed and the historical conversations are rewritten, and the rewritten statements and key parameters (mandatory parameters) are obtained. The rewritten statements and key parameters (mandatory parameters) are combined as prompt information and input into the big model for plug-in identification. The target plug-in is selected from multiple candidate plug-ins. If the big model has not determined the plug-in, the operation is used to guide the plug-in to interact with the target object again.

[0313] Among them, the specific implementation process of determining the target plug-in based on the intent is as follows Fig. 9As shown, since the target object can obtain session information (session) by multiple interactions, the same user is the same session. It can be based on the recent (exemplary as n times) historical conversations of the session, or combined with the intention recognized by the session most recently. In addition, it can also be combined with historical conversations within a preset period of time, and pending events. The aforementioned multiple historical information can select at least one historical information as reference information, and combine the pending statements to obtain the intention recognition of this conversation. The goal of this recognition is to obtain required parameters, optional parameters, and target plug-ins. The content in the target can be gradually identified. It is necessary to determine the target plug-in first. When the target plug-in is accurately obtained, the necessary parameters of the plug-in are determined. If the necessary parameters of the plug-in exist, it is necessary to determine whether the necessary parameters are complete. When all the necessary parameters are determined, it is determined whether there is permission. If there is permission to call, the plug-in is called to execute the task, and the processing result of the plug-in is returned to the target object. In the case of not accurately obtaining the target plug-in, the question and answer plug-in is called to inquire the target object. After receiving a positive answer, the subsequent operation is performed. If necessary parameters are detected to be missing, the target object is interacted with again to obtain all necessary parameters before subsequent operations are performed. If it is determined that there is no call permission, it is necessary to apply for call permission. When a process reaches a preset number of times (for example, 3 times), the target object is informed to contact the QA on duty.

[0314] Based on the method of enabling the target object to use natural language dialogue in the embodiment of the present disclosure, the delivery task is completed. In the related art, the delivery task is implemented, such as Fig.10 As shown, under this delivery model, the entire project iteration process is based on the GUI graphical interface for interaction, and must be executed in sequence according to the established standards, delivery process (coding, self-testing, compilation, access, testing, release, and online) script-style operations. This is usually a solidified, regular, strict, and even somewhat complex operation method and execution process. R&D has to go through a complex process, such as: consulting learning materials, code development, local self-testing (which may involve the use of multiple testing tools), integration into pipeline access testing, troubleshooting causes of abnormal failures, communicating with test duty, submitting tests, publishing, online observation, confirming abnormal interception indicators, etc. This process often requires tedious operations on multiple platforms / tools, such as: demand space platform, local development platform, integrated pipeline, test plan, environment deployment, submission platform, online platform, indicator monitoring platform, etc. In the process of each demand delivery, two core problems may be faced:

[0315] (1) During the access and testing process: Who will test, what to test, how to test, and whether it can be put online?

[0316] For this problem, QA is usually required to have strong code comprehension ability, rich business knowledge accumulation and experience accumulation.

[0317] (2) During the launch and deployment process: What to observe, how to observe, whether there are any abnormalities, and how to stop losses?

[0318] This problem tests RD's self-awareness, system and business capabilities, and troubleshooting experience.

[0319] In response to problem (1), the intelligent delivery engine has achieved the transition from human intervention, human formulation, and human decision-making to AI insight, AI planning, automatic execution, and AI decision-making: based on a large amount of test delivery experience, we have trained a large delivery expert-level model for multiple sub-directions such as risk insight, test planning, decision approval, monitoring item generation, decision alarm, and decision stop loss for information flow services. Combined with the guiding ideology of the four-step method for building delivery agent applications (insight, planning, execution, and planning), we have built an intelligent delivery engine for information flow services to drive demand for unattended delivery: the questions of who will test, what to test, and how to test are handed over to offline test agents for resolution; the questions of what to observe online, how to observe, whether it is abnormal, and whether to stop loss are handed over to online test agents for resolution.

[0320] Regarding question (2): The intelligent delivery assistant is used to transform the entire delivery process from being oriented towards multiple platforms to being oriented towards the intelligent delivery assistant. The conversational intelligent delivery assistant based on LUI understands queries through large models, integrates the core key paths in the integrated delivery process, and connects all underlying platform capabilities in the entire process from development self-testing to delivery and launch. It makes the testing tool copilot online, and intelligentizes process management. Based on the knowledge background of delivery experts, it builds delivery problem location and question-answering intelligent entities. It is a personal assistant beside the RD, standing by anytime, anywhere, in real time and without interruption to solve problems encountered by the RD in the delivery process, provide help, and assist the RD to complete the entire delivery process smoothly using natural language, thereby improving delivery efficiency. The human-friendly interactive method greatly improves the project delivery experience.

[0321] Based on the two types of intelligent entities, the intelligent delivery engine and the intelligent delivery assistant proposed in the embodiments of the present disclosure, the traditional delivery model is broken to build a new human-machine collaborative delivery model: using LUI natural language as the basis for interaction, the knowledge background of a series of delivery experts such as business background, code engineering, and testing are instilled into the big model, and the big model's own understanding, generation, logic, and memory capabilities are used to achieve risk identification and detection, test planning, and enable RD to use the most simple way of human beings-natural language to complete the entire delivery process. Based on the AI ​​native application <Conversational LUI Intelligent Delivery System>, the "traditional script-style delivery model" is transformed into the "human-machine collaborative natural dialogue delivery model", which lowers the delivery threshold and improves delivery efficiency. The entire delivery process has achieved a transformation from the "traditional script-style" to the "human-machine collaborative natural dialogue" model.

[0322] Based on the same technical concept, a delivery device 1100 is proposed in the embodiment of the present disclosure, such as Fig.11 As shown, including:

[0323] A generating module 1101 is used to generate a target test plan for the function to be delivered;

[0324] An interaction module 1102 is used to guide the target object to interact in a natural language manner based on the target test plan to obtain an interaction result;

[0325] The execution module 1103 is used to execute the target test plan when the interaction result indicates to execute the target test plan, and obtain the target test result of the function to be delivered.

[0326] In some embodiments, the target test plan includes at least one of the following:

[0327] A first test plan for performing offline testing on the function to be delivered;

[0328] A second test plan for performing online testing on the function to be delivered.

[0329] In some embodiments, the generating module comprises:

[0330] a determining unit, configured to determine, when the target test plan includes the first test plan, a potential risk of the function to be delivered;

[0331] The first generating unit is used to generate the first test plan for the potential risk of the function to be delivered based on the big model.

[0332] In some embodiments, the determining unit is configured to:

[0333] The potential risk of the function to be delivered is determined based on a preset risk analysis strategy.

[0334] In some embodiments, the determining unit is configured to:

[0335] Identify the potential risk from the description information of the function to be delivered based on preset rules; and / or,

[0336] The potential risks are identified from the description information of the functions to be delivered based on the big model.

[0337] In some embodiments, the determining unit is configured to:

[0338] Performing semantic analysis on the functional requirement text of the function to be delivered to obtain the key requirements of the function to be delivered; and

[0339] Perform white box testing on the code of the function to be delivered to obtain the first type of risk points;

[0340] Analyze the key points of the requirement, the first type of risk points, and the functional characteristics of the function to be delivered based on the preset rules to obtain the second type of risk points; the description information includes the key points of the requirement, the first type of risk points, and the functional characteristics;

[0341] The potential risks include the first-category risk points and the second-category risk points.

[0342] In some embodiments, the determining unit is used to:

[0343] Performing semantic analysis on the functional requirement text of the function to be delivered to obtain the key requirements of the function to be delivered; and

[0344] Perform white box testing on the code of the function to be delivered to obtain the first type of risk points;

[0345] Analyze the key points of the requirements, the first type of risk points, and the functional characteristics of the functions to be delivered based on the large model to obtain third type of risk points; the description information includes the key points of the requirements, the first type of risk points, and the functional characteristics;

[0346] The potential risks include the first-category risk points and the third-category risk points.

[0347] In some embodiments, the first generating unit is used to:

[0348] Generate recommended input information based on the white box test results of the function to be delivered;

[0349] Analyze the recommended input information and the potential risks based on the large model to generate the first test plan;

[0350] The first test plan includes the recommended input information, test scenarios and test tasks.

[0351] In some embodiments, the generating module comprises:

[0352] an analyzing unit, configured to analyze a test result of the first test plan based on a potential risk of the function to be delivered by using the large model when the target test plan includes the second test plan;

[0353] The second generating unit is used to generate the second test plan based on the test result of the first test plan when it is determined that the test result of the first test plan is a test pass.

[0354] In some embodiments, the analysis unit is used to:

[0355] Query the historical test data of the function to be delivered to obtain the historical data acquisition result;

[0356] The historical data acquisition result and the potential risk are input into the big model to obtain a decision result of the big model on the test result of the first test plan.

[0357] In some embodiments, the decision result includes at least one of the following:

[0358] Indicative information on whether the function to be delivered has passed the offline test;

[0359] The coverage of the test results for potential risks;

[0360] Indicative information about the risk points that have been resolved among the potential risks;

[0361] Risk level of residual risk;

[0362] whether additional testing is needed;

[0363] Potential failures;

[0364] Solutions to the potential failures described.

[0365] In some embodiments, the second generating unit is used to:

[0366] In the event that the business system where the function to be delivered is located is changed, obtaining target risks in the test results of the first test plan whose risk levels are lower than a preset level and which have not been resolved by offline testing;

[0367] Determining the target risk as an online key monitoring risk in the second test plan;

[0368] The second test plan is generated by analyzing the online key monitoring risks and the requirement change knowledge of the functions to be delivered based on the large model.

[0369] In some embodiments, the second test plan includes at least one of the following:

[0370] Key monitoring items:

[0371] Added monitoring items for new business strategies;

[0372] Alarm information processing rules.

[0373] In some embodiments, the second generating unit is used to:

[0374] When a business indicator of the business system where the function to be delivered is located changes, obtaining a system state of the business system and a knowledge graph of the business system;

[0375] The system status of the business system, the knowledge graph and the functional requirement knowledge of the function to be delivered are processed based on the big model to generate the second test plan.

[0376] In some embodiments, the second test plan includes at least one of the following:

[0377] Key abnormal indicators;

[0378] Recommended alarm thresholds;

[0379] Key inspection targets.

[0380] In some embodiments, the system further includes a first processing module for:

[0381] Based on the relevant processing parameters of the alarm information in the second test plan, the test result of the second test plan is processed.

[0382] In some embodiments, the interaction module includes:

[0383] An interaction unit, configured to guide the target object to provide the information to be supplemented based on a natural language when the target object is required to provide the information to be supplemented in the target test plan;

[0384] An updating unit is used to update the target test plan in response to the information to be supplemented.

[0385] In some embodiments, the system further includes a screening unit for determining the information to be supplemented based on the following method:

[0386] Based on the recommended input information in the target test plan, obtaining the recommended input information from a known information set;

[0387] In the case that the information to be supplemented in the recommended input information is missing from the known information set, the target object is guided to provide the information to be supplemented in a natural language manner.

[0388] In some embodiments, the updating unit is used to:

[0389] Extracting supplementary parameters from the information to be supplemented based on the required parameters and optional parameters required by the target node in the delivery process;

[0390] The target test plan is improved based on the supplementary parameters.

[0391] In some embodiments, the system further includes a second processing module for:

[0392] Obtain the sentences to be processed provided by the target object based on natural language;

[0393] Determine a target plug-in for processing the statement to be processed;

[0394] The to-be-processed statement is processed based on the target plug-in.

[0395] In some embodiments, the second processing module is specifically used to:

[0396] Performing intent recognition on the sentence to be processed to obtain a target intent;

[0397] Using a large model to assign at least one candidate plug-in to the target intent;

[0398] According to the assigned priority of the at least one candidate plug-in, interact with the at least one candidate plug-in to determine a plug-in with the highest priority among the candidate plug-ins capable of processing the statement to be processed as the target plug-in.

[0399] In some embodiments, the second processing module is specifically used to:

[0400] Extracting keyword slot information from the sentence to be processed;

[0401] Rewrite the sentence to be processed based on the keyword slot information to obtain a rewritten sentence;

[0402] Intent recognition is performed on the rewritten sentence to obtain the target intent.

[0403] In some embodiments, when the target plug-in is a delivery process management plug-in, the second processing module is specifically configured to:

[0404] Based on the required parameters and optional parameters required by the target plug-in, supplementary parameters are extracted from the statement to be processed.

[0405] In some embodiments, the second processing module is specifically used to:

[0406] Extracting the mandatory parameters and the optional parameters from the statement to be processed;

[0407] In the case that at least one sub-parameter among the required parameters is missing in the statement to be processed, the target object is guided to supplement the at least one sub-parameter through multiple rounds of dialogue.

[0408] In some embodiments, a third processing module is further included, configured to:

[0409] Obtaining test results of the target test plan;

[0410] Based on the test results, determine whether to proceed to the next delivery stage.

[0411] In some embodiments, a fourth processing module is further included, configured to:

[0412] Based on obtaining the parameters required to execute the target test plan, it is determined whether a start condition of the target test plan is met.

[0413] In some embodiments, a fifth processing module is further included, configured to:

[0414] Determine the problem description information of the function to be delivered;

[0415] Based on the problem description information, determine the problem expert model;

[0416] The problem description information is analyzed based on the problem expert model to obtain the problematic code snippet and repair suggestions.

[0417] In some embodiments, a sixth processing module is further included, configured to:

[0418] Interact with the target object to complete offline testing of the target function.

[0419] In some embodiments, a seventh processing module is further included, configured to:

[0420] Interact with the target object and provide delivery knowledge to the target object.

[0421] In some embodiments, an eighth processing module is further included, configured to:

[0422] Interact with the target object and provide the target object with an operation method of the delivery system.

[0423] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0424] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0425] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0426] Fig.12 A schematic block diagram of an example electronic device 1200 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0427] like Fig.12 As shown, the device 1200 includes a computing unit 1201, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1202 or a computer program loaded from a storage unit 1208 into a random access memory (RAM) 1203. In the RAM 1203, various programs and data required for the operation of the device 1200 can also be stored. The computing unit 1201, the ROM 1202, and the RAM 1203 are connected to each other via a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.

[0428] A number of components in the device 1200 are connected to the I / O interface 1205, including: an input unit 1206, such as a keyboard, a mouse, etc.; an output unit 1207, such as various types of displays, speakers, etc.; a storage unit 1208, such as a disk, an optical disk, etc.; and a communication unit 1209, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1209 allows the device 1200 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0429] The computing unit 1201 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1201 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1201 performs the various methods and processes described above, such as the payment method. For example, in some embodiments, the payment method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1208. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1200 via the ROM 1202 and / or the communication unit 1209. When the computer program is loaded into the RAM 1203 and executed by the computing unit 1201, one or more steps of the payment method described above may be performed. Alternatively, in other embodiments, the computing unit 1201 may be configured to execute the payment method in any other appropriate manner (e.g., by means of firmware).

[0430] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0431] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0432] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0433] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0434] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0435] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0436] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0437] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A delivery method comprising: Generate target test plans for the functions to be delivered; Based on the target test plan, guiding the target object to interact in a natural language manner to obtain an interaction result; In the case where the interaction result indicates to execute the target test plan, the target test plan is executed to obtain the target test result of the function to be delivered.

2. The method according to claim 1, wherein the target test plan comprises at least one of the following: A first test plan for performing offline testing on the function to be delivered; A second test plan for performing online testing on the function to be delivered.

3. The method according to claim 2, wherein: The target test plan is generated for the functions to be delivered, including: In a case where the target test plan includes the first test plan, determining potential risks of the function to be delivered; The first test plan is generated based on the big model for the potential risk of the function to be delivered.

4. The method according to claim 3, wherein: The generating the first test plan based on the big model for the potential risk of the function to be delivered includes: Generate recommended input information based on the white box test results of the function to be delivered; Analyze the recommended input information and the potential risks based on the large model to generate the first test plan; The first test plan includes the recommended input information, test scenarios and test tasks.

5. The method according to claim 2, wherein: The target test plan is generated for the functions to be delivered, including: In a case where the target test plan includes the second test plan, using the large model to analyze the test results of the first test plan based on the potential risks of the function to be delivered; When it is determined that the test result of the first test plan is a passed test, the second test plan is generated based on the test result of the first test plan.

6. The method according to claim 5, wherein: The step of generating the second test plan based on the test result of the first test plan includes: In the event that the business system where the function to be delivered is located is changed, obtaining target risks in the test results of the first test plan whose risk levels are lower than a preset level and which have not been resolved by offline testing; Determining the target risk as an online key monitoring risk in the second test plan; The second test plan is generated by analyzing the online key monitoring risks and the requirement change knowledge of the functions to be delivered based on the large model.

7. The method according to claim 5, wherein: The step of generating the second test plan based on the test result of the first test plan includes: When a business indicator of the business system where the function to be delivered is located changes, obtaining a system state of the business system and a knowledge graph of the business system; The system status of the business system, the knowledge graph and the functional requirement knowledge of the function to be delivered are processed based on the big model to generate the second test plan.

8. The method according to claim 1, further comprising: Obtain the sentences to be processed provided by the target object based on natural language; Determine a target plug-in for processing the statement to be processed; The to-be-processed statement is processed based on the target plug-in.

9. The method according to claim 8, wherein: The determining of a target plug-in for processing the statement to be processed includes: Performing intent recognition on the sentence to be processed to obtain a target intent; Using a large model to assign at least one candidate plug-in to the target intent; According to the assigned priority of the at least one candidate plug-in, interact with the at least one candidate plug-in to determine a plug-in with the highest priority among the candidate plug-ins capable of processing the statement to be processed as the target plug-in.

10. The method according to claim 9, wherein: The performing intent recognition on the sentence to be processed to obtain the target intent includes: Extracting keyword slot information from the sentence to be processed; Rewrite the sentence to be processed based on the keyword slot information to obtain a rewritten sentence; Intent recognition is performed on the rewritten sentence to obtain the target intent.

11. A delivery system, comprising an intelligent delivery engine and an intelligent delivery assistant, wherein: The intelligent delivery engine is used to generate a target test plan for the function to be delivered, and send the target test plan to the intelligent delivery assistant; The intelligent delivery assistant is used to guide the target object to interact in a natural language manner based on the target test plan to obtain an interaction result; The intelligent delivery engine is further configured to execute the target test plan when the interaction result indicates to execute the target test plan, and obtain the target test result of the function to be delivered.

12. The system according to claim 11, wherein the target test plan comprises at least one of the following: A first test plan for performing offline testing on the function to be delivered; A second test plan for performing online testing on the function to be delivered.

13. The system according to claim 12, wherein: The offline test agent is used to: Generate recommended input information based on the white box test results of the function to be delivered; Analyze the recommended input information and the potential risks based on the large model to generate the first test plan; The first test plan includes the recommended input information, test scenarios and test tasks.

14. The system according to claim 12, wherein: In the case where the target test plan includes the second test plan, the intelligent delivery engine includes an offline test agent and an online test agent, wherein: The offline test agent is used to analyze the test results of the first test plan based on the potential risks of the function to be delivered by using the large model; The online test agent is used to generate the second test plan based on the test result of the first test plan when it is determined that the test result of the first test plan is a passed test.

15. The system of claim 14, wherein: The online test agent is specifically used for: In the event that the business system where the function to be delivered is located is changed, obtaining target risks in the test results of the first test plan whose risk levels are lower than a preset level and which are not resolved by online linear testing; Determining the target risk as an online key monitoring risk in the second test plan; The second test plan is generated by analyzing the online key monitoring risks and the requirement change knowledge of the functions to be delivered based on the large model.

16. The system of claim 14, wherein: The agent is tested on the baseline, specifically for: When a business indicator of the business system where the function to be delivered is located changes, obtaining a system state of the business system and a knowledge graph of the business system; The system status of the business system, the knowledge graph and the functional requirement knowledge of the function to be delivered are processed based on the big model to generate the second test plan.

17. The system according to claim 11, wherein the intelligent delivery assistant is further configured to: Obtain the sentences to be processed provided by the target object based on natural language; Determine a target plug-in for processing the statement to be processed; The to-be-processed statement is processed based on the target plug-in.

18. The system of claim 17, wherein: The intelligent delivery assistant is specifically used for: Performing intent recognition on the sentence to be processed to obtain a target intent; Using a large model to assign at least one candidate plug-in to the target intent; According to the assigned priority of the at least one candidate plug-in, interact with the at least one candidate plug-in to determine a plug-in with the highest priority among the candidate plug-ins capable of processing the statement to be processed as the target plug-in.

19. A delivery device comprising: The generation module is used to generate target test plans for the functions to be delivered; An interaction module, used to guide the target object to interact in a natural language manner based on the target test plan to obtain an interaction result; An execution module is used to execute the target test plan when the interaction result indicates to execute the target test plan, so as to obtain the target test result of the function to be delivered.

20. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 10.

21. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-10.

22. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 10.

Images