Language model-based regression test method and device for business system

Abstract

The invention provides a regression testing method and device of a business system based on a language model. According to the method, in the online operation process of a business system based on a language model, online real business requests received by the business system are collected; based on the service request, constructing a regression test case for the service system; and according to a received test request for the service system, performing a regression test on the service system through the regression test case. The timeliness and diversity of the regression test case are effectively ensured, and the accuracy and reliability of the test result are further improved.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the technical field of regression testing, and in particular, to a regression testing method and apparatus for a business system based on a language model. Background Art

[0002] With the rapid development of large language model technology, language model-based applications have penetrated into diverse fields such as natural language processing, intelligent recommendations, and intelligent question-answering. Due to the complexity of language model architectures and the breadth of application scenarios, the stability and reliability of business systems based on language models are facing higher requirements. In this context, regression testing has become an essential step in ensuring the continued high-quality operation of business systems. Its goal is to ensure functional integrity and performance compliance throughout the iteration process of business systems through systematic verification.

[0003] Currently, business system testing is typically performed using manual testing or semi-automated tools. Test cases used in regression testing are typically constructed based on fixed input data and output results to achieve end-to-end testing of model applications. However, in real-world applications, business requests are often diverse and constantly updated as user needs change and business systems upgrade. Test cases constructed based on fixed input data are less diverse and timely, making it difficult to guarantee the reliability and accuracy of test results. Summary of the Invention

[0004] In view of this, one or more embodiments of this specification provide the following technical solutions:

[0005] According to a first aspect of one or more embodiments of this specification, a regression testing method for a business system based on a language model is proposed, comprising:

[0006] During the online operation of the language model-based business system, real online business requests received by the business system are collected;

[0007] Based on the business request, construct regression test cases for the business system;

[0008] According to the received test request for the business system, regression testing is performed on the business system using the regression test case.

[0009] According to a second aspect of one or more embodiments of this specification, a regression testing apparatus for a business system based on a language model is proposed, comprising:

[0010] A collection module, configured to collect real online business requests received by a language model-based business system during its online operation;

[0011] A construction module, configured to construct a regression test case for the business system based on the business request;

[0012] The testing module is used to perform regression testing on the business system through the regression test case according to the received test request for the business system.

[0013] According to a third aspect of one or more embodiments of this specification, an electronic device is proposed, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor implements the steps of the above-mentioned language model-based business system regression testing method by running the executable instructions.

[0014] According to a fourth aspect of one or more embodiments of this specification, a computer-readable storage medium is proposed, on which computer instructions are stored. When the instructions are executed by a processor, the steps of the regression testing method of the business system based on the language model are implemented as described above.

[0015] According to a fifth aspect of one or more embodiments of this specification, a computer program product is proposed, including a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned regression testing method for a business system based on a language model.

[0016] It can be seen from the above embodiments that this specification collects real online business requests received by the business system based on the language model during its online operation; constructs regression test cases for the business system based on the business requests; and performs regression testing on the business system through the regression test cases based on the received test requests for the business system.

[0017] In this method, the business requests received by the business system based on the language model can be collected during its online operation, and regression test cases of the business system can be constructed based on the business requests. The input and output of the business system in the regression test cases constructed in this way are determined based on the dynamic data during online operation. As user needs change and the business system is updated, the regression test cases constructed based on this method will also be updated accordingly, effectively ensuring the timeliness and diversity of the regression test cases, and further improving the accuracy and reliability of the test results. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a flowchart of a regression testing method for a business system based on a language model provided by an exemplary embodiment;

[0019] Figure 2 This is a schematic diagram of a process for constructing a regression test case provided by an exemplary embodiment;

[0020] Figure 3 This is a schematic diagram of a regression testing process provided by an exemplary embodiment;

[0021] Figure 4 is a structural diagram of a device provided by an exemplary embodiment;

[0022] Figure 5 This is a block diagram of a regression testing method for a business system based on a language model provided by an exemplary embodiment. DETAILED DESCRIPTION

[0023] To make the objectives, technical solutions, and advantages of this specification more clear, the following will clearly and completely describe the technical solutions of this specification in conjunction with the specific embodiments of this specification and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this specification.

[0024] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and corresponding operation entrances are provided for users to choose to authorize or refuse.

[0025] Language model-based business systems are intelligent software or tools built on large-scale pre-trained language models, capable of understanding and generating complex language. Their core goal is to leverage the powerful computing power and vast knowledge of large models to empower applications with natural interaction, dynamic decision-making, and cross-domain knowledge integration.

[0026] The multi-layered structure of large models, their dependent sub-models, Retrieval-Augmented Generation (RAG), and the numerous application programming interfaces (APIs) that comprise them, results in a complex architecture with numerous change points. Factors such as the quality of model training data, the allocation of computing resources, and the stability of the dependent systems can all lead to anomalies. These issues typically require online monitoring or user feedback to identify them.

[0027] Most current business systems offer multimodal input capabilities, such as voice, and their language generalization and rewriting capabilities are gradually improving, allowing for greater diversity in user requests. Furthermore, with the introduction of RAG capabilities, most business systems can respond to time-sensitive information. However, traditional regression testing typically involves preparing fixed data in advance, then requesting the business system and verifying the results based on the response. For large-scale application systems, it's difficult to prepare for diverse, spoken user input and highly real-time requests. This results in traditional regression testing methods being unable to automatically update these more time-sensitive requests in real time, resulting in weak coverage.

[0028] In addition, the business interfaces of large-scale business systems are usually relatively simple, and the response information is mostly text-based. In addition to the output port, other deterministic verification points cannot be built, making it difficult to adapt to complex application architectures. It also cannot cover the backup capability verification function after complex link anomalies.

[0029] Based on this, this specification provides a regression testing method for business systems. By collecting the data required to construct regression test cases during the actual reasoning process, it realizes the dynamic construction and flexible configuration of regression test cases, effectively improves the flexibility and timeliness of regression testing, realizes efficient and accurate automated regression testing, reduces the occurrence of potential anomalies, and improves the reliability and stability of the business system.

[0030] The technical solutions provided by the embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0031] Figure 1 This is a flowchart of a regression testing method for a business system based on a language model provided by an exemplary embodiment, including:

[0032] S100: During the online operation of a language model-based business system, real online business requests received by the business system are collected.

[0033] In this specification, the execution entity for implementing a regression testing method for a business system can be a server on a business platform, or alternatively, a terminal device such as a laptop or desktop computer, or a client device installed on such a terminal device. For ease of description, the following description of a regression testing method for a business system provided in this specification will only use the server as the execution entity.

[0034] A business system based on a language model (such as a Large Language Model (LLM)) can include multiple network modules, each of which corresponds to a sub-model with a specific function or a network node in the language model. Furthermore, the business system can support multiple external functions, which can be implemented by invoking the functional components corresponding to these external functions during business execution.

[0035] During the online operation of a business system, users can input business requests into the business system, and the business system can output corresponding response information based on the business request. The business request can be a query request for information, including a query statement input by the user. If the business system supports the RAG function, the response information can be the response language generated by the large language model based on the RAG search results.

[0036] In the process of the business system generating response information based on business requests, the business system will generate a large amount of intermediate data. These intermediate data may include: data output by at least some network modules in the business system (such as key sub-models, key engineering nodes, etc.), such as multiple rounds of rewriting results (that is, the results obtained by optimizing the original query through multiple rounds of prompts), domain results (such as domain classification obtained by calling sub-models in vertical fields such as medical care, finance, and law), intention results (the results of intent classification of user queries, such as "query weather", "perform operation", "query location", etc.) and retrieval results (that is, the actual content retrieved from the information database based on the query), etc.

[0037] In addition, intermediate data can also include external call data of the business system calling external functional components (such as RAG, APIs, etc.), such as the call results of RAG (queried knowledge base identifier, document metadata, etc.) and the call results of APIs.

[0038] It should be noted that the intermediate data may only include one of the data output by at least part of the network module and the data called to other program components. Of course, it may also include both.

[0039] In the above process, the server can collect the real online business requests received by the business system, the response information output by the business system based on the business requests, and the intermediate data generated by the business system in the process of outputting the response information, and store this information in the database.

[0040] Among them, since the reply information returned by the language model-based business system to the user end is returned in multiple asynchronous streaming forms, that is, only one segment of streaming data is returned each time, the server can monitor all the streaming data output by the business system for the reply information (except for the last output data stream that carries an end marker), summarize each segment of streaming data and write it into a log file, and store the log file.

[0041] In actual applications, the server can capture the asynchronous streaming data returned by the business system through the preset middleware, and insert log points in its service export layer (such as Nginx or API Gateway) to record the streaming data of business requests and response information.

[0042] Since the amount of intermediate data is large, every time the server detects that the business system generates intermediate data, it immediately stores it in the database, thereby achieving real-time return of the intermediate data.

[0043] In practical applications, the acquisition of intermediate data can be achieved by modifying the business system. For example, the output of the network layer can be captured by registering hook functions and other tools with the business system (language model).

[0044] S102: Constructing a regression test case for the business system based on the business request.

[0045] After collecting the business requests, reply information and intermediate data, the server can manually screen and / or model the business requests, reply information and intermediate data to obtain processed business requests, reply information and intermediate data, and then construct regression test cases based on the processed business requests, reply information and intermediate data.

[0046] For example, the server can filter out abnormal data (badcase) in business requests, reply information and intermediate data through manual screening, automatic labeling of large models (such as marking badcase), or a combination of the two. For example, it can filter out reply information that does not match or has a low degree of match with the business request and the intermediate data generated in the process of generating the reply information, incorrect reply information fed back by users during the online operation of the business system and the intermediate data generated in the process of generating the reply information, and retain data with higher accuracy as benchmark data for regression test cases.

[0047] In this specification, the regression test case may include test input data (i.e., Request) and test benchmark data (i.e., Checker). The server can construct test input data for the business system based on preset input parameters and the query text (i.e., query) carried in the collected business request.

[0048] The above input parameters may include: Location Based Services (LBS), user information, etc. In addition, default fields may be reserved in the test input data, and these default fields may be automatically supplemented by the business system during the actual regression test.

[0049] Additionally, the server can construct test benchmark data for the business system based on the response information and intermediate data. This test benchmark data is used to verify test results and can include processed and filtered correct response information and intermediate data, such as feature fields in business requests, intent classifications in intermediate data, rewrite results, RAG call information, API call information, and more.

[0050] Regression test cases can also include other data, such as use case category data (i.e., type). This type of data is used to categorize regression test cases, making it easier to identify different use case sets during continuous integration. In practical applications, the categories used to categorize regression test cases can be the categories of the business fields corresponding to the regression test cases, such as healthcare, finance, education, and scientific research.

[0051] It should be added that since the server will collect business requests, response information and intermediate data multiple times, the server can build multiple regression test cases based on the collected data. After each regression test case is built, it can be added to the preset use case queue in sequence.

[0052] This makes it possible to generate regression test cases based on test input data and test benchmark data.

[0053] Of course, the above test benchmark data can also be constructed based on the reply information only. In this case, there is no need to obtain the intermediate data generated in the process of generating the reply information.

[0054] For ease of understanding, this manual provides a schematic diagram of the process of building a regression test case, as shown in the following example: Figure 2 shown.

[0055] Figure 2 The present invention is a schematic diagram of a process for constructing a regression test case provided by an exemplary embodiment.

[0056] After receiving a business request, the business system can determine whether the request contains an inspection tag. If not, it indicates that the request is an actual business request entered by the user. During the actual operation of the business system, the server can collect and store online data such as query text, intermediate data, and response information. This data is then manually screened or processed using a large model in the database, and a set of regression test cases is constructed based on this processed data and other manually entered data.

[0057] S104: Perform regression testing on the business system using the regression test case according to the received test request for the business system. After receiving the test request for the business system, the server may perform regression testing on the business system using the regression test case according to the test request.

[0058] In actual applications, the server can read the regression test cases in sequence based on the test request and in the order in which the regression test cases are arranged in the use case queue to perform regression testing on the business system.

[0059] In addition, when the regression test case contains use case category data, the server can sequentially read regression test cases that match the category information carried in the test request from the use case queue to perform regression testing on the business system.

[0060] While the business system is running online, the server can perform regression testing on the business system using regression test cases. In this case, the server can determine whether the business request currently received by the business system contains a specific inspection mark field. If so, the current business request is a test request, and the regression test case can be invoked to test the business system. If the current business request does not contain the inspection mark field, it indicates that the business request is an actual business request entered by the user, so regression testing is not performed. The business system can then generate a response based on the actual business request and return it to the user.

[0061] Of course, in this specification, the server can also set up an offline business system, which is the same as the business system running online. In this way, while the online business system executes actual business requests, the offline business system can be regression tested through regression test cases.

[0062] In the above process, the server can call the regression test case based on the regression test request, and supplement the default fields in the test input data, such as adding experimental parameters, timestamps, etc., to obtain the test input data of the business system.

[0063] Specifically, the server can input the test input data into the business system so that the business system outputs reply information based on the test input data as the reply information to be tested, and obtains the intermediate data generated by the business system in the process of generating the reply information to be tested as the intermediate data to be tested, and then determines the test comparison data of the business system based on the reply information to be tested and the intermediate data to be tested.

[0064] The server can then perform regression testing on the business system based on the deviation between the test comparison data and the test benchmark data.

[0065] In this specification, the business system can be regression tested by one or more of rule verification, large model verification, and manual verification.

[0066] During the large model verification process, the server can input the combination of test comparison data and test benchmark data into multiple comparison models respectively, obtain the comparison results between the test comparison data output by each comparison model and the test benchmark data, and then determine the regression test results for the business system from the comparison results output by multiple comparison models.

[0067] For example, the server can use the majority rule to determine if the majority of comparison models output the same regression test result. Alternatively, the server can aggregate the regression test results output by each comparison model to obtain the target test result.

[0068] After the server aggregates the streaming data, it can call the callback interface of the automation platform through an asynchronous thread to return the response information to be tested and the intermediate data to be tested, and store them in the database for use in subsequent asynchronous result verification.

[0069] After getting the test results, the server can generate a test report based on the test results. When the test results are abnormal, the test report can contain abnormal location information, which is used to indicate which module or link of the business system has the abnormality. The server can then return the test report to the user via email, IM, etc. For ease of understanding, this manual provides a schematic diagram of the regression test process, such as Figure 3 shown.

[0070] Figure 3 This is a schematic diagram of a regression testing process provided by an exemplary embodiment.

[0071] After receiving a service request, the business system can determine whether the request carries a check mark. If so, it indicates that the request is a pre-built test request. The server can then call the regression test case in the test case queue based on the test request to test the business system and re-stream the test data for storage. The verification results can then be determined through rule verification, model verification, manual verification, and other methods. A test report based on the verification results is then generated and returned to the user.

[0072] As can be seen from the above content, this solution ensures that large models can continue to run stably and reliably after multiple iterations and updates by efficiently building a set of regression use cases, intelligent result verification, and a testing process that adapts to the large model application architecture.

[0073] The business system based on the language model collects the business requests it receives during its online operation, and constructs regression test cases for the business system based on the business requests. The input and output of the business system in the regression test cases constructed in this way are determined based on the dynamic data during online operation. As user needs change and the business system is updated, the regression test cases constructed based on this method will also be updated accordingly, effectively ensuring the timeliness and diversity of the regression test cases and improving the accuracy and reliability of the test results.

[0074] In addition, the data used to construct regression test cases not only includes the response information output by the business system, but also includes the intermediate data generated in the process of generating the response information. The regression test cases constructed based on these intermediate data can accurately test and locate the behavior or function of a specific module in the large model application. It is suitable for complex business system structures and further ensures the reliability of the test results.

[0075] Figure 4 This is a schematic structural diagram of a device provided by an exemplary embodiment. Figure 4 At the hardware level, the device includes a processor 402, an internal bus 404, a network interface 406, a memory 408, and a non-volatile memory 410. Of course, it may also include hardware required for other functions. One or more embodiments of this specification can be implemented based on software, such as the processor 402 reading the corresponding computer program from the non-volatile memory 410 into the memory 408 and then running it. Of course, in addition to software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0076] Please refer to Figure 5,The regression testing device of business system based on language model can be applied to, Figure 4 The device shown in the figure is used to implement the technical solution of this specification. The regression testing device of the business system based on the language model may include:

[0077] The collection module 500 is used to collect real online business requests received by the business system based on the language model during its online operation;

[0078] A construction module 502 is used to construct a regression test case for the business system based on the business request;

[0079] The testing module 504 is configured to perform regression testing on the business system using the regression test case according to the received test request for the business system.

[0080] Optionally, before constructing a regression test case for the business system based on the business request, the collection module 500 is further configured to collect response information output by the business system in response to the business request, as well as intermediate data generated by the business system in the process of outputting the response information; wherein the intermediate data includes data output by at least part of the network module in the business system and / or external call data;

[0081] The construction module 502 is specifically configured to construct the regression test case based on the service request, the response information, and the intermediate data.

[0082] Optionally, the collection module 500 is specifically used to, after monitoring the business system outputting all the streaming data of the reply information, summarize each segment of the streaming data and write it into a log file, and store the log file; and, when monitoring the business system generating any intermediate data in the process of outputting the reply information, store the intermediate data.

[0083] Optionally, the construction module 502 is specifically used to manually screen and / or model process the business request, the reply information and the intermediate data to obtain processed business requests, reply information and intermediate data; and construct the regression test case based on the processed business requests, reply information and intermediate data.

[0084] Optionally, the testing module 504 is specifically configured to perform regression testing on the business system using the regression test case according to a received test request for the business system during the online operation of the business system.

[0085] Optionally, the test module 504 is further configured to determine, during the online operation of the business system, whether the currently received business request contains a patrol flag field; if so, determine that the currently received business request is the test request.

[0086] Optionally, the construction module 502 is specifically configured to construct test input data for the business system based on preset input parameters and the query text carried in the business request, and to construct test benchmark data for the business system based on the response information and the intermediate data;

[0087] The regression test case is generated according to the test input data and the test benchmark data.

[0088] Optionally, the test module 504 is specifically used to input the test input data into the business system so that the business system outputs reply information based on the test input data as the reply information to be tested, and to obtain the intermediate data generated by the business system in the process of generating the reply information to be tested as the intermediate data to be tested; determine the test comparison data of the business system based on the reply information to be tested and the intermediate data to be tested; and perform regression testing on the business system based on the deviation between the test comparison data and the test benchmark data.

[0089] Optionally, the test module 504 is specifically used to input the combination of the test comparison data and the test benchmark data into multiple comparison models respectively, to obtain the comparison results between the test comparison data and the test benchmark data output by each comparison model; and to determine the regression test results for the business system from the comparison results output by the multiple comparison models.

[0090] Optionally, the construction module 502 is specifically configured to construct a plurality of regression test cases for the business system based on the business request, and add each regression test case to a preset case queue after the construction is completed;

[0091] The testing module 504 is specifically configured to read the regression test cases in sequence according to the test request and in the order in which the regression test cases are arranged in the use case queue to perform regression testing on the business system.

[0092] Based on the same concept as the above method, this specification also provides an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein the processor implements the steps of the method described in any of the above embodiments by running the executable instructions.

[0093] Based on the same concept as the above method, this specification also provides a computer-readable storage medium on which computer instructions are stored. When the instructions are executed by a processor, the steps of the method described in any of the above embodiments are implemented.

[0094] Based on the same concept as the above method, this specification also provides a computer program product, including a computer program / instruction, which implements the steps of the method described in any of the above embodiments when executed by a processor.

Claims

1. A regression testing method for a business system based on a language model, comprising: During the online operation of the language model-based business system, real online business requests received by the business system are collected; Based on the business request, construct regression test cases for the business system; According to the received test request for the business system, regression testing is performed on the business system using the regression test case.

2. The method according to claim 1, before constructing a regression test case for the business system based on the business request, the method further comprises: Collecting reply information output by the business system according to the business request, as well as intermediate data generated by the business system in the process of outputting the reply information; wherein the intermediate data includes data output by at least part of the network module in the business system and / or external call data; Based on the business request, a regression test case is constructed for the business system, specifically including: The regression test case is constructed based on the business request, the response information and the intermediate data.

3. The method according to claim 2, wherein collecting the response information output by the business system according to the business request and the intermediate data generated by the business system in the process of outputting the response information specifically comprises: After monitoring the business system to output all the streaming data of the reply information, each segment of the streaming data is summarized and written into a log file, and the log file is stored. In addition, when it is monitored that the business system generates any intermediate data in the process of outputting the reply information, the intermediate data is stored.

4. The method according to claim 2, constructing the regression test case based on the business request, the response information, and the intermediate data, specifically comprising: Manually screening and / or model processing the business request, the response information, and the intermediate data to obtain processed business requests, response information, and intermediate data; The regression test case is constructed according to the processed business request, response information and intermediate data.

5. The method according to claim 1, wherein, based on the received test request for the business system, regression testing is performed on the business system using the regression test case, specifically comprising: During the online operation of the business system, regression testing is performed on the business system using the regression test case according to the received test request for the business system.

6. The method according to claim 5, before performing regression testing on the business system using the regression test case, the method further comprises: During the online operation of the business system, determining whether the currently received business request contains an inspection mark field; If so, it is determined that the currently received service request is the test request.

7. The method according to claim 2, constructing a regression test case for the business system based on the business request, the response information, and the intermediate data, specifically comprising: Constructing test input data for the business system based on preset input parameters and the query text carried in the business request, and constructing test benchmark data for the business system based on the response information and the intermediate data; The regression test case is generated according to the test input data and the test benchmark data.

8. The method according to claim 7, wherein the regression test case is used to perform regression testing on the business system, specifically comprising: Inputting the test input data into the business system so that the business system outputs response information according to the test input data as the response information to be tested, and obtaining intermediate data generated by the business system in the process of generating the response information to be tested as the intermediate data to be tested; Determining test comparison data of the business system according to the response information to be tested and the intermediate data to be tested; Perform regression testing on the business system based on the deviation between the test comparison data and the test benchmark data.

9. The method according to claim 8, wherein the regression test is performed on the business system according to the deviation between the test comparison data and the test benchmark data, specifically comprising: Inputting the combination of the test comparison data and the test reference data into a plurality of comparison models respectively, and obtaining a comparison result between the test comparison data and the test reference data output by each comparison model; A regression test result for the business system is determined from the comparison results output by the multiple comparison models.

10. The method according to claim 1, constructing a regression test case for the business system based on the business request, specifically comprising: Based on the business request, construct multiple regression test cases for the business system, and after each regression test case is constructed, add it to a preset test case queue; Performing regression testing on the business system through the regression test case specifically includes: According to the test request, the regression test cases are read in sequence according to the order in which the regression test cases are arranged in the use case queue to perform regression testing on the business system.

11. A regression testing device for a business system based on a language model, comprising: A collection module, configured to collect real online business requests received by a language model-based business system during its online operation; A construction module, configured to construct a regression test case for the business system based on the business request; The testing module is used to perform regression testing on the business system through the regression test case according to the received test request for the business system.

12. An electronic device comprising: processor; A memory for storing processor-executable instructions; wherein the processor implements the steps of the method according to any one of claims 1 to 10 by executing the executable instructions.

13. A computer-readable storage medium having computer instructions stored thereon, wherein when the instructions are executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

14. A computer program product comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 10.

Images