Test review method and device

Abstract

The invention provides a test review method which can be applied to the technical field of artificial intelligence. The method comprises the following steps: aiming at a target business demand, based on a large language model, determining a target test case for realizing the target business demand in N to-be-screened test cases, and determining a target execution log generated by executing the target test case in M to-be-screened execution logs; the N and the M are positive integers; and for the target business demand, the target test case and the target execution log, determining a test review result based on a large language model. The invention further provides a test review device and equipment, a storage medium and a program product.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to a testing and evaluation method and apparatus. Background Technology

[0002] Currently, testing of test cases often requires manual test review. This includes evaluating test results to determine if the test cases cover the test requirements and whether all possible scenarios are covered during test execution. However, manual test review is inefficient. Summary of the Invention

[0003] In view of the above problems, this application provides a test review method, apparatus, equipment, media and program product to improve the efficiency of test review.

[0004] According to a first aspect of this application, a test review method is provided, comprising: for a target business requirement, based on a large language model, determining a target test case for implementing the target business requirement from N test cases to be screened, and determining a target execution log generated by executing the target test case from M execution logs to be screened; wherein N and M are positive integers;

[0005] Based on the target business requirements, the target test cases, and the target execution logs, the test review results are determined using a large language model.

[0006] Optionally, the above method further includes: clustering the obtained business requirements, test cases, and execution logs to obtain clustering results; the N test cases to be screened include test cases in the clustering results that belong to the same cluster as the target business requirement; the M execution logs to be screened include execution logs in the clustering results that belong to the same cluster as the target business requirement.

[0007] Optionally, the above method further includes: updating a preset knowledge graph based on the target business requirement, the target test case, and the target execution log; the preset knowledge graph is used to represent at least the relationship between any two of the following: business requirement, test case, and execution log; the step of determining the test review result based on a large language model for the target business requirement, the target test case, and the target execution log includes: determining the test review result based on a large language model according to the updated preset knowledge graph for the target business requirement, the target test case, and the target execution log.

[0008] Optionally, the updated preset knowledge graph includes at least one of the following: a target requirement node for representing the target business requirement; a target test case node for representing the target test case; a target log node for representing the target execution log; an edge between the target requirement node and the target test case node for representing that the target test case is used to implement the target business requirement; and an edge between the target test case node and the target log node for representing that the target execution log is generated by executing the target test case.

[0009] Optionally, determining the test review result based on a large language model for the target business requirement, the target test case, and the target execution log includes: determining the test review result based on a large language model for the target business requirement, the target test case, the target execution log, and business-related personnel information; the business-related personnel information includes at least one of the following: personnel information related to the target business requirement, personnel information related to the target test case, and personnel information related to the target execution log.

[0010] Optionally, the above method further includes: updating a preset knowledge graph based on the target business requirement, the target test case, the target execution log, and business-related personnel information; the preset knowledge graph is used to represent at least the relationship between any two of the following: business requirement, test case, execution log, and business-related personnel; determining the test review result based on a large language model for the target business requirement, the target test case, the target execution log, and business-related personnel information, including: determining the test review result based on a large language model according to the updated preset knowledge graph for the target business requirement, the target test case, the target execution log, and business-related personnel information.

[0011] Optionally, the preset knowledge graph includes at least one of the following: a requirement node for representing business requirements; a test case node for representing test cases; a log node for representing execution logs; a personnel node for representing business-related personnel; an object node for representing test objects; a defect node for representing defect types; an edge between a requirement node and a test case node for representing that the test case represented by the test case node is used to implement the business requirement represented by the requirement node; an edge between a test case node and a log node for representing that executing the test case represented by the test case node generates the execution log represented by the log node; an edge between a test case node and an object node for representing that the test case represented by the test case node is used to test the test object represented by the object node; an edge between a log node and a defect node for representing that the execution log represented by the log node contains the defect type represented by the defect node; and an edge between a personnel node and any other node for representing that the business-related personnel represented by the personnel node are related to the content represented by the other node.

[0012] Optionally, determining the test review result based on the updated preset knowledge graph and the large language model includes: determining the test review result based on the updated preset knowledge graph and the current historical update status of the preset knowledge graph and the large language model.

[0013] Optionally, the step of determining the test review result based on the updated preset knowledge graph and the large language model includes: determining the test review result based on the updated nodes in the updated preset knowledge graph and other nodes and / or other edges associated with the updated nodes.

[0014] Optionally, the test review results may include at least one of the following: coverage of the target test cases, implementation of the target business requirements, test results of the target test cases, test risk information, supplementary suggestions for test cases, test defect information, and root cause analysis results of test defects.

[0015] A second aspect of this application provides a test evaluation apparatus, comprising:

[0016] The filtering unit is used to determine, based on a large language model, target test cases for testing the target business requirements from N test cases to be filtered, and target execution logs generated by executing the target test cases from M execution logs to be filtered; where N and M are positive integers.

[0017] The review unit is used to determine the test review results based on the target business requirements, the target test cases, and the target execution logs, using a large language model.

[0018] Optionally, the filtering unit is further configured to: cluster the acquired business requirements, test cases, and execution logs to obtain clustering results; the N test cases to be filtered include test cases in the clustering results that belong to the same cluster as the target business requirement; the M execution logs to be filtered include execution logs in the clustering results that belong to the same cluster as the target business requirement.

[0019] Optionally, the review unit is further configured to: update a preset knowledge graph based on the target business requirement, the target test case, and the target execution log; the preset knowledge graph is used to represent at least the relationship between any two of the following: business requirement, test case, and execution log; the review unit is configured to: determine the test review result based on a large language model, according to the updated preset knowledge graph, for the target business requirement, the target test case, and the target execution log.

[0020] Optionally, the updated preset knowledge graph includes at least one of the following: a target requirement node for representing the target business requirement; a target test case node for representing the target test case; a target log node for representing the target execution log; an edge between the target requirement node and the target test case node for representing that the target test case is used to implement the target business requirement; and an edge between the target test case node and the target log node for representing that the target execution log is generated by executing the target test case.

[0021] Optionally, the review unit is used to: determine the test review result based on a large language model for the target business requirement, the target test case, the target execution log, and business-related personnel information; the business-related personnel information includes at least one of the following: personnel information related to the target business requirement, personnel information related to the target test case, and personnel information related to the target execution log.

[0022] Optionally, the review unit is further configured to: update a preset knowledge graph based on the target business requirement, the target test case, the target execution log, and business-related personnel information; the preset knowledge graph is used to represent at least the relationship between any two of the following: business requirement, test case, execution log, and business-related personnel; the review unit is configured to: determine the test review result based on a large language model, according to the updated preset knowledge graph, for the target business requirement, the target test case, the target execution log, and the business-related personnel information.

[0023] Optionally, the preset knowledge graph includes at least one of the following: a requirement node for representing business requirements; a test case node for representing test cases; a log node for representing execution logs; a personnel node for representing business-related personnel; an object node for representing test objects; a defect node for representing defect types; an edge between a requirement node and a test case node for representing that the test case represented by the test case node is used to implement the business requirement represented by the requirement node; an edge between a test case node and a log node for representing that executing the test case represented by the test case node generates the execution log represented by the log node; an edge between a test case node and an object node for representing that the test case represented by the test case node is used to test the test object represented by the object node; an edge between a log node and a defect node for representing that the execution log represented by the log node contains the defect type represented by the defect node; and an edge between a personnel node and any other node for representing that the business-related personnel represented by the personnel node are related to the content represented by the other node.

[0024] Optionally, the review unit is used to: determine the test review result based on the updated preset knowledge graph and the current historical update status of the preset knowledge graph, using a large language model.

[0025] Optionally, the review unit is used to: determine the test review result based on the updated nodes in the updated preset knowledge graph, as well as other nodes and / or other edges associated with the updated nodes, based on the large language model.

[0026] Optionally, the test review results may include at least one of the following: coverage of the target test cases, implementation of the target business requirements, test results of the target test cases, test risk information, supplementary suggestions for test cases, test defect information, and root cause analysis results of test defects.

[0027] A third aspect of this application provides an electronic device comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0028] A fourth aspect of this application also provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0029] The fifth aspect of this application also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method. Attached Figure Description

[0030] The above-mentioned contents, other objects, features and advantages of this application will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0031] Figure 1 This illustration schematically depicts an application scenario of a test evaluation method and apparatus according to an embodiment of this application.

[0032] Figure 2 A flowchart illustrating a test review method according to an embodiment of this application is shown schematically;

[0033] Figure 3 This schematically illustrates a structural block diagram of a test evaluation apparatus according to an embodiment of this application; and

[0034] Figure 4 A block diagram schematically illustrates an electronic device suitable for implementing a test review method according to an embodiment of this application. Detailed Implementation

[0035] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0036] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0037] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0038] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0039] Currently, testing of test cases often requires manual test review. For example, this review assesses the test results to determine whether the test cases fulfill business requirements and whether all possible scenarios are covered during test execution.

[0040] However, manual testing and review is less efficient.

[0041] Embodiments of this application provide a test review method. In this method, a large language model can be used to parse business requirements, test cases, and logs, automatically performing test reviews. Specifically, automatic test reviews can be performed based on a large language model, using prompt words combined with information from business requirements, test cases, and the logs themselves.

[0042] For example, the prompt could be, "Please analyze the coverage of the test results for the following business requirements, test cases, and logs." This allows for automated test review based on a large language model.

[0043] This method takes into account the relationships between business requirements, test cases, and logs, thereby enabling the discovery of these relationships. This facilitates automated test review using large language models, improving the accuracy and efficiency of test review.

[0044] Business requirements, such as a "user login function," can be reflected in test cases that include input data from multiple different users and the corresponding theoretical login results. For example, data entered by a normal user would correspond to a "login successful" result, while incorrect data would correspond to a "login failed" result. Executing test cases generates corresponding log data, which can include the input data and the actual login result, allowing for analysis and determination of the test results. This leads to test reviews, such as using test results to determine whether the business requirement has been fulfilled.

[0045] Specifically, determining the relationship can involve identifying the test cases corresponding to the business requirements, i.e., the test cases used to implement the business requirements; it can also involve identifying the logs corresponding to the test cases, i.e., the logs generated when the test cases are executed; or it can involve identifying the logs related to the business requirements, i.e., the logs related to the business requirements, which can be used to determine whether the business requirements have been implemented, etc. For example, the logs obtained when the test cases used to implement the business requirements are executed can be used to determine the execution results of the test cases, thereby determining whether the test requirements have been implemented.

[0046] Accordingly, when determining specific relationships, a large language model can be used. Specifically, for any target business requirement, the large language model can be used to determine the target test cases used to implement the business requirement from multiple test cases, and further, the large language model can be used to determine the target logs obtained from executing the target test cases from the logs.

[0047] The above methods enable automated test reviews through large language models, improving review efficiency and reducing manual intervention. Furthermore, large language models can be used to determine the relationships between business requirements, test cases, and logs, facilitating automated test reviews and further enhancing their efficiency and accuracy.

[0048] It should be noted that the methods and apparatus disclosed in the embodiments of this application can be used in the field of artificial intelligence technology, and also in the field of fintech. For example, for the test review of financial software, the above methods can be used for automated test review; it can also be applied to any field other than fintech, such as for the test review of video software or communication software. The application fields of the methods and apparatus disclosed in the embodiments of this application are not limited.

[0049] In the technical solution of this application, the user information (including but not limited to user personal information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, and displayed data) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with relevant laws, regulations, and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entry points for users to choose to authorize or refuse.

[0050] In scenarios involving automated decision-making using personal information, the methods, devices, and systems provided in this application all offer users corresponding entry points for choosing to agree to or reject the automated decision-making results. If the user chooses to reject, the process proceeds to the expert decision-making stage. Here, "automated decision-making" refers to the activity of automatically analyzing and evaluating an individual's behavioral habits, interests, or economic, health, and credit status through computer programs, and then making a decision. Here, "expert decision-making" refers to the activity of making decisions by personnel who specialize in a particular field, possess specialized experience, knowledge, and skills, and have reached a certain level of professional expertise.

[0051] Figure 1 The illustration shows an application scenario of a test evaluation method and apparatus according to an embodiment of this application.

[0052] like Figure 1As shown, application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 can be used as a medium to provide a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0053] Users can use the first terminal device 101, the second terminal device 102, or the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, or the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0054] The first terminal device 101, the second terminal device 102, or the third terminal device 103 can be various electronic devices with a display screen and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0055] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, or the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0056] It should be noted that the test review method provided in this application embodiment can generally be executed by server 105, or by first terminal device 101, second terminal device 102, or third terminal device 103.

[0057] For example, automatic test review can be performed in server 105 using a large language model, or it can be performed in the first terminal device 101, the second terminal device 102, or the third terminal device 103 using a large language model. Alternatively, the first terminal device 101, the second terminal device 102, or the third terminal device 103 can interact with server 105 and use the large language model in server 105 to perform automatic review, and the review results can be fed back to the terminal devices.

[0058] Accordingly, the test evaluation apparatus provided in this application embodiment can generally be located in server 105, or it can be located in the first terminal device 101, the second terminal device 102, or the third terminal device 103. The test evaluation method provided in this application embodiment can also be executed by a server or server cluster that is different from server 105 but can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Accordingly, the test evaluation apparatus provided in this application embodiment can also be located in a server or server cluster that is different from server 105 but can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.

[0059] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0060] Figure 2 A flowchart illustrating a test review method according to an embodiment of this application is shown schematically.

[0061] like Figure 2 As shown, the method flow of this embodiment may include operations S210 to S220. This method flow is not limited to a single executing entity; it can be any electronic device or any software, such as a user terminal, a server, a server device, etc.

[0062] Optionally, the executing entity may have a large language model deployed locally. For example, a server with a large language model can execute this method flow for automatic test review. The executing entity may also be able to call or interact with the large language model. For example, a user can interact with the large language model through a software application installed on their terminal, send test-related data to the large language model for automatic test review, and obtain the test review results fed back by the large language model.

[0063] This method flow may include operation S210 and operation S220.

[0064] In operation S210, based on the large language model, target test cases are determined from N test cases to be screened to implement the target business requirements, and target execution logs generated by executing the target test cases are determined from M execution logs to be screened; N and M are positive integers.

[0065] In operation S220, test review results are determined based on a large language model, targeting business requirements, test cases, and execution logs.

[0066] This methodology can automate test reviews using large language models, improving the efficiency of test reviews and reducing manual intervention.

[0067] This methodology can also identify the relationships between business requirements, test cases, and execution logs through a large language model, enabling automated test review using the large language model and improving the efficiency and accuracy of test review.

[0068] The embodiments of this application do not limit the specific structure and form of the large language model. Optionally, the large language model can be an intelligent agent, a multimodal large language model, etc. Specifically, it can be based on intelligent agents for automatic test review, or it can be based on intelligent agents to determine the correlation between business requirements, test cases, and execution logs.

[0069] The embodiments of this application do not limit whether the large language models in S210 and S220 are the same. Optionally, the structures of the large language models in S210 and S220 can be the same or different. The same large language model can be used in S210 and S220, or different large language models can be used.

[0070] The embodiments of this application are not limited to the target business requirement. The target business requirement can be any business requirement. For ease of description, any business requirement addressed will be referred to as the target business requirement. Specific business requirements include, for example, user login function requirements, payment function requirements, or recommendation function requirements, etc.

[0071] This methodology is designed for automated test review of a single target business requirement, along with related test cases and execution logs. It can be understood that this methodology can also be used for automated test review of multiple different business requirements. Specifically, it can be executed in parallel for each of these different business requirements, and the large language model can also determine the associated test cases and execution logs for automated test review based on multiple input business requirements.

[0072] The embodiments of this application do not limit the source of the target business requirements. Optionally, the target business requirements may be extracted from a requirements document or determined based on user input. The embodiments of this application also do not limit the specific form of the target business requirements; they may be in document form, specifically a requirements document; they may be in text form; or they may be in multimodal form, including multimodal content such as images, voice, and text. Accordingly, the target business requirements can be parsed using a large language model or a multimodal large language model.

[0073] The embodiments of this application do not limit the source of the test cases and execution logs to be filtered. Optionally, test cases can be extracted from test case documents provided by testers, determined based on user input, or retrieved from logs during test case execution, or determined based on user input. The embodiments of this application also do not limit the specific format of the test cases and execution logs. Specifically, they can be in document form or text form. Accordingly, a large language model can be used to parse the test cases and execution logs to be filtered.

[0074] In a specific example, since software development and testing are often handled by different business teams, and the actual testing process typically requires collaboration among multiple personnel, business requirements, test cases, and execution logs are often scattered across multiple devices. Different personnel also have different testing content and habits, resulting in the collected business requirements, test cases, and execution logs often being out of order and making it difficult to intuitively determine their relationships. Therefore, a large language model can be used to parse business requirements, test cases, and execution logs, automatically determining their relationships, thereby enabling automated test review.

[0075] The following explanations address S210 and S220 respectively.

[0076] 1. In operation S210, based on the large language model and the target business requirements, target test cases are determined from N test cases to be screened to implement the target business requirements, and target execution logs generated by executing the target test cases are determined from M execution logs to be screened; N and M are positive integers.

[0077] The embodiments of this application do not limit the specific way of calling the large language model. Optionally, the executing entity may call the local large language model to perform operation S210; or the executing entity may call the large language model deployed on other devices to determine the relationship between business requirements, test cases and execution logs, and obtain the output results of the large language model to determine the target test cases and target execution logs associated with the target business requirements.

[0078] In a specific example, the executing entity might transmit the target business requirements, N test cases to be screened, and M execution logs to be screened via a large language model application or an intelligent agent application to the corresponding server. The server then identifies the target test cases and target execution logs and returns them to the executing entity. Specifically, this could involve returning the identifiers of the target test cases and the target execution logs.

[0079] The embodiments of this application do not limit the input content of the large language model. Optionally, the target business requirements, N test cases to be filtered, and M execution logs to be filtered can be input into the large language model; alternatively, the target business requirements, N test cases to be filtered, M execution logs to be filtered, and corresponding prompt words can be input into the large language model to determine the target test cases and target execution logs.

[0080] In a specific example, the target business requirement, N test cases to be filtered, and M execution logs to be filtered can be combined with prompts and input into the large language model. Specifically, this could be: "The business requirement is" + target business requirement (which can be in document format) + "The test cases to be filtered are" + N test cases to be filtered (which can be in text or document format) + "The execution logs to be filtered are" + M execution logs to be filtered (which can be in text or document format) + "Please filter out the test cases used to implement the business requirement, and further filter out the execution logs of the filtered test cases," input into the large language model. The large language model can then parse and match the target business requirement, N test cases to be filtered, and M execution logs to be filtered based on the prompts to determine the relationships between them.

[0081] The embodiments of this application do not limit the relationship between business requirements, test cases, and execution logs. Optionally, a single business requirement may correspond to one or more test cases; a single test case may correspond to one or more execution logs. For a single business requirement, multiple test cases can be designed to cover various scenarios. For a single test case, it can be executed multiple times or different branch scenarios can be executed, generating corresponding multiple execution logs. Therefore, correspondingly, one or more target test cases can be determined for a single target business requirement, and one or more target execution logs can be determined for a single target test case. Specifically, based on a large language model, one or more target test cases for implementing the target business requirement can be determined from N test cases to be screened, and one or more target execution logs generated by executing the target test cases can be determined from M execution logs to be screened. The embodiments of this application do not limit the specific number of target business requirements, target test cases, and target execution logs.

[0082] Alternatively, business requirements and execution logs can also be correlated, and the execution logs used to implement business requirements can be determined. Specifically, the target execution log used to implement the target business requirement can be determined from M execution logs to be filtered.

[0083] It should be noted that if zero target test cases or zero target execution logs are identified, S220 can still be executed for automated test review. The corresponding test review result will be, for example, "No relevant test cases found" or "No relevant execution logs found."

[0084] The embodiments of this application do not limit the source of the test cases and execution logs to be filtered. Optionally, N test cases and M execution logs to be filtered can be obtained directly, specifically from the development and testing process or from user input.

[0085] In one optional embodiment, the collected business requirements, test cases, and execution logs can be filtered first to determine the target business requirements, N test cases to be filtered, and M execution logs to be filtered, before executing S210. This embodiment can reduce the amount of input data for the large language model and improve the efficiency of the large language model in determining relationships through the filtering operation.

[0086] Understandably, in a specific example, the collected business requirements, test cases, and execution logs can be directly input into the large language model. The large language model can then use any one of these business requirements as a target business requirement to determine the associated target test cases and target execution logs. Alternatively, the collected business requirements, test cases, and execution logs can be filtered to identify potentially related target business requirements, N test cases, and M execution logs (N and M are merely illustrative and not limited to a specific number in practice), and then input into the large language model. This reduces the amount of input data for the large language model and improves its efficiency in determining relationships.

[0087] The embodiments of this application do not limit the specific screening method. Optionally, business requirements, test cases, and execution logs can be classified according to their attributes to identify potentially related business requirements, test cases, and execution logs; alternatively, they can be screened based on a large language model to identify potentially related business requirements, test cases, and execution logs; or they can be clustered based on the collected business requirements, test cases, and execution logs, considering those belonging to the same cluster as potentially related; or they can be parsed to identify related information such as business identifiers to determine potentially related business requirements, test cases, and execution logs; or they can determine the similarity between any two of the business requirements, test cases, and execution logs to identify potentially related business requirements, test cases, and execution logs based on similarity.

[0088] It is understood that the above screening methods can be combined with each other, and the embodiments of this application do not limit the specific combination method. Optionally, after classifying according to the attributes of business requirements, test cases, and execution logs, further screening can be performed by clustering for business requirements, test cases, and execution logs in one category of the classification results.

[0089] The following explains the clustering filtering method in detail. Optionally, clustering can be performed on the obtained business requirements, test cases, and execution logs to obtain clustering results; among them, N test cases to be filtered can include test cases belonging to the same cluster as the target business requirements in the clustering results; M execution logs to be filtered can include execution logs belonging to the same cluster as the target business requirements in the clustering results.

[0090] In this embodiment, clustering can be used to identify test cases and execution logs that may be related to the target business requirements, thereby reducing the amount of input data for the large language model and improving the efficiency and accuracy of the large language model in determining the relationship.

[0091] The embodiments of this application do not limit the method and source of obtaining business requirements, test cases and execution logs. Specifically, they can be obtained from the development and testing process or from user input.

[0092] The embodiments of this application do not limit the specific clustering method. Optionally, features can be extracted from business requirements, test cases, and execution logs. The extracted features can be in the same feature space, so clustering can be performed on the extracted features to determine the clustering results. It can be assumed that business requirements, test cases, and execution logs belonging to the same cluster in the clustering results may have a correlation.

[0093] The embodiments of this application do not limit the specific method for determining the test cases and execution logs to be screened. Optionally, the test cases and execution logs to be screened can be determined directly based on the clustering results, or the results of multiple screening methods can be combined to determine the test cases and execution logs to be screened.

[0094] Optionally, the N test cases to be screened may include: N test cases belonging to the same cluster as the target business requirement in the clustering results; the M execution logs to be screened may include: M execution logs belonging to the same cluster as the target business requirement in the clustering results. Thus, the N test cases and M execution logs to be screened can be directly determined based on the clustering results. Furthermore, optionally, in addition to the test cases belonging to the same cluster as the target business requirement in the clustering results, the N test cases to be screened may also include other test cases, for example, test cases determined through other screening methods; in addition to the execution logs belonging to the same cluster as the target business requirement in the clustering results, the M execution logs to be screened may also include other execution logs.

[0095] The following explains the filtering method for classification. Optionally, considering that business requirements, test cases, and execution logs can all have attributes, they can be classified according to attributes to identify potentially related business requirements, test cases, and execution logs. Attributes include, for example, time attributes, business attributes, organizational attributes, etc.

[0096] For example, business requirements can have a development time point attribute, test cases can have a design time point attribute, and execution logs can have a generation time point attribute. According to the testing process, test cases are generally designed after business requirements are developed, and execution logs are generally generated when test cases are executed after design. Therefore, for potentially related business requirements, test cases, and execution logs, the general order from front to back is: business requirement development time point, test case design time point, and execution log generation time point. The relationships between time attributes can be used for filtering.

[0097] For example, business requirements, test cases, and execution logs generally correspond to business category attributes, specifically broad business categories such as user business, consumption business, video business, etc. Each broad business category can be further subdivided into multiple specific business requirements; for instance, the video business category could include video playback requirements, video recommendation requirements, video cover display requirements, etc. Business requirements, test cases, and execution logs that may have related relationships generally belong to the same broad business category, thus allowing for filtering based on business category attributes.

[0098] For example, business requirements, test cases, and execution logs generally correspond to organizational attributes, that is, the organization to which they belong. For instance, they might belong to multiple subsidiaries within the same group, or multiple business teams within the same company. Business requirements, test cases, and execution logs that may have related relationships usually belong to the same organization, thus allowing for filtering based on organizational attributes.

[0099] The following explains the similarity-based filtering method in detail. Optionally, for business requirements, test cases, and execution logs, if there is a relationship, there may be a lot of similar content in the process or description. For example, for a business requirement representing a user login function, the business requirement itself may contain the keyword "user login," the test case may contain multiple "user login" keywords, and the execution log may contain multiple pieces of information representing user login, such as "login." Thus, business requirements, test cases, and execution logs that may have a relationship can be identified based on similarity. This embodiment does not limit the specific method of calculating similarity. Optionally, text similarity calculation can be used, or the similarity between any two of the business requirements, test cases, and execution logs can be determined through a large language model.

[0100] It is understood that the above-described examples of screening methods are merely illustrative and can be combined with each other. The embodiments of this application do not limit the specific screening methods or combinations thereof. For example, clustering can be performed based on similarity, or clustering can be performed after classification based on attributes, etc.

[0101] By using the above filtering method, the target business requirements, as well as the test cases and execution logs to be filtered, can be identified. These can then be input into the large language model to determine the relationships, reducing the amount of input data for the large language model and improving the efficiency of test review.

[0102] Second, in operating S220, the test review results are determined based on the large language model, targeting the target business requirements, target test cases, and target execution logs.

[0103] The embodiments of this application do not limit the specific way of calling the large language model. Optionally, the executing entity may call the local large language model to perform operation S220; alternatively, the executing entity may call the large language model deployed in other devices to perform automatic test review and obtain the output results of the large language model.

[0104] In a specific example, the executing entity may transmit the target business requirements, target test cases, and target execution logs to the corresponding server through a large language model application or an intelligent agent application. The server will then conduct automatic test reviews and return the determined test review results to the executing entity.

[0105] The embodiments of this application do not limit the input content of the large language model. Optionally, the target business requirements, target test cases, and target execution logs can be input into the large language model; alternatively, the target business requirements, target test cases, target execution logs, and corresponding prompt words can be input into the large language model for automatic test review.

[0106] In a specific example, the target business requirement, target test cases, and target execution logs can be combined with prompts and input into the large language model. Specifically, this could be: "The business requirement requiring review is" + target business requirement (which can be in document format) + "The test cases requiring review are" + target test cases (which can be in text or document format, and can be one or more) + "The execution logs requiring review are" + target execution logs (which can be in text or document format, and can be one or more) + specific test review requirements (e.g., determining whether the test cases implement the business requirement, the coverage of the business requirement, etc.). The large language model can then automatically perform test reviews based on the prompts for the target business requirement, target test cases, and target execution logs. Specifically, it can parse the semantics of the target business requirement, target test cases, and target execution logs and perform automatic test reviews.

[0107] The embodiments of this application do not limit the input content when the large language model performs automated test review. Optionally, in addition to the target business requirements, target test cases, and target execution logs, other input content may be included to enable the large language model to perform test review more comprehensively and accurately.

[0108] Optionally, other input content may include information about business-related personnel, specifically information about personnel who propose business requirements, design test cases, execute test cases, or modify the test object (software or code). This personnel information may include personnel identifiers for easy identification, or other personnel information such as historical development information for test review.

[0109] Optionally, other input content can also include the object tested by the test cases, such as code or software. For ease of description, the object tested by the test cases is called the test object, and the object tested by the target test cases is called the target test object. This allows for the integration of information about the target test object, such as code semantics, code modification details, code structure, and error messages, into automated test reviews of large language models.

[0110] Optionally, other inputs can also include information on risk types or defect types that occur during the testing process, such as test error messages. This facilitates automated test review using large language models.

[0111] In one optional embodiment, the test review result is determined based on a large language model, taking into account the target business requirements, target test cases, and target execution logs. Specifically, this can be achieved by determining the test review result based on a large language model, taking into account the target business requirements, target test cases, target execution logs, and relevant personnel information. The relevant personnel information may include at least one of the following: personnel information related to the target business requirements, personnel information related to the target test cases, and personnel information related to the target execution logs.

[0112] This embodiment can improve the comprehensiveness and accuracy of information for automatic testing and review of large language models by introducing information of business-related personnel.

[0113] The embodiments of this application do not limit the scope to business-related personnel. Optionally, personnel related to the target business requirements may include those who develop the target business requirements, those who modify the target business requirements, those who review the target business requirements, etc. Personnel related to the target test cases may include those who design the target test cases, those who modify the target test cases, those who review the target test cases, etc. Personnel related to the target execution log may include those who execute the target test cases. In addition, business-related personnel may also include personnel related to the test object, such as those who develop the target test object, those who modify the target test object, etc.

[0114] The embodiments of this application do not limit the specific content of the business-related personnel information. Optionally, the business-related personnel information may include the identifier of the business-related personnel to facilitate subsequent location, or it may include the characteristics of the business-related personnel, such as development experience, business expertise, etc., to facilitate more comprehensive testing and review of the large language model.

[0115] Optionally, the input content for testing and reviewing the aforementioned large language model may also include the relationships between the input content. For example, in addition to the relationships between target business requirements, target test cases, and target execution logs, it may also include the relationships between business-related personnel and other input content, etc., thus allowing for a more comprehensive test review by combining these relationships with the input to the large language model. The aforementioned information on business-related personnel may include relevant business requirements, test cases, execution logs, or test objects, etc.

[0116] The embodiments of this application do not limit the form of the input content to the large language model. Optionally, text representing the relationship can be directly input into the large language model, or it can be input into the large language model in the form of a graph structure.

[0117] In one optional embodiment, a knowledge graph can be used to integrate information such as business requirements, test cases, and execution logs, providing it to a large language model for automated test review, thereby improving the comprehensiveness and accuracy of the test review. Specifically, the knowledge graph can be in the form of a graph structure.

[0118] Optionally, a pre-defined knowledge graph can be updated based on the determined target business requirements, target test cases, and target execution logs. The pre-defined knowledge graph can at least be used to represent the relationship between any two of the following: business requirements, test cases, and execution logs. For the target business requirements, target test cases, and target execution logs, the test review results are determined based on a large language model. Specifically, for the target business requirements, target test cases, and target execution logs, the test review results are determined based on the updated pre-defined knowledge graph and a large language model.

[0119] In this embodiment, a knowledge graph can be used to represent the relationship between any two of the following: business requirements, test cases, and execution logs. This allows for the automatic test review of target business requirements, target test cases, and target execution logs by combining information from a pre-defined knowledge graph. This increases the amount of input information for the test review and improves the comprehensiveness and accuracy of the test review.

[0120] The embodiments of this application do not limit the information represented by the preset knowledge graph. Optionally, the preset knowledge graph can be used to represent the relationship between business requirements, test cases and execution logs, or the relationship between business requirements and test cases, or the relationship between test cases and execution logs, etc.

[0121] The embodiments of this application do not limit the structure of the preset knowledge graph. Optionally, the preset knowledge graph may include: requirement nodes for representing business requirements; use case nodes for representing test cases; and log nodes for representing execution logs. Edges between nodes can be used to represent relationships. Specifically, the edge between a requirement node and a use case node can be used to represent that the test case represented by the use case node is used to implement the business requirement represented by the requirement node. The edge between a use case node and a log node can be used to represent that executing the test case represented by the use case node generates the execution log represented by the log node.

[0122] The embodiments of this application do not limit the specific content of the preset knowledge graph. Optionally, the preset knowledge graph may include accumulated business requirements, test cases, and execution logs, as well as accumulated relationships. Specifically, the preset knowledge graph can be updated by adding new business requirements, test cases, and execution logs, and their corresponding relationships, and / or by updating business requirements, test cases, and execution logs, and their corresponding relationships, in order to accumulate test information and improve the comprehensiveness of test information.

[0123] In a specific example, for a single business requirement, a test can be performed first, storing the corresponding business requirement, test cases, execution logs, and related relationships in a pre-defined knowledge graph. Subsequently, as the business requirement is updated, another test is performed, updating the pre-defined knowledge graph with the corresponding updated business requirements, test cases, execution logs, and related relationships. New business requirements, test cases, execution logs, and related relationships can also be updated and stored in the pre-defined knowledge graph. The pre-defined knowledge graph can contain a large amount of test information, which can be provided to a large language model, improving the comprehensiveness and accuracy of test reviews.

[0124] The embodiments of this application are not limited to the updating method for the preset knowledge graph. Optionally, updating the preset knowledge graph may include at least one of the following: adding, deleting, and modifying nodes in the preset knowledge graph, adding, deleting, and modifying edges in the preset knowledge graph, etc. Specifically, it may update the information represented by the nodes, or update the information represented by the edges, etc.

[0125] Optionally, the preset knowledge graph can be updated based on the target business requirements, target test cases, and target execution logs with existing relationships determined by the large language model in S210. This embodiment does not limit the specific update method. Optionally, the updated preset knowledge graph may include at least one of the following: target requirement nodes for representing target business requirements; target test case nodes for representing target test cases; target log nodes for representing target execution logs; edges between target requirement nodes and target test case nodes for representing that target test cases are used to implement target business requirements; and edges between target test case nodes and target log nodes for representing that executing target test cases generates target execution logs. Accordingly, specifically updating the preset knowledge graph may involve updating its nodes and edges to include at least one of the above, which can be used to represent target business requirements, target test cases, and target execution logs, as well as the corresponding relationships.

[0126] This embodiment can update the preset knowledge graph with related target business requirements, target test cases and target execution logs, thereby increasing the comprehensiveness of the test information in the preset knowledge graph and facilitating the improvement of the comprehensiveness and accuracy of test review.

[0127] The specific number of target business requirements, target test cases, and target execution logs is not limited. Optionally, automated test reviews can be performed on the identified target business requirements, one or more identified target test cases, and one or more identified target execution logs. Alternatively, a pre-defined knowledge graph can be updated based on the identified target business requirements, one or more identified target test cases, and one or more identified target execution logs. Each target business requirement corresponds to a single requirement node; different target test cases can correspond to different test case nodes, and different test case nodes can be used to represent different target test cases; different target execution logs can correspond to different log nodes, and different log nodes can be used to represent different target execution logs.

[0128] In one alternative embodiment, the knowledge graph can be updated by incorporating information from business stakeholders, and the test review can be conducted based on the updated knowledge graph.

[0129] Optionally, a pre-defined knowledge graph can be updated based on target business requirements, target test cases, target execution logs, and information on business-related personnel. The pre-defined knowledge graph is used to represent at least the relationship between any two of the following: business requirements, test cases, execution logs, and business-related personnel. For the target business requirements, target test cases, target execution logs, and information on business-related personnel, the test review result is determined based on a large language model. Specifically, this can be done by determining the test review result based on the updated pre-defined knowledge graph and the large language model, based on the target business requirements, target test cases, target execution logs, and information on business-related personnel.

[0130] In this embodiment, updating the preset knowledge graph based on information about business-related personnel can improve the comprehensiveness of the preset knowledge graph, thereby facilitating the improvement of the comprehensiveness and accuracy of large language model testing and review.

[0131] For explanations from business-related personnel, please refer to other embodiments.

[0132] Understandably, the pre-defined knowledge graph can represent the relationships between business-related personnel and other content, such as the relationship between business-related personnel and at least one of the following: business requirements, test cases, and execution logs.

[0133] In one optional embodiment, the preset knowledge graph can store various types of information, such as business requirements, test cases, execution logs, defect types, personnel information, etc. The relationships between these information can also be stored in the preset knowledge graph. Regarding the information in the preset knowledge graph, the embodiments of this application do not limit the specific structure of the preset knowledge graph.

[0134] Optionally, the preset knowledge graph may include at least one of the following: requirement nodes for representing business requirements; test case nodes for representing test cases; log nodes for representing execution logs; personnel nodes for representing business-related personnel; object nodes for representing test objects; defect nodes for representing defect types; edges between requirement nodes and test case nodes, indicating that the test cases represented by the test case node are used to implement the business requirements represented by the requirement node; edges between test case nodes and log nodes, indicating that executing the test cases represented by the test case node generates the execution log represented by the log node; edges between test case nodes and object nodes, indicating that the test cases represented by the test case node are used to test the test objects represented by the object node; edges between log nodes and defect nodes, indicating that the execution log represented by the log node contains the defect type represented by the defect node; and edges between personnel nodes and any other node, indicating that the business-related personnel represented by the personnel node are related to the content represented by the other node.

[0135] This embodiment can improve the comprehensiveness of information in the preset knowledge graph by representing information content through nodes and representing the relationships between information content through edges between nodes, which facilitates the improvement of the comprehensiveness and accuracy of testing and reviewing large language models.

[0136] It is understood that the preset knowledge graph may also contain other information and other relationships, and the above embodiments are only used for illustrative purposes.

[0137] The embodiments of this application do not limit the specific method by which the large language model performs test reviews based on a preset knowledge graph. Optionally, the preset knowledge graph can be input into the large language model for test reviews targeting target business requirements, target test cases, and target execution logs. Specifically, the updated preset knowledge graph can be input into the large language model. Optionally, the large language model can also use the preset knowledge graph as reference data or retrieval data, and retrieve relevant information from the preset knowledge graph based on target business requirements, target test cases, and target execution logs for test reviews.

[0138] Optionally, the preset knowledge graph can be updated, and the different versions of the preset knowledge graph before and after the update can also contain test information, which can then be used as a basis for test review through a large language model. For example, as the preset knowledge graph is updated, the versions before and after the update can be used to represent changes in nodes or edges, allowing for test review based on these changes. Therefore, optionally, the test review result can be determined based on the updated preset knowledge graph and the current historical update status of the preset knowledge graph, using the large language model.

[0139] This embodiment can improve the comprehensiveness of the information in the preset knowledge graph by introducing the historical update information of the preset knowledge graph for automatic review, which facilitates the improvement of the comprehensiveness and accuracy of the large language model test review.

[0140] This embodiment does not limit the specific content of the historical update information of the preset knowledge graph. Optionally, it may include the full historical update information of the preset knowledge graph, or it may include the historical update information of the preset knowledge graph within a preset time period. Specifically, it may include different historical versions of the preset knowledge graph, or it may include update information of the preset knowledge graph. For example, for a basic version of the preset knowledge graph, the recorded node update information and edge update information.

[0141] Specifically, the updated preset knowledge graph, along with its current historical update status, can be input into the large language model for testing and evaluation.

[0142] Optionally, information in the preset knowledge graph can be filtered, and test reviews can be conducted based on the filtered content, which can improve the efficiency of test reviews for large language models. The specific filtering method is not limited. Optionally, it can be to filter out information in the preset knowledge graph that is related to the target business requirements, target test cases, or target execution logs, or it can be to filter out information in the preset knowledge graph that has been updated.

[0143] Optionally, the test review results can be determined based on the updated preset knowledge graph and the large language model. Specifically, the test review results can be determined based on the updated nodes in the updated preset knowledge graph and other nodes and / or other edges associated with the updated nodes.

[0144] This embodiment can preset the information selected in the knowledge graph for testing and review, which can improve the efficiency of testing and reviewing large language models.

[0145] The updated nodes can be related to target business requirements, target test cases, and target execution logs. This allows information related to target business requirements, target test cases, and target execution logs to be selected from a pre-defined knowledge graph for test review, thereby improving the comprehensiveness and accuracy of the test review.

[0146] The embodiments of this application do not limit the form and content of the test review results. Optionally, the test review results may include at least one of the following: coverage of the target test cases, implementation of the target business requirements, test results of the target test cases, risk information of the tests, supplementary suggestions for the test cases, defect information of the tests, and root cause analysis results of the test defects.

[0147] This embodiment can improve the comprehensiveness of test review by explicitly defining the content of the test review results. Specifically, it can use prompt words to call a large language model to output the content of the aforementioned test review results.

[0148] It is understood that the test review results may also include other information, and the above embodiments are merely illustrative. Optionally, the test review results may be presented to users in the form of a report.

[0149] Optionally, test review results can be obtained by analyzing a large language model combined with information from a pre-defined knowledge graph, such as the relationships between nodes. For example, for defect information in tests, personnel information in the pre-defined knowledge graph can be used to locate and identify relevant personnel, thus facilitating subsequent defect correction. Supplementary suggestions for test cases can also be provided by using historical updates in the pre-defined knowledge graph to determine if similar business requirements and test cases exist. Regarding test case coverage, information from all execution log nodes associated with the test case in historical updates of the pre-defined knowledge graph can be used to determine the coverage of the test case. Regarding the implementation status of business requirements, all test cases associated with the business requirement in historical updates of the pre-defined knowledge graph, along with the full execution logs, can be used to determine whether the business requirement has been implemented. Alternatively, for test defects, the root cause of the defect can be determined by combining the node changes in the preset knowledge graph of historical updates. For example, for the test defect "execution error", the historical version of the test object and the historical version of the test case can be determined in the preset knowledge graph to determine whether the error occurred during a certain version change, thereby locating the relevant personnel.

[0150] For ease of understanding, this application also provides an application embodiment.

[0151] Test review can specifically involve analyzing test cases, execution results, and log data. This analysis, combined with information such as tester information, application modification information, and test cycle, assesses test quality and identifies potential risks, including incomplete test content and test delays. Traditional test reviews rely on manually writing test cases and assertions, which is relatively inefficient.

[0152] This application example provides an automated test review method based on a large language model. By inputting tester and test case information during the test review phase, the method analyzes test quality by associating it with application modification points, development sites, patch delivery status, test start time, and test result checks, and provides alerts indicating potential risks.

[0153] In this application example, the system can identify the categories of input data (business requirements, test cases, logs, etc.) and analyze the relationships between different categories of input data (through contextual analysis, feature extraction, and association combination). Then, it can automatically perform test reviews based on a large language model, using an attention mechanism to determine the relationships between different categories. Combining the relationships between "requirements-use cases-persons-logs (test results)" and historical defects, it determines review metrics and results, generates a review report, and provides recommended solutions.

[0154] In this application embodiment, each document can also be split and vectorized to determine the vectors of various requirements, use cases, logs, personnel information, etc., and then clustered. For any cluster in the clustering results, it is determined that the documents corresponding to the vectors are related. The relationships between the contents of these documents are analyzed through a large language model to construct the relationship of "requirement-use case-log-personnel", which can be specifically formed into a knowledge graph.

[0155] Based on the large language model, automated test reviews can be performed on the constructed relationships to determine whether business requirements are met, whether all branches of test cases are covered, etc., generating corresponding reports. The aforementioned relationship graph can also be output to facilitate the identification of relevant personnel. Furthermore, it can be combined with historical defects to determine the reasons for test failures. The generated knowledge graph can be stored and updated based on new test information, such as new business requirements and new test cases. The updated knowledge graph results, along with historical update information, enable the large language model to perform more accurate automated test reviews.

[0156] In this application example, after document clustering, the relationship between any two of "requirements, test cases, logs, and personnel" can be determined using a large language model for documents within the cluster. Based on the determined relationships, the knowledge graph can be updated for automated test review, enabling the identification of relevant personnel and analysis of failure causes, etc.

[0157] The technical solution of this application embodiment mainly includes three steps.

[0158] Step 1: Use the requirements document, test cases, execution logs, and information on developers and testers as raw, multi-type input data. This data exists in various formats, including unstructured, structured, and semi-structured data. Specifically, before reviewing the test results, enter the necessary data files for the review, including modified requirements documents, test cases and related test patches, test assignments, test cycles, application developers, and execution logs.

[0159] Specifically, you can input raw document data (text, images, PDFs, scanned documents, and other unstructured data), and collect data from relevant systems, databases, API interfaces, and other channels based on the target data.

[0160] A hierarchical classification mechanism can be used to identify document types and content. First, design a primary classification (business-level classification) and secondary classifications (business requirement documents, test cases, transaction logs, and developer information). Then, perform multiple rounds of structured data extraction. First, extract the requirement documents and test cases, and then extract the corresponding key transaction fields (transaction name, transaction process, upstream and downstream application information, etc.) from the content based on the requirement and test case content.

[0161] By using cross-modal unified encoding, multi-source data such as text, images, and tables are mapped to a unified vector space. This facilitates the quantization and association of heterogeneous data (such as images and text) that were originally not directly comparable during subsequent processing through vector operations. A requirement use case vector library is generated.

[0162] Step 2: Cross-modal relationship mining. Establish mapping relationships between requirement content, test case content, test log content, testers, and developers.

[0163] The current test log content is parsed using a multimodal large model, and developers are mapped accordingly. The first step is to parse the logs, extract the transaction code and service name from the test logs, and extract the code developers recorded in the code repository. This transforms the relationship between transactions, testers, and developers into a vector space, which is then stored to form a transaction developer vector library.

[0164] Based on data such as code commit history (number of lines of code changes, number of defect fixes), test execution statistics (pass rate of test cases, defect detection rate), and log errors (transaction processing anomalies, frequency of anomaly triggers), a vector structure for development quality, test quality, and test case risk level is determined through a multimodal large model and stored as a quality level vector library.

[0165] The vector dimension design for development quality can include [number of lines of code changes, number of defect fixes, module coupling coefficient, and recent activity]. The vector dimension design for test quality can include [test case pass rate, defect detection rate, regression efficiency, and false positive suppression rate]. The vector dimension design for test case risk can include [requirement criticality, module dependency depth, historical defect rate, and number of log errors].

[0166] The system inputs standardized vectors and uses vector analysis and similarity calculations to establish relationships between entities, determining core node and edge information. Node information primarily includes requirement nodes, test case nodes, test log nodes, personnel nodes, transaction code nodes, and defect / issue nodes. Edge information mainly includes structural association edges (e.g., requirement-test case association, test case-code association), process traceability edges (e.g., tester-test log, developer-code program), and quality impact edges (e.g., defect-test log). This information is stored in a graph database. The output graph includes explicit links in the requirement-test case-person relationship (e.g., the chain relationship of requirement ID → test case ID → log executor ID) and implicit links in test quality (e.g., the number of development defects / issues, defect detection rate, etc.).

[0167] Real-time correlation updates: When new logs are added or requirements change, incremental updates to the correlation graph are triggered through an event-driven rule engine.

[0168] Step 3: Output test quality assessment results. The review report generation module includes two aspects: (1) Basic test information. Generate test case coverage, test result risk rating, and supplementary test scenario suggestions based on graph information. (2) Test problem report. Analyze the reasons for the lack of test case coverage (different situations such as test omissions or development program problems). Based on causal reasoning algorithms (such as Bayesian networks), locate the root cause chain of test failure, such as requirement change → related test cases not updated → execution failure → the person in charge is tester X, log L1 is associated with developer P → P last modified the relevant code.

[0169] This application example utilizes multimodal output fusion technology to improve quality review efficiency and risk localization accuracy, generating remediation plans with specific test cases and significantly enhancing the effectiveness of test recommendations. It can also leverage causal reasoning based on event chains, historical defect pattern matching and time-series analysis, and the fusion of cross-modal features from text, code, and time-series logs. Utilizing context window expansion technology of a large model, it achieves long-term event chain analysis, enabling root cause localization of test anomalies. Furthermore, it can utilize a dual-link structure of explicit business flow and implicit quality flow within a knowledge graph, facilitating the identification of relevant test personnel and the assessment of test risks. Through cross-role association vectorization, the relationships between developers, testers, and transaction code can be mapped to relationships within a unified vector space.

[0170] Based on the above method embodiments, this application also provides a test evaluation device embodiment. Figure 3 A schematic block diagram of a test evaluation apparatus according to an embodiment of this application is shown. Figure 3 As shown, the test review apparatus 300 of this embodiment includes a screening unit 310 and a review unit 320. For further explanation of this apparatus embodiment, please refer to the explanations of other embodiments.

[0171] The filtering unit 310 is used to determine, based on a large language model, target test cases for testing the target business requirements from N test cases to be filtered, and target execution logs generated by executing the target test cases from M execution logs to be filtered; N and M are positive integers. In one embodiment, the filtering unit 310 can be used to perform the operation S210 described above, which will not be repeated here.

[0172] The review unit 320 is used to determine the test review results based on the target business requirements, target test cases, and target execution logs, using a large language model. In one embodiment, the review unit 320 can be used to perform the operation S220 described above, which will not be repeated here.

[0173] Optionally, the filtering unit 310 is also used to: cluster the obtained business requirements, test cases and execution logs to obtain clustering results; N test cases to be filtered include test cases in the clustering results that belong to the same cluster as the target business requirements; M execution logs to be filtered include execution logs in the clustering results that belong to the same cluster as the target business requirements.

[0174] Optionally, the review unit 320 is also used to: update the preset knowledge graph based on the target business requirements, target test cases, and target execution logs; the preset knowledge graph is used to represent at least the relationship between any two of the following: business requirements, test cases, and execution logs; the review unit 320 is used to: determine the test review results based on the updated preset knowledge graph and the large language model for the target business requirements, target test cases, and target execution logs.

[0175] Optionally, the updated preset knowledge graph includes at least one of the following: a target requirement node for representing the target business requirement; a target test case node for representing the target test case; a target log node for representing the target execution log; an edge between the target requirement node and the target test case node for representing the target test case for implementing the target business requirement; and an edge between the target test case node and the target log node for representing the execution of the target test case to generate the target execution log.

[0176] Optionally, the review unit 320 is used to: determine the test review results based on a large language model, taking into account the target business requirements, target test cases, target execution logs, and information on business-related personnel; the information on business-related personnel includes at least one of the following: information on personnel related to the target business requirements, information on personnel related to the target test cases, and information on personnel related to the target execution logs.

[0177] Optionally, the review unit 320 is also used to: update the preset knowledge graph based on the target business requirements, target test cases, target execution logs, and business-related personnel information; the preset knowledge graph is used to represent at least the relationship between any two of the following: business requirements, test cases, execution logs, and business-related personnel; the review unit 320 is used to: determine the test review results based on the updated preset knowledge graph and the large language model, for the target business requirements, target test cases, target execution logs, and business-related personnel information.

[0178] Optionally, the preset knowledge graph includes at least one of the following: a requirement node for representing business requirements; a test case node for representing test cases; a log node for representing execution logs; a personnel node for representing business-related personnel; an object node for representing test objects; a defect node for representing defect types; an edge between a requirement node and a test case node for representing that the test case represented by the test case node is used to implement the business requirement represented by the requirement node; an edge between a test case node and a log node for representing that executing the test case represented by the test case node generates the execution log represented by the log node; an edge between a test case node and an object node for representing that the test case represented by the test case node is used to test the test object represented by the object node; an edge between a log node and a defect node for representing that the execution log represented by the log node contains the defect type represented by the defect node; and an edge between a personnel node and any other node for representing that the business-related personnel represented by the personnel node are related to the content represented by the other node.

[0179] Optionally, the review unit 320 is used to: determine the test review results based on the updated preset knowledge graph and the current historical update status of the preset knowledge graph, using a large language model.

[0180] Optionally, the review unit 320 is used to: determine the test review result based on the large language model according to the updated nodes in the updated preset knowledge graph, as well as other nodes and / or other edges associated with the updated nodes.

[0181] Optionally, the test review results may include at least one of the following: coverage of target test cases, implementation of target business requirements, test results of target test cases, risk information of tests, supplementary suggestions for test cases, defect information of tests, and root cause analysis results of test defects.

[0182] According to embodiments of this application, any plurality of modules in the screening unit 310 and the review unit 320 can be combined into one module, or any one of the modules can be split into multiple modules. Alternatively, at least a portion of the functionality of one or more of these modules can be combined with at least a portion of the functionality of other modules and implemented in one module. According to embodiments of this application, at least one of the screening unit 310 and the review unit 320 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any appropriate combination of any of these three implementation methods. Alternatively, at least one of the screening unit 310 and the review unit 320 can be at least partially implemented as a computer program module, which, when run, can perform corresponding functions.

[0183] Figure 4 A block diagram schematically illustrates an electronic device suitable for implementing a test review method according to an embodiment of this application.

[0184] like Figure 4 As shown, an electronic device 900 according to an embodiment of this application includes a processor 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage portion 908 into a random access memory (RAM) 903. The processor 901 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 901 may also include onboard memory for caching purposes. The processor 901 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this application.

[0185] RAM 903 stores various programs and data required for the operation of electronic device 900. Processor 901, ROM 902, and RAM 903 are interconnected via bus 904. Processor 901 executes various operations of the method flow according to embodiments of this application by executing programs in ROM 902 and / or RAM 903. It should be noted that the programs may also be stored in one or more memories other than ROM 902 and RAM 903. Processor 901 may also execute various operations of the method flow according to embodiments of this application by executing programs stored in said one or more memories.

[0186] According to embodiments of this application, the electronic device 900 may further include an input / output (I / O) interface 905, which is also connected to a bus 904. The electronic device 900 may also include one or more of the following components connected to the input / output (I / O) interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the input / output (I / O) interface 905 as needed. A removable medium 911, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 910 as needed so that computer programs read from it can be installed into the storage section 908 as needed.

[0187] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.

[0188] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this application, the computer-readable storage medium may include ROM 902 and / or RAM 903 and / or one or more memories other than ROM 902 and RAM 903 described above.

[0189] Embodiments of this application also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to cause the computer system to implement any of the method embodiments provided in the embodiments of this application.

[0190] When the computer program is executed by the processor 901, it performs the functions defined in the system / apparatus of this application embodiment. According to the embodiments of this application, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0191] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via the communication section 909, and / or installed from a removable medium 911. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0192] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 909, and / or installed from the removable medium 911. When the computer program is executed by the processor 901, it performs the functions defined in the system of this application embodiment. According to the embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0193] According to embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0194] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two connected blocks may actually be executed substantially in parallel, or they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0195] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application.

Claims

1. A test review method, characterized in that, The method includes: Based on the large language model, target test cases for implementing the target business requirements are determined from N test cases to be screened, and target execution logs generated by executing the target test cases are determined from M execution logs to be screened; where N and M are positive integers. Based on the target business requirements, the target test cases, and the target execution logs, the test review results are determined using a large language model.

2. The method according to claim 1, characterized in that, The method further includes: Cluster the obtained business requirements, test cases, and execution logs to obtain clustering results; The N test cases to be screened include: test cases in the clustering results that belong to the same cluster as the target business requirement; The M execution logs to be screened include: execution logs that belong to the same cluster as the target business requirement in the clustering results.

3. The method according to claim 1, characterized in that, The method further includes: Based on the target business requirements, the target test cases, and the target execution logs, update the preset knowledge graph; the preset knowledge graph is used to represent at least the relationship between any two of the following: business requirements, test cases, and execution logs; The process of determining test review results based on a large language model, taking into account the target business requirements, the target test cases, and the target execution logs, includes: Based on the target business requirements, the target test cases, and the target execution logs, the test review results are determined using the updated preset knowledge graph and a large language model.

4. The method according to claim 3, characterized in that, The updated preset knowledge graph includes at least one of the following: Target requirement node for characterizing the target business requirement; target test case node for characterizing the target test case; target log node for characterizing the target execution log; The edge between the target requirement node and the target use case node is used to represent that the target test case is used to implement the target business requirement; as well as The edge between the target test case node and the target log node is used to represent the generation of the target execution log by executing the target test case.

5. The method according to claim 1, characterized in that, The process of determining test review results based on a large language model, taking into account the target business requirements, the target test cases, and the target execution logs, includes: Based on the target business requirements, the target test cases, the target execution logs, and information on relevant personnel, the test review results are determined using a large language model. The business-related personnel information includes at least one of the following: personnel information related to the target business requirements, personnel information related to the target test cases, and personnel information related to the target execution logs.

6. The method according to claim 5, characterized in that, The method further includes: Based on the target business requirements, the target test cases, and the target execution logs, as well as information on business-related personnel, the preset knowledge graph is updated; the preset knowledge graph is used to represent at least the relationship between any two of the following: business requirements, test cases, execution logs, and business-related personnel. The process of determining test review results based on a large language model, taking into account the target business requirements, the target test cases, the target execution logs, and information on relevant personnel, includes: Based on the target business requirements, target test cases, target execution logs, and information on business-related personnel, the test review results are determined using a large language model and an updated preset knowledge graph.

7. The method according to claim 3 or 6, characterized in that, The preset knowledge graph includes at least one of the following: Requirement nodes are used to represent business requirements; test case nodes are used to represent test cases; log nodes are used to represent execution logs; personnel nodes are used to represent business-related personnel; object nodes are used to represent test objects; and defect nodes are used to represent defect types. The edge between the requirement node and the use case node is used to indicate that the test case represented by the use case node is used to implement the business requirement represented by the requirement node. The edge between the test case node and the log node is used to represent: the test case represented by the test case node generates the execution log represented by the log node. The edge between a test case node and an object node indicates that the test case represented by the test case node is used to test the test object represented by the object node. The edge between a log node and a defect node is used to indicate that the execution log represented by the log node contains the defect type represented by the defect node. An edge between a personnel node and any other node is used to indicate that the business-related personnel represented by the personnel node are related to the content represented by the other node.

8. The method according to claim 3 or 6, characterized in that, The step of determining the test review results based on the updated preset knowledge graph and the large language model includes: Based on the updated preset knowledge graph and the current historical update status of the preset knowledge graph, the test review results are determined based on the large language model.

9. The method according to claim 3 or 6, characterized in that, The step of determining the test review results based on the updated preset knowledge graph and the large language model includes: Based on the updated nodes in the updated preset knowledge graph, as well as other nodes and / or other edges associated with the updated nodes, the test review results are determined based on the large language model.

10. The method according to claim 1, characterized in that, The test review results include at least one of the following: The coverage of the target test cases, the implementation status of the target business requirements, the test results of the target test cases, the risk information of the tests, the supplementary suggestions for the test cases, the defect information of the tests, and the root cause analysis results of the defects of the tests.

11. A test and evaluation device, characterized in that, The device includes: The filtering unit is used to determine, based on a large language model, target test cases for testing the target business requirements from N test cases to be filtered, and target execution logs generated by executing the target test cases from M execution logs to be filtered; where N and M are positive integers. The review unit is used to determine the test review results based on the target business requirements, the target test cases, and the target execution logs, using a large language model.

12. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 10.

13. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 10.

14. A computer program product comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 10.

Images