Test method and device of intelligent dialogue product, medium, equipment and program product

Abstract

The invention discloses a test method and device of an intelligent dialogue product, a medium, equipment and a program product, relates to the technical field of large language models, large models, agents and artificial intelligence, and aims to improve the test coverage rate and test accuracy of the intelligent dialogue product. The test method of the intelligent dialogue product comprises the steps of determining process information of a to-be-tested workflow of the intelligent dialogue product, wherein the intelligent dialogue product is accessed to a first machine learning model; determining a page screenshot of a to-be-tested page in the intelligent dialogue product, and determining a first action based on the page screenshot, the first action being an action which can be executed by the intelligent dialogue product on the to-be-tested page; determining at least one second action through a second machine learning model according to the process information and the first action, the second action being an action which can be executed by the intelligent dialogue product in the to-be-tested workflow and the to-be-tested page; and determining a target action from the at least one second action, and controlling the intelligent dialogue product to execute the target action.

Description

Technical Field

[0001] The present disclosure relates to the fields of large language models, large models, intelligent agents and artificial intelligence technology, and in particular, to a testing method, device, medium, equipment and program product for an intelligent dialogue product. Background Art

[0002] With the rapid development of machine learning models, more and more software products are connected to machine learning models such as Large Language Models (LLMs). In the testing process of such software products, due to the uncertainty of the input and output of machine learning models, multiple workflow paths and branches may appear, making it difficult to ensure that the test is carried out according to the expected workflow, which in turn affects the coverage and accuracy of the test. Summary of the invention

[0003] This summary is provided to introduce concepts in a brief form that will be described in detail in the detailed description below. This summary is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0004] In a first aspect, the present disclosure provides a method for testing an intelligent dialogue product, the method comprising: Determining process information of a workflow to be tested of the intelligent dialogue product, wherein the intelligent dialogue product is connected to a first machine learning model; Determine a page screenshot of a page to be tested in the intelligent dialogue product, and determine a first action based on the page screenshot, where the first action is an action that the intelligent dialogue product can perform on the page to be tested; Determine, by a second machine learning model, at least one second action according to the process information and the first action, where the second action is an action that the intelligent dialogue product can perform in the workflow to be tested and the page to be tested; A target action is determined from the at least one second action, and the intelligent dialogue product is controlled to execute the target action.

[0005] In a second aspect, the present disclosure provides a testing device for an intelligent dialogue product, the testing device for the intelligent dialogue product comprising: A first determination module is used to determine process information of a workflow to be tested of the intelligent dialogue product, wherein the intelligent dialogue product is connected to a first machine learning model; A second determination module is used to determine a page screenshot of the page to be tested in the intelligent dialogue product, and determine a first action based on the page screenshot, where the first action is an action that the intelligent dialogue product can perform on the page to be tested; A third determination module is used to determine at least one second action according to the process information and the first action through a second machine learning model, where the second action is an action that the intelligent dialogue product can perform in the workflow to be tested and the page to be tested; The fourth determination module is used to determine a target action from the at least one second action and control the intelligent dialogue product to perform the target action.

[0006] In a third aspect, the present disclosure provides a computer-readable medium having a computer program stored thereon, which implements the steps of the method described in the first aspect when the program is executed by a processing device.

[0007] In a fourth aspect, the present disclosure provides an electronic device, including: a storage device having a computer program stored thereon; A processing device is used to execute the computer program in the storage device to implement the steps of the method in the first aspect.

[0008] In a fifth aspect, the present disclosure provides a computer program product, comprising a computer program, which implements the steps of the method described in the first aspect when executed by a processor.

[0009] Through the above technical solution, based on the page screenshot of the page to be tested in the intelligent dialogue product, the first action that the intelligent dialogue product can perform on the page to be tested can be determined, and then the first machine learning model can be used to determine at least one second action that the intelligent dialogue product can perform in the workflow to be tested and the page to be tested according to the first action and the process information of the workflow to be tested of the intelligent dialogue product, and finally the intelligent agent to be tested is controlled to perform the target action determined from the at least one second action. By adopting this method, the intelligent dialogue product can be controlled to automatically perform actions that can be performed in both the workflow to be tested and the page to be tested, which is equivalent to testing the intelligent dialogue product according to the workflow to be tested, thereby ensuring that the intelligent dialogue product connected to the machine learning model is automatically tested according to the expected workflow, thereby improving the test coverage and test accuracy.

[0010] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale. In the drawings: Figure 1is a flowchart of a testing method for an intelligent dialogue product according to an exemplary embodiment of the present disclosure; Figure 2 is a schematic diagram of a test page according to an exemplary embodiment of the present disclosure; Figure 3 is a process diagram of a testing method for an intelligent dialogue product according to an exemplary embodiment of the present disclosure; Figure 4 is a schematic diagram of a process of determining a first action according to an exemplary embodiment of the present disclosure; Figure 5 is a schematic diagram showing a prompt word and an example sample according to an exemplary embodiment of the present disclosure; Figure 6 is a schematic diagram of a processing process of a sandbox system according to an exemplary embodiment of the present disclosure; Figure 7 is a structural block diagram of a testing device for an intelligent dialogue product according to an exemplary embodiment of the present disclosure; Figure 8 It is a schematic structural diagram of an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0012] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0013] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0014] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0015] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0016] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0017] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0018] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0019] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0020] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0021] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0022] At the same time, it is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0023] With the rapid development of machine learning models, more and more software products are connected to machine learning models such as Large Language Models (LLMs). For example, intelligent dialogue products connected to large language models can help users interact in natural language, reduce the difficulty of using software products, and thus obtain more accurate, efficient and personalized services and support, and improve user experience.

[0024] For intelligent dialogue products that are connected to large language models, relevant technologies are generally tested using black-box testing and white-box testing. Among them, black-box testing is a testing method that does not require understanding the internal workflow of intelligent dialogue products. Testers test their functions by interacting with intelligent dialogue products. It is usually crowdsourcing, allowing a large number of different testers to participate in the testing. White-box testing is a testing method that requires testers to have an in-depth understanding of the internal workflow of intelligent dialogue products, and to verify the functions of intelligent dialogue products through preset test cases.

[0025] The above test method only supports static functional testing and lacks support for dynamic and complex testing processes such as large language models and other machine learning models. Therefore, it is impossible to fully verify the performance and user experience of intelligent dialogue products connected to large language models. This is mainly reflected in the following aspects: 1. Process uncertainty: The testing process of intelligent dialogue products is uncertain, and black-box testing and white-box testing are difficult to fully cover all test cases. Since intelligent dialogue products need to process various user inputs, there may be multiple dialogue paths and branches, which increases the difficulty of testing. In addition, user input may involve different language expressions, context changes, and the diversity of questions, all of which make the dialogue process complex and unpredictable.

[0026] 2. Performance of work branching processes: Some work branching processes built into intelligent dialogue products are difficult to intuitively reflect in product form, and it is difficult for testers to fully understand and grasp all possible workflow paths and branches. For example, dialogue branches may include multiple links such as different user question identification, context processing, and response generation, and each link may produce different dialogue branches and paths. Traditional testing methods are difficult to fully cover these complex dialogue branching processes, thus affecting the comprehensiveness and accuracy of the test.

[0027] 3. Limitations of testing methods: Due to the diversity and uncertainty of user input, it is difficult to cover all possible dialogue paths and branches using black box testing, resulting in low test case coverage. Especially when faced with complex dialogue scenarios, black box testing often cannot fully verify the performance and stability of intelligent dialogue products. In addition, since the dialogue is dynamic, historical chat records and context will affect the progress of the dialogue, and it is difficult to ensure that the test is carried out according to the expected workflow using white box testing. This not only increases the complexity of the test, but also requires testers to have certain professional knowledge and skills.

[0028] 4. Testing efficiency: Since intelligent dialogue products involve a large number of workflow paths and branches, black box testing requires a lot of manpower and time to test, while white box testing requires writing a large number of test scripts and test cases. The testing efficiency of both methods is not high.

[0029] In view of this, the present disclosure provides a testing method, apparatus, medium, equipment and program product for an intelligent dialogue product to improve the test coverage and test accuracy of an intelligent dialogue product connected to a machine learning model.

[0030] The embodiments of the present disclosure are further explained below with reference to the accompanying drawings.

[0031] Figure 1 is a flowchart of a method for testing an intelligent dialogue product according to an exemplary embodiment of the present disclosure, referring to Figure 1 , the test method may include the following steps: S101: Determine process information of a workflow to be tested of an intelligent dialogue product, where the intelligent dialogue product is connected to a first machine learning model.

[0032] For example, a workflow can be understood as a series of interrelated tasks and steps in a business process or operational process. These tasks and steps are executed in a certain order and according to rules to achieve specific business goals or results. The specific order or results can be determined based on the business needs and processing logic of the intelligent dialogue product, and the present disclosure does not limit this.

[0033] For example, the first machine learning model can be a large language model, or other neural network models, deep learning models, etc., which are not limited in the present disclosure. Taking an intelligent dialogue product connected to a large language model as an example, it is possible to interact with the user based on the large language model.

[0034] S102: Determine a page screenshot of the page to be tested in the intelligent dialogue product, and determine a first action based on the page screenshot, where the first action is an action that the intelligent dialogue product can perform on the page to be tested.

[0035] For example, the page to be tested for the intelligent dialogue product may be as follows: Figure 2 The chatbot page for interacting with the user shown in the figure is determined according to the actual business scenario, and this disclosure does not limit this. An action can be understood as an atomic interaction operation in the intelligent conversation product that can affect user data (such as chat records, process status, system status, etc.). Figure 2 Taking the dialogue robot page shown as an example, the action can be an action of communicating with the robot through the dialogue chat data box, an action of clicking on the controls in the message card, an action of closing the page or the function box provided by the page, an action of resetting or starting the session, an action of triggering some initialization or scenario-based entry calls, etc., which is determined according to the actual business scenario and is not limited in this regard in the present disclosure.

[0036] S103: Determine at least one second action through a second machine learning model according to the process information and the first action, where the second action is an action that the intelligent dialogue product can perform in the workflow to be tested and the page to be tested.

[0037] For example, the second machine learning model may be a large language model, or other neural network models, deep learning models, etc., which is not limited in the present disclosure.

[0038] For example, the actions that can be performed by the page to be tested and the workflow to be tested of the intelligent dialogue product change dynamically with the dialogue. Therefore, during the testing process, the actions that can be currently executed can be determined based on the page screenshot of the page to be tested and the process information of the workflow to be tested, so that the test process proceeds according to the expected workflow.

[0039] S104: Determine a target action from at least one second action, and control the intelligent dialogue product to execute the target action.

[0040] For example, after the intelligent dialogue product executes the target action, a page screenshot of the page to be tested of the intelligent dialogue product can be re-obtained, and steps S101-S104 can be repeated until the preset test completion conditions are met, such as the test coverage reaches a preset threshold, etc., which is not limited in this disclosure.

[0041] During the testing of the above intelligent dialogue products, Figure 2 As shown in the figure, the chatbot page can automatically answer questions without human intervention to enter content in the input box. It can automatically generate and cover a variety of chat paths and branches, so as to realize automatic testing of intelligent chat products connected with machine learning models, improve test efficiency and test coverage. It can also handle various types of input content and complex chat scenarios, ensuring the stability and consistency of intelligent chat products in complex scenarios.

[0042] It is important to understand that during the testing process, a virtual user can be constructed to interact with the intelligent dialogue product, such as Figure 2 The conversation content on the right side can be understood as the conversation content sent by the virtual user, which simulates the conversation interaction between the actual user and the conversation robot in the actual business scenario.

[0043] By using the above method, the intelligent dialogue product can be controlled to automatically execute actions that can be executed in the workflow to be tested and the page to be tested, which is equivalent to testing the intelligent dialogue product according to the workflow to be tested, thereby ensuring that the intelligent dialogue product connected to the machine learning model is automatically tested according to the expected workflow, thereby improving the test coverage and test accuracy.

[0044] In a possible manner, determining a first action based on a page screenshot includes: performing image recognition on the page screenshot to obtain a page recognition result; recalling target knowledge associated with the page to be tested from a knowledge base based on the page recognition result, the knowledge base storing product knowledge associated with the intelligent dialogue product, the product knowledge being obtained by parsing at least one of the requirement document information, page design information, and instruction information of the intelligent dialogue product; at least inputting the page recognition result and the target knowledge into a third machine learning model to obtain a first action output by the third machine learning model, the third machine learning model being used to output an action that the intelligent dialogue product can perform based on the input content.

[0045] For example, the third machine learning model may be a large language model, or other neural network models, deep learning models, etc., which is not limited by the present disclosure.

[0046] For example, Figure 3 As shown, the page screenshot can be image recognized to determine the content included in the page to be tested, such as the text content corresponding to the question, the content of the control used for interaction, etc. Figure 4 As shown, the page screenshot can be subjected to image preprocessing, attention analysis, image conversion into text, etc., and can be processed by a pre-trained image recognition model to obtain a page recognition result, which is not limited in the present disclosure.

[0047] In addition, since the pages of intelligent dialogue products may include pictures, text, audio and other content, in addition to identifying page screenshots, you can also obtain the content included in the page and perform multimodal processing based on a multimodal model to obtain page recognition results. The specific results can be determined based on the actual business scenario, and the present disclosure does not impose any restrictions on this.

[0048] For example, continue to refer to Figure 3, perform knowledge recall based on the page recognition results. For example, when the page includes control 1, you can recall knowledge related to control 1, such as the description of control 1 in the requirement document, so that the model can understand and analyze it and use the actions related to control 1 as actions that can be performed by the intelligent dialogue product.

[0049] For example, Figure 4 As shown, it is possible to obtain demand documents, other content used to supplement the explanation of demand documents, page design drafts, usage documents, permission information, label information and other content related to intelligent dialogue products, which can be determined according to the actual application scenario, and the present disclosure does not limit this. Then, the obtained content is parsed to obtain different heterogeneous data, such as table parsing, content parsing, page or document layout parsing, etc., and the different heterogeneous data are stored in the corresponding database, such as data in the form of vectors stored in the vector library, data in the form of key-value pairs stored in the ES library (Elasticsearch, search server), graph data stored in the Graph library, etc., which can be determined according to the actual application scenario, and the present disclosure does not limit this.

[0050] For example, the knowledge recalled from different databases can be sorted to obtain target knowledge, and then prompt words can be constructed based on the page recognition results and the target knowledge. The prompt words are used to prompt the third machine learning model to determine the actions that the intelligent dialogue product can perform according to the input content, and then the prompt words are input into the third machine learning model, such as Figure 4 The model A shown in the figure obtains the actions that the intelligent dialogue product can perform on the page to be tested output by the model A. Thus, the actions that the intelligent dialogue product can perform on the page to be tested can be determined based on the page screenshot.

[0051] In a possible manner, the testing method of the intelligent dialogue product further includes: determining the first historical interaction information of the intelligent dialogue product during the testing process. At least inputting the page recognition result and the target knowledge into the third machine learning model to obtain the first action output by the third machine learning model, including: inputting the page recognition result, the target knowledge and the first historical interaction information into the third machine learning model to obtain the first action output by the third machine learning model.

[0052] For example, Figure 4 As shown, historical interaction information of the intelligent dialogue product during the test process can also be obtained. It should be understood that during the test process, historical interaction information can refer to information generated by historical interactions between virtual users and intelligent dialogue products. The historical interaction information of the intelligent dialogue product can be historical dialogue content, which can help the model perform context processing, better understand the problem content of the current page, or avoid outputting actions that have been executed in the historical test process, etc. The specific information can be determined according to the actual application scenario, and the present disclosure does not limit this.

[0053] By combining historical interaction information to determine the actions that the intelligent conversation product can perform on the page to be tested, the test efficiency and test coverage can be improved. In addition, the intelligent conversation product can dynamically adjust the conversation process based on the historical conversation content and context to ensure that the actions performed can simulate the real interaction between the user and the conversation robot.

[0054] It is worth noting that the third machine learning model can be pre-trained based on content related to the intelligent dialogue product so that the third machine learning model can determine the actions that can be performed by the intelligent dialogue product based on the input content.

[0055] In a possible manner, the workflow to be tested includes at least one process node, and the process information of the workflow to be tested of the intelligent dialogue product is determined, including: converting the workflow to be tested of the intelligent dialogue product into process information in a preset format, and the preset format is a format that can be understood by the second machine learning model; wherein the process information includes at least one of a node identifier, a node type, a node content, and a transfer condition corresponding to each process node, the node type represents a functional type of the process node, the node content represents a display content output by the execution process node, and the transfer condition represents a condition and path from the process node to other process nodes.

[0056] For example, Figure 3 As shown, the workflow to be tested can be converted into process information in a preset format to facilitate subsequent models to process and analyze the process information. The preset format can be JSON format, RDF format, etc., which can be selected according to needs, and the present disclosure does not limit this.

[0057] For example, the workflow to be tested of the intelligent dialogue product may include one or more process nodes, each process node may represent a dialogue state, and the connection between the process nodes may represent the possible path of the dialogue. Accordingly, the process information may include at least one of the node identification, node type, node content and transfer condition corresponding to each process node, such as Figure 5 It should be understood that for ease of understanding Figure 5 The content shown is described in natural language, which can be a format that the model can understand in practical applications.

[0058] Among them, the node identifier represents the unique identifier of the process node, for example, it can be a number. The node type represents the functional type of the process node, for example, it can be a conditional judgment node, a question node, an answer node, etc., which can be determined according to the actual business scenario, and the present disclosure does not limit this. The node content represents the display content output by the execution process node, such as the question text content, the answer option content, etc., which can be determined according to the actual business scenario, and the present disclosure does not limit this. The transfer condition represents the conditions and paths from the current process node to other process nodes.

[0059] In a possible manner, the training process of the second machine learning model includes: obtaining first sample flow information and example samples, wherein the example samples include second sample flow information and sample actions corresponding to the second sample flow information; determining target prompt words based at least on the first sample flow information and the example samples, wherein the target prompt words are used to prompt the second machine learning model to understand the first sample flow information based on the example samples and generate at least one action; and inputting the target prompt words into the second machine learning model so that the second machine learning model learns through the example samples.

[0060] For example, a sample workflow for testing may be obtained, and the sample action flow may be converted into a format to obtain corresponding sample process information, such as Figure 5 As shown, based on the sample process information and corresponding executable actions of a part of the sample workflow as example samples, the sample process information of another part of the sample workflow as training samples, and then constructing the target prompt words, the target prompt words are input into the second machine learning model, so that the second machine learning model can learn through the example samples and guide the model to understand the structure and processing logic of the workflow. The trained second machine model can output the actions that the intelligent dialogue product can perform in the workflow based on the input workflow process information.

[0061] It should be understood that if Figure 3 As shown, the second machine learning model can be Figure 3 Model B shown, in actual use, needs to output the actions that the intelligent dialogue product can perform on the page to be tested and the workflow to be tested based on the process information and the first action, and the corresponding prompt words need to be modified. For example, the prompt words can be "Please analyze the following actions and process information, and enter the executable actions...", so that model B can directly output the actions that the intelligent dialogue product can perform on the page to be tested and the workflow to be tested. The specific actions can be determined according to the actual scenario, and the present disclosure does not limit this. Each action that can be executed by the intelligent dialogue product represents the next action that can be executed in the current dialogue state, including different ways of answering questions, asking new questions, changing topics, etc. The present disclosure does not limit this.

[0062] Of course, model B can also be used to output the third action that the intelligent dialogue product can perform in the workflow to be tested, and then the intersection of the first action and the third action can be used as the action that the intelligent dialogue product can perform on the page to be tested and the workflow to be tested. The specific action can be determined based on the actual scenario, and the present disclosure does not limit this.

[0063] It should be noted that, in practical applications, example samples may also be input to improve the robustness of the model, and the present disclosure does not impose any restrictions on this.

[0064] In a possible manner, the testing method of the intelligent dialogue product also includes: generating a first test request in response to the intelligent dialogue product executing a target action; determining that the first test request carries a preset test field and a first business operation corresponding to the first test request; when the first business operation includes a data operation on the data storage system of the intelligent dialogue product, routing the data operation to a shadow system corresponding to the data storage system according to the field value of the preset test field, and performing the data operation based on the shadow system.

[0065] It is worth noting that in order to ensure the stability and security of the production system where the intelligent dialogue product is officially deployed, Figure 3 As shown, a sandbox system for testing can be set up, and the intelligent dialogue product can be tested based on the sandbox system without affecting the data and services of the production system actually used by users.

[0066] It should be understood that after an action is performed on the front-end page of the intelligent dialogue product, a corresponding data request will be sent to the back-end. For example, if the front-end sends "query xx", a data query request will be sent to the back-end to query the content related to xx in the data storage system corresponding to the intelligent dialogue product. Correspondingly, after the target action is performed during the test, the corresponding test request will also be triggered.

[0067] For example, a preset test field can be encapsulated in the protocol layer of the test request. Whenever a test request is sent to the sandbox system, the protocol layer checks whether the test request contains the field. The value of the field can be an identifier for distinguishing different sandbox environments. Figure 6 As shown, the sandbox gateway can be annotated based on preset test fields. For example, the preset test fields can be bound to the thread corresponding to the test request through the thread context. In this way, when the thread is subsequently executed for business processing, routing decisions can be made based on the value of the preset test field, and data operations on the persistence layer can be redirected to the shadow system for execution instead of the data storage system corresponding to the production system. The persistence layer can be a database, file system, etc.

[0068] The shadow system may include resources such as a shadow database and a shadow file system. These resources have the same resource structure as the production system, but the data is independent. The configuration of the shadow system can be adjusted as needed to better simulate the production system, and this disclosure does not add anything new to this.

[0069] It is worth noting that all operations performed in the shadow system are independent of the production system. Therefore, even if a large number of read and write operations are performed in the shadow system, it will not have any impact on the data and services of the production system, thus ensuring the security and reliability of the test.

[0070] In addition, in order to ensure the effectiveness and security of the sandbox system, detailed logging and detection can be performed on the operations in the shadow system. Through logging and detection, potential problems can be discovered and resolved in a timely manner to ensure the stability of the sandbox system.

[0071] It should be understood that in the testing process of intelligent dialogue products, time factors often play a crucial role. For example, some batch tasks must be executed in the early morning, or some business logic will only be triggered at a specific time (such as 9 am to 5 pm), that is, some functions need to be executed in a specific time period. For example, after the user completes an operation, a reminder or notification needs to be sent after a certain period of time, that is, some functions in intelligent dialogue products involve delayed notifications. For example, some caching mechanisms or scheduled tasks need to run continuously for a long time, that is, some functions of intelligent dialogue products need to be run for a long time to verify their stability and performance.

[0072] Therefore, in a possible manner, the testing method of the intelligent dialogue product also includes: generating a second test request in response to the intelligent dialogue product executing a target action; determining a second business operation corresponding to the second test request; when the second business operation includes a time-related operation performed at a specific time, calling a hook function to obtain time information, the time information includes custom moment information and / or custom time flow rate, the hook function is used to intercept time-related operations, and read the returned time information from the configuration file or query the time information by calling a query interface; and executing time-related operations according to the time information.

[0073] For example, the time travel function can be implemented by using a hook function, such as the getTime method of the Hook system. The Hook system is a method for intercepting and modifying system calls. Through the getTime method of the Hook system, the operation of obtaining the current time can be intercepted and the configured time value can be returned. In this way, when performing time-related operations, the configured simulation time will be used instead of the actual current time, thereby realizing time simulation and travel.

[0074] For example, the time to be simulated can be configured through a configuration file or a query interface. For example, the current time can be configured as a custom moment, or a custom time flow rate can be configured, such as 10 minutes actually elapse per second, which can be determined according to needs, and the present disclosure does not limit this. In this way, these specific time periods can be simulated to ensure that the intelligent dialogue products operate normally under these time conditions, and the delay effect can be quickly simulated, such as verifying the correctness of the notification function, and the time process can be accelerated, such as quickly verifying the performance of the intelligent dialogue product under long-term operation, etc.

[0075] For example, when the intelligent dialogue product is started or the test begins, the time information of the configuration file or query interface can be read, and these settings can be implanted into the intelligent dialogue product through the hook function. In this way, all time-related operations will be performed based on the configured time during the operation of the system. In addition, the time information can be adjusted dynamically. For example, the tester can modify the configuration file at any time during the test or adjust the time information through the interface. The intelligent dialogue product will update the time simulation in real time, thereby achieving more flexible time travel, so that the changes in time can be simulated arbitrarily in the test phase, verifying the performance of the intelligent dialogue product under various time conditions, and ensuring the stability and correctness of the intelligent dialogue product under different time conditions.

[0076] In a possible manner, determining a target action from at least one second action includes: determining second historical interaction information of an intelligent dialogue product during a test process; determining an expected reward value after the intelligent dialogue product executes each second action in a current test state based on the second historical interaction information, the expected reward value being used to characterize the difference between the current test state and the next test state after executing the second action; and preferentially determining the second action with the largest reward value as the target action through a preset selection strategy.

[0077] For example, when the number of second actions is one, the second action can be directly determined as the target action. When the number of second actions is multiple, it is necessary to determine a target action from multiple second actions. For example, a second action can be randomly selected as the target action, or a Markov process can be used to calculate the reward value of each second action, and then the second action with the largest reward value is preferentially determined as the target action based on a preset selection strategy, etc. The specific determination can be based on demand, and the present disclosure does not limit this.

[0078] For example, the Markov process is a mathematical model used to describe the probability of a system transitioning between different states. For ease of description, each second action is represented as a state, and then the transition probability in the test workflow is calculated using a reinforcement learning algorithm. Taking the above-mentioned intelligent dialogue product as an example, each second action is represented as a state, and the transition between states is determined by the conditions and paths in the dialogue workflow. By analyzing historical interaction information, such as historical dialogue data and historical user operations, the transition probability of each second action can be calculated.

[0079] For example, the transition probability of each second action reflects the possibility and expected effect of taking the corresponding second action under the current test state, which can be represented by the difference between the current test state and the next test state after executing the second action. It should be noted that the test state can be the user state of a virtual user, and a user state can include multiple sub-states, each of which can be represented in the form of a key-value pair sequence. For example, a user state can be represented by {"timestamp": xxx, "whether to input": yes, "whether it is the first answer": no, "number of dialogue turns": 5, ...}, which can be determined according to the actual business scenario, and the present disclosure does not limit this.

[0080] For example, a reward function for calculating the expected reward value can be constructed based on indicators such as user feedback, conversation completion, and user satisfaction, so as to evaluate the effect of each second action through the reward function. The specific value can be determined according to needs, and the present disclosure does not impose any restrictions on this.

[0081] For example, the target action can be selected through a reinforcement learning algorithm such as Q-learning or a policy gradient algorithm. For example, the target action can be decided among multiple second actions based on the greedy algorithm (ε-greedy) in Q-learning, that is, the action with the highest expected reward value can be selected as the target action most of the time, but there is a small probability (ε) that any possible action will be randomly selected as the target action among multiple second actions. Of course, the action with the largest expected reward value can also be preferentially executed in other ways similar to the greedy algorithm, and the present disclosure does not limit this.

[0082] In related technologies, after the test is completed, the analysis and feedback of the test results often need to be done manually, which is not only time-consuming and labor-intensive, but also prone to subjective bias and omissions. Since the conversation process of intelligent dialogue products involves a large amount of text data, manual analysis and feedback are difficult, and it is difficult to ensure the accuracy and timeliness of the test results.

[0083] Therefore, the embodiment of the present disclosure can automatically generate a visual test flowchart after the test is completed, and can generate a visual dialogue flowchart of the intelligent dialogue product to help testers intuitively understand and grasp all possible paths and branches of the intelligent dialogue product, thereby improving test efficiency and accuracy.

[0084] Based on the same concept, the present disclosure provides a testing device for an intelligent dialogue product, such as Figure 7 As shown, the testing device 700 of the intelligent dialogue product includes: A first determination module 701 is used to determine process information of a workflow to be tested of an intelligent dialogue product, wherein the intelligent dialogue product is connected to a first machine learning model; A second determination module 702 is used to determine a page screenshot of the page to be tested in the intelligent dialogue product, and determine a first action based on the page screenshot, where the first action is an action that the intelligent dialogue product can perform on the page to be tested; A third determination module 703 is used to determine at least one second action according to the process information and the first action by using a second machine learning model, where the second action is an action that the intelligent dialogue product can perform in the workflow to be tested and the page to be tested; The fourth determination module 704 is used to determine a target action from the at least one second action and control the intelligent dialogue product to perform the target action.

[0085] Optionally, the second determination module 702 is used to: perform image recognition on the page screenshot to obtain a page recognition result; recall target knowledge associated with the page to be tested from a knowledge base based on the page recognition result, the knowledge base storing product knowledge associated with the intelligent dialogue product, the product knowledge being obtained by parsing at least one of the requirement document information, page design information, and instruction information of the intelligent dialogue product; input at least the page recognition result and the target knowledge into a third machine learning model to obtain a first action output by the third machine learning model, and the third machine learning model is used to output an action that the intelligent dialogue product can perform based on the input content.

[0086] Optionally, the testing device 700 of the intelligent dialogue product further includes: A fifth determination module, configured to determine first historical interaction information of the intelligent dialogue product during the test process; The second determining module 702 is used for: The page recognition result, the target knowledge and the first historical interaction information are input into a third machine learning model to obtain a first action output by the third machine learning model.

[0087] Optionally, the workflow to be tested includes at least one process node, and the first determining module 701 is used to: Converting the workflow to be tested of the intelligent dialogue product into process information in a preset format, where the preset format is a format that can be understood by the second machine learning model; Among them, the process information includes at least one of the node identification, node type, node content and transfer condition corresponding to each of the process nodes, the node type represents the functional type of the process node, the node content represents the display content output by executing the process node, and the transfer condition represents the conditions and paths from the process node to other process nodes.

[0088] Optionally, the training process of the second machine learning model includes: Acquire first sample process information and an example sample, wherein the example sample includes second sample process information and a sample action corresponding to the second sample process information; Determine a target prompt word at least according to the first sample process information and the example sample, wherein the target prompt word is used to prompt the second machine learning model to understand the first sample process information according to the example sample and generate at least one action; The target prompt word is input into the second machine learning model so that the second machine learning model learns through the example samples.

[0089] Optionally, the testing device 700 for the intelligent dialogue product further includes a shadow module, and the shadow module is used to: In response to the intelligent dialogue product executing the target action, generating a first test request; Determining that the first test request carries a preset test field and a first service operation corresponding to the first test request; When the first business operation includes a data operation on the data storage system of the intelligent dialogue product, the data operation is routed to a shadow system corresponding to the data storage system according to the field value of the preset test field, and the data operation is performed based on the shadow system.

[0090] Optionally, the testing device 700 for the intelligent dialogue product further includes a time module, and the time module is used to: In response to the intelligent dialogue product performing the target action, generating a second test request; Determining a second business operation corresponding to the second test request; When the second business operation includes a time-related operation performed at a specific time, calling a hook function to obtain time information, the time information includes custom moment information and / or custom time flow rate, the hook function is used to intercept the time-related operation, and read and return the time information from a configuration file or query the time information by calling a query interface; The time-related operation is performed according to the time information.

[0091] Optionally, the fourth determining module 704 is used to: Determining second historical interaction information of the intelligent dialogue product during the testing process; Determine, according to the second historical interaction information, an expected reward value after the intelligent dialogue product performs each of the second actions in the current test state, wherein the expected reward value is used to represent the difference between the current test state and the next test state after performing the second action; The second action with the largest reward value is preferentially determined as the target action through a preset selection strategy.

[0092] Based on the same concept, an embodiment of the present disclosure also provides a computer-readable medium on which a computer program is stored, and when the program is executed by a processing device, the steps of the testing method for the above-mentioned intelligent dialogue product are implemented.

[0093] Based on the same concept, the present disclosure also provides an electronic device, which may include: a storage device having a computer program stored thereon; A processing device is used to execute the computer program in the storage device to implement the steps of the above-mentioned testing method for the intelligent dialogue product.

[0094] Based on the same concept, an embodiment of the present disclosure also provides a computer program product, including a computer program, which implements the steps of the above-mentioned testing method for the intelligent dialogue product when executed by a processor.

[0095] Reference below Figure 8 , which shows a schematic diagram of the structure of an electronic device 800 suitable for implementing the embodiment of the present disclosure. The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 8 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0096] like Figure 8As shown, the electronic device 800 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage device 808 to a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the electronic device 800 are also stored. The processing device 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0097] Typically, the following devices may be connected to the I / O interface 805: an input device 806 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 807 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 808 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 809. The communication device 809 may allow the electronic device 800 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 8 The electronic device 800 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0098] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 809, or installed from a storage device 808, or installed from a ROM 802. When the computer program is executed by the processing device 801, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0099] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0100] In some embodiments, any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol) can be used for communication, and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0101] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0102] The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by the electronic device, the electronic device: determines the process information of the workflow to be tested of the intelligent dialogue product, and the intelligent dialogue product is connected to a first machine learning model; determines a page screenshot of the page to be tested in the intelligent dialogue product, and determines a first action based on the page screenshot, and the first action is an action that the intelligent dialogue product can perform on the page to be tested; determines at least one second action through a second machine learning model according to the process information and the first action, and the second action is an action that the intelligent dialogue product can perform in the workflow to be tested and the page to be tested; determines a target action from the at least one second action, and controls the intelligent dialogue product to perform the target action.

[0103] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0104] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0105] The modules involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a module does not, in some cases, limit the module itself.

[0106] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0107] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0108] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0109] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0110] Although the subject matter has been described in language specific to structural features and / or method logic actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims. Regarding the device in the above embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.

Claims

1. A testing method for an intelligent dialogue product, characterized in that: The testing method of the intelligent dialogue product includes: Determining process information of a workflow to be tested of the intelligent dialogue product, wherein the intelligent dialogue product is connected to a first machine learning model; Determine a page screenshot of a page to be tested in the intelligent dialogue product, and determine a first action based on the page screenshot, where the first action is an action that the intelligent dialogue product can perform on the page to be tested; Determine, by a second machine learning model, at least one second action according to the process information and the first action, where the second action is an action that the intelligent dialogue product can perform in the workflow to be tested and the page to be tested; A target action is determined from the at least one second action, and the intelligent dialogue product is controlled to execute the target action.

2. The method for testing an intelligent dialogue product according to claim 1, characterized in that: The determining a first action based on the page screenshot includes: Performing image recognition on the page screenshot to obtain a page recognition result; Recalling target knowledge associated with the page to be tested from a knowledge base according to the page recognition result, wherein the knowledge base stores product knowledge associated with the intelligent dialogue product, and the product knowledge is obtained by parsing at least one of requirement document information, page design information, and instruction information of the intelligent dialogue product; At least the page recognition result and the target knowledge are input into a third machine learning model to obtain a first action output by the third machine learning model, and the third machine learning model is used to output an action that the intelligent dialogue product can perform based on the input content.

3. The method for testing an intelligent dialogue product according to claim 2, characterized in that: The testing method of the intelligent dialogue product also includes: Determining first historical interaction information of the intelligent dialogue product during the testing process; The first action of inputting at least the page recognition result and the target knowledge into a third machine learning model to obtain an output of the third machine learning model includes: The page recognition result, the target knowledge and the first historical interaction information are input into a third machine learning model to obtain a first action output by the third machine learning model.

4. The method for testing an intelligent dialogue product according to claim 1, characterized in that: The workflow to be tested includes at least one process node, and determining the process information of the workflow to be tested of the intelligent dialogue product includes: Converting the workflow to be tested of the intelligent dialogue product into process information in a preset format, where the preset format is a format that can be understood by the second machine learning model; Among them, the process information includes at least one of the node identification, node type, node content and transfer condition corresponding to each of the process nodes, the node type represents the functional type of the process node, the node content represents the display content output by executing the process node, and the transfer condition represents the conditions and paths from the process node to other process nodes.

5. The method for testing an intelligent dialogue product according to claim 1, characterized in that: The training process of the second machine learning model includes: Acquire first sample process information and an example sample, wherein the example sample includes second sample process information and a sample action corresponding to the second sample process information; Determine a target prompt word at least according to the first sample process information and the example sample, wherein the target prompt word is used to prompt the second machine learning model to understand the first sample process information according to the example sample and generate at least one action; The target prompt word is input into the second machine learning model so that the second machine learning model learns through the example samples.

6. The method for testing an intelligent dialogue product according to any one of claims 1 to 5, characterized in that: The testing method of the intelligent dialogue product also includes: In response to the intelligent dialogue product executing the target action, generating a first test request; Determining that the first test request carries a preset test field and a first service operation corresponding to the first test request; When the first business operation includes a data operation on the data storage system of the intelligent dialogue product, the data operation is routed to a shadow system corresponding to the data storage system according to the field value of the preset test field, and the data operation is performed based on the shadow system.

7. The method for testing an intelligent dialogue product according to any one of claims 1 to 5, characterized in that: The testing method of the intelligent dialogue product also includes: In response to the intelligent dialogue product performing the target action, generating a second test request; Determining a second business operation corresponding to the second test request; When the second business operation includes a time-related operation performed at a specific time, calling a hook function to obtain time information, the time information includes custom moment information and / or custom time flow rate, the hook function is used to intercept the time-related operation, and read and return the time information from a configuration file or query the time information by calling a query interface; The time-related operation is performed according to the time information.

8. The method for testing an intelligent dialogue product according to any one of claims 1 to 5, characterized in that: The determining of the target action from the at least one second action comprises: Determining second historical interaction information of the intelligent dialogue product during the testing process; Determine, according to the second historical interaction information, an expected reward value after the intelligent dialogue product performs each of the second actions in the current test state, wherein the expected reward value is used to represent the difference between the current test state and the next test state after performing the second action; The second action with the largest reward value is preferentially determined as the target action through a preset selection strategy.

9. A testing device for an intelligent dialogue product, characterized in that: The testing device of the intelligent dialogue product includes: A first determination module is used to determine process information of a workflow to be tested of the intelligent dialogue product, wherein the intelligent dialogue product is connected to a first machine learning model; A second determination module is used to determine a page screenshot of the page to be tested in the intelligent dialogue product, and determine a first action based on the page screenshot, where the first action is an action that the intelligent dialogue product can perform on the page to be tested; A third determination module is used to determine at least one second action according to the process information and the first action through a second machine learning model, where the second action is an action that the intelligent dialogue product can perform in the workflow to be tested and the page to be tested; The fourth determination module is used to determine a target action from the at least one second action and control the intelligent dialogue product to perform the target action.

10. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processing device, the steps of the method according to any one of claims 1 to 8 are implemented.

11. An electronic device, characterized in that: include: a storage device having a computer program stored thereon; A processing device, configured to execute the computer program in the storage device to implement the steps of the method according to any one of claims 1 to 8.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images