Question and answer generation model training method and device, computer equipment and storage medium

Abstract

The invention relates to a question and answer generation model training method and device, computer equipment, a storage medium and a computer program product. The method comprises the steps that problem elements under multiple dimensions in a business system are determined, and the problem elements are obtained by dividing business data in the business system according to the multiple dimensions; generating verification data based on the question elements through a large language model, wherein the verification data comprises a verification question and a verification answer for the verification question; constructing sample data based on the question elements and the verification data, and training a question and answer generation model through the sample data to obtain a trained question and answer generation model; and the trained question and answer generation model is used for outputting target verification data for the construction based on the input target question elements, a target verification question included in the target verification data and a target verification answer for the target verification question, and is used for performing model verification for a question and answer model in the service system. By adopting the method, the accuracy of model verification can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a question-answering generation model training method, apparatus, computer equipment, storage medium and computer program product, and a question-answering data processing method, apparatus, computer equipment, storage medium and computer program product. Background Art

[0002] In the field of artificial intelligence, model verification is a key step to ensure that the model can accurately and reliably perform its intended functions in actual application scenarios. In order to evaluate the performance of the model, it can be verified by constructing a question-answer pair dataset. Specifically, by designing a series of questions and their corresponding correct answers, simulating user input and expected output in actual application scenarios, the model's understanding, reasoning, and generation capabilities can be fully tested. However, in the current method of model verification by constructing a question-answer pair dataset, the quality of the constructed question-answer pair dataset is unstable, resulting in low accuracy in model verification. Summary of the invention

[0003] Based on this, it is necessary to provide a question and answer generation model training method, device, computer equipment, computer readable storage medium and computer program product, and a question and answer data processing method, device, computer equipment, storage medium and computer program product that can improve the accuracy of model verification in response to the above-mentioned technical problems.

[0004] In a first aspect, the present application provides a question-answer generation model training method. The method comprises:

[0005] Determine the problem elements in multiple dimensions of the business system. Problem elements are obtained by dividing the business data in the business system according to multiple dimensions;

[0006] Generate verification data based on question elements through a large language model, where the verification data includes verification questions and verification answers to the verification questions;

[0007] Construct sample data based on question elements and verification data, train the question-answer generation model through the sample data, and obtain a trained question-answer generation model;

[0008] The trained question-answering generation model is used to output target verification data for the construction based on the input target question elements. The target verification data includes target verification questions and target verification answers to the target verification questions, which are used to perform model verification on the question-answering model in the business system.

[0009] In a second aspect, the present application also provides a question-answer generation model training device. The device comprises:

[0010] The problem element determination module is used to determine the problem elements in multiple dimensions of the business system. The problem elements are obtained by dividing the business data in the business system according to multiple dimensions.

[0011] A verification data acquisition module, used to generate verification data based on question elements through a large language model, wherein the verification data includes verification questions and verification answers to the verification questions;

[0012] A sample data training module is used to construct sample data based on question elements and verification data, train the question-answer generation model through the sample data, and obtain a trained question-answer generation model;

[0013] The trained question-answering generation model is used to output target verification data for the construction based on the input target question elements. The target verification data includes target verification questions and target verification answers to the target verification questions, which are used to perform model verification on the question-answering model in the business system.

[0014] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of the above question-answer generation model training method when executing the computer program.

[0015] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above question-answer generation model training method when executed by a processor.

[0016] In a fifth aspect, the present application also provides a computer program product. The computer program product includes a computer program, which implements the steps of the above question-answer generation model training method when executed by a processor.

[0017] The above-mentioned question and answer generation model training method, device, computer equipment, storage medium and computer program product determine the question elements obtained by dividing the business data in the business system according to multiple dimensions, generate verification data including verification questions and verification answers based on the question elements through a large language model, and construct sample data to train the question and answer generation model based on the question elements and the verification data. The trained question and answer generation model can output target verification data for model verification of the question and answer model according to the input target question elements. Using the large language model to generate verification data based on the question elements to train the question and answer generation model can ensure the comprehensiveness of the target verification data output by the trained question and answer generation model, thereby improving the accuracy of model verification of the question and answer model based on the target verification data.

[0018] In a sixth aspect, the present application provides a method for processing question and answer data. The method comprises:

[0019] Obtain target question elements and a pre-trained question-answer generation model, where the question-answer generation model is trained according to the question-answer generation model training method described above;

[0020] The target question elements are input into the question-answering generation model to obtain the target verification data output by the question-answering generation model. The target verification data includes the target verification questions and the target verification answers to the target verification questions, which are used to perform model verification on the question-answering model in the business system.

[0021] In a seventh aspect, the present application also provides a question-answer data processing device. The device comprises:

[0022] A model acquisition module is used to obtain target question elements and a pre-trained question-answer generation model, where the question-answer generation model is trained according to the question-answer generation model training method described above;

[0023] The verification data acquisition module is used to input the target question elements into the question and answer generation model to obtain the target verification data output by the question and answer generation model. The target verification data includes the target verification questions and the target verification answers to the target verification questions, which are used to perform model verification on the question and answer model in the business system.

[0024] In an eighth aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the above question-and-answer data processing method are implemented.

[0025] In a ninth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the above question-and-answer data processing method are implemented.

[0026] In a tenth aspect, the present application further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the above question-and-answer data processing method are implemented.

[0027] The above-mentioned question and answer data processing method, device, computer equipment, storage medium and computer program product obtain target question elements and a pre-trained question and answer generation model, and obtain target verification data output by the question and answer generation model for model verification of the question and answer model by inputting the target question elements into the question and answer generation model. The question and answer generation model is trained with verification data generated based on question elements using a large language model, which can ensure the comprehensiveness of the target verification data output by the question and answer generation model, thereby improving the accuracy of model verification of the question and answer model based on the target verification data. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0029] Figure 1 An application environment diagram of a question-answer generation model training method and a question-answer data processing method in one embodiment;

[0030] Figure 2 A schematic diagram of a process flow of a question-answer generation model training method in one embodiment;

[0031] Figure 3 A schematic diagram of a process for training a question-answer generation model in one embodiment;

[0032] Figure 4 Schematic diagram of a process of question and answer data processing in one embodiment;

[0033] Figure 5 A schematic diagram of a traditional model validation process;

[0034] Figure 6 A flow chart of another traditional model validation process;

[0035] Figure 7 A schematic diagram of dividing a user question into question elements in one embodiment;

[0036] Figure 8 A schematic diagram of dividing the problem elements into three dimensions in one embodiment;

[0037] Fig. 9 A flowchart of a question-answer generation model training method in another embodiment;

[0038] Fig.10 A schematic diagram of generating verification data by using a large language model in one embodiment;

[0039] Fig.11 A schematic diagram of training a small model using verification data generated by a large language model in one embodiment;

[0040] Fig.12 A structural block diagram of a question-answer generation model training device in one embodiment;

[0041] Fig.13 is a structural block diagram of a question and answer data processing device in one embodiment;

[0042] Fig.14 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0044] The question-answer generation model training method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be set up separately, integrated on the server 104, or placed on the cloud or other servers. The user can send the problem elements under multiple dimensions in the business system to the server 104 through the terminal 102, and the problem elements can be obtained by the server 104 in advance according to multiple dimensions for the business data in the business system. After the server 104 obtains the problem elements, it can generate verification data including verification questions and verification answers based on the problem elements through a large language model. The server 104 can build a sample data training question and answer generation model based on the problem elements and the verification data. The trained question and answer generation model can output the target verification data for model verification of the question and answer model for the input target question elements. In some embodiments, the server 104 can deploy the trained question and answer generation model to the terminal 102, so that the user can build the target verification data for the question and answer model through the question and answer generation model deployed in the terminal 102, so as to perform model verification on the question and answer model through the target verification data.

[0045] The question-answer data processing method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. The user can send the target question element to the server 104 through the terminal 102, and the server 104 can input the target question element into the question-answer generation model to obtain the target verification data output by the question-answer generation model for model verification of the question-answer model. The server 104 can also return the obtained target verification data to the terminal 102, so that the terminal 102 performs model verification on the question-answer model in the business system based on the target verification data.

[0046] The terminal 102 may be, but is not limited to, various desktop computers, laptop computers, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart vehicle-mounted devices, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers, or may be a cloud server.

[0047] In an exemplary embodiment, Figure 2 As shown in Figure 2, a question-answer generation model training method is provided, which is applied to Figure 1 The server in the example is used to illustrate, including the following steps 202 to 206. Among them:

[0048] Step 202, determining problem elements in multiple dimensions in the business system, where the problem elements are obtained by dividing the business data in the business system according to multiple dimensions.

[0049] Among them, the question-answering generation model is an artificial neural network model used to generate corresponding question-answering data according to input data. The question-answering data output by the question-answering generation model can be used to verify the model of the question-answering model, and specifically can verify the understanding ability, reasoning ability and generation ability of the question-answering model. The question-answering model can answer the input questions and output the corresponding answers. The input questions can include natural language. The question-answering model can realize semantic understanding of natural language questions and output the corresponding answers. For example, the question-answering model can extract key elements from the input questions, and the key elements can be used as the answers of the question-answering model so that users can quickly understand the key content in the natural language questions. The business system can be a system that implements various business functions, such as enterprise management systems, banking business systems, audit systems, transaction systems and other systems. Question elements are elements used to construct question-answering data, which can be divided into multiple dimensions, such as time, analysis dimension and measurement value.

[0050] Exemplarily, the server can pre-acquire the business data in the business system and divide the business data according to different dimensions, and the business data under each dimension can be configured with corresponding question elements. For example, in an e-commerce enterprise system, the business data can be divided according to the three dimensions of time, analysis dimension and measurement value for business data to construct various question elements, such as the time dimension can include various time-related question elements such as purchase time, delivery time, freeze time, purchase time, order time, confirmation receipt time, etc., and the analysis dimension can include various question elements such as product category, customer name, supplier name, customer type, channel type, etc., and the measurement value can include various question elements such as sales, order volume, average unit price, confirmation delivery quantity, etc. The business data of the business system can be divided into multiple dimensions, and at least one question element can be constructed under each dimension. The question element is used to construct corresponding question and answer data, so as to perform model verification on the question and answer model through the question and answer data.

[0051] The server may determine question elements under different dimensions in the business system, such as the server may determine multiple dimensions of the business system, and select from the respective question elements of each dimension to obtain question elements for constructing question and answer data. In some embodiments, different business systems may be divided into different dimensions, and different dimensions may include different question elements. The server determines the question elements of the business system, and the constructed question and answer data may be used for model verification processing of the question and answer model in the business system.

[0052] Step 204: Generate verification data based on the question elements using the large language model, where the verification data includes the verification question and the verification answer to the verification question.

[0053] Among them, Large Language Models (LLM) is a natural language processing (NLP) model based on deep learning, which is designed to understand and generate natural language text. Large language models are artificial intelligence models trained with a large amount of data. They use deep learning technology to capture the complexity and diversity of language. They are based on language operations and are used to understand, generate and process text data. The verification data includes verification questions and verification answers to the verification questions. The verification data can be used as question-and-answer data for model verification of the question-and-answer model.

[0054] Optionally, the server may call a large language model to process the problem elements through the large language model and generate corresponding verification data. For example, the server may input the problem elements into the large language model, and the large language model may output verification data including verification questions and verification answers. The verification data may be used to perform model verification on the question-answering model. In some embodiments, after the server inputs the problem elements into the large language model, the large language model may output at least one verification data, each of which includes a verification question and a verification answer to the verification question. The number of verification data output by the large language model can be flexibly configured according to actual needs, such as being set according to the number of problem elements, that is, the more problem elements input into the large language model, the more verification data can be generated. For example, problem elements of different dimensions may have different degrees of importance, and different weights may be configured accordingly, based on which the number of generated verification data may be controlled. Specifically, a standard number of verification data can be configured, that is, each question element input into the large language model can generate a standard number of verification data, and then corresponding weights are configured for different question elements, so that the number of verification data generated can be a weighted number of the standard number and the weight, so that different numbers of verification data can be generated for question elements of different importance, so as to ensure the model verification reliability of the question and answer model.

[0055] Step 206, construct sample data based on the question elements and the verification data, train the question and answer generation model through the sample data, and obtain a trained question and answer generation model; the trained question and answer generation model is used to output the constructed target verification data based on the input target question elements, and the target verification data includes target verification questions and target verification answers to the target verification questions, which are used to perform model verification on the question and answer model in the business system.

[0056] Among them, the sample data is used to train the question-answer generation model, so that the role of the large language model can be achieved through the trained question-answer generation model, that is, the trained question-answer generation model can generate corresponding verification data according to the input question elements. Exemplarily, the server can construct sample data based on the question elements and verification data, such as the question elements that generate the verification data can be used as input samples, and the verification data can be used as output samples to construct sample data. The server can be trained based on the sample data to obtain a trained question-answer generation model. The trained question-answer generation model can generate corresponding target verification data for the input target question elements, and the target verification data includes target verification questions and target verification answers for the target verification questions. The model verification can be performed for the question-answer model in the business system through the target verification questions and target verification answers. In some embodiments, the server can directly train and initially construct an untrained question-answer generation model based on the sample data, or it can fine-tune the pre-trained general model based on the sample data to obtain a trained question-answer generation model.

[0057] In some embodiments, when the target verification data is used to perform model verification on the question-answering model, the target verification question in the target verification data can be input into the question-answering model, and the question-answering model can output the corresponding model answer. The server can compare the model answer output by the question-answering model with the target verification answer in the target verification data, and the question-answering processing effect of the question-answering model can be determined based on the comparison result. For example, the question-answering model can include an NL2SQL (Natural Language to Structured Query Language) model, that is, the NL2SQL model can take the input target verification question as a natural language question, and output the answer to the natural language question, that is, output the corresponding model answer. The model answer can include the conversion result of converting the input target verification question into the structured query language, so that the language conversion effect of the NL2SQL model can be verified through the target verification data.

[0058] In the above-mentioned question and answer generation model training method, question elements are determined by dividing the business data in the business system according to multiple dimensions, and verification data including verification questions and verification answers are generated based on the question elements through a large language model. Sample data is constructed according to the question elements and the verification data to train the question and answer generation model. The trained question and answer generation model can output target verification data for model verification of the question and answer model based on the input target question elements. Using a large language model to generate verification data based on the question elements to train the question and answer generation model can ensure the comprehensiveness of the target verification data output by the trained question and answer generation model, thereby improving the accuracy of model verification of the question and answer model based on the target verification data.

[0059] In an exemplary embodiment, verification data is generated based on question elements through a large language model, and the verification data includes verification questions and verification answers to the verification questions, including: obtaining at least one question element combination based on question elements, and the question element combination includes at least one question element under at least one dimension; inputting the question element combination into the large language model for question and answer generation, and obtaining verification data output by the large language model.

[0060] Among them, the question element combination is obtained by combining the question elements of different dimensions in the business system, each question element combination may include at least one question element, and at least one question element may belong to at least one dimension. Exemplarily, the server can combine the question elements to construct at least one question element combination, and each question element combination may include at least one question element. By constructing the question element combination, each question element can be fully mined to generate different verification data. In some embodiments, the server can select question elements from different dimensions respectively, and combine the selected question elements to obtain the question element combination. The server can input the question element combination into the large language model, and the large language model generates questions and answers for the questions, and outputs the corresponding verification data.

[0061] Further, constructing sample data based on question elements and verification data includes: constructing sample data based on question elements included in the question element combination, verification questions included in the verification data, and verification answers to the verification questions.

[0062] Optionally, the server may construct sample data based on the verification data and the question elements included in the question element combination that generates the verification data. In some embodiments, the question element combination constructed by the server based on the question elements may include multiple, each question element combination may generate at least one verification data by a large language model, each verification data has a corresponding question element combination, and the server may construct sample data based on the question element combination and the corresponding verification data, thereby constructing multiple sample data. In some embodiments, the server may screen the obtained question element combination and the corresponding verification data, such as the server may evaluate the question element combination and the corresponding verification data, specifically, the question element combination and the corresponding verification data may be evaluated by a pre-trained data evaluation model to determine the data quality score of the question element combination and the corresponding verification data, and when the data quality score reaches the quality score threshold, it may be considered that the question element combination and the corresponding verification data can be used for model training, thereby constructing sample data.

[0063] In this embodiment, the server obtains a question element combination based on question elements, so that a large language model generates corresponding verification data based on the question element combination, and constructs sample data based on the verification data and the question elements included in the question element combination for generating the verification data. The question elements can be used to construct rich and diverse sample data to ensure the training effect of the question and answer generation model, thereby improving the accuracy of model verification for the question and answer model based on the target verification data output by the question and answer generation model.

[0064] In an exemplary embodiment, sample data is constructed based on question elements and verification data, including: obtaining a pre-trained target question-answering model; inputting the verification questions included in the verification data into the target question-answering model to obtain model answers to the verification questions output by the target question-answering model; matching the model answers with the verification answers included in the verification data to obtain matching results; and constructing sample data based on the matching verification data and the question elements corresponding to the matching verification data that are matched according to the matching results.

[0065] The target question-answering model is a pre-trained question-answering model that can answer the input questions and output the corresponding answers. The model answer is the answer output by the target question-answering model for the input verification questions. The matching result can represent the difference between the model answer output by the target question-answering model and the verification answer in the verification data.

[0066] Optionally, the server can obtain a pre-trained target question-answering model, and input the verification questions in the verification data into the target question-answering model, and the target question-answering model processes the input verification questions, such as answering the input verification questions to output the corresponding model answers. The server can match the model answers with the verification answers included in the verification data, and specifically can compare the model answers with the verification answers to obtain a matching result. When the matching result indicates a match, it indicates that for the same verification question, the model answer output by the target question-answering model is consistent with the corresponding verification answer generated by the large language model, that is, the verification data generated by the large language model can be used for effective model verification of the question-answering model, so the verification data can be used as training data for training the question-answering generation model. Based on the matching result, the server can determine that the matching result is the matching verification data and the question elements that generate the matching verification data, and the server can construct sample data based on the matching verification data and the corresponding question elements. For multiple verification data generated by the large language model, the server can input the verification questions in each verification data into the target question-answering model respectively, and match the verification answers in the verification data with the model answers output by the target question-answering model to obtain corresponding matching results. The server traverses each verification data and can filter based on the matching results of each verification data to filter out matching verification data with matching results from each verification data. The server can construct sample data based on the matching verification data and the question elements used to generate the matching verification data.

[0067] In this embodiment, the server screens the verification data through a pre-trained target question and answer model to select matching verification data with high reliability to construct sample data. The verification data can be filtered to obtain high-quality verification data to ensure the training effect of the question and answer generation model, thereby improving the accuracy of model verification of the question and answer model based on the target verification data output by the question and answer generation model.

[0068] In an exemplary embodiment, Figure 3 As shown, the process of training the question-answer generation model, that is, training the question-answer generation model through sample data to obtain a trained question-answer generation model, includes:

[0069] Step 302, obtaining the question-answer generation model to be trained.

[0070] Among them, the question and answer generation model to be trained is the question and answer generation model that needs to be trained. The purpose of the question and answer generation model is to generate corresponding question and answer data for the input question elements, and the generated question and answer data is used to verify the question and answer model in the business system. Optionally, the server can construct a question and answer generation model to be trained, and the question and answer generation model can be constructed using various neural network algorithms or deep learning algorithms according to actual needs. In some embodiments, the question and answer generation model to be trained can also include a pre-trained general model, so that the constructed sample data can be used to fine-tune the general model using the LoRA (Low-Rank Adaptation, low-rank fine-tuning) fine-tuning method, so that the general model is further trained using the sample data to obtain a question and answer generation model that meets the training needs.

[0071] Step 304, input the question elements in the sample data into the question-answer generation model to be trained, and obtain question-answer data output by the question-answer generation model to be trained, wherein the question-answer data includes question data and answer data for the question data.

[0072] Among them, the sample data may include question elements and verification data generated based on the question elements. For each sample data, the server can input the question elements in the sample data into the question and answer generation model to be trained, and the question and answer generation model to be trained will process the question elements and output the obtained question and answer data. The question and answer data may include question data and answer data generated based on the question elements.

[0073] Step 306, based on the data difference between the question and answer data and the verification data in the sample data, the question and answer generation model to be trained is updated and then continued to be trained to obtain a trained question and answer generation model.

[0074] Among them, data differences are used to characterize the differences between the question and answer data and the verification data in the sample data, which may specifically include question data differences between the question data in the question and answer data and the verification questions in the sample data, as well as answer data differences between the question answer data in the question and answer data and the verification answers in the sample data.

[0075] Optionally, the server may compare the question and answer data with the verification data included in the sample data to determine the data difference between the question and answer data and the verification data in the sample data. For example, the server may compare the question data in the question and answer data with the verification question in the sample data to obtain the question data difference; the server may also compare the question and answer data in the question and answer data with the verification answer in the sample data to obtain the answer data difference, and the server may combine the question data difference and the answer data difference to obtain the data difference between the question and answer data and the verification data. In some embodiments, respective weights may be configured for the question data difference and the answer data difference, and the server may weightedly fuse the question data difference and the answer data difference based on the respective configured weights, thereby obtaining the data difference between the question and answer data and the verification data in the sample data. The server may update the question and answer generation model to be trained based on the data difference, such as updating the model parameters of the question and answer generation model to be trained to obtain an updated question and answer generation model. The server may continue training based on the updated question and answer generation model, such as obtaining the next sample data for training, until the training is completed to obtain the trained question and answer generation model.

[0076] In this embodiment, the server uses constructed sample data to train the question and answer generation model. The sample data is constructed by generating verification data based on question elements according to a large language model. This can ensure the comprehensiveness of the target verification data output by the trained question and answer generation model, thereby improving the accuracy of model verification for the question and answer model based on the target verification data.

[0077] In an exemplary embodiment, based on the data difference between the question and answer data and the verification data in the sample data, the question and answer generation model to be trained is updated and then continued to be trained to obtain a trained question and answer generation model, including: based on the data difference between the question and answer data and the verification data in the sample data, the question and answer generation model to be trained is updated and then continued to be trained until a first training end condition is met to obtain an intermediate question and answer generation model; obtaining reinforcement training question elements, inputting the reinforcement training question elements into the intermediate question and answer generation model to obtain reinforcement training question and answer data output by the intermediate question and answer generation model; constructing negative sample data through the reinforcement training question elements and the reinforcement training question and answer data; and performing reinforcement training on the intermediate question and answer generation model based on the negative sample data until a second training end condition is met to obtain a trained question and answer generation model.

[0078] Among them, the first training end condition is used to determine whether to end the training process of the question and answer generation model using the constructed sample data. The first training end condition can be configured according to actual needs, such as reaching a preset number of training times, traversing each sample data, and the accuracy of the trained question and answer generation model meets the requirements. After the first training end condition is met, the intermediate question and answer generation model can be obtained, and further training can be performed on the intermediate question and answer generation model. The enhanced training question element is a question element used for enhanced training of the intermediate question and answer generation model, which can be obtained based on the sample data and can also be pre-constructed according to actual needs. The enhanced training question and answer data is the question and answer data generated and output by the intermediate question and answer generation model for the enhanced training question element. The negative sample data is constructed based on the enhanced training question element and the enhanced training question and answer data, which can be the question and answer data and the corresponding question elements output by the intermediate question and answer generation model that do not meet the requirements.

[0079] Exemplarily, the server can update the model parameters of the question-answer generation model to be trained based on the data difference between the question-answer data and the verification data in the sample data, and then continue training until the first training end condition is met. For example, after traversing each sample data, the training of this stage can be ended to obtain the intermediate question-answer generation model. The server can obtain the pre-constructed reinforcement training question elements, and input the reinforcement training question elements into the intermediate question-answer generation model for question-answer data processing, and the intermediate question-answer generation model outputs the corresponding reinforcement training question-answer data, which may include question data and corresponding answer data. The server can construct negative sample data based on the reinforcement training question elements and the reinforcement training question-answer data, such as the server can screen the reinforcement training question elements and the reinforcement training question-answer data, and screen out the reinforcement training question-answer data that does not correspond to the reinforcement training question elements, or screen out the reinforcement training question-answer data that does not correspond to the question data and the answer data in the reinforcement training question-answer data, to construct negative sample data. The server can perform intensive training on the intermediate question and answer generation model based on the negative sample data. Specifically, the server can input the negative sample data into the intermediate question and answer generation model, and the intermediate question and answer generation model outputs the corresponding question and answer data. The server updates the intermediate question and answer generation model based on the data difference between the question and answer data in the negative sample data and the question and answer data output by the intermediate question and answer generation model, and then continues training until the second training end condition is met. For example, after traversing all the negative sample data, the server can end the training and obtain a trained question and answer generation model.

[0080] In this embodiment, the server uses the verification data constructed based on the large language model as positive samples to train the intermediate question and answer generation model, and performs intensive training on the intermediate question and answer generation model by constructing negative samples, which can ensure the training effect of the question and answer generation model and is conducive to improving the accuracy of model verification of the question and answer model based on the target verification data.

[0081] In an exemplary embodiment, determining problem elements in a business system under multiple dimensions includes: obtaining various business data in the business system; performing dimensional division based on respective data fields of each business data to obtain multiple dimensions of the business system; and dividing each business data according to multiple dimensions to obtain problem elements under multiple dimensions.

[0082] Exemplarily, the server can obtain business data in the business system, each business data includes different data fields, and the server can divide the dimensions of each business data according to the data fields to construct multiple different dimensions in the business system. The server can divide the business data according to each dimension to divide each business data into corresponding dimensions. For example, the business system can be divided into time dimension, analysis dimension and measurement value dimension. The server can divide each business data into corresponding dimensions according to the respective attributes of each business data. For example, the time dimension can include business data of various data fields such as modification date, creation date, purchase date, approval date, etc. The server can obtain problem elements under multiple dimensions based on the divided business data.

[0083] In this embodiment, the server divides the business data in the business system into dimensions according to data fields, and divides each business data into corresponding question elements according to the divided dimensions, so as to use the large language model to construct verification data based on the question elements to train the question and answer generation model, which can ensure the comprehensiveness of the target verification data output by the trained question and answer generation model, thereby improving the accuracy of model verification of the question and answer model based on the target verification data.

[0084] In an exemplary embodiment, Figure 4 As shown, a question-answering data processing method is provided, and the method is applied to Figure 1 The server in the example is used to illustrate, including the following steps 402 to 404. Among them:

[0085] Step 402, obtaining target question elements and a pre-trained question-answer generation model, where the question-answer generation model is trained according to a question-answer generation model training method.

[0086] The pre-trained question-answer generation model may be pre-trained using the question-answer generation model training method described above, and the pre-trained question-answer generation model may output target verification data for the construction based on the input target question elements, and the target verification data includes target verification questions and target verification answers for the target verification questions, which are used to perform model verification on the question-answer model in the business system. The target question elements may be question elements configured by the user that need to generate target verification data.

[0087] Optionally, the server may obtain target question elements, which may be configured by the user according to actual needs. The server may obtain a pre-trained question-answer generation model, such as when the server receives the target question elements sent by the user through the terminal, it may further obtain a pre-trained question-answer generation model.

[0088] Step 404, input the target question elements into the question and answer generation model, and obtain the target verification data output by the question and answer generation model. The target verification data includes the target verification questions and the target verification answers to the target verification questions, which are used to perform model verification on the question and answer model in the business system.

[0089] Exemplarily, the server may input the target question elements into the question-answering generation model, and the question-answering generation model may generate verification data based on the target question elements, and output the generated target verification data. The target verification data includes the target verification question and the target verification answer to the target verification question, and the server may perform model verification on the question-answering model in the business system based on the target verification data.

[0090] In the above-mentioned question and answer generation model training method, target question elements and a pre-trained question and answer generation model are obtained, and by inputting the target question elements into the question and answer generation model, target verification data output by the question and answer generation model for model verification of the question and answer model is obtained. The question and answer generation model is trained with verification data generated based on question elements using a large language model, which can ensure the comprehensiveness of the target verification data output by the question and answer generation model, thereby improving the accuracy of model verification of the question and answer model based on the target verification data.

[0091] The present application also provides an application scenario, which applies the above-mentioned question and answer data processing method and question and answer data processing method. Specifically, the question and answer data processing method and the question and answer data processing method are applied in the application scenario as follows:

[0092] With the advent of the AI ​​(Artificial Intelligence) era, there are more and more application scenarios for AI models. It is very important to provide an efficient AI model application scenario verification model. At present, the verification of AI application scenarios still relies on a lot of manual input, which is not only inefficient and high-investment, but also the accuracy of the verification results is not enough. It depends on the professional ability of the personnel and the person in charge of the work. In order to solve this accuracy problem, some companies have adopted cross-AI application scenario verification, which has increased a lot of labor costs. Figure 5 As shown, the verification of AI model application scenarios is mostly questions and conclusions, which are generally created manually for AI models and specified application scenarios to verify the problems of the AI ​​models and obtain corresponding model verification results.

[0093] Among them, question-answer pairs are the most common verification method for data involved in training, fine-tuning, and verification. However, in the process of creating question-answer pairs, the traditional solution is to create questions through a large amount of manual input, and the quality of the created questions depends on the professional understanding of the personnel in the AI ​​application scenario. Figure 6 As shown in the figure, the creation of questions and answers, as well as the question verification process, rely on the professionalism and input of personnel. The verification set standard cannot be well controlled, and the breadth and depth of the verification set cannot be guaranteed to be stable. The knowledge background of different builders will have a large difference in quality and coverage. Moreover, the efficiency of verification depends on the number of people invested, resulting in relatively inefficient model verification. In addition, the accuracy cannot be well controlled, and there are cases where human errors lead to errors in the answers to the constructed question and answer pairs.

[0094] In the question-and-answer data processing method and the question-and-answer data processing method provided in the embodiment of the present application, through a large number of problem analysis and summarization, it is determined that any complete question depends on three elements, namely, the time dimension, the analysis dimension, and the metric dimension. Based on this underlying paradigm, in the document processing application scenario, the data of various documents can be divided into time, dimension, and metric dimension, and then randomly and freely combined to cover the questions of various scenarios. By classifying and disassembling the business system fields in the application scenario, multiple fields of different dimensions are selected in a permutation and combination manner, and rewritten using a large model to simulate user questions. Based on these problem elements, they are rewritten into more colloquial user questions through a large language model, so that user questions can be manufactured in large quantities. The specific server can construct a corresponding verification data set based on the MECE (Mutually Exclusive Collectively Exhaustive, no duplication and no omission) standard, and can specifically exhaustively permutate and combine user question fields to achieve the effect of no duplication and no omission, and can automatically generate answers to questions as a verification set to verify the NL2SQL model answer.

[0095] like Figure 7As shown in the figure, different user questions can be divided into different question elements, which are the core information of the user question. For example, for the user question "What is the unit price including tax when the supplier's bulk delivery date is in the past year and the corresponding group supplier of the processor is XXXXX and the specified supplier's location name is 2?", the analysis results after word segmentation can include question elements such as "supplier delivery date, corresponding group supplier of the processor, and the unit price including tax of the specified supplier's location name". In a specific application, such as Figure 8 As shown in the figure, for the order verification assessment, the user questions after word segmentation can be divided into three types of question elements: time (creation date, review date, modification date...), dimension (department, employee, organization...), and measurement value (purchase unit price, purchase quantity, purchase amount...). The problem construction model involved can be problem model = analysis dimension + time dimension + measurement value dimension. For example, the user question can be constructed based on the paradigm of purchasing department (A) + today's review + total purchase order amount.

[0096] Furthermore, if Fig. 9 As shown, in the question-and-answer data processing method and the question-and-answer data processing method provided in the embodiments of the present application, in the application scenario verification of the AI ​​model, a model can be created based on the problem, such as the paradigm of analysis dimension + time dimension + metric dimension, and the large language model LLM is used to perform multiple rewrites to obtain multiple different questions and corresponding answers, so as to achieve the creation of questions and answer processing. The server can perform question verification based on the question verification model. The paradigm of the question verification model can specifically include various question elements of the created user question. The server can perform model verification on the AI ​​model to obtain corresponding model verification results.

[0097] Specifically, for the processing of creating user questions, such as Fig.10 As shown, the server can obtain verification data through the large language model LLM, including verification questions and verification answers. Specifically, the server can divide the various business data in the business system into dimensions, and divide the various business data into different dimensions. The dimensions specifically include time dimension, measurement dimension and analysis dimension. The server can combine the question elements under each dimension and rewrite them through the large language model LLM to generate corresponding verification data, which includes verification questions generated based on the question elements and verification answers to the verification questions. Fig.11As shown, the server can construct training data based on the verification data obtained through the large language model LLM, and train the small model with the training data constructed with the same specifications. The trained small model can be used as a trained question and answer generation model to generate target verification data for model verification through the trained small model. Specifically, the user questions and correct answers after the model is rewritten are used as SFT (Supervised Fine-Tuning) annotated data for positive samples, and they are used as question and answer combinations to adjust the general model through the LoRA fine-tuning method to enhance its ability to understand and execute commands in ERP (Enterprise Resource Planning) scenarios. After the fine-tuning is completed, the question and answer pairs are regenerated to collect Bad Cases as negative samples, and the iteration is completed through reinforcement learning.

[0098] The question-and-answer data processing method and the question-and-answer data processing method provided in the embodiments of the present application can systematically and automatically complete the verification of AI model application scenarios, and can construct a training framework including model construction and automatic iteration according to various business fields, and can train small parameter models according to specific business fields and user scenarios to achieve customized effects for thousands of people, which can effectively improve performance and accuracy, and can also reduce the company's investment in labor costs.

[0099] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0100] Based on the same inventive concept, the embodiment of the present application also provides a question-answer generation model training device for implementing the question-answer generation model training method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more question-answer generation model training device embodiments provided below can refer to the limitations of the question-answer generation model training method above, and will not be repeated here.

[0101] In an exemplary embodiment, Fig.12As shown, a question-answer generation model training device 1200 is provided, comprising: a question element determination module 1202, a verification data acquisition module 1204 and a sample data training module 1206, wherein:

[0102] The problem element determination module 1202 is used to determine the problem elements in multiple dimensions in the business system. The problem elements are obtained by dividing the business data in the business system according to multiple dimensions;

[0103] A verification data acquisition module 1204 is used to generate verification data based on question elements through a large language model, wherein the verification data includes a verification question and a verification answer to the verification question;

[0104] A sample data training module 1206 is used to construct sample data based on question elements and verification data, train the question-answer generation model through the sample data, and obtain a trained question-answer generation model;

[0105] The trained question-answering generation model is used to output target verification data for the construction based on the input target question elements. The target verification data includes target verification questions and target verification answers to the target verification questions, which are used to perform model verification on the question-answering model in the business system.

[0106] In an exemplary embodiment, the verification data acquisition module 1204 is further used to obtain at least one question element combination based on question elements, wherein the question element combination includes at least one question element under at least one dimension; the question element combination is input into the large language model for question and answer generation to obtain verification data output by the large language model; the sample data training module 1206 is further used to construct sample data based on the question elements included in the question element combination, the verification questions included in the verification data, and the verification answers to the verification questions.

[0107] In an exemplary embodiment, the sample data training module 1206 is also used to obtain a pre-trained target question-answering model; input the verification questions included in the verification data into the target question-answering model to obtain model answers to the verification questions output by the target question-answering model; match the model answers with the verification answers included in the verification data to obtain matching results; and construct sample data based on the matching verification data and the question elements corresponding to the matching verification data that are matched with the matching results.

[0108] In an exemplary embodiment, the sample data training module 1206 is also used to obtain a question-answer generation model to be trained; input question elements in the sample data into the question-answer generation model to be trained, and obtain question-answer data output by the question-answer generation model to be trained, wherein the question-answer data includes question data and answer data for the question data; based on the data difference between the question-answer data and the verification data in the sample data, the question-answer generation model to be trained is updated and training is continued to obtain a trained question-answer generation model.

[0109] In an exemplary embodiment, the sample data training module 1206 is also used to update the question and answer generation model to be trained according to the data difference between the question and answer data and the verification data in the sample data, and then continue training until the first training end condition is met to obtain an intermediate question and answer generation model; obtain reinforcement training question elements, input the reinforcement training question elements into the intermediate question and answer generation model, and obtain the reinforcement training question and answer data output by the intermediate question and answer generation model; construct negative sample data through the reinforcement training question elements and the reinforcement training question and answer data; and perform reinforcement training on the intermediate question and answer generation model based on the negative sample data until the second training end condition is met to obtain a trained question and answer generation model.

[0110] In an exemplary embodiment, the problem element determination module 1202 is also used to obtain various business data in the business system; perform dimensional division based on the data fields of each business data to obtain multiple dimensions of the business system; divide each business data according to multiple dimensions to obtain problem elements under multiple dimensions.

[0111] Based on the same inventive concept, the embodiment of the present application also provides a question and answer data processing device for implementing the question and answer data processing method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more question and answer data processing device embodiments provided below can refer to the limitations of the question and answer data processing method above, and will not be repeated here.

[0112] In an exemplary embodiment, Fig.13 As shown, a question-answer data processing device 1300 is provided, comprising: a model acquisition module 1302 and a verification data acquisition module 1304, wherein:

[0113] Model acquisition module 1302, used to acquire target question elements and a pre-trained question-answer generation model, where the question-answer generation model is trained according to the question-answer generation model training method described above;

[0114] The verification data acquisition module 1304 is used to input the target question elements into the question and answer generation model to obtain the target verification data output by the question and answer generation model. The target verification data includes the target verification questions and the target verification answers to the target verification questions, which are used to perform model verification on the question and answer model in the business system.

[0115] Each module in the above-mentioned question-answer generation model training device or question-answer data processing device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0116] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Fig.14 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store various data involved in the question-answer generation model training method and / or the question-answer data processing method. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a question-answer generation model training method and / or a question-answer data processing method is implemented.

[0117] Those skilled in the art will understand that Fig.14 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0118] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0119] In one embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0120] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0121] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0122] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0123] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0124] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A question-answer generation model training method, characterized in that: The method comprises: Determine problem elements in a business system under multiple dimensions, wherein the problem elements are obtained by dividing business data in the business system according to the multiple dimensions; Generate verification data based on the question elements by using a large language model, wherein the verification data includes a verification question and a verification answer to the verification question; Constructing sample data based on the question elements and the verification data, training a question-answer generation model through the sample data, and obtaining a trained question-answer generation model; The trained question-and-answer generation model is used to output target verification data for the construction based on the input target question elements. The target verification data includes target verification questions and target verification answers to the target verification questions, which are used to perform model verification on the question-and-answer model in the business system.

2. The method according to claim 1, characterized in that The generating verification data based on the question elements by using a large language model, wherein the verification data includes a verification question and a verification answer to the verification question, includes: Combining the problem elements to obtain at least one problem element combination, wherein the problem element combination includes at least one problem element under at least one dimension; Inputting the question element combination into a large language model to generate questions and answers, and obtaining verification data output by the large language model; The constructing sample data based on the question elements and the verification data includes: Sample data is constructed based on the question elements included in the question element combination, the verification question included in the verification data, and the verification answer to the verification question.

3. The method according to claim 1, characterized in that The constructing sample data based on the question elements and the verification data includes: Get the pre-trained target question answering model; Inputting the verification question included in the verification data into the target question-answering model to obtain a model answer for the verification question output by the target question-answering model; Matching the model answer with the verification answer included in the verification data to obtain a matching result; Sample data is constructed based on the matching verification data that is matched by the matching result and the question elements corresponding to the matching verification data.

4. The method according to claim 1, characterized in that: The step of training the question-answer generation model by using the sample data to obtain a trained question-answer generation model includes: Get the question-answer generation model to be trained; Inputting the question elements in the sample data into the question-answer generation model to be trained, and obtaining question-answer data output by the question-answer generation model to be trained, wherein the question-answer data includes question data and answer data for the question data; According to the data difference between the question and answer data and the verification data in the sample data, the question and answer generation model to be trained is updated and then continued to be trained to obtain a trained question and answer generation model.

5. The method according to claim 4, characterized in that The updating of the question-answer generation model to be trained and continuing the training according to the data difference between the question-answer data and the verification data in the sample data to obtain a trained question-answer generation model includes: According to the data difference between the question and answer data and the verification data in the sample data, the question and answer generation model to be trained is updated and then continued to be trained until the first training end condition is met, thereby obtaining an intermediate question and answer generation model; Obtaining reinforcement training question elements, inputting the reinforcement training question elements into the intermediate question and answer generation model, and obtaining reinforcement training question and answer data output by the intermediate question and answer generation model; Constructing negative sample data by using the reinforcement training question elements and the reinforcement training question and answer data; Based on the negative sample data, intensive training is performed on the intermediate question and answer generation model until a second training end condition is met, thereby obtaining a trained question and answer generation model.

6. The method according to any one of claims 1 to 5, characterized in that: Determining the problem elements in multiple dimensions of the business system includes: Obtain various business data in the business system; Performing dimension division based on the data fields of each of the business data to obtain multiple dimensions of the business system; The business data are divided according to the multiple dimensions to obtain problem elements under the multiple dimensions.

7. A method for processing question and answer data, characterized in that: The method comprises: Obtain target question elements and a pre-trained question-answer generation model, wherein the question-answer generation model is trained according to the question-answer generation model training method according to any one of claims 1 to 6; The target question elements are input into the question-answer generation model to obtain target verification data output by the question-answer generation model. The target verification data includes target verification questions and target verification answers to the target verification questions, which are used to perform model verification on the question-answer model in the business system.

8. A question-answer generation model training device, characterized in that: The device comprises: A problem element determination module is used to determine the problem elements under multiple dimensions in the business system, wherein the problem elements are obtained by dividing the business data in the business system according to the multiple dimensions; A verification data acquisition module, used to generate verification data based on the question elements through a large language model, wherein the verification data includes a verification question and a verification answer to the verification question; A sample data training module, used to construct sample data based on the question elements and the verification data, train the question-answer generation model through the sample data, and obtain a trained question-answer generation model; The trained question-and-answer generation model is used to output target verification data for the construction based on the input target question elements. The target verification data includes target verification questions and target verification answers to the target verification questions, which are used to perform model verification on the question-and-answer model in the business system.

9. A question-answer data processing device, characterized in that: The device comprises: A model acquisition module, used to acquire target question elements and a pre-trained question-answer generation model, wherein the question-answer generation model is trained according to the question-answer generation model training method according to any one of claims 1 to 6; A verification data acquisition module is used to input the target question elements into the question and answer generation model to obtain the target verification data output by the question and answer generation model. The target verification data includes the target verification questions and the target verification answers to the target verification questions, which are used to perform model verification on the question and answer model in the business system.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images