Question and answer sentence quality inspection method, device and related products

Abstract

The embodiments of the present application disclose a method, device and related products for quality inspection of question and answer statements based on artificial intelligence technology. When a question and answer statement pair needs to be quality inspected for matching, it is only necessary to input the question and answer statement pair into a trained question and answer statement quality inspection model. Since the question and answer statement quality inspection model has the function of determining the matching between the question statement and the answer statement, the model can be used to determine whether the question statement and the answer statement in the input question and answer statement pair match. The output result can then determine whether the question and answer statement pair receiving the quality inspection has passed the quality inspection. It can be seen that the present application realizes automatic quality inspection of question and answer statements with a trained question and answer statement quality inspection model, improves the quality inspection efficiency of question and answer statements, and reduces the quality inspection cost. At the same time, the solution enables full quality inspection, thereby greatly improving the coverage of quality inspection.

Description

Technical Field

[0001] The present application relates to the field of natural language processing technology, and in particular to a question and answer sentence quality inspection method, device and related products. Background Art

[0002] Questions and answers between people and between people and machines occur in many scenarios in people's daily lives. Take the e-commerce scenario as an example. When people inquire about product information on a shopping platform, human or electronic customer service will respond to the consumer's questions. For example, a consumer provides personal body dimensions and asks the customer service for suitable clothing sizes. Accordingly, the customer service recommends clothing sizes to the consumer; a consumer inquires about the warranty period of a certain product, and the customer service replies with the warranty period of the product; a consumer asks about the use of a certain product, and the customer service provides the correct use of the product.

[0003] Whether it is a manual answer or a machine answer, there is a possibility that the answer does not match the question. For example, wrong answers, missing answers, or perfunctory answers. In order to ensure the quality of customer service, it is usually necessary to conduct quality inspection on the matching of question and answer statements. In the existing technology, manual quality inspection of question and answer statements is adopted, but the efficiency and cost of manual quality inspection of question and answer statements are low. On the other hand, due to the poor efficiency of manual quality inspection and consideration of quality inspection costs, manual quality inspection of question and answer statements is usually carried out by sampling, so the coverage rate of the question and answer statements tested is low. For example, it is difficult to conduct full quality inspection on the matching of question and answer statements by sampling 5% or 10% of the question and answer statements. Summary of the invention

[0004] The embodiments of the present application provide a method, device and related products for quality inspection of question and answer sentences to improve the quality inspection efficiency of question and answer sentences, reduce the quality inspection cost, and make full quality inspection feasible.

[0005] In view of this, the first aspect of the present application provides a question and answer sentence quality inspection method, the method comprising:

[0006] Obtaining a question-answer sentence pair to be inspected, the question-answer sentence pair to be inspected comprising a question sentence and an answer sentence having a question-answer relationship;

[0007] Inputting the question sentence and the answer sentence into the question-answer sentence quality inspection model to determine the matching of the question sentence and the answer sentence through the question-answer sentence quality inspection model;

[0008] Get the output result of the question and answer sentence quality inspection model. The output result is used to indicate whether the question and answer sentence pair to be inspected has passed the quality inspection.

[0009] A second aspect of the present application provides a question and answer sentence quality inspection device, the device comprising:

[0010] A sentence pair acquisition unit, used to acquire a question-answer sentence pair to be inspected, wherein the question-answer sentence pair to be inspected includes a question sentence and an answer sentence having a question-answer relationship;

[0011] A sentence pair quality inspection unit, used for inputting the question sentence and the answer sentence into the question and answer sentence quality inspection model, so as to determine the matching of the question sentence and the answer sentence through the question and answer sentence quality inspection model;

[0012] The output result acquisition unit is used to obtain the output result of the question and answer sentence quality inspection model through quality inspection results. The output result is used to indicate whether the question and answer sentence pair to be quality inspected has passed the quality inspection.

[0013] A third aspect of the present application provides a question and answer sentence quality inspection device, the device comprising a processor and a memory:

[0014] The memory is used to store the program code and transmit the program code to the processor;

[0015] The processor is used to execute the steps of the question and answer statement quality inspection method of the first aspect as described above according to the instructions in the program code.

[0016] The fourth aspect of the present application provides a computer-readable storage medium, which is used to store program code, and the program code is used to execute the question and answer sentence quality inspection method of the first aspect mentioned above.

[0017] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0018] In the embodiment of the present application, a method, device, equipment and storage medium for quality inspection of question and answer statements are provided. In the present application, when a question and answer statement pair for quality inspection matching is required, the question and answer statement pair only needs to be input into a pre-trained question and answer statement quality inspection model. Since the question and answer statement quality inspection model has the function of determining the matching between the question statement and the answer statement, the model can be used to determine whether the question statement and the answer statement in the input question and answer statement pair match. Furthermore, the result output by the model is used to determine whether the question and answer statement pair receiving the quality inspection has passed the quality inspection. For example, when the model outputs that the matching probability of the question statement and the answer statement is 0.9 and the probability of mismatch is 0.1, it can be determined based on the relative size of the two probabilities that the matching quality inspection of the question statement and the answer statement in the question and answer statement pair after receiving the quality inspection of this model has failed. It can be seen that the present application realizes automatic quality inspection of question and answer statements with a trained question and answer statement quality inspection model, improves the quality inspection efficiency of question and answer statements, and reduces the quality inspection cost. At the same time, the scheme makes full quality inspection achievable, thereby greatly improving the coverage of quality inspection. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A flow chart of a method for quality inspection of question and answer sentences provided in an embodiment of the present application;

[0020] Figure 2 Schematic diagram of the question-answer relationship between question sentences and answer sentences;

[0021] Figure 3 A schematic diagram of positive samples of multiple question-answer sentence pairs;

[0022] Figure 4A A schematic diagram of a method for constructing negative samples for question-answering sentences;

[0023] Figure 4B A schematic diagram of another implementation method for constructing negative samples for question-answering sentences;

[0024] Figure 4C A schematic diagram of a method for constructing negative samples for question-answering sentences;

[0025] Figure 5 A schematic diagram of constructing negative samples from positive samples extracted from work orders archived in different batches in an embodiment of the present application;

[0026] Fig. 6A A schematic diagram of the structure of a question-and-answer sentence quality inspection model provided in an embodiment of the present application;

[0027] Figure 6B A schematic diagram of the structure of another question-and-answer sentence quality inspection model provided in an embodiment of the present application;

[0028] Figure 6C A schematic diagram of the structure of another question-answer sentence quality inspection model provided in an embodiment of the present application;

[0029] Figure 7 A schematic diagram of a scenario of multiple quality inspection question and answer sentence pairs in automatic and manual modes provided in an embodiment of the present application;

[0030] Figure 8 A schematic diagram of the structure of a question and answer sentence quality inspection device provided in an embodiment of the present application;

[0031] Fig. 9 A schematic diagram of the structure of a server for question and answer sentence quality inspection provided in an embodiment of the present application;

[0032] Fig.10 A schematic diagram of the structure of a terminal for question and answer sentence quality inspection provided in an embodiment of the present application. DETAILED DESCRIPTION

[0033] As mentioned above, the quality inspection of the matching of question statements and answer statements is usually done manually. Manual sampling has the problems of low detection coverage, low efficiency and high cost. Many platforms that provide consulting services often need to meet non-real-time and fragmented user consulting services, and this type of business can generate a large number of work orders. Work orders are archived and collected through asynchronous channels, and a large number of question and answer statements can usually be extracted from the work orders. Faced with such a large number of work orders and question and answer statements, it is necessary to provide a more efficient, low-cost and high-coverage question and answer statement quality inspection solution. Based on this demand, the present application provides a solution for automating the quality inspection of question and answer statements. The following is an introduction and explanation of the implementation of the technical solution in combination with the embodiments and drawings.

[0034] In the embodiments of the present application, artificial intelligence technology is used to implement model training and application. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0035] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, mechatronics and other technologies. Artificial intelligence software technologies mainly include computer vision technology, speech processing technology, natural language processing technology, as well as machine learning / deep learning, autonomous driving, smart transportation and other major directions.

[0036] Among them, natural language processing (NLP) is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between people and computers using natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field will involve natural language, that is, the language people use in daily life, so it is closely related to the study of linguistics. Natural language processing technology usually includes text processing, semantic understanding, machine translation, robot question answering, knowledge graph and other technologies.

[0037] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0038] In the embodiment of the present application, the trained model is trained using a machine learning method, and natural language processing technology is used in the construction of samples and in the processing of part of the text within the model.

[0039] The above-mentioned question and answer sentence quality inspection method can be applied to a processing device that has a communication relationship with the screen, such as a terminal device or server with a question and answer sentence quality inspection function. The method is executed independently by a terminal device or a server, and can also be applied to a network scenario in which a terminal device and a server communicate, and the terminal device and the server cooperate to run. Among them, the terminal device can be a mobile phone, a desktop computer, a personal digital assistant (PDA for short), a tablet computer, a smart home appliance, an intelligent voice interaction device and other devices. The server can be understood as an application server or a Web server. In actual deployment, the server can be an independent physical server or a server cluster or distributed system composed of multiple physical servers. The terminal and the server can be directly or indirectly connected via wired or wireless communication, and this application is not limited here.

[0040] For ease of explanation, the following embodiments use a terminal device as an exemplary execution subject to describe the solution in detail.

[0041] See also Figure 1 , which is a flow chart of a question and answer sentence quality inspection method provided in an embodiment of the present application. Figure 1 The question and answer sentence quality inspection methods shown include:

[0042] S101: The terminal device obtains a question-answer sentence pair to be inspected.

[0043] The question-answer sentence pair to be inspected refers to the question-answer sentence pair that needs to be inspected by the technical solution provided by the embodiment of the present application. The question-answer sentence pair to be inspected includes a question sentence and a reply sentence having a question-answer relationship. Figure 2 A schematic diagram of the question-answer relationship between question statements and answer statements is provided. Figure 2As shown, the customer service side responded to the question statement Q1 on the user side, and the response statement is represented by A1; similarly, the customer service side responded to the question statement Q2 on the user side, and the response statement is represented by A2. Even if the response statement A1 may be a perfunctory response or an incorrect response, the response statement A1 also constitutes a response relationship with the question statement Q1; the question-answer relationship between A2 and Q2 is the same. Therefore, the question statement Q1 and the response statement A2 constitute a pair of question-answer statement pairs to be inspected, and the question statement Q2 and the response statement A2 also constitute a pair of question-answer statement pairs to be inspected. For any pair of question-answer statement pairs to be inspected, the subsequent step S102 can be performed for quality inspection.

[0044] An exemplary implementation method for obtaining question-answer sentence pairs to be quality-checked is introduced below.

[0045] Taking the asynchronous system as an example, multiple work orders are archived. There may be some noise in these work orders. When only the work orders that require manual responses from quality inspection are needed, the noise can be work orders that are not manually responded to. In addition, if the corpus itself in the work order is unclear, then the significance of quality inspection is also small, and this type of work order can also be regarded as noise. When denoising, the terminal device can specifically use some image processing or natural language processing technology to realize the recognition and screening of noise. One thing that needs to be explained is that if the original work order data is encoded, the work order data needs to be decoded before denoising.

[0046] For the sake of explanation and distinction, the work orders retained after denoising are called target work orders. The question and answer pairs to be inspected are extracted from the target work orders. Specifically, the terminal device can extract question information and answer information with question and answer relationships from the target work order. The question information can include the contents of multiple fields, including: question description, question type, and question details. The contents of these fields are combined to form a question statement. The answer information can be the content of the customer service's reply to the question raised by the user, and the reply content can be directly combined into a reply statement.

[0047] In the embodiment of the present application, the terminal device improves the effectiveness of the question-answer pair to be inspected by screening the target work order. In addition, by combining the question statement with the question description, question type and question details, the types of question information included in the question statement are enriched, thereby improving the quality of the question statement and the reliability of subsequent quality inspection based on this question statement.

[0048] S102: The terminal device inputs the question sentence and the answer sentence into the question and answer sentence quality inspection model to determine the matching of the question sentence and the answer sentence through the question and answer sentence quality inspection model.

[0049] In an embodiment of the present application, the question and answer sentence quality inspection model is a model obtained by pre-training before executing the method of this embodiment. After training, this model can process and analyze the input question sentence and answer sentence, and calculate the probability of mutual matching and mismatch of the input question sentence and answer sentence. Among them, mutual matching can specifically refer to a matching degree greater than a preset threshold, and mismatch can specifically refer to a matching degree less than or equal to a preset threshold. The sum of the two probabilities is 1. Therefore, the output result is essentially a binary classification result. The matching degree can be calibrated in a variety of different ways. For example, the matching degree of different numerical values ​​can be calibrated according to a manually set method. The preset threshold can be set according to the actual requirements for the accuracy of the model quality inspection. For example, the more accurate the quality inspection result of the model for the matching of question and answer sentences is, the higher the preset threshold can be set.

[0050] To facilitate understanding of the implementation of the model function, an exemplary training method of the question-answer sentence quality inspection model is introduced below. In order to realize the binary classification function of the model, the terminal device constructs two different types of samples, one of which is a positive sample of a question-answer sentence pair and the other is a negative sample of a question-answer sentence pair. The positive sample of the question-answer sentence pair includes a question sentence and a reply sentence that have a question-answer relationship and match each other, and the negative sample of the question-answer sentence pair includes an unmatched question sentence and a reply sentence. In the training stage, the positive sample of the question-answer sentence pair and the negative sample of the question-answer sentence pair are input into the initial quality inspection model respectively. The parameters in the initial quality inspection model will be adjusted with the training process. Specifically, according to the difference between the output result of the initial quality inspection model and the label of the sample input into the initial quality inspection model, the initial quality inspection model is adjusted until the training cutoff condition is met to obtain the question-answer sentence quality inspection model. The label of the positive sample of the question-answer sentence pair is the first category, and the corresponding matching probability of the first category should be 1, and the probability of mismatch is 0. The label of the negative sample of the question-answer sentence pair is the second category, and the corresponding matching probability of the second category should be 0, and the probability of mismatch is 1. The matching probability and mismatching probability refer to the probability that the input question-answer pair belongs to the first category (the question statement and the answer statement match each other and pass the quality inspection) and the probability that it belongs to the second category (the question statement and the answer statement do not match each other and fail the quality inspection). Accordingly, the output results of the model include the matching probability and the mismatching probability.

[0051] In the above embodiments, the training cutoff condition may include various forms, such as the number of training iterations reaching a preset number, or the loss value being less than a preset value, etc. The training cutoff condition may be set according to actual needs, so the training cutoff condition is not limited here.

[0052] The terminal device learns to adjust the model coefficients through the selection and iterative training of the above samples. The question and answer sentence pair quality inspection model finally obtained can realize the function of determining whether the question sentence and the answer sentence in the input question and answer sentence pair to be inspected match.

[0053] S103: The terminal device obtains the output result of the question and answer sentence quality inspection model, and the output result is used to indicate whether the question and answer sentence pair to be inspected has passed the quality inspection.

[0054] Specifically, when the matching probability of the question statement and the answer statement in the output result is greater than the non-matching probability, it indicates that the question-answer statement pair to be inspected has passed the quality inspection, and the question statement and the answer statement match each other after the automatic quality inspection of the terminal device of this solution. When the matching probability of the question statement and the answer statement in the output result is less than the non-matching probability, it indicates that the question-answer statement pair to be inspected has failed the quality inspection, and the question statement and the answer statement do not match each other after the automatic quality inspection of this solution.

[0055] In the technical solution introduced in the above embodiment, the quality inspection model for question-answer pairs obtained through pre-training can realize automatic quality inspection of the matching between question statements and answer statements with question-answer relationship. Since the quality inspection can be completed automatically in this process, there is no need for manual review one by one, which reduces the dependence on manual quality inspection, thereby improving the efficiency of quality inspection and reducing the cost of quality inspection. In addition, since the automated quality inspection of question-answer pairs overcomes the problems of low efficiency and high cost of manual quality inspection, it can achieve higher coverage of quality inspection. For example, a full quality inspection of the matching of question statements and answer statements in massive work orders can be achieved.

[0056] In the model training process introduced above, it is mentioned that positive samples of question-answer sentence pairs and negative samples of question-answer sentence pairs need to be trained. The quantity and quality of samples are usually related to the training effect of the model and the functional reliability of the model after training. If the samples are manually labeled, the cost of labeling will be very high. In order to reduce the labeling cost, in an embodiment of the present application, the construction of negative samples of question-answer sentence pairs can be achieved in an unsupervised manner. For example, it is assumed by default that the question-answer sentence pairs obtained from historical work orders are all positive samples, that is, they contain question statements and answer statements that have a question-answer relationship and match each other. The construction of negative samples can be achieved by disrupting the relationship between the question statements and the answer statements in the positive samples of multiple question-answer sentence pairs and re-pairing them.

[0057] Figure 3 is a schematic diagram of multiple question-answer sentence pairs. Figure 3As shown, the positive sample of the question-answering statement is represented as: {Q1, A1}, {Q2, A2}…{Qn, An}, where Q represents the question statement and A represents the answer statement. In practical applications, the method of re-pairing the question-answering relationship between the question statement and the answer statement in the positive sample of multiple question-answering statements can be performed randomly or according to a certain rule. For example, the question statement (or answer statement) in a certain question-answering statement pair positive sample is replaced with the question statement (or answer statement) of another random other question-answering statement pair positive sample; or, according to the sequence number rule and the preset numerical interval x (x is a non-zero integer), the question statement (or answer statement) with the sequence number w+x in the positive sample is replaced with the position of the question statement (or answer statement) with the sequence number w, where w is a positive integer. Of course, the above-listed random or regular re-pairing of the question-answering relationship to obtain negative samples is only an example, and this is not limited in the embodiments of the present application. In addition, the scope of re-pairing the question-answering relationship can be all question-answering statement pairs positive samples, or part of the question-answering statement pairs positive samples. For the scenario where some question-answer sentences need to re-pair the question-answer relationship for positive samples to form negative samples, in the specific implementation, these positive samples can be selected randomly or based on factors such as the work order filing time and question type.

[0058] The following describes an implementation method in which a terminal device filters some positive samples of question-answer pairs to construct negative samples of question-answer pairs. In one possible implementation method, multiple target sample pairs are first selected from the constructed positive samples of question-answer pairs. The screening method may be screening according to a preset probability. For example, for each, a preset probability is used to determine whether to use it as a target sample pair to generate a negative sample. As an example, the preset probability may be 0.7, and of course, the preset probability may also be set according to the actual number of sample generation requirements. Assuming that 10 million positive samples are constructed, about 7 million negative samples are generated according to the preset probability. For the target sample pairs mentioned above, a variety of methods can be used to process and generate negative samples. The role of negative samples is to change the matching of the question statement and the answer statement in the original target sample pair, for example, by changing the question-answer relationship between the question statement and the answer statement in the target sample pair, the target sample pair with matching question statements and answer statements can be changed to a mismatched negative sample. Thus, the negative sample has the utility of training the question-answer statement quality inspection model of the previous text.

[0059] The following introduces several optional implementation methods for processing target sample pairs.

[0060] In one processing method of the example, the terminal device replaces the question sentence in the target sample pair with the question sentence in other question and answer sentence positive samples to obtain a negative sample of the question and answer sentence pair. Figure 4AThe following is a schematic diagram of an implementation method for constructing negative samples of question-answering sentence pairs. Assuming {Q1, A1} is the target sample pair, the negative sample of the question-answering sentence pair is represented as {Qn, A1}, where Qn represents the question sentence of the positive sample of other question-answering sentence pairs other than the target sample pair {Q1, A1}.

[0061] In another processing method of the example, the terminal device replaces the answer statement in the target sample pair with the answer statement in other question and answer sentence positive samples to obtain a negative sample of the question and answer sentence pair. Figure 4B The following is a schematic diagram of another implementation method for constructing negative samples of question-answering sentences. Assuming {Q1, A1} is the target sample pair, the negative sample of the question-answering sentence pair is represented as {Q1, Am}, where Am represents the reply sentence of the positive sample of other question-answering sentence pairs other than the target sample pair {Q1, A1}.

[0062] In another exemplary processing method, the terminal device reverses the question sentence and the answer sentence in the target sample pair to obtain a negative sample of the question-answer sentence pair. Figure 4C A schematic diagram of an implementation method for constructing negative samples of question-answer pairs. Assuming {Q1, A1} is the target sample pair, the generated negative sample of question-answer pairs is represented as {A1, Q1}. Combined with the above example processing method, it can be seen that the question-answer relationship between the question sentence and the answer sentence in the negative sample of the question-answer pair obtained based on the target sample pair is re-paired compared to the original target sample pair.

[0063] In the above embodiments, the terminal device realizes data enhancement by selecting target sample pairs and then constructing negative samples of question-answer pairs. By constructing negative samples of question-answer pairs based on positive samples of question-answer pairs, sample labeling under unsupervised learning is realized, which saves the cost of manual labeling of data and improves the efficiency of manual labeling. In the above embodiments, the terminal device can use the question-answer pairs extracted from the historical work orders as positive samples, and can also use the positive samples (all or part) that have been manually labeled as the data basis for constructing negative samples.

[0064] In actual applications, the correlation between the positive samples of question-answering sentences constructed by the terminal device through the asynchronous system from the work orders archived in the same batch may be stronger. In order to improve the quality of the constructed negative samples of question-answering sentences, the terminal device can improve the quality of negative samples when constructing the negative samples of question-answering sentences, thereby promoting the effect of the trained model. Figure 5 Schematic diagram of constructing negative samples from positive samples extracted from work orders archived in different batches in the embodiment of the present application. Figure 5As shown in the figure, the answer sentences in the positive samples extracted from some batches of archived work orders are replaced by the answer sentences in the positive samples extracted from other batches of archived work orders. That is, the aforementioned target sample pairs and other question-answer sentence positive samples come from work orders archived in different batches of the asynchronous system. For example Figure 5 The first pair of positive samples obtained from the first batch of archives is used as the target sample pair, and the question sentence Q11 in the target sample pair and the reply sentence A57 of the positive sample of the question-answer sentence pair obtained from the fifth batch of archives are reorganized into the negative sample of the question-answer sentence pair {Q11, A57}; the second pair of positive samples obtained from the first batch of archives is used as the target sample pair, and the question sentence Q12 in the target sample pair and the reply sentence A98 of the positive sample obtained from the ninth batch of archives are reorganized into the negative sample of the question-answer sentence pair {Q12, A98}.

[0065] The following introduces the structure of the question-answer sentence quality inspection model to help you understand how the model processes the input data.

[0066] The QA sentence quality inspection model includes: input layer, encoding layer, classification input layer and classification layer. From the input end to the output end of the model, there are input layer, encoding layer, classification input layer and classification layer in order, that is, the output of the input layer is used as the input of the encoding layer; the output of the encoding layer is used as the input of the classification input layer; the output of the classification input layer is used as the input of the classification layer. The output of the classification layer is used as the output of the entire model. Fig. 6A A schematic diagram of the structure of a question-answer sentence quality inspection model provided in an embodiment of the present application. It can be understood that the relationship between the layers in the structure is the same as the relationship between the layers in the structure of the question-answer sentence quality inspection model obtained after training.

[0067] In practical applications, the input layer in the question-answering sentence quality inspection model can process the question sentence and the answer sentence in the input question sentence separately, and then pass the processing results to the subsequent layers for processing. Figure 6B A structural diagram of another question-answer sentence quality inspection model is also provided. Figure 6B As shown, in the positive or negative sample of the question-answer sentence, the question sentence and the answer sentence need to be input separately at different input ends of the input layer.

[0068] The specific functions of each layer in the model are introduced below.

[0069] The input layer is used to segment the question sentence and the answer sentence of the question and answer sentence pair to be inspected, convert the segmentation result of the question sentence into a question word vector, and convert the segmentation result of the answer sentence into an answer word vector. Figure 6C A structural diagram of another question-answer sentence quality inspection model provided in an embodiment of the present application. Figure 6CThe input layer specifically includes the word segmentation processing layer and the word embedding layer. The word segmentation processing layer performs word segmentation on the input sentence, and the word embedding layer is used to complete the conversion of the word segmentation result to the word vector. Figure 6C The u1~un shown represent the word segmentation results of the question sentence, and v1~vm represent the word segmentation results of the answer sentence.

[0070] The encoding layer is used to encode the question word vector to obtain the question vector, and to encode the answer word vector to obtain the answer vector. The encoding layer includes a combination of one or more of the following encoders: CNN (Convolutional Neural Networks) encoder, LSTM (Long Short-Term Memory) encoder, BiLSTM (Bidirectional Long Short-Term Memory) encoder, BERT (a pre-trained model proposed by Google AI Research Institute) encoder, RoBERTa (an improved version of BERT) encoder or Word-Average (word vector average model) encoder. When a variety of different encoders are used, it is convenient for the model to learn more different knowledge during the training process and the use process, thereby improving the generalization ability of the model. For example, the encoding layer uses a CNN encoder and an LSTM encoder, where the CNN encoder can capture N-gram information, while the LSTM can encode text information relatively easily. In practical applications, there is no limitation on the combination and application of encoders.

[0071] exist Figure 6C In the illustrated model structure, the encoding layer uses a CNN encoder and an LSTM encoder, and also includes a splicing layer, the function of which is represented by a splicing symbol. The CNN encoder is used to encode the question word vector through the CNN structure to obtain a first question vector, and to encode the answer word vector through the CNN structure to obtain a first answer vector. The LSTM encoder is used to encode the question word vector through the LSTM structure to obtain a second question vector, and to encode the answer word vector through the LSTM structure to obtain a second answer vector. Figure 6C From the splicing symbols and the output content after splicing, it can be seen that the splicing layer is used to splice the first question vector and the second question vector into a question vector Qemb, and is used to splice the first reply vector and the second reply vector into a reply vector Aemb.

[0072] The classification input layer further expands the input content of the classification layer based on the question vector and the answer vector finally output by the encoding layer. Specifically, the classification input layer performs calculations based on the question vector and the answer vector to obtain two new vectors, called the first vector and the second vector.

[0073] Specifically, the classification input layer is used to process based on the question vector and the answer vector to obtain a first vector and a second vector, and then concatenate the question vector, the answer vector, the first vector and the second vector to obtain a concatenated vector. The first vector is used to represent the difference between the question vector and the answer vector, and the second vector is used to represent the relationship between the question vector and the answer vector.

[0074] In an optional implementation, the classification input layer obtains the absolute value of the difference between the question vector and the answer vector as the first vector based on the question vector and the answer vector. The classification input layer obtains the cross-correlation result between the question vector and the answer vector as the second vector based on the question vector and the answer vector. The concatenated vector can be expressed as follows:

[0075] [Qemb, Aemb, abs(Qemb-Aemb), Qemb*Aemb]

[0076] Where abs(Qemb-Aemb) represents the first vector, and Qemb*Aemb represents the second vector. The first and second vectors in the concatenated vector will provide richer information in the classification layer, which is convenient for improving the classification accuracy of the model, achieving the ultimate quality inspection purpose, and improving the quality inspection accuracy. The classification layer is used to learn based on the concatenated vector, predict the probability of mutual matching and mismatch between the question statement and the answer statement in the question and answer sentence pair to be inspected, and output the predicted probability.

[0077] See also Figure 6C The model structure shown in the figure may specifically include: a vector space fusion layer, a vector space conversion layer and a softmax function in the classification layer. The vector space fusion layer may ignore the order of the question vector and the answer vector based on the concatenated vector, especially the first vector, and realize the combination of the relationship between the question vector and the answer vector through the second vector. The vector space conversion layer is a hidden layer of the model learning classification. The softmax function is used to obtain various prediction probabilities, that is, the probability of matching and mismatching between the question statement and the answer statement.

[0078] pass Figure 6C It is not difficult to find that in the embodiment of the present application, a Figure 6C The Infersent-based question-answer sentence quality inspection model shown in the figure implements the quality inspection of the matching of the question-answer sentence to be inspected. Figure 6CIn the example, the encoding layer uses two encoders, CNN and LSTM, to respectively output word vectors of the input layer. As mentioned above, other forms of single-type encoders or encoders of different types can also be used. The trained question-answer sentence pair quality inspection model corresponding to different forms of encoding layers is subsequently tested using a test set. Among them, the test set can be a question-answer sentence pair extracted by collecting work orders in real time. After testing with the test set, the accuracy of the test set quality inspection results of the models corresponding to different forms of encoding layers is shown in Table 1 below.

[0079] Table 1 Comparison table of model test set quality inspection results accuracy and coding layer form

[0080]

[0081]

[0082] BiLSTM and Self-Attention (the structural unit of BERT) usually take context into consideration. Since the work order texts archived by asynchronous systems are usually long and contain a large number of useless words, models using relatively complex encoder structures are more likely to have overfitting problems. Models using Word-Average encoders have better generalization performance because Word-Average works better for tasks such as topic classification. In practical applications, you may not need to pay attention to the entire text, but only to certain words or sentences, and whether the topics of the question and answer are consistent.

[0083] Combined with the actual test results, the inventors also found that for the word embedding layer in the input layer, when the encoding layer uses a non-BERT series encoder, the word embedding layer can use Word2Vec (a model for converting words into vectors), Glove model, and ELMo model, and through testing, the quality inspection model obtained by training has better results.

[0084] For the output results of the model obtained through the above embodiments, in order to further increase the reliability of quality inspection and reduce the recognition error rate of the question-answer mismatch problem in the work order, the mismatched question-answer sentence pairs can also be submitted to manual review to obtain the final quality inspection results after multiple reviews.

[0085] Figure 7 The following is a schematic diagram of the scenario of multiple quality inspection question and answer sentence pairs in automatic and manual methods. Figure 7As shown, the question-answer sentence pair quality inspection device first obtains the question-answer sentence pair to be inspected, and then inputs it into the question-answer sentence quality inspection model to obtain the output result of the question-answer sentence quality inspection model. When the output result indicates that the quality inspection fails, the device then provides the question-answer sentence pair to be inspected to the manual quality inspection platform. After manual review, the manual quality inspection platform will feedback the manual quality inspection result of the question-answer sentence pair to the question-answer sentence pair quality inspection device. After the question-answer sentence pair quality inspection device obtains the manual quality inspection result of the manual quality inspection platform for the question-answer sentence pair to be inspected, if it is determined that the manual quality inspection result indicates that the question-answer sentence pair to be inspected fails the manual quality inspection, the question-answer sentence pair to be inspected is reported to the business system. The front-end business system can perform business processing on the question-answer sentence pair. For example, the work order from which the question-answer sentence pair comes is queried to obtain the account information provided by the user in the business system, so as to take measures such as apology and re-reply, thereby improving the user's question-answer experience.

[0086] Based on the question and answer sentence quality inspection method described in the above embodiment, the present application also provides a question and answer sentence quality inspection device. The specific implementation of the device is described below in conjunction with the embodiments and drawings. Figure 8 The figure shows the structure diagram of the question and answer sentence quality inspection device. Figure 8 In the device structure shown, it can be seen that the question and answer sentence quality inspection device includes:

[0087] A sentence pair acquisition unit 81 is used to acquire a question-answer sentence pair to be checked, wherein the question-answer sentence pair to be checked includes a question sentence and an answer sentence having a question-answer relationship;

[0088] A sentence pair quality inspection unit 82, used for inputting the question sentence and the answer sentence into a question and answer sentence quality inspection model, so as to determine the matching of the question sentence and the answer sentence through the question and answer sentence quality inspection model;

[0089] The output result acquisition unit 83 is used for obtaining the output result of the question and answer sentence quality inspection model through quality inspection result, and the output result is used to indicate whether the question and answer sentence pair to be quality inspected has passed the quality inspection.

[0090] Optionally, the question-answer sentence quality inspection device further includes: a model training unit. The model training unit is used to perform the following operations to train and obtain a question-answer sentence pair quality inspection model:

[0091] Constructing positive samples of question-answer sentence pairs and negative samples of question-answer sentence pairs, wherein the positive samples of question-answer sentence pairs include question sentences and answer sentences that have a question-answer relationship and match each other, and the negative samples of question-answer sentence pairs include question sentences and answer sentences that do not match each other;

[0092] Inputting the positive sample of the question-answer sentence pair and the negative sample of the question-answer sentence pair into the initial quality inspection model respectively;

[0093] According to the difference between the output result of the initial quality inspection model and the label of the sample input into the initial quality inspection model, the initial quality inspection model is adjusted until the training cutoff condition is met to obtain the question and answer sentence quality inspection model.

[0094] Optionally, the model training unit is used to disrupt the question-answer relationship between the question sentences and the answer sentences in the positive samples of the multiple question-answer sentence pairs and re-pair them to obtain the negative samples of the question-answer sentence pairs.

[0095] Optionally, the model training unit is used to select a plurality of target sample pairs from the constructed positive samples of the question-answer sentence pairs; and process the target sample pairs in any of the following ways to obtain negative samples of the question-answer sentence pairs:

[0096] Replace the question sentence in the target sample pair with the question sentence in other positive question and answer sentence samples to obtain a negative sample of the question and answer sentence pair; or replace the answer sentence in the target sample pair with the answer sentence in other positive question and answer sentence samples to obtain a negative sample of the question and answer sentence pair; or reverse the question sentence and the answer sentence in the target sample pair to obtain a negative sample of the question and answer sentence pair.

[0097] Optionally, the question and answer sentence quality inspection device further includes:

[0098] A first sending unit is configured to provide the question-answer sentence pair to be inspected to a manual quality inspection platform after obtaining the output result of the question-answer sentence quality inspection model when the output result indicates that the question-answer sentence pair to be inspected fails the quality inspection;

[0099] A receiving unit, used for obtaining a manual quality inspection result of the manual quality inspection platform on the question-answer sentence pair to be inspected;

[0100] The second sending unit is used to report the question and answer sentence pair to be checked to the business system when the manual quality inspection result indicates that the question and answer sentence pair to be checked fails the manual quality inspection, so that the business system performs business processing on the question and answer sentence pair to be checked.

[0101] Optionally, the statement pair acquisition unit 81 includes:

[0102] A denoising subunit is used to remove noise from the generated work orders to obtain the target work orders;

[0103] An information extraction subunit is used to extract question information and answer information with a question-answer relationship in a target work order;

[0104] The sentence construction subunit is used to use the question information to form the question sentence in the question-answer sentence pair to be checked, and use the answer information to form the answer sentence in the question-answer sentence pair to be checked.

[0105] Fig. 9 : This is a schematic diagram of a server structure provided in an embodiment of the present application. The server 900 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 922 (for example, one or more processors) and a memory 932, and one or more storage media 930 (for example, one or more mass storage devices) storing application programs 942 or data 944. Among them, the memory 932 and the storage medium 930 can be short-term storage or permanent storage. The program stored in the storage medium 930 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the central processing unit 922 can be configured to communicate with the storage medium 930 and execute a series of instruction operations in the storage medium 930 on the server 900.

[0106] The server 900 may also include one or more power supplies 926, one or more wired or wireless network interfaces 950, one or more input and output interfaces 958, and / or one or more operating systems 941, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0107] The steps performed by the server in the above embodiment can be based on the Fig. 9 The server structure shown.

[0108] The CPU 922 is used to execute the following steps:

[0109] Obtaining a question-answer sentence pair to be checked, wherein the question-answer sentence pair to be checked includes a question sentence and an answer sentence having a question-answer relationship;

[0110] Inputting the question sentence and the answer sentence into a question-answer sentence quality inspection model to determine the matching of the question sentence and the answer sentence through the question-answer sentence quality inspection model;

[0111] The output result of the question and answer sentence quality inspection model is obtained, and the output result is used to indicate whether the question and answer sentence pair to be quality inspected has passed the quality inspection.

[0112] The present application embodiment also provides another question and answer sentence quality inspection device, such as Fig.10For the sake of convenience, only the parts related to the embodiments of the present application are shown. For specific technical details not disclosed, please refer to the method part of the embodiments of the present application. The terminal can be any terminal device including a mobile phone, a tablet computer, a personal digital assistant (English full name: Personal Digital Assistant, English abbreviation: PDA), a sales terminal (English full name: Point of Sales, English abbreviation: POS), a car computer, etc., taking the terminal as a mobile phone as an example:

[0113] Fig.10 FIG. 1 is a block diagram showing a partial structure of a mobile phone related to a terminal provided in an embodiment of the present application. Fig.10 The mobile phone includes: a radio frequency (RF) circuit 1010, a memory 1020, an input unit 1030, a display unit 1040, a sensor 1050, an audio circuit 1060, a wireless fidelity (WiFi) module 1070, a processor 1080, and a power supply 1090. Those skilled in the art can understand that Fig.10 The mobile phone structure shown in the figure does not constitute a limitation on the mobile phone, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0114] Combine the following Fig.10 A detailed introduction to the various components of the mobile phone:

[0115] The RF circuit 1010 can be used for receiving and sending signals during information transmission or calls. In particular, after receiving the downlink information of the base station, it is sent to the processor 1080 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit 1010 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (full name: LowNoiseAmplifier, English abbreviation: LNA), a duplexer, etc. In addition, the RF circuit 1010 can also communicate with the network and other devices through wireless communication. The above-mentioned wireless communications may use any communication standard or protocol, including but not limited to Global System of Mobile communications (Global System of Mobile communication, English abbreviation: GSM), General Packet Radio Service (General Packet Radio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, English abbreviation: CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (Long Term Evolution, English abbreviation: LTE), e-mail, Short Messaging Service (SMS), etc.

[0116] The memory 1020 can be used to store software programs and modules. The processor 1080 executes various functional applications and data processing of the mobile phone by running the software programs and modules stored in the memory 1020. The memory 1020 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 1020 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0117] The input unit 1030 can be used to receive input digital or character information, and to generate key signal input related to the user settings and function control of the mobile phone. Specifically, the input unit 1030 may include a touch panel 1031 and other input devices 1032. The touch panel 1031, also known as a touch screen, can collect the user's touch operation on or near it (such as the user's operation on the touch panel 1031 or near the touch panel 1031 using any suitable object or accessory such as a finger, stylus, etc.), and drive the corresponding connection device according to a pre-set program. Optionally, the touch panel 1031 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into contact coordinates, and then sends it to the processor 1080, and can receive and execute commands sent by the processor 1080. In addition, the touch panel 1031 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic waves. In addition to the touch panel 1031, the input unit 1030 may further include other input devices 1032. Specifically, the other input devices 1032 may include but are not limited to one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like.

[0118] The display unit 1040 can be used to display information input by the user or information provided to the user and various menus of the mobile phone. The display unit 1040 may include a display panel 1041. Optionally, the display panel 1041 may be configured in the form of a liquid crystal display (full name in English: Liquid Crystal Display, English abbreviation: LCD), an organic light-emitting diode (full name in English: Organic Light-Emitting Diode, English abbreviation: OLED), etc. Further, the touch panel 1031 may cover the display panel 1041. When the touch panel 1031 detects a touch operation on or near it, it is transmitted to the processor 1080 to determine the type of touch event. Subsequently, the processor 1080 provides a corresponding visual output on the display panel 1041 according to the type of touch event. Although in Fig.10 In the embodiment, the touch panel 1031 and the display panel 1041 are used as two independent components to realize the input and output functions of the mobile phone, but in some embodiments, the touch panel 1031 and the display panel 1041 can be integrated to realize the input and output functions of the mobile phone.

[0119] The mobile phone may also include at least one sensor 1050, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 1041 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 1041 and / or the backlight when the mobile phone is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors that can be configured in the mobile phone, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be repeated here.

[0120] The audio circuit 1060, the speaker 1061, and the microphone 1062 can provide an audio interface between the user and the mobile phone. The audio circuit 1060 can transmit the received audio data to the speaker 1061 after converting the received audio data into an electrical signal, which is converted into a sound signal for output; on the other hand, the microphone 1062 converts the collected sound signal into an electrical signal, which is received by the audio circuit 1060 and converted into audio data, and then the audio data is output to the processor 1080 for processing, and then sent to another mobile phone through the RF circuit 1010, or the audio data is output to the memory 1020 for further processing.

[0121] WiFi is a short-range wireless transmission technology. The mobile phone can help users send and receive emails, browse web pages and access streaming media through the WiFi module 1070. It provides users with wireless broadband Internet access. Fig.10 A WiFi module 1070 is shown, but it is understandable that it is not an essential component of the mobile phone and can be omitted as needed without changing the essence of the invention.

[0122] The processor 1080 is the control center of the mobile phone. It uses various interfaces and lines to connect various parts of the entire mobile phone. It executes various functions of the mobile phone and processes data by running or executing software programs and / or modules stored in the memory 1020 and calling data stored in the memory 1020. Optionally, the processor 1080 may include one or more processing units; preferably, the processor 1080 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 1080.

[0123] The mobile phone also includes a power supply 1090 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 1080 through a power management system, so that the power management system can manage functions such as charging, discharging, and power consumption.

[0124] Although not shown, the mobile phone may also include a camera, a Bluetooth module, etc., which will not be described in detail here.

[0125] In the embodiment of the present application, the processor 1080 included in the terminal also has the following functions:

[0126] Obtaining a question-answer sentence pair to be checked, wherein the question-answer sentence pair to be checked includes a question sentence and an answer sentence having a question-answer relationship;

[0127] Inputting the question sentence and the answer sentence into a question-answer sentence quality inspection model to determine the matching of the question sentence and the answer sentence through the question-answer sentence quality inspection model;

[0128] The output result of the question and answer sentence quality inspection model is obtained, and the output result is used to indicate whether the question and answer sentence pair to be quality inspected has passed the quality inspection.

[0129] An embodiment of the present application also provides a computer-readable storage medium for storing program code, which is used to execute any implementation of the question and answer sentence quality inspection method described in the aforementioned embodiments.

[0130] An embodiment of the present application also provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute any one of the methods for quality inspection of question and answer statements described in the aforementioned embodiments.

[0131] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0132] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0133] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0134] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0135] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (full name in English: Read-Only Memory, English abbreviation: ROM), random access memory (full name in English: Random Access Memory, English abbreviation: RAM), disk or optical disk and other media that can store program codes.

[0136] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A question-answer sentence quality inspection method, characterized in that: include: Obtain a question-answer sentence pair to be inspected, wherein the question-answer sentence pair to be inspected includes a question sentence and an answer sentence having a question-answer relationship, and each of the question sentences includes content of multiple fields, and the fields include: question description, question type, and question details; Input the question sentence and the reply sentence into a question-answer sentence quality inspection model, so as to perform word segmentation on the input question sentence and the reply sentence respectively through the question-answer sentence quality inspection model, convert the word segmentation result of the question sentence into a question word vector, convert the word segmentation result of the reply sentence into a reply word vector, encode the question word vector to obtain a question vector, encode the reply word vector to obtain a reply vector, obtain the absolute value of the difference between the question vector and the reply vector as a first vector, obtain the cross-correlation result between the question vector and the reply vector as a second vector, splice the question vector, the reply vector, the first vector and the second vector to obtain a spliced ​​vector, and determine the matching of the question sentence and the reply sentence based on the spliced ​​vector; Obtaining an output result of the question-answer sentence quality inspection model, wherein the output result is used to indicate whether the question-answer sentence pair to be inspected has passed the quality inspection; The question-answer sentence quality inspection model is trained by the following steps: A positive sample of a question-answer sentence pair is obtained from the historical work orders archived in each batch, wherein the positive sample of the question-answer sentence pair includes a question sentence and an answer sentence having a question-answer relationship and a matching degree greater than a preset threshold, and the question sentence and the answer sentence in the positive sample of the question-answer sentence pair from the historical work orders archived in different batches are reorganized to obtain a negative sample of the question-answer sentence pair; The positive samples of the question-answer sentence pairs and the negative samples of the question-answer sentence pairs are respectively input into the initial quality inspection model. According to the difference between the output result of the initial quality inspection model and the labels of the samples input into the initial quality inspection model, the initial quality inspection model is adjusted until the training cutoff condition is met, so as to obtain the question-answer sentence quality inspection model.

2. The method according to claim 1, characterized in that: The question and answer sentences in the positive samples of the question and answer sentence pairs from the historical work orders archived in different batches are reorganized to obtain the negative samples of the question and answer sentence pairs, including: Select multiple target sample pairs from the positive samples of question-answer pairs from historical work orders archived in different batches; The target sample pair is processed in any of the following ways to obtain a negative sample of the question-answer sentence pair: Replace the question sentences in the target sample pairs archived in a batch with the question sentences in other question-answer sentence positive samples archived in other batches to obtain the negative samples of the question-answer sentence pairs; or, The answer statements in the target sample pairs archived in one batch are replaced with answer statements in other question-answer sentence positive samples archived in other batches to obtain negative samples of the question-answer sentence pairs.

3. The method according to claim 1, characterized in that: The question-answer sentence quality inspection model includes: an input layer, an encoding layer, a classification input layer and a classification layer; The input layer is used to segment the question sentence and the answer sentence of the input question-answer sentence pair to be inspected, convert the segmentation result of the question sentence into a question word vector, and convert the segmentation result of the answer sentence into an answer word vector; The encoding layer is used to encode the question word vector to obtain a question vector, and is used to encode the answer word vector to obtain an answer vector; The classification input layer is used to process based on the question vector and the answer vector to obtain a first vector and a second vector, and to concatenate the question vector, the answer vector, the first vector and the second vector to obtain a concatenated vector; The classification layer is used to predict the probability that the matching degree between the question sentence and the answer sentence in the question and answer sentence pair to be inspected is greater than the preset threshold and the probability that the matching degree is less than or equal to the preset threshold based on the concatenated vector learning, and output the predicted probability.

4. The method according to claim 3, characterized in that The coding layer includes a combination of one or more of the following encoders: CNN encoder, LSTM encoder, BiLSTM encoder, BERT encoder, RoBERTa encoder, or Word-Average encoder.

5. The method according to claim 4, characterized in that The encoding layer includes the CNN encoder, the LSTM encoder and a concatenation layer; The CNN encoder is used to encode the question word vector through the CNN structure to obtain a first question vector, and is used to encode the answer word vector through the CNN structure to obtain a first answer vector; The LSTM encoder is used to encode the question word vector through the LSTM structure to obtain a second question vector, and is used to encode the answer word vector through the LSTM structure to obtain a second answer vector; The concatenation layer is used to concatenate the first question vector and the second question vector into the question vector, and is used to concatenate the first answer vector and the second answer vector into the answer vector.

6. The method according to any one of claims 1 to 5, characterized in that: When the output result indicates that the question and answer sentence to be checked fails the quality check, after obtaining the output result of the question and answer sentence quality check model, the method further includes: Providing the question-answer sentence pairs to be inspected to the manual quality inspection platform; Obtaining manual quality inspection results of the manual quality inspection platform on the question-answer sentence pair to be inspected; When the manual quality inspection result indicates that the question and answer pair to be inspected fails the manual quality inspection, the question and answer pair to be inspected is reported to a business system so that the business system performs business processing on the question and answer pair to be inspected.

7. The method according to any one of claims 1 to 5, characterized in that: The step of obtaining the question-answer sentence pair to be inspected includes: Remove noise from the generated work orders and obtain the target work orders; Extract question information and answer information with question-answer relationship in the target work order; The question information is used to form the question statement in the question-answer statement pair to be checked, and the answer information is used to form the answer statement in the question-answer statement pair to be checked.

8. A question-answer sentence quality inspection device, characterized in that: include: A sentence pair acquisition unit is used to acquire a question-answer sentence pair to be checked, wherein the question-answer sentence pair to be checked includes a question sentence and an answer sentence having a question-answer relationship, and each question sentence includes content of multiple fields, wherein the fields include: question description, question type, and question details; A sentence pair quality inspection unit, used for inputting the question sentence and the reply sentence into a question-answer sentence quality inspection model, so as to perform word segmentation on the input question sentence and the reply sentence respectively through the question-answer sentence quality inspection model, converting the word segmentation result of the question sentence into a question word vector, converting the word segmentation result of the reply sentence into a reply word vector, encoding the question word vector to obtain a question vector, encoding the reply word vector to obtain a reply vector, obtaining the absolute value of the difference between the question vector and the reply vector as a first vector, obtaining the cross-correlation result between the question vector and the reply vector as a second vector, splicing the question vector, the reply vector, the first vector and the second vector to obtain a spliced ​​vector, and determining the matching of the question sentence and the reply sentence based on the spliced ​​vector; An output result acquisition unit, used for obtaining the output result of the question and answer sentence quality inspection model through quality inspection results, wherein the output result is used to indicate whether the question and answer sentence pair to be quality inspected has passed the quality inspection; A model training unit is used to obtain positive samples of question-answer sentence pairs from historical work orders archived in each batch, wherein the positive samples of question-answer sentence pairs include question sentences and answer sentences having a question-answer relationship and a matching degree greater than a preset threshold, and to reorganize the question sentences and answer sentences in the positive samples of question-answer sentence pairs from historical work orders archived in different batches to obtain negative samples of question-answer sentence pairs; the positive samples of question-answer sentence pairs and the negative samples of question-answer sentence pairs are respectively input into an initial quality inspection model, and the initial quality inspection model is adjusted according to the difference between the output result of the initial quality inspection model and the label of the sample input into the initial quality inspection model until the training cutoff condition is met to obtain the question-answer sentence quality inspection model.

9. The device according to claim 8, characterized in that The model training unit is used to select multiple target sample pairs from the positive samples of question-answer sentence pairs from historical work orders archived in different batches; replace the question sentences in the target sample pairs archived in one batch with the question sentences in other positive samples of question-answer sentence pairs archived in other batches to obtain negative samples of the question-answer sentence pairs; Alternatively, the answer statements in the target sample pairs archived in one batch are replaced with answer statements in other question-answer statement positive samples archived in other batches to obtain negative samples of the question-answer statement pairs.

10. The device according to claim 8, characterized in that The question-answer sentence quality inspection model includes: an input layer, an encoding layer, a classification input layer and a classification layer.

11. The device according to claim 10, characterized in that The coding layer includes a combination of one or more of the following encoders: CNN encoder, LSTM encoder, BiLSTM encoder, BERT encoder, RoBERTa encoder, or Word-Average encoder.

12. The device according to claim 11, characterized in that The encoding layer includes the CNN encoder, the LSTM encoder and a concatenation layer; The CNN encoder is used to encode the question word vector through the CNN structure to obtain a first question vector, and is used to encode the answer word vector through the CNN structure to obtain a first answer vector; The LSTM encoder is used to encode the question word vector through the LSTM structure to obtain a second question vector, and is used to encode the answer word vector through the LSTM structure to obtain a second answer vector; The concatenation layer is used to concatenate the first question vector and the second question vector into the question vector, and is used to concatenate the first answer vector and the second answer vector into the answer vector.

13. The device according to any one of claims 8 to 12, characterized in that The device also includes: A first sending unit is configured to provide the question-answer sentence pair to be inspected to a manual quality inspection platform after obtaining the output result of the question-answer sentence quality inspection model when the output result indicates that the question-answer sentence pair to be inspected fails the quality inspection; A receiving unit, used for obtaining a manual quality inspection result of the manual quality inspection platform on the question-answer sentence pair to be inspected; The second sending unit is used to report the question and answer sentence pair to be checked to the business system when the manual quality inspection result indicates that the question and answer sentence pair to be checked fails the manual quality inspection, so that the business system performs business processing on the question and answer sentence pair to be checked.

14. The device according to any one of claims 8 to 12, characterized in that The statement pair acquisition unit includes: A denoising subunit is used to remove noise from the generated work orders to obtain the target work orders; An information extraction subunit is used to extract question information and answer information with a question-answer relationship in a target work order; The sentence construction subunit is used to use the question information to form the question sentence in the question-answer sentence pair to be checked, and use the answer information to form the answer sentence in the question-answer sentence pair to be checked.

15. A question-answer sentence quality inspection device, characterized in that: The device comprises a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the question and answer sentence quality inspection method described in any one of claims 1-7 according to the instructions in the program code.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program code, and the program code is used to execute the question and answer sentence quality inspection method described in any one of claims 1-7.

Images