A business document-driven intelligent customer service construction method

Abstract

A method for constructing intelligent customer service driven by business documents, characterized in that: step 1, collecting the user's current input q, business document D and conversation history, encoding and pooling them to obtain their sentence-level representation; step 2, converting the sentence-level representation D of the business document into s Each item in is regarded as a discourse unit, and the association between the sentence level representation q currently input by the user and each discourse unit is established in turn, and each discourse unit is associated with the sentence level representation H of the dialogue history. s The association between them is used to realize document structure analysis and dialogue status tracking in turn; step 3, classifying the business documents and dialogue history based on the association relationship obtained in step 2, and reasoning about the current answer to generate the optimal strategy #imgabs0#The method of the present invention draws on rhetorical structure theory to realize corpus association and accurate reasoning, overcomes the problem of information slots, represents the system dialogue with the interactive state of discourse units, and smoothly drives the dialogue.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing, and more specifically, to a business document-driven intelligent customer service construction method. Background Art

[0002] The intelligent customer service system is an industry-oriented intelligent platform developed on the basis of large-scale knowledge processing, suitable for large-scale knowledge processing, natural language understanding, knowledge management, automatic question-answering system, reasoning and other technical industries. At present, the intelligent customer service system can not only provide enterprises with fine-grained knowledge management technology, but also establish a fast and effective technical means based on natural language for communication between enterprises and massive users, and at the same time provide enterprises with the statistical analysis information required for refined management. Its huge commercial value and academic significance have attracted the research interest of industry and academia.

[0003] In the prior art, intelligent customer service is mainly implemented through slot filling technology. Specifically, slot filling technology can first be used by business experts to define the service field, user intent, information slots and slot constraints to build the task ontology of intelligent customer service, and annotate large-scale conversation data based on the defined ontology. Based on large-scale annotated data, a machine learning model is trained to automatically identify user intent and fill in information slots. When all necessary information slots are filled, the customer service task is considered to be successfully completed.

[0004] This slot-filling-based intelligent customer service technology has two problems: first, the time and manpower required to construct the task body and label the data are expensive, and domain migration is difficult, resulting in a long time and high cost for the implementation of the intelligent customer service system; second, the information slot set is a flat structure, and different information slots are independent of each other, making it difficult to describe more complex customer service tasks in actual applications.

[0005] In addition, customer service methods that use business description documents to implement human-computer dialogue tasks are also common in the prior art. For example, the patent text CN112328774A discloses a method for implementing a multi-document task-based human-computer dialogue task, part of which discloses a multi-document dialogue task in the film field. This task allows the model to actively ask questions based on the knowledge of the movie description document and guess the movie the user wants based on the user's reply. This task uses the document as the knowledge source of the dialogue system. Background document Feng S, Wan H, Gunasekara C, et al. doc2dial: A Goal-Oriented Document-Grounded Dialogue Dataset [C] / / Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020. proposes a document-based goal-oriented (Goal-Oriented) doc2dial task (a document-based dialogue dataset that can retrieve sentences related to the question from the document and generate reasonable replies based on the results of the previous step) to build a multi-document, multi-domain dialogue system. However, at present, these document-based task-based dialogues mostly focus on using documents as external knowledge of the dialogue system to complete relevant information retrieval or recommendation, and it is difficult to complete complex reasoning and dialogue decisions.

[0006] On the other hand, business documents are very important business development guides in the automated integrated management platform, and they often contain a large number of complex business specifications, business processes or constraints. For example, as a type of business document, the business specification document includes complex constraints and procedures, such as "If the student's family has suffered a natural disaster in the past two years, they can apply for government relief compensation. The compensation amount for students in area A is within 10,000 yuan, and the compensation amount for students in area B is within 20,000 yuan." If these complex contents can be fully utilized, the intelligent customer service function can be fully expanded.

[0007] However, in order to answer the user's question, "Can I apply for government funding?", the intelligent customer service system usually needs to fully understand the user's question and the content described in the business document, and when necessary, actively ask the user for information and answer it step by step. For example, the intelligent customer service needs to fully understand the business document and correctly express the relationship between the basic chapter units in the document, and actively ask the user to add more information, such as "Has the family suffered from natural disasters?", "The year of the natural disaster?", "Where is the student located?", etc. The intelligent customer service system also needs to further formulate interactive behaviors based on the user's answer until the user's dialogue goal is achieved.

[0008] In order to enable the intelligent customer service system to proactively obtain necessary information from users, the dialogue model must know: 1. What requirements or processes does the current business have? 2. What are the sequence, trigger conditions, and constraints between these requirements or processes?

[0009] In order to solve the above problems, a technical solution for constructing an intelligent information consultation system based on government documents as background knowledge is proposed in the background literature Sam, Minner, Greg, et al. Using a phoneanswering machine to provide feedback to rural parents and improve student behavior [J]. Rural Special Education Quarterly, 1988, 9 (2): 33-36. A background literature Gao Y, Wu CS, Joty S, et al. EMT: Explicit Memory Tracker with Coarse-to-Fine Reasoning for Conversational Machine Reading [J]. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020. An explicit memory tracker (EMT) model is proposed, and the model is optimized in terms of the coarse granularity of sentence division and the lack of modeling of sentence node relationships. The Discern model (a recognition model) is proposed, and the relationship between the dialogue history and the document discourse unit is established using the self-attention mechanism, which achieves better results than EMT. However, all of the above methods lack sufficient parsing and representation of the document structure, making it difficult to accurately track the global dialogue state.

[0010] In summary, building a task-based dialogue system based on business description documents requires solving two major challenges. First, how to construct the ontology knowledge of the dialogue task through documents. Second, how to guide the dialogue through the dialogue ontology knowledge constructed from documents.

[0011] In view of the above problems, the present invention proposes a business document driven intelligent customer service construction method. Summary of the invention

[0012] In order to solve the deficiencies in the prior art, the purpose of the present invention is to provide a business document driven intelligent customer service construction method.

[0013] The present invention draws on the Rhetorical Structure Theory (RST) to model the dialogue ontology into several discourse units (also known as elementary discourse units, EDU), and realizes the graph structure through the association between these discourse units. In addition, the present invention also draws on the task-based dialogue method of slot filling technology to represent the dialogue state of the system as the interaction state on each discourse unit, thereby realizing the selection of the optimal strategy in the current answer, thereby driving the smooth progress of the dialogue.

[0014] The present invention adopts a document-driven task-oriented dialogue model (DG-TOD, Document Grounded Task-Oriented Dialogue), which can conduct multiple rounds of dialogues based on natural language with users with business documents as the background, so as to answer questions raised by users. Specifically, the model in the present invention takes business documents, user current input and dialogue history as input, and outputs the current answer in natural language form. Inspired by the slot-filling task-based human-computer dialogue method, the document-driven task-oriented dialogue model in the present invention adopts six different modules to implement the method in the present invention in sequence. These six modules include a discourse unit division module, a text encoding module, a document structure analysis module, a dialogue state tracking module, a dialogue strategy reasoning module and a natural language generation module.

[0015] Specifically, the discourse unit division module can divide the text content that undertakes a specific task into discourse units with punctuation as the main segmentation feature. Subsequently, the text encoding module can encode the corresponding text in the form of character splicing to obtain the representation of various text words, and use the pooling algorithm to convert the text word level representation into the sentence level representation. The document structure analysis module can establish an association between the user's current input and the business document, thereby realizing the analysis of the business document structure. The dialogue state tracking module can associate the business document with the dialogue history, thereby realizing the tracking of the dialogue state. The dialogue strategy reasoning module can infer some discourse units that best match the user's current input and the association between discourse units based on the pre-analysis of the document structure analysis module and the dialogue state tracking module, thereby generating the optimal strategy. The natural language generation module can generate the current answer expressed in natural language according to the optimal strategy, thereby realizing the construction and output of intelligent customer service.

[0016] The present invention adopts the following technical solution.

[0017] A method for constructing intelligent customer service driven by business documents, wherein the method comprises the following steps: Step 1, collecting the user's current input Q, business document D and conversation history H, encoding and pooling them, so as to obtain the sentence level representation q of the user's current input Q, the sentence level representation D of the business document s and the sentence-level representation H of the dialogue history H s ; Step 2, represent the sentence level of the business document D s Each item in is regarded as a discourse unit, and the association between the sentence-level representation q of the user's current input Q and each discourse unit is established in turn, and the association between each discourse unit and the sentence-level representation H of the dialogue history H is established. s The association between them is realized in turn, thereby realizing document structure analysis and dialogue state tracking; Step 3, based on the association relationship obtained in Step 2, the business documents and dialogue history H are classified, and the current answer is inferred by the reasoning method to generate the optimal strategy As the current answer.

[0018] According to the rhetorical structure theory, the user's current input Q, business document D and conversation history H are divided into discourse units; the divided discourse units are encoded after character splicing, and the encoding is achieved through the RoBERTa model.

[0019] Preferably, the sentence level representation q of the current input is obtained by: collecting the current input of the user Among them, L is the sample sequence of the user's current input Q The sample length is d, and d is the encoding length; the user's current input Q is pooled to obtain the sentence level representation of the user's current input Q = Pooling() is the pooling formula.

[0020] Preferably, collect business documents Wherein, M is the total length of the business document, and n is the number of discourse units in the business document; the business document D is pooled to obtain the sentence-level representation of the business document Among them, e1,…,e n is a discourse unit in the business document.

[0021] Preferably, the conversation history is collected Among them, N

[0022] is the total length of the conversation history H, and m is the number of dialogue segments; the dialogue history H is pooled to obtain the sentence-level representation of the dialogue history H

[0023] Among them, h1,…,h m is the dialogue segment in the dialogue history H.

[0024] Preferably, in step 2, an attention network model is used to generate a sentence-level representation q of the user's current input Q and a sentence-level representation D of the business document. s Each discourse unit e1,…,e n The relationship between them; in step 2.1.2, the text structure encoder is used to calculate the matrix [e1,…,e n ,q] between any two nodes of the edge relationship type L k ; Step 2.1.3, based on the edge relationship strength A k and edge relationship type L k , for discourse units e1,…,e n The graph structure is associated with the sentence-level representation q of the user's current input Q, and an updated representation of the business document based on discourse units is generated.

[0025] Preferably, the text structure encoder is a relation-excited graph encoder or a structure-excited graph encoder.

[0026] Preferably, the relation-motivated graph encoder is: a linear rectification function is used to calculate the query vector and the key-value vector obtained in the multi-head attention mechanism to obtain a matrix [e1,…,e n ,q] between any two nodes of the edge relationship type L k .

[0027] Preferably, the structure-excited graph encoder is: Assume that the matrix [e1,…,e n ,q], the basis of the vector space of the edge between any two nodes is The query vector Q obtained using the multi-head attention mechanism k and the key-value vector K k , the weight w of the basis of the vector space of the edge k,ixjz Calculate; where ix and jx are e1,…,e n The numbers of any two nodes in the edge-based vector space L base and weight w k,ixjx , using a multi-head attention mechanism for the matrix [e1,…,e n ,q] to update.

[0028] Preferably, the basis L of the edge-based vector space base and weight w k,ixjx , using a multi-head attention mechanism for the matrix [e1,…,e n ,q] to update the matrix at time t-1 Calculate the edge relationship type L k , where k is the number of heads in the multi-head attention mechanism, L k The basis L of the vector space of edges base and weight w ij The product of; using multi-head attention mechanism to update the matrix [e1,…,e n ,q], each vector e i The representation of each attention head A feedforward neural network is used to fuse the representation of each attention head to update the matrix [e1,…,e n ,q], each vector e i The expression at time t

[0029] Preferably, the matrix [e1,…,e n ,q] The convergence condition for the update is the hyperparameter T d ; When the convergence condition T is met d After that, the updated representation of the business document based on the discourse unit is obtained

[0030] Preferably, in step 2, a self-attention mechanism is used to generate a sentence-level representation D of the business document. s Each discourse unit e1,…,e n and each dialogue segment h1,…,h in the dialogue history H m The relationship between in, is the representation of the discourse unit in the business document after being associated with the conversation history H, is the dialogue paragraph in the dialogue history H after being associated with the discourse unit, T i is the number of layers of the self-attention mechanism.

[0031] Step 3 also includes, based on the association relationship obtained in step 2, using a classifier to classify the states of the discourse units and calculating the state vector of each discourse unit.

[0032] Preferably, the status classification of the discourse unit includes a conforming state, a conflicting state and a pending state.

[0033] Preferably, step 3 also includes: step 3.1, inputting the updated representation of the business document obtained in step 2 based on the discourse unit into the reasoner Representation of discourse units in business documents associated with the conversation history H The state vector s of each discourse unit j ; Step 3.2, based on the reasoning of the reasoner, obtain the text library related to the user's current input Q; Step 3.3, extract the optimal strategy from the text library through the pooling algorithm, classification algorithm and maximum function in sequence To achieve the current answer.

[0034] Preferably, the types of current answers include affirmative, negative and interrogative types; wherein, an affirmative or negative current answer is generated based on the conforming state of the discourse unit in the optimal strategy; and an interrogative current answer is generated based on the pending state of the discourse unit in the optimal strategy.

[0035] Preferably, the cross entropy loss of the classifier in the process of classifying the state of the discourse unit is calculated, and the loss is recorded as L state ; Calculate the optimal strategy for extracting from the text base The cross entropy loss of the reasoner in the process is recorded as L strategy ; for L state and L strategy The sum is performed to obtain the loss function of the intelligent customer service method.

[0036] Preferably, the current answer also includes using a neural network decoder to optimize the strategy Decode to obtain the natural language representation of the current answer and output it as intelligent customer service.

[0037] The beneficial effect of the present invention is that, compared with the prior art, the business document-driven intelligent customer service construction method of the present invention can refer to the rhetorical structure theory to model the dialogue ontology into discourse units, thereby realizing the association between the business document, the user's current input Q and the dialogue history based on discourse units, and realizing the reasoning of the current answer based on the association. The method of the present invention effectively overcomes the problem that the traditional information slot is difficult to represent complex dialogue tasks, and represents the dialogue state of the system as the interactive state on each discourse unit, thereby smoothly driving the entire dialogue.

[0038] The beneficial effects of the present invention also include:

[0039] 1. The present invention selects the RoBERTa model for encoding. This model is an innovation of the BERT model. It improves the pre-training method and adopts a dynamic mask, which makes the training time longer and the training data more. Since the calculation methods in the present invention are all performed on discourse units, the training results obtained by this model can better meet the requirements of the algorithm in the present invention and have stronger adaptability.

[0040] 2. Different from the content in the prior art, in order to obtain more accurate customer service results, not only the user's current input and business documents are used, but also the conversation history between the customer service system and the user is taken into account, so that the method of the present invention can provide both user needs and business functions at the same time, making the results more accurate.

[0041] 3. In the present invention, two different attention mechanisms are selected and used in the two association processes. Business documents are generated based on professional methods and can be pre-organized before the algorithm. Therefore, although there is a lot of content, a simple splicing method can be used to achieve the consideration and aggregation of multiple different aspects of information. However, the system cannot accurately predict the content of the conversation history and the logical properties between multiple conversation paragraphs in these conversation contents, so it is necessary to adopt a dynamic weight generation method to obtain more accurate results. It can be seen that the method of the present invention takes into account the characteristics of two different text contents at the same time, while taking into account the aggregation amount of information and the accuracy of the calculation results.

[0042] 4. In order to evaluate whether the algorithm of the present invention is accurate, the present invention performs estimation in both the process of determining the status classification of the discourse unit and the process of determining the type of the current answer, thereby fully realizing the accuracy of the evaluation method. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 The present invention is a flowchart of the steps of a business document-driven intelligent customer service construction method. DETAILED DESCRIPTION

[0044] The present application is further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present application.

[0045] Figure 1 The following is a flow chart of the steps of a business document driven intelligent customer service construction method of the present invention. Figure 1 As shown, a business document-driven intelligent customer service construction method includes steps 1 to 3.

[0046] Step 1: Collect and encode the user's current input Q, business document D, and conversation history H to obtain the sentence-level representation q of the user's current input Q and the sentence-level representation d of the business document. s and the sentence-level representation H of the dialogue history H s .

[0047] In order to implement the input of the document-driven task-based dialogue model in the present invention, the text content to be input can first be segmented and reconstructed based on the rhetorical structure theory.

[0048] Specifically, Rhetorical Structure Theory (RST) believes that a text is composed of multiple parts, which are also called discourse units. Discourse units are mostly presented in a core-satellite relationship, and also include a core-nucleus relationship. Relative to the author's communicative intention, the discourse unit that occupies a core position is in a relatively important position, and there are many different discourse relationships between the core discourse unit and the satellite discourse unit, such as cause, result, solution, condition, and evaluation.

[0049] Specifically, the user's current input Q, business document D and conversation history H are divided into discourse units according to rhetorical structure theory; the divided discourse units are encoded after character splicing, and the encoding is achieved through the RoBERTa model.

[0050] In the present invention, when using rhetorical structure theory to preliminarily divide the content of a text, such as a chapter, the chapter can be preliminarily segmented using punctuation marks and other contents as the main features, thereby decomposing the chapter into multiple independent discourse units. In the subsequent algorithm of the present invention, these discourse units are regarded as unit elements describing the business, and encoding, association and reasoning are realized.

[0051] After the preliminary division of the text is achieved, the character splicing form of these texts is obtained, and they are encoded based on the character splicing form. In the present invention, since the user's current input Q, business documents and conversation history H are selected as inputs respectively, the character splicing forms of the user's current input Q, business documents and conversation history H can be obtained respectively, and these contents are input into the encoder for encoding. In the present invention, the RoBERTa model is selected for encoding. This model is an innovation of the BERT model. It improves the pre-training method and adopts dynamic masks, which makes the training time longer and the training data more. Since the calculation methods in the following text of the present invention are all carried out for discourse units, the training results obtained by this model can better meet the requirements of the algorithm in the present invention and are more adaptable.

[0052] After the present invention obtains the word-level representation of the user's current input Q, business document and conversation history H, it can be assumed that the user's current input Q is a question, the business document is n discourse units, and the conversation history H contains m conversations that have occurred, so the user's current input Q, business document and conversation history H are respectively represented as follows after encoding: and Specifically, L, M, and D are the sample sequences of the user's current input Q. The sample length, the total length of the business document and the total length of the conversation history H, where d is the encoding length of the RoBERTa model.

[0053] Specifically, the sentence level representation q of the user's current input Q is obtained by: collecting the user's current input Among them, L is the sample sequence of the user's current input Q The sample length is d, and d is the encoding length; the user's current input Q is pooled to obtain the sentence level representation of the user's current input Q Pooling() is the pooling formula.

[0054] It can be understood that, in the present invention, a pooling method is used to obtain a sentence-level representation of the user's current input Q. A similar method can also be used to obtain a sentence-level representation of a business document and a conversation history H.

[0055] Specifically, collect business documents in, n is the number of discourse units in the business document; the business document D is pooled to obtain the sentence level representation of the business document Among them, e1,…,e n It is a discourse unit in business documents.

[0056] In addition, collect conversation history in, m is the number of dialogue segments; the dialogue history H is pooled to obtain the sentence-level representation of the dialogue history H Among them, h1,…,h m is the dialogue segment in the dialogue history H.

[0057] Step 2: Represent the sentence level of the business document as D s Each item in is regarded as a discourse unit, and the association between the sentence-level representation q of the user's current input Q and each discourse unit is established in turn, and the association between each discourse unit and the sentence-level representation H of the dialogue history H is established. s The association between them is realized in turn to realize document structure analysis and dialogue state tracking.

[0058] First, rhetorical structure theory believes that the structural relationship of a document can be composed of the semantic relationship between each basic discourse unit. Therefore, the basic discourse unit can be summarized into a hierarchical tree structure, which can ensure the coherence of the text. In order to construct the relationship between discourse units in this way, it is necessary to first construct the semantic relationship between different discourse units, and then use the graph structure to realize the representation of this relationship.

[0059] In order to realize the semantic representation of different discourse units in the graph structure, the present invention adopts the attention network model. Specifically, in step 2, the attention network model is used to generate the sentence level representation q of the user's current input Q and the sentence level representation D of the business document. s Each discourse unit e1,…,e n The correlation between them; in step 2.1.1, a multi-head attention mechanism is used to calculate the matrix [e1,…,e m ,q]’s attention matrix A k , which is used to characterize the strength of the edge relationship between any two nodes in the matrix; Step 2.1.2, using the text structure encoder, calculate the matrix [e1,…,e n ,q] between any two nodes of the edge relationship type L k ; Step 2.1.3, based on the edge relationship strength A k and the edge relationship type L k , for discourse units e1,…,e n The graph structure is associated with the sentence-level representation q of the user's current input Q, and an updated representation of the business document based on discourse units is generated.

[0060] It can be understood that in the present invention, a multi-head attention mechanism is first used to form a matrix of the discourse units of the business document generated in step 1 and the sentence-level representation of the user's current input Q, and the attention algorithm is used to solve the attention matrix. The present invention adopts a multi-head attention mechanism, and the number of heads is k, so the present invention adopts A k To represent the attention matrix.

[0061] The calculation formula of the attention matrix is: Among them, softmax() is a classification function, Q k is the query value of the k-head attention mechanism, and K k is the key value of the k-head attention mechanism, and and That is to say, k Identifies the dimensions of the text content in each header.

[0062] In the present invention, if the acquired attention matrix A k By splicing and linear transformation, multiple learning nodes can be obtained, that is, e1,…,e n The relationship strength matrix A between each of them. Each value in the relationship strength matrix can represent the edge relationship strength between any two nodes.

[0063] Next, the present invention adopts the chapter structure encoder to the matrix [e1,…,e n ,q] to obtain the edge relationship type between each learning node. This is because the connection relationship between multiple discourse units extracted from the text content is not only related to the strength of the connection, but also directly related to the type of connection. For this reason, after obtaining the edge relationship strength, the present invention also realizes the calculation of the edge relationship type.

[0064] The method of the present invention provides two paragraph structure encoders for realizing different edge relationship type learning and representation methods, namely, a relation-powered graph (RPG) encoder and a structure-powered graph (SPG) encoder.

[0065] Among them, the relation-inspired graph encoder can assume that the relationship between two nodes is determined by the semantics of the two nodes. Therefore, by modeling the number of layers of the stacked network, the hierarchical relationship between multiple learning nodes can be obtained.

[0066] Specifically, a linear rectification function is used to calculate the query vector and key value vector obtained in the multi-head attention mechanism to obtain the matrix [e1,…,e n,q] between any two nodes of the edge relationship type L k .

[0067] According to the calculation results of the query value and key value obtained by the multi-head attention mechanism model in the previous link, that is, and The tensor L can be used k To indicate the edge relationship type between two nodes. In this formula, there is a corresponding edge relationship type between any two nodes ix and jx In this formula, ReLU() is the linear rectification function, q k,ix ||k k,jx is the concatenation of the corresponding values ​​in the query value vector and the key value vector, is the slope of the linear rectification function, h k is the vector dimension of the edge space. Usually, in order to fully realize the comparison between every two nodes, d k =h k , that is, the edge space is the sum of the spaces of the two nodes. In addition, It can be understood as the intercept of the linear rectification function.

[0068] The edge relationship type L obtained in this way k The calculation process is relatively simple and fast. However, the problem with this algorithm is that the edge relationship types learned are poorly interpretable and difficult to correspond to the relationship types in rhetorical structure theory, and thus it is difficult to determine the correctness of the learned relationship types. Therefore, the present invention also provides a second encoder, namely, a structure-motivated graph encoder.

[0069] In the structure-excited graph encoder, assume that the matrix [e1,…,e n ,q], the basis of the vector space of the edge between any two nodes is Among them, ix and jx are e1,…,e n The numbers of any two nodes in the query vector Q obtained by the multi-head attention mechanism k and the key-value vector K k , the weight w of the basis of the vector space of the edge k,ixjx Calculate; based on the basis L of the edge vector space base and the weight w k,ixjx , using a multi-head attention mechanism for the matrix [e1,…,e n ,q] to update.

[0070] It can be understood that in the present invention, based on the relation-motivated graph encoder, it can be further assumed that the spatial dimension of the edge is finite. In other words, given a specific expectation and a specific application field of the corpus, the number of types of basic relationships between two nodes is limited. As mentioned above, the types of basic relationships between two nodes can be cause relationships, result relationships, and so on. Thus, the vector space of the edge can be assumed to be composed of s h-dimensional linear vectors. Further, these s h-dimensional linear vectors can be defined as basis vectors.

[0071] When obtaining the vector representation, we can first calculate the weight of each basis vector, and then multiply the basis vector by a matrix composed of multiple weights to obtain the edge vector between every two nodes in the multiple nodes, and thus deduce the edge relationship type between every two nodes in the multiple nodes.

[0072] Specifically, we can first assume that the basis of the edge vector space is Then the input node matrix is Thus, the weight of the basis corresponding to each edge is calculated. The calculation formula for the weight of the basis corresponding to the edge between two nodes ix and jx is

[0073] w k,ixjx =softmax(W k,E (q k,ix ||k k,jx )+b)

[0074] In this formula, q k,ix ||k k,jx are the query value of node ix and the key value of node jx obtained by the k-head attention mechanism. After solving the weight of the basis through the classification algorithm, the weight is multiplied by the basis vector matrix to obtain the value of the edge relationship type of each edge.

[0075] Specifically, Among them, k,ixjx Is the edge relationship type L k The value of the element in the ixth row and jxth column of the matrix. k Finally, the representation of the edge relationship type can be combined with the node representation to use a multi-head attention mechanism to deal with [e1,…,e n ,q] to update.

[0076] Specifically, in, is the value item (Value) calculated using the multi-head attention mechanism, is the edge relationship type calculated in the previous process. The two are connected and solved. a ixjx ∈A. Through this method, the value items of all other nodes that have edge relationships with node ix can be connected to the edge relationships, and the multi-head attention mechanism can be used to generate weights and perform weighted calculations to obtain the representation of the new node ix based on each head in the multi-head attention mechanism.

[0077] Subsequently, the present invention uses a feedforward neural network (FNN) to fuse multiple heads. The formula FFN(x) = max(0, xW1+b1)W2+b2 is a general hypothesis function of a feedforward neural network. The network usually adopts a multi-layer recursive form, so that neurons in adjacent layers are fully connected instead of cross-layer connections to achieve modeling. Therefore, in this hypothesis function, there are multiple layers of fully connected layers.

[0078] In the present invention, the calculation formula of FNN used is: In this formula, are all t-1, that is, the matrix [e1,…,e n ,q] in e ix The value of the vector, that is, the representation of the node ix at the previous moment, is calculated after multi-head attention. There are k representations in the content of this connection. Multiply by the coefficient W o After the calculation of the feedforward neural network model, the new representation of the node ix at the next moment, that is, at time t, is obtained. In this formula, Among them, d i is the dimension of the internal vector in the feedforward neural network model. In one embodiment of the present invention, d i =4d.

[0079] After multiple iterations in the manner described above in the present invention, the result of the updated matrix can be obtained, that is, the updated new representation of each discourse unit.

[0080] It should be noted that the matrix [e1,…,e n ,q] The convergence condition for the update can be a preset hyperparameter T d , the method for obtaining the hyperparameter can refer to any hyperparameter method setting in the prior art. Since the present invention does not involve this part of the improvement, it will not be described in detail in this article; when the convergence condition T is met d After that, the updated representation of the business document based on the discourse unit is obtained

[0081] The method of the present invention will first perform an initialization process before the model starts. During the initialization process, the number of iterations can be reasonably set according to the needs of the algorithm. In one embodiment of the present invention, a maximum number of iterations T can be set. d , when the number of iterations is equal to the maximum number of iterations, the iteration process can be ended and the representation of the updated discourse unit can be generated.

[0082] The subsequent algorithm will perform the next application or operation on the updated discourse unit. Therefore, in order to distinguish it from the content before the update, it is recorded as This matrix not only includes the content of the original discourse unit, but also should include the edge relationship content between multiple discourse units and between the user's current input Q.

[0083] Secondly, in step 2, the self-attention mechanism is used to generate the sentence-level representation D of the business document s Each discourse unit e1,…,e n and each dialogue segment h1,…,h in the dialogue history H m The relationship between in, is a representation of a discourse unit in the business document after being associated with the conversation history H, is the dialogue paragraph in the dialogue history H after being associated with the discourse unit, T i is the number of layers of the self-attention mechanism.

[0084] It should be noted that, in the present invention, after obtaining the association relationship between the user's current input Q and the business document, that is, the representation of the discourse unit in the updated business document Afterwards, a similar approach may be used to associate dialogue paragraphs with discourse units.

[0085] In the present invention, unlike the previous association content, the self-attention mechanism is used in the second association process. Different from the content in the prior art, the present invention selects two different attention mechanisms in two different association processes. This is because, in order to obtain customer service results more accurately, the present invention not only uses the user's current input Q and business documents, but also takes into account the conversation history H between the customer service system and the user. However, although the content of the business document and the user's conversation history H are both textual content, their characteristics are very different. Since the business document is generated based on a professional method and can be pre-organized before the algorithm, although there are a lot of contents, a simple splicing method can be used to achieve the consideration and aggregation of multiple different aspects of information. The conversation history H is different. The system cannot accurately predict the content of the conversation history H and the logical attributes between multiple conversation paragraphs in these conversation contents. Therefore, the self-attention mechanism can be used to achieve the dynamic generation of the relationship weights between longer input sequences, so that the generated current answer results are more accurate. It can be seen that the method of the present invention simultaneously considers the characteristics of two different text contents, and takes into account the aggregation amount of information and the accuracy of the calculation results.

[0086] Since the calculation method of the self-attention mechanism is a common content in this field, it will not be described in detail here. This method can obtain the representation of the conversation history H inside the system. It should be noted that this representation is based on the representation of each chapter unit. That is, after the self-attention mechanism model is passed, Among them, T i is the number of layers of the self-attention mechanism.

[0087] Step 3: Classify the discourse units based on the association relationship obtained in step 2, and use the reasoning method to reason about the current answer to generate the optimal strategy. As the current answer.

[0088] Specifically, in order to classify the discourse units, the discourse units and the user's current input Q

[0089] To confirm the relationship between , we can consider collecting the representations of the discourse units associated with the user's current input Q in this step. and the representation of discourse units associated with the dialogue history H

[0090] Step 3 also includes, based on the association relationship obtained in step 2, using a classifier to classify the states of the discourse units and calculating the state vector of each discourse unit.

[0091] The classifier used in the present invention is Softmax(). Based on this classifier and the representations of two different discourse units input into the classifier, the classification of discourse units can be obtained. Specifically, the probability function In this formula, for and The iyth link in

[0092] After calculation by this classifier, we get the vector The specific method of classifying this vector is to combine it with the trainable parameters Perform multiplication to obtain For example, in the present invention, the state classification of the discourse unit includes a conforming state, a conflicting state, and a pending state. iy Three different types of differentiation can be obtained.

[0093] Step 3 also includes, step 3.1, inputting the updated representation of the business document based on the discourse unit obtained in step 2 into the reasoner Representation of discourse units in business documents associated with the conversation history H The state vector s of each dialogue segment j ; Step 3.2, based on the reasoning of the reasoner, obtain the text library related to the user's current input Q; Step 3.3, extract the optimal strategy from the text library through the pooling algorithm, classification algorithm and maximum function in sequence To achieve the current answer.

[0094] After classifying the discourse units, the current dialogue strategy can be inferred based on the structure of the business document and the content of the dialogue history H. The input of the reasoner in the present invention may include the updated representation of the discourse unit and And the state vector s of each discourse unit j That is, the initial input to the reasoner is

[0095] The reasoning process of the reasoner is iterative. For the time t, the reasoning result in the present invention is obtained based on the result of the previous time t-1, that is, the calculation formula is Where jy is the current node, and iy is all other nodes other than jy. By calculating and summing the relationship between this node and other nodes associated with it, the output of the inference engine at the current iteration can be obtained. s After the step of reasoning, reasoning can be realized and the representation of the user's current input Q and the representation of the discourse unit can be updated. The representation of the matrix where the discourse unit is located is now updated to The representation of the discourse unit at this time can be used as a discourse library for subsequent use in generating the current answer.

[0096] Specifically, the present invention can extract the optimal strategy from the discourse library by using average pooling, classification algorithm and maximum function in sequence. The calculation formula is:

[0097]

[0098] In this formula, Calculate the formula to get the value of the optimal strategy.

[0099] The types of current answers include affirmative, negative and interrogative. Among them, an affirmative or negative current answer is generated based on the conforming state of the discourse unit in the optimal strategy. An interrogative current answer is generated based on the pending state of the discourse unit in the optimal strategy.

[0100] It should be noted that the optimal strategy can be a clear answer to the user's question. For example, when the optimal strategy can accurately determine whether the discourse unit is in a suitable state, it provides a "YES" or "NO" answer. However, in most cases, the optimal strategy cannot determine the state of the discourse unit in the optimal strategy. At this time, the discourse unit is in a pending state. Therefore, the optimal strategy can generate a related question to the user, thereby further narrowing the scope according to the user's reply until an accurate answer is generated. In this case, the content constructed by the customer service is of the "Inquiry" type.

[0101] It should be noted that each discourse in the discourse library is obtained based on discourse units in business documents and dialogue paragraphs in the dialogue history H. In order to evaluate whether the algorithm of the present invention is accurate, the present invention also provides a loss function calculation method of cross entropy loss.

[0102] Specifically, since the present invention performs estimation in both the process of determining the state classification of the discourse unit and the process of determining the type of the current answer, the loss function of the present invention is also obtained by summing the losses in these two steps.

[0103] Calculate the loss of the classifier in the process of classifying the state of the discourse unit, and record the loss as L state ; Calculate the optimal strategy to extract the above from the text library The cross entropy loss of the reasoner in the process is recorded as L strategy ; for L state and L strategy The sum is performed to obtain the loss function of the intelligent customer service method.

[0104] Specifically, L totalloss =L strategy +λL state , the total loss is the weighted sum of the two, where the proportional relationship is also called the harmonic factor and its value range is 0≤λ≤1.

[0105] In this formula, It can be seen that L state The distance between each actual distribution and the expected distribution can be obtained, where

[0106] on the other hand, This formula is the loss function of the optimal strategy, which is solved using cross entropy loss. Where S is the correct strategy label.

[0107] In this way, not only the loss of the classifier in the process of state classification can be taken into account, but also the optimal strategy can be extracted from the text library. The cross entropy loss of the reasoner in the process is reduced, thereby fully realizing the accuracy of the evaluation method.

[0108] Current answers also include using a neural network decoder to find the optimal strategy Decode to obtain the natural language representation of the current answer and output it as intelligent customer service.

[0109] It is understandable that in order to realize the answer to the user, it is necessary to reconstruct the content in the optimal strategy by natural language processing to generate human language that the user can understand. After this step, the final current answer is generated, and the answer is output as the result of the intelligent customer service construction.

[0110] The beneficial effect of the present invention is that, compared with the prior art, the business document-driven intelligent customer service construction method of the present invention can refer to the rhetorical structure theory to model the dialogue ontology into discourse units, thereby realizing the association between business documents, current user input and dialogue history based on discourse units, and realizing the reasoning of the current answer based on the association. The method of the present invention effectively overcomes the problem that traditional information slots are difficult to represent complex dialogue tasks, and represents the dialogue state of the system as the interactive state on each discourse unit, thereby smoothly driving the entire dialogue.

[0111] The applicant of the present invention has made a detailed explanation and description of the implementation examples of the present invention in conjunction with the drawings in the specification. However, those skilled in the art should understand that the above implementation examples are only preferred implementation schemes of the present invention, and the detailed description is only to help readers better understand the spirit of the present invention, but not to limit the scope of protection of the present invention. On the contrary, any improvements or modifications based on the inventive spirit of the present invention should fall within the scope of protection of the present invention.

Claims

1. A business document driven intelligent customer service construction method, characterized in that: The method comprises the following steps: Step 1: Collect the user's current input Q, business document D and conversation history H, encode and pool them to obtain the sentence level representation q of the user's current input Q and the sentence level representation D of the business document. s and the sentence-level representation H of the conversation history H s ; The sentence level representation q of the current user input Q is obtained as follows: Collect the user's current input Where L is the sample sequence of the user's current input Q The sample length is , and d is the encoding length; Pool the user's current input Q to obtain a sentence-level representation of the user's current input Q Pooling() is the pooling formula; Collect the business documents Wherein, M is the total length of the business document, and n is the number of discourse units in the business document; Pooling is performed on the business document D to obtain a sentence-level representation of the business document Among them, e1,...,e n is a discourse unit in the business document; Collect the conversation history Where N is the total length of the conversation history H, and m is the number of dialogue segments; Pooling the conversation history H to obtain a sentence-level representation of the conversation history H Among them, h1, ..., h m is a dialogue segment in the dialogue history H; Step 2: The sentence level representation D of the business document s Each item in is regarded as a discourse unit, and the association between the sentence level representation q of the user's current input Q and each discourse unit is established in sequence, and the association between each discourse unit and the sentence level representation H of the dialogue history H is established. s The association between them is realized in turn to realize document structure analysis and dialogue state tracking; The attention network model is used to generate the sentence level representation q of the user's current input Q and the sentence level representation D of the business document s Each discourse unit e1,...,e n The relationship between them; Step 2.1.1, use the multi-head attention mechanism to calculate the matrix [e1, ..., e n ,q]’s attention matrix A k , the attention matrix A k Used to represent the matrix [e1, ..., e n , the edge relationship strength between any two nodes in q]; Step 2.1.2, using the text structure encoder, calculate the matrix [e1, ..., e n , q] between any two nodes of the edge relationship type L k ; Step 2.1.3, based on the edge relationship strength A k and the edge relationship type L k , for the discourse units e1,...,e n The graph structure is associated with the sentence level representation q of the user's current input Q, and an updated representation of the business document based on the discourse unit is generated. The self-attention mechanism is used to generate the sentence-level representation D of the business document. s Each discourse unit e1,...,e n Each dialogue segment h1, ..., h in the dialogue history H m The relationship between in, is a representation of a discourse unit in the business document associated with the conversation history H, is a dialogue paragraph in the dialogue history H associated with the discourse unit, T i is the number of layers of the self-attention mechanism; Step 3: Classify the business document and the conversation history H based on the association relationship obtained in step 2, and use the reasoning method to reason about the current answer to generate the optimal strategy. As said current answer; The step 3 also includes: Step 3.1: inputting the updated representation of the business document based on the discourse unit obtained in step 2 into the reasoner Representation of the discourse unit in the business document associated with the conversation history H The state vector s of each discourse unit j ; The state vector s of each of the discourse units j The result is calculated by classifying the states of the discourse units using a classifier based on the association relationship obtained in step 2; Step 3.2, based on the reasoning of the reasoner, obtaining a text library related to the user's current input Q; Step 3.3, extracting the optimal strategy from the text library through the pooling algorithm, classification algorithm and maximum function in sequence To achieve the current answer; The optimal strategy is extracted from the discourse library by using average pooling, classification algorithm and maximum function in sequence. The optimal strategy is: In the formula, h is the dimension of the inference engine input.

2. A method for constructing intelligent customer service driven by business documents according to claim 1, characterized in that: According to the rhetorical structure theory, the user's current input Q, business document D and conversation history H are divided into discourse units; The divided discourse units are encoded after character splicing, and the encoding is achieved through the RoBERTa model.

3. A method for constructing intelligent customer service driven by business documents according to claim 1, characterized in that: The text structure encoder is a relation-excited graph encoder or a structure-excited graph encoder.

4. A method for constructing intelligent customer service driven by business documents according to claim 2, characterized in that: The relation-excited graph encoder is: The query vector and key value vector obtained in the multi-head attention mechanism are calculated using a linear rectification function to obtain the matrix [e1, ..., e n , q] between any two nodes of the edge relationship type L k .

5. A method for constructing intelligent customer service driven by business documents according to claim 2, characterized in that: The structure-excited graph encoder is: Assume that the matrix [e1, ..., e n ,q], the basis of the vector space of the edge between any two nodes is The query vector Q obtained using the multi-head attention mechanism k and the key-value vector K k , the weight w of the basis of the edge vector space k，ixjx Calculate; where ix and jx are e1, ..., e n The numbers of any two nodes in ; The basis L of the vector space based on the edge base and the weight w k，ixjx , a multi-head attention mechanism is used to analyze the matrix [e1, ..., e n ,q] to update.

6. A method for constructing intelligent customer service driven by business documents according to claim 5, characterized in that: The basis L of the edge-based vector space base and the weight w k，ixjx , a multi-head attention mechanism is used to analyze the matrix [e1, ..., e n ,q] to update specifically include: For the matrix at time t-1 Calculate the edge relationship type L k , where k is the number of heads of the multi-head attention mechanism, L k is the basis L of the vector space of the edge base and the weight w ij The product of The multi-head attention mechanism is used to update the matrix [e1, ..., e n ,q], each vector e i The representation of each attention head A feedforward neural network is used to fuse the representations of each attention head to update the matrix [e1, ..., e n ,q], each vector e i The expression at time t 7. A method for constructing intelligent customer service driven by business documents according to claim 5, characterized in that: The matrix [e1, ..., e n ,q] The convergence condition for the update is the hyperparameter T d ; When the convergence condition T is met d After that, the updated representation of the business document based on the discourse unit is obtained 8. A method for constructing intelligent customer service driven by business documents according to claim 5, characterized in that: The state classification of the discourse unit includes a conforming state, a conflicting state and a pending state.

9. A method for constructing intelligent customer service driven by business documents according to claim 8, characterized in that: The types of the current answers include affirmative, negative and inquiry; among them, generating a current answer of an affirmative type or a negative type based on the conformity status of the discourse unit in the optimal strategy; A current answer of the inquiry type is generated based on the pending status of the discourse unit in the optimal strategy.

10. A method for constructing intelligent customer service driven by business documents according to claim 1, characterized in that: Calculate the cross entropy loss of the classifier in the process of classifying the state of the discourse unit, and record the loss as L state ; Calculate the optimal strategy to extract the optimal strategy from the text library The cross entropy loss of the reasoner in the process is recorded as L strategy ; For L state and L strategy The sum is performed to obtain the loss function of the intelligent customer service method.

11. A method for constructing intelligent customer service driven by business documents according to claim 1, characterized in that: The current answer also includes, The optimal strategy is decoded using a neural network Decode to obtain the natural language representation of the current answer and output it as intelligent customer service.

Images