Human-computer interaction method, human-computer interaction model training method and electronic device

Abstract

Provided in the embodiments of the present application are a human-computer interaction method, a human-computer interaction model training method and an electronic device. The human-computer interaction method comprises: determining a knowledge graph corresponding to input question data, and by means of a graph model, obtaining, from the knowledge graph, node data corresponding to a node matching the question data; on the basis of the node data, generating input data which can be accepted by a first interactive language model, wherein the first interactive language model is a language model used for executing an open-ended task; and on the basis of the input data and the question data, performing answer generation by means of the first interactive language model, so as to generate answer data corresponding to the question data. By means of the embodiments of the present application, not only can the accuracy of task processing be ensured, but the capability of an interactive language model processing an open-ended task can also be fully utilized, such that the solution has better openness and can be flexibly applicable to various scenarios and environments.

Description

Human-computer interaction and model training method, and electronic equipment

[0001] This application claims priority to the Chinese patent application filed on January 15, 2024, with application number 202410056450.6 and invention name “Human-computer interaction and model training method thereof, and electronic device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] Embodiments of the present application relate to the field of artificial intelligence technology, and in particular to a human-computer interaction method, a training method for a human-computer interaction model, and an electronic device. Background Art

[0003] In the field of artificial intelligence, two key models are graph models (GMs) and interactive language models capable of performing open-ended tasks, such as large language models (LLMs). Large language models are deep learning models trained on massive amounts of text data. They possess powerful open-ended task processing capabilities, enabling them to generate natural language text, deeply understand its meaning, and handle various natural language tasks (such as text summarization, question-answering, and translation). Graph models, on the other hand, excel at predefined tasks and are largely incapable of performing open-ended tasks.

[0004] To enable graph models to leverage the capabilities of interactive language models with open-ended task execution capabilities, such as LLMs, to improve their task performance, one approach, as shown in Figure 1, first processes the graph data using the LLM, leveraging its extensive knowledge to enhance the textual attributes of nodes. The GM then generates predictions. While this approach enables the GM to accurately predict predefined tasks, it still cannot handle open-ended tasks.

[0005] Therefore, how to effectively combine interactive language models such as LLM and GM to fully utilize the ability of interactive language models to handle open-ended tasks has become an urgent problem to be solved. Summary of the Invention

[0006] In view of this, an embodiment of the present application provides a human-computer interaction and model training solution to at least partially solve the above problems.

[0007] According to a first aspect of an embodiment of the present application, a human-computer interaction method is provided, comprising: determining a knowledge graph corresponding to input question data, and obtaining node data corresponding to nodes in the knowledge graph that match the question data through a graph model; generating input data acceptable to a first interactive language model based on the node data, wherein the first interactive language model is a language model for performing open-ended tasks; and generating a reply based on the input data and the question data through the first interactive language model to generate reply data corresponding to the question data.

[0008] According to a second aspect of an embodiment of the present application, a human-computer interaction model training method is provided, comprising: obtaining node attribute text of a sample node in a knowledge graph sample, and generating a node representation of the sample node through a pre-trained graph model; based on the node attribute text, generating a node description text for describing the sample node through a generator; constructing modal alignment data based on the node representation and the node description text; using the modal alignment data, performing a first-stage training on a translation model that connects the graph model and a pre-trained first interactive language model to obtain a translation model capable of converting the output of the graph model into input acceptable to the first interactive language model; wherein the first interactive language model is a language model for performing open-ended tasks.

[0009] According to a third aspect of an embodiment of the present application, another human-computer interaction method is provided, including: determining a form text attribute graph corresponding to an input form question, and obtaining node data corresponding to nodes in the form text attribute graph that match the form question through a graph model, wherein the nodes in the form text attribute graph correspond to form entities, and the form text attributes of the nodes are used to describe form structure information and / or form content information; based on the node data, generating input data that can be accepted by a first interactive language model, wherein the first interactive language model is a language model for performing open tasks; based on the input data and the form question, generating a reply through the first interactive language model to generate reply data corresponding to the form question.

[0010] According to a fourth aspect of an embodiment of the present application, another human-computer interaction method is provided, including: determining a user text attribute graph corresponding to an input user information question, and obtaining node data corresponding to nodes in the user text attribute graph that match the user information question through a graph model, wherein the nodes in the user text attribute graph correspond to user entities, and the user text attributes of the nodes are used to describe the user's attribute information; based on the node data, generating input data that can be accepted by a first interactive language model, wherein the first interactive language model is a language model for performing open tasks; based on the input data and the user information question, generating a reply through the first interactive language model to generate reply data corresponding to the user information question.

[0011] According to the fifth aspect of the embodiments of the present application, an electronic device is provided, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other through the communication bus; the memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to the method described in the first aspect, the second aspect, the third aspect or the fourth aspect.

[0012] According to a fourth aspect of an embodiment of the present application, a computer storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the method described in the first aspect, the second aspect, the third aspect, or the fourth aspect is implemented.

[0013] According to the solution provided by the embodiment of the present application, in order to effectively combine and utilize the capabilities of the interactive language model and the graph model, after determining the knowledge graph corresponding to the question data, the node data of the nodes in the knowledge graph that match the question data is first extracted through the graph model GM. Although GM can determine the node data more accurately, it lacks the ability to generate open-ended responses, and the node data cannot be accepted by the interactive language model (i.e., the first interactive language model) such as LLM, and there is a modal difference between the two. Therefore, in the embodiment of the present application, based on the determined node data, input data that can be accepted by the interactive language model (i.e., the first interactive language model) such as LLM is generated, and the interactive language model (i.e., the first interactive language model) generates open-ended responses based on the input data and the question data using its own open task processing capabilities. As a result, the human-computer interaction solution of the embodiment of the present application can not only ensure the accuracy of task processing, but also make full use of the ability of the interactive language model to process open tasks, so that the solution has good openness and can be flexibly applied to various scenarios and environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0015] FIG1 is a schematic diagram of a combination of LLM and GM in the related art;

[0016] FIG2 is a schematic diagram of an exemplary system applicable to the embodiment of the present application;

[0017] FIG3A is a flowchart of the steps of a human-computer interaction model training method according to the first embodiment of the present application;

[0018] FIG3B is a schematic structural diagram of a human-computer interaction model in the embodiment shown in FIG3A ;

[0019] FIG3C is a schematic diagram of an example of a training process in the embodiment shown in FIG3A ;

[0020] FIG4A is a flowchart of the steps of a human-computer interaction method according to the second embodiment of the present application;

[0021] FIG4B is a schematic diagram of an example scenario in the embodiment shown in FIG4A ;

[0022] FIG4C is a schematic diagram of another example scenario in the embodiment shown in FIG4A ;

[0023] FIG5 is a schematic structural diagram of an electronic device according to a third embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field should fall within the scope of protection of the embodiments of the present application.

[0025] The specific implementation of the embodiment of the present application is further explained below in conjunction with the accompanying drawings of the embodiment of the present application.

[0026] Figure 2 shows an exemplary system applicable to the embodiments of the present application. As shown in Figure 2, the system 100 may include a cloud service 102, a communication network 104 and / or one or more user devices 106, with Figure 2 showing multiple user devices as an example.

[0027] The cloud server 102 can be any suitable device for storing information, data, programs, and / or any other suitable type of content, including but not limited to a distributed storage system, a server cluster, a computing cloud server cluster, etc. In some embodiments, the cloud server 102 can perform any suitable function. For example, in some embodiments, the cloud server 102 can be used to conduct question-and-answer interactions based on a graph model and an interactive language model. As an optional example, in some embodiments, after receiving question data, the cloud server 102 can determine the corresponding knowledge graph and, using the graph model, extract node data for nodes that match the question data. Based on the node data, the cloud server 102 can then generate input data acceptable to a first interactive language model. The first interactive language model can then generate open-ended responses based on the input data and the question data. The first interactive language model is a language model used to perform open-ended tasks. As another example, in some embodiments, the cloud server 102 can receive question data sent from a user device 106 and return generated response data to the user device 106. As another example, in some embodiments, the cloud server 102 can also train the models involved in the aforementioned question-and-answer interactions.

[0028] In some embodiments, the communication network 104 can be any suitable combination of one or more wired and / or wireless networks. For example, the communication network 104 can include any one or more of the following: the Internet, an intranet, a wide area network (WAN), a local area network (LAN), a wireless network, a digital subscriber line (DSL) network, a frame relay network, an asynchronous transfer mode (ATM) network, a virtual private network (VPN), and / or any other suitable communication network. The user device 106 can be connected to the communication network 104 via one or more communication links (e.g., communication link 112), and the communication network 104 can be linked to the cloud service end 102 via one or more communication links (e.g., communication link 114). The communication link can be any communication link suitable for transmitting data between the user device 106 and the cloud service end 102, such as a network link, a dial-up link, a wireless link, a hard-wired link, any other suitable communication link, or any suitable combination of such links.

[0029] The user device 106 may include any one or more user devices suitable for interacting with a user. In some embodiments, the user device 106 may receive question data in text form input by the user, or receive question data in voice form and convert it into text form, and then send it to the cloud service end 102. As an optional example, the user device 106 also receives reply data returned by the cloud service end 102 based on the question data, and displays or plays the reply data to the user. In some embodiments, the user device 106 may include any suitable type of device. For example, in some embodiments, the user device 106 may include a mobile device, a tablet computer, a laptop computer, a desktop computer and / or any other suitable type of user device.

[0030] Based on the above system, this application describes the above human-computer interaction scheme through multiple embodiments. To facilitate understanding of the scheme of the embodiments of this application, the following first describes the training process of the human-computer interaction model, and then describes the human-computer interaction method based on the trained human-computer interaction model.

[0031] Example 1

[0032] The following describes the training method of the human-computer interaction model according to an embodiment of the present application in conjunction with FIG. 3A to FIG. 3C .

[0033] First, referring to Figure 3B, a schematic diagram of the structure of a human-computer interaction model according to an embodiment of the present application is shown. As shown in Figure 3B, the human-computer interaction model includes a graph model, a generator, a translation model, and a first interactive language model.

[0034] in:

[0035] Graph models are used to process graph-structured data, such as knowledge graph data. They can be implemented as any appropriate graph neural network model structure, including but not limited to GraphSAGE (Graph SAmple and aggreGatE) model, GNN (Graph Neural Network), GCN (Graph Convolutional Networks) model, and so on.

[0036] The generator is mainly used in the model training phase, and is used to construct a node description text for describing the node's own characteristics, node neighbor characteristics, and the commonalities between the two based on the node attribute text collected from the knowledge graph. The node description text can be used together with the node representation generated by the graph model to construct modal alignment data for translation model training, so that in the model reasoning phase, the human-computer interaction model can use its translation model to generate input that can be accepted by the first interactive language model based on the node representation output by the graph model and the question data. It can be seen that in the model reasoning phase, the generator is no longer required to participate in the processing. In one feasible way, the generator can be implemented based on the second interactive language model. Because the input of the generator is the node attribute text, it can be directly processed as input by the second interactive language model. Optionally, the second interactive language model can be a trained language model that is higher-order than the first interactive language model and can perform open tasks, including but not limited to an LLM model.

[0037] The translation model is connected between the graph model and the first interactive language model to perform modality alignment between the two, thereby converting the output of the graph model into input acceptable to the first interactive language model. Exemplarily, the translation model can be implemented based on an encoder structure.

[0038] The first interactive language model is used to perform open-ended task processing, such as open-ended response generation, based on the output of the translation model (carrying information of the graph nodes). Exemplarily, the first interactive language model can also be implemented as an LLM model.

[0039] Below, based on the above model structure, the training method of the human-computer interaction model of the embodiment of the present application is described.

[0040] 3A , there is shown a flowchart of the steps of a method for training a human-computer interaction model according to the first embodiment of the present application.

[0041] The training method of the human-computer interaction model of this embodiment includes the following steps:

[0042] Step S202: Obtain the node attribute text of the sample node in the knowledge graph sample, and generate the node representation of the sample node through the pre-trained graph model.

[0043] One of the purposes of training the human-computer interaction model in this embodiment is to bridge the modal differences between the graphical model and the first interactive language model. Therefore, for the graphical model in the human-computer interaction model, a pre-trained graphical model can be used, and the parameters of the graphical model are frozen during the training process, that is, the parameters of the graphical model remain unchanged during the training process of the human-computer interaction model.

[0044] In one feasible approach, a knowledge graph can be implemented as a TAG (Text Attributed Graph). A TAG is a type of graph that uses text as node attributes. In addition to the graph's structural information, it also provides important textual information about the node through its corresponding text attributes. Therefore, it is necessary to consider both the graph's structural information and its textual information, as well as the relationship between the two. However, precisely because of this, the attribute information of its nodes can be directly collected and used.

[0045] Specifically in this embodiment, TAG is represented as The graph model takes GNN as an example, represented by g θ (A, X). Where, A node set representing N nodes of TAG; A∈{0,1} N×N , represents the adjacency matrix of TAG; for Each node υ in is associated with a sequential text feature, denoted as s υ . And for g representing GNN θ (A, X), θ represents the set of learnable parameters, and X represents the node representation processed by shallow methods such as BoW model (bag of words model) or skip-gram model (skip-gram model). Taking GraphSAGE as an example, a fixed number of neighbor nodes N(υ) are sampled around the target node υ, and then the previous layer vector of the target node υ is added. and the aggregated neighbor node vector Splicing, get the local graph information for the target node υ As follows:

[0046] Among them, σ() represents the activation function, W k Represents the weight matrix. Then, the pre-trained graph model gθ * Local graph information for nodes Encode and get the node representation of the target node υ, denoted as z υ , g θ * (A, X)v.

[0047] The node attribute text of the sample node can be directly obtained through the attribute information in the text form of the node.

[0048] Step S204: Based on the node attribute text, a generator generates a node description text for describing the sample node.

[0049] In order for the subsequent first interactive language model to effectively understand the node representation output by the graph model, modal alignment data needs to be constructed, and the node description text used to describe the sample nodes is an important component of the modal alignment data.

[0050] For a particular node, while its node attribute text can represent the node's attributes, it still cannot fully and objectively express the node's characteristics. Therefore, in one feasible approach, a generator can generate node information text used to represent the characteristics of the sample node itself, neighbor information text used to represent the characteristics of the sample node's neighbor nodes, and common information text used to represent the commonalities between the sample node and its neighbor nodes based on the node attribute text of the sample node and the node attribute text of the sample node's neighbor nodes. Based on the node information text, neighbor information text, and common information text, a node description text is generated to describe the sample node. This node description text can more comprehensively and objectively represent the characteristics of the node.

[0051] As previously mentioned, the generator can be implemented using a second interactive language model that can perform open-ended tasks. Based on this, the process of generating node description text can be implemented as follows: based on the node attribute text of the sample node, the second interactive language model in the generator performs node feature analysis and processing to generate node information text that characterizes the sample node's own characteristics; based on the node attribute text of the sample node's neighbor nodes, the second interactive language model performs neighbor node feature analysis and processing to generate neighbor information text that characterizes the sample node's neighbor node characteristics; based on the node information text and neighbor information text, the second interactive language model performs node commonality analysis and processing to generate commonality information text that characterizes the commonalities between the sample node and its neighbor nodes. The second interactive language model itself has relatively strong reasoning and open-ended task execution capabilities. Based on this, one feasible approach is to use the second interactive language model to construct high-quality node description text based on the CoT (Chain of Thought) method. CoT generates output by inducing step-by-step reasoning of the dialogue agent. It is an improved prompting strategy that can improve the performance of second interactive language models, such as LLM, in complex reasoning tasks. In this way, the obtained node description text can be made more accurate.

[0052] Taking the second interactive language model as the second LLM type as an example, an exemplary generation process of guiding the second LLM to construct node description text along the above three key dimensions is as follows:

[0053] Node information text: In TAG, each node has node attribute text, which is regarded as one of the features of each node. It should be noted that in this embodiment, node attribute text is used to describe the attributes of each node in TAG. Node attribute text can be converted into text vector form through a bag-of-words model or a word vector model. Based on this, the generator can use LLM to summarize and analyze the text vector of the node to obtain the description information of the node's own features, which is called node information text and is recorded as

[0054] Neighbor information text: A subset of the neighboring nodes of the current sample node can be randomly extracted and their representations can be aggregated to generate a neighbor information vector. For example, the node attribute text of the sample node itself and the neighbor nodes can be fused together by serial splicing and fusion, and then LLM can be used to perform neighbor node feature analysis to summarize and analyze the neighbor nodes. The attributes of the neighbor node are used to obtain the description information of the neighbor node characteristics, which is called the neighbor information text and is recorded as

[0055] Commonality information text: Considering that GraphSAGE is run as a low-pass filter, in order to indicate the similarity between the sample node and its neighbor nodes for smoothing purposes, this example will also guide LLM to integrate and summarize the common information between the two. For example, LLM can be used according to and Perform node commonality analysis to infer the relationship between the sample node v and its neighbor nodes The commonality between them is obtained, and the common information text is recorded as

[0056] Having obtained and After that, connect and You can build high-quality node description text t v ,Right now

[0057] Step S206: constructing modal alignment data based on the node representation and node description text of the sample node.

[0058] After obtaining the node representation z of the sample node υ and node description text t υ After that, the modal alignment data can be constructed Among them, N P is a subset of node N.

[0059] Step S208: Using the modality alignment data, a first-stage training is performed on a translation model connecting the graph model and the pre-trained first interactive language model to obtain a translation model capable of converting the output of the graph model into input acceptable to the first interactive language model.

[0060] The first interactive language model is a language model used to perform open tasks, including but not limited to LLM.

[0061] In this step, the graph model and the first interactive language model are both pre-trained models. Due to the modal differences between the two, the first interactive language model cannot directly understand the node representation. Therefore, in this embodiment, the translation model between the graph model and the first interactive language model is trained based on the constructed modal alignment data, so that the trained translation model can not only convert the output of the graph model into input acceptable to the first interactive language model, such as input in text form or input in token form, but also make the node representation of the node closer to its node description text, so as to achieve accurate conversion from node representation to text representation.

[0062] To this end, one feasible approach is to perform feature extraction on the node representations and node description text in the modality alignment data to obtain the corresponding node feature vectors and node text vectors. The translation model is then trained in the first phase based on the attention processing results of the node feature vectors and node text vectors. This attention processing can produce more effective feature vectors.

[0063] Optionally, based on the attention processing results of the node feature vector and the node text vector, training the translation model may include: performing self-attention processing on the preset learnable word element vector and the text word element vector corresponding to the node text vector to obtain the corresponding learnable word element attention features and node text attention features; performing cross-attention processing on the learnable word element attention features and the node feature vector to obtain the node attention features; and performing the first stage of training the translation model based on the difference between the node attention features and the node text attention features. This approach allows for more effective feature extraction while also effectively fusing the two features to obtain a more comprehensive and objective feature vector.

[0064] In a specific example, for a pair (z v ,t v ), using two encoders, denoted as f z (·) and f t (·), to extract the language features for alignment. Among them, for the node description text t v , using the text encoder f t(·) (such as BERT) to extract features, that is, f t (t v ) to obtain the corresponding node text vector T v =f t (t v ). Among them, f t (·) can be implemented as an encoder containing 12 layers of Transformer blocks. v , you can use the encoder f z (·) for feature extraction. z (·) can also be implemented as an encoder based on the Transformer architecture. In the specific implementation, the encoder can take M learnable token vectors as input. The M learnable token vectors are called query tokens and are expressed as Via encoder f z (·) processing, output M features Moreover, H υ =f z (Q, z υ ).

[0065] In order to extract z υ Zhong and t υ The most relevant information, in the specific implementation, Q is designed to use the self-attention layer for self-attention processing, and then through the cross attention layer and z υ Perform cross attention processing to v interact and coordinate f t and f z The self-attention layer between υ to interact.

[0066] Based on the above process, the node attention feature H can be obtained υ and node text attention feature T υ , and then, H can be calculated according to the preset loss function, such as the cross entropy loss function, etc. υ and T υ The difference is the loss value, and the translation model is trained in the first stage according to the difference until the termination condition of the first stage training is met, such as reaching a preset number of training times, or the loss value reaches a preset threshold, etc.

[0067] Through the first stage of training, the translation model can serve as an effective modal conversion bridge between the graph model and the interactive language model. However, in order to make the interaction between the translation model and the interactive language model smoother and to further improve the modal conversion accuracy of the translation model, in one feasible way, a two-stage training method is adopted for the human-computer interaction model of the embodiment of the present application. Based on this, on the basis of completing the first stage of training, the modal alignment data can be further used as input, and the modal alignment data can be processed by the translation model that has completed the first stage of training, and the processing results can be converted into parameters that can be accepted by the first interactive language model; the first interactive language model generates responses based on the parameters, and based on the generated responses, the translation model undergoes second stage training.

[0068] In a specific implementation, when the reply is generated based on the parameter by the first interactive language model, the parameter can be used as the prompt data of the first interactive language model, combined with the question sample data, as the input of the first interactive language model; the reply is generated based on the input by the first interactive language model. Then, based on the generated reply, the translation model is trained in the second stage. The data output by the translation model is introduced into the interactive language model in the form of prompt data, and used as the input of the interactive language model together with the question sample data. Then, based on the generated reply, the translation model is tuned again in combination with the modal alignment data input to the translation model, i.e., the second stage of training. In this way, the node representation output by the graph model can be better aligned with the input vector acceptable to the first interactive language model through the translation model, so that the first interactive language model can generate corresponding reply data more accurately and effectively.

[0069] Below, the training process of the above-mentioned human-computer interaction model is exemplified in the form of a specific example, as shown in Figure 3C. In the example shown in Figure 3C, the example of the graph model is a pre-trained GraphSAGE model (illustrated as a "frozen graph model" in Figure 3C), the example of the first interactive language model is a pre-trained LLM model (specifically ChatGLM2-6B, illustrated as a "frozen first LLM" in Figure 3C), the example of the generator (illustrated as "Producer" in Figure 3C) is based on the second interactive language model (specifically an LLM model, illustrated as a "second LLM" in Figure 3C) structure implementation, the example of the translation model (illustrated as "Translator" in Figure 3C) is based on the encoder structure implementation, and the knowledge graph used for training is a knowledge graph in the form of TAG. In this example, there is no restriction on the specific knowledge field of the knowledge graph, and it can be set and selected by those skilled in the art according to actual needs.

[0070] This exemplary training process includes:

[0071] Step 1: Obtain a pre-trained graph model.

[0072] Pre-training of the graph model, specifically GraphSAGE in this example, can be accomplished using conventional methods and will not be detailed in this embodiment of the present application. The pre-trained graph model is capable of executing predefined tasks. In this example, the pre-trained GraphSAGE is capable of processing graph-structured data, including but not limited to extracting node features from graph-structured data such as knowledge graphs. For example, using the knowledge graph as a tag, GraphSAGE is capable of extracting node features from the tag.

[0073] For example, GraphSAGE can sample a fixed number of neighbor nodes N(υ) around the target node υ, and then replace the previous layer vector of the target node υ with and the aggregated neighbor node vector Splicing, get the local graph information for the target node υ As follows:

[0074] Among them, σ() represents the activation function, W k Represents the weight matrix. Then, the pre-trained graph model g θ * Local graph information for nodes Encode and get the node representation of the target node v, recorded as z v , g θ * (A,X) v .

[0075] It should be noted that, regardless of the first or second stage of training, the parameters of GraphSAGE remain unchanged, as shown in the “frozen graph model” in FIG3C .

[0076] Step 2: Obtain the pre-trained first interactive language model.

[0077] In this example, the first interactive language model is specifically the first LLM. The first LLM is pre-trained on a broad text corpus, acquiring a wealth of knowledge. During the first phase of training, the LLM's parameters remain unchanged, illustrated as the "frozen first LLM" in Figure 3C. Taking the specific LLM CHATGLM2-6B as an example, it employs an autoregressive blank-filling approach involving a "destruction and reconstruction" strategy, in which portions of the original text are masked (destructed) and subsequently predicted (reconstructed). Through extensive training on a large corpus, CHATGLM2-6B has acquired the ability to retain a wealth of knowledge and provide reasonable answers to queries.

[0078] Process 3: The processing of the generator in the first stage of training.

[0079] In this example, the Producer is implemented using the second LLM. When generating high-quality node description text, it builds the node description text based on the CoT method from three dimensions: node information text, neighbor information text, and common information text. The second LLM is also a pre-trained LLM.

[0080] Specifically:

[0081] Node information text: In TAG, each node has node attribute text, which is considered as one of the features of each node. In this example, node attribute text is used to describe the attributes of each node in TAG. Node attribute text can be converted into text vector form through a bag-of-words model or a word vector model. Based on this, the generator can use the second LLM to summarize and analyze the text vector of the node to obtain the description information of the node's own features, which is called node information text and recorded as

[0082] Neighbor information text: A subset of the neighboring nodes of the current sample node can be randomly sampled and their representations can be aggregated to generate a neighbor information vector. For example, the node attribute text of the sample node itself and the neighbor nodes can be fused together by weighted summation or serial fusion, and then the second LLM can be used to perform neighbor node feature analysis to summarize and analyze the neighbor nodes. The attributes of the neighbor node are used to obtain the description information of the neighbor node characteristics, which is called the neighbor information text and is recorded as

[0083] Commonality information text: Considering that GraphSAGE is run as a low-pass filter, in order to indicate the similarity between the current node and its neighboring nodes for smoothing purposes, this example will also guide the second LLM to integrate and summarize the common information between the two. For example, the second LLM can be used according to and Perform node commonality analysis to infer the current node v and its neighbor nodes The commonality between them is obtained, and the common information text is recorded as

[0084] Having obtained and After that, connect and You can build high-quality node description text t υ ,Right now

[0085] From the above process, it can be seen that for the second LLM used to implement the generator, its input is the node attribute text of a node collected from the knowledge graph and the node attribute text of the node's neighboring nodes. After the reasoning and analysis of the second LLM, the second LLM will output the node description text t v .

[0086] In the specific process of generating the above-mentioned text, the second LLM will process the prompt data such as the prompt data in Figure 3C to generate a corresponding reply, and determine the node description text based on the reply. Among them, for the generation of node information text, the exemplary prompt data is such as "Please summarize the information of the node", then based on the prompt data and the attribute text of the node, the reply obtained by the second LLM processing can be used as the node information text, such as "The user's interest is..."; for the generation of neighbor information text, the exemplary prompt data is such as "Please summarize the information of the neighbor nodes", then based on the prompt data and the attribute text of the neighbor nodes of the node, the reply obtained by the second LLM processing can be used as the neighbor information text, such as "The neighbor's interest is..."; for common information text, the exemplary prompt data is such as "Please summarize their commonalities", then based on the above-mentioned node information text and neighbor information text, the reply obtained by the second LLM processing can be used as the common information text, such as "Their commonalities are...".

[0087] After obtaining the node representation z v and node description text t v After that, the modal alignment data (z υ ,t υ ) to train a translation model using the modality-aligned data.

[0088] Process 4: The processing of the translation model in the first stage of training.

[0089] There is a modality difference between the trained graph model, such as GraphSAGE, and the first interactive language model, such as the first LLM. The first LLM cannot directly understand the node representations output by GraphSAGE. That is, the node representation vectors and the first LLM's input token vectors are of different sizes and have different feature spaces. To address this discrepancy, the present embodiment introduces a translation model, illustrated as Translator in Figure 3C, which aims to align the graph model and the interactive language model by converting the learned node representations into tokens.

[0090] Although in a simpler way, a simple trainable projection matrix can be applied to z υ Convert the tokens into language feature space and align their dimensions with the dimensions of the word vector space in the first LLM. However, this simple conversion method makes it difficult to represent the node zυ The complex information contained in the nodes is extracted and translated into natural language, and it is difficult to generalize to unseen nodes.

[0091] To this end, in the translation model Translator of this example, for a pair (z υ , t υ ), using two encoders, denoted as f z (·) and f t (·), to extract the language features for alignment. Among them, for the node description text t υ , using the text encoder f t (·) (such as BERT) to extract features, that is, f t (t v ) to obtain the corresponding node text vector T υ =f t (t υ ). Among them, f t (·) can be implemented as an encoder containing 12 layers of Transformer blocks. υ , you can use the encoder f z (·) for feature extraction. z (·) can also be implemented as an encoder based on the Transformer architecture. In specific implementation, the encoder can take M learnable token vectors as input. The M learnable token vectors are called query tokens, which are shown as Q in Figure 3C. z (·) processing, output M features Moreover, H v =f z (Q,z v ).

[0092] In order to extract z υ Zhong and t υ The most relevant information, in the specific implementation, Q is designed to use the self-attention layer for self-attention processing, and then through the cross attention layer and z υ Perform cross attention processing to v interact and coordinate f t and f z The self-attention layer between v to interact.

[0093] Based on the above process, the node attention feature H can be obtained v and node text attention feature T v , and then, H can be calculated according to the preset loss function, such as the cross entropy loss function, etc. υ and Tυ The difference is the loss value (shown as "first-stage loss" in Figure 3C). The translation model is trained in the first stage according to the difference until the termination condition of the first stage training is met, such as reaching a preset number of training times, or the loss value reaches a preset threshold, etc.

[0094] During specific training, the node representation z output by the pre-trained graph model GraphSAGE is first υ Freeze, only train the translation model Translator. In order to align H υ and (T υ [CLS]TOKEN vector, which can be accepted by the first LLM as input), first, adopt the method of maximizing H υ and Mutual information to align H υ and For example, H can be calculated first υ and The pairwise similarity between each TOKEN is calculated, and the highest similarity is selected as the similarity score between the two. Then the similarity of the positive sample pair is compared with the similarity of the negative sample pair. Secondly, the training goal of Translator is to generate a text describing a node based on a given representation text. Specifically, it can be achieved by f z The query token in (·) is Q Extract given z υ The basic information of f is then seamlessly passed to f through the shared self-attention layer (“Shared Self-Attention” in Figure 3C). z The text token in (·) (shown as “Description Tokens” in Figure 3C) can be replaced with [DEC]TOKEN in the text generation task. v Generated text features H v The feature T corresponding to the actual node description text v The cross entropy loss between ,Q is forced to capture the difference between t v Related z v More details in v and T v Matching is performed to learn alignment at a finer granularity. Then, each TOKENh v,i ∈H v With T v [CLS]TOKEN Concatenate them and then feed them into a binary classifier and calculate the matching score by averaging the logits of all queries. The first stage of Translator training is performed based on this matching score.

[0095] Through this first stage of training, Translator can align the node representation and text output by the graph model.

[0096] Process 5: The processing of the translation model in the second stage of training.

[0097] In the second stage of training, the linear layer can be used to transform the output of the Translator, the TOKEN vector H v , projected into the same dimension as the word vector of the first LLM. The projected vector can be regarded as a soft prompt, which is connected with the human language instruction (illustrated as "Please summarize the interests of the user and his neighbors..." in Figure 3C) as the input of the first LLM. Then, through the generative learning of the first LLM, the modality alignment data is used to tune the Translator parameters. In this way, the node representation z v It can be aligned with the word vector of the pre-trained first LLM.

[0098] The output of the first LLM is response data for human language instructions (also referred to as question data in the embodiment of the present application). The response data and the node description text output by the generator can be compared, and the difference between the two (i.e., the loss value, illustrated as "second-stage loss" in Figure 3C) can be calculated based on a preset loss function. The Translator can then be fine-tuned based on the loss value to achieve parameter tuning.

[0099] Through the second stage of training, Translator can connect the graph model with the interactive language model (such as GraphSAGE-the first LLM).

[0100] As can be seen, in this example, for the lightweight Translator, a two-stage joint training paradigm is used to gradually bridge the gap between the graph model and the interactive language model. In the first stage of training, the Translator is trained to output the node representation z from the graph model. v Extracted with t v Most relevant H v In the second stage of training, generative learning is performed by connecting the output of the Translator to the frozen interactive language model to continue training the Translator so that its output can be understood by the interactive language model.

[0101] From the above, in the training process of this example, based on the output of the pre-trained graph model, the generator first textualizes the target node, its neighbor nodes and the commonalities between the two to construct modal alignment data, that is, aligned (node ​​representation and node description text) pairs. In the first stage of training, (node ​​representation, node description text) pairs are used for alignment training. In order to make the node representation better conform to the text prompts and instructions, the Translator is bridged with the first LLM. Then, the (node ​​representation and node description text) pairs are used to fine-tune the Translator in combination with the generative learning of the first LLM. As a result, the human-computer interaction model obtained after two stages of training can be extended to the invisible node representation of the reasoning stage, and open tasks can be achieved through dialogue. Therefore, through the training process of the human-computer interaction model of this example, the graph model GraphSAGE can be modally aligned with the first interactive language model, namely LLM, to fully utilize the effective performance of LLM on open tasks.

[0102] When the human-computer interaction model obtained by training in Example 1 enters the inference stage, it can be used to implement the corresponding human-computer interaction. The human-computer interaction process is described below through specific examples.

[0103] Example 2

[0104] 4A , there is shown a flowchart of the steps of a human-computer interaction method according to the second embodiment of the present application.

[0105] The human-computer interaction method of this embodiment includes the following steps:

[0106] Step S302: Determine the knowledge graph corresponding to the input question data, and obtain node data corresponding to the nodes in the knowledge graph that match the question data through the graph model.

[0107] In this embodiment, the input question data can be question data sent by a user device and input by a user through the user device. The user device can be any appropriate user device, including but not limited to intelligent interactive devices such as smart speakers, mobile terminals, etc. In addition, different knowledge graphs correspond to knowledge in different fields, industries, services, or applications. For a certain question data, the overall knowledge graph required can be determined based on the field to which the question data belongs. However, since a single question data may only involve part of the knowledge graph, such as a branch or a sub-branch in the knowledge graph, in this embodiment, the knowledge graph corresponding to the input question data can be an overall knowledge graph, or it can be the part of the knowledge graph related to the question data in a certain overall knowledge graph. When specifically determining the part of the knowledge graph related to the question data, a traversal method or a node similarity matching method can be used to first determine the node that matches the question data, and then determine the associated nodes that have an associated relationship with the node, and use the partial knowledge graph formed by the node and the associated nodes as the knowledge graph corresponding to the question data.

[0108] When the knowledge graph corresponding to the question data is implemented in TAG form, this step can be implemented as follows: determining the text attribute graph corresponding to the input question data and identifying the nodes in the text attribute graph that match the question data; obtaining the text attribute data of the matching nodes; and extracting features from the node text attribute data using the graph model to obtain node vectors representing the node data. This approach, on the one hand, makes it easier to align the node data in the TAG with the interactive language model, facilitating conversion processing suitable for the interactive language model; and on the other hand, it also facilitates feature extraction from the graph model.

[0109] When the human-computer interaction model trained in Example 1 is used to perform this step, node features of TAG can be extracted by its graph model to obtain node vectors representing node data, such as z v .

[0110] Step S304: Based on the node data, generate input data that can be accepted by the first interactive language model.

[0111] The first interactive language model is a language model used to perform open tasks, including but not limited to LLM.

[0112] Specifically in this embodiment, the first interactive language model is mainly used for human-computer question-answering interaction based on graph data (such as knowledge graph), but as mentioned above, the node data output by the graph model cannot be directly processed by the first interactive language model. Therefore, it needs to be converted to generate input data that can be accepted by the first interactive language model, such as text data.

[0113] Based on this, in one feasible manner, this step can be implemented as follows: obtaining corresponding text features based on the node vector corresponding to the node data; and mapping the text features to text input data that can be accepted by the first interactive language model.

[0114] Among them, based on the node vector corresponding to the node data, obtaining the corresponding text feature can be achieved by: performing cross-attention calculation on the node vector corresponding to the node data to obtain the corresponding text feature.

[0115] When the human-computer interaction model obtained by training in Example 1 is used, this step can be implemented by the translation model therein, such as: through the cross attention layer of the translation model, the node vector corresponding to the node data, i.e., z v Cross-attention processing is performed to obtain corresponding text features. Since the interactive language model can accept text data, the text features can be converted to obtain their corresponding text data, that is, text input data that can be accepted by the first interactive language model.

[0116] Step S306: Based on the input data and the question data, a response is generated by using the first interactive language model to generate response data corresponding to the question data.

[0117] Among them, the question data can be used as the human-computer interaction instruction of the first interactive language model, and the input data can be used as the graph node data corresponding to the human-computer interaction instruction, so that the first interactive language model performs open task processing for the human-computer interaction instruction based on the graph node data and generates corresponding reply data.

[0118] It can be seen that through this embodiment, in order to effectively combine and utilize the capabilities of the interactive language model and the graph model, after determining the knowledge graph corresponding to the question data, the node data of the nodes in the knowledge graph that match the question data is first extracted through the graph model GM. Although GM can determine the node data more accurately, it lacks the ability to generate open-ended responses, and the node data cannot be accepted by the interactive language model (i.e., the first interactive language model) such as LLM. There is a modal difference between the two. Therefore, in this embodiment, based on the determined node data, input data that can be accepted by the interactive language model (i.e., the first interactive language model) such as LLM is generated. The interactive language model (i.e., the first interactive language model) generates open-ended responses based on the input data and the question data using its own open task processing capabilities. As a result, the human-computer interaction solution of this embodiment can not only ensure the accuracy of task processing, but also make full use of the ability of the interactive language model to process open tasks, making the solution more open and flexible and applicable to various scenarios and environments.

[0119] The following is an illustrative example of the above human-computer interaction process using specific scenarios.

[0120] In an example scenario of human-computer interaction based on a knowledge graph corresponding to a form, the knowledge graph is implemented as a form TAG. Each node in the TAG corresponds to a form, and its attribute text includes but is not limited to the form's metadata (i.e., the form's structural information) and content data (i.e., the form's content information).

[0121] Based on this, the human-computer interaction method may include: determining a form text attribute graph TAG corresponding to an input form question, and obtaining node data corresponding to a node in the form TAG that matches the form question through a graph model, wherein the node in the form TAG corresponds to a form entity, and the form text attribute of the node is used to describe form structure information and / or form content information; based on the node data, generating input data that can be accepted by a first interactive language model, wherein the first interactive language model is a language model for performing open tasks; based on the input data and the form question, generating a reply through the first interactive language model to generate reply data corresponding to the form question.

[0122] Behind all kinds of human-computer interactions are massive amounts of various data, and in many cases, these data are stored in the form of forms. Due to the correlation between data, usually a form is associated with one or more other forms. Based on this, a graph structure can be used to represent and associate these forms. For example, TAG is used to represent and associate these forms in this example. Among them, a node in TAG represents a form entity, and the edges between different nodes represent the association relationship between different form entities. For each node, attribute text can be used to record its structural information and / or content information. Therefore, based on the TAG, through the human-computer interaction model in the embodiment of the present application, human-computer question-and-answer interaction of various form data can be performed.

[0123] When the human-computer interaction model trained in Example 1 is used to implement this example, as shown in Figure 4B , assuming the form question is "Please help me find the downstream form that uses Form A," the device hosting the human-computer interaction model will first determine the corresponding form tag based on the form question. In this example, the tag is set to TAG-1.

[0124] On this basis, the human-computer interaction model will process the nodes involved in the form question in TAG-1 through its graph model such as GraphSAGE to extract the corresponding node data. In this example, it is the node vector z v . Furthermore, the node vector z v It will be input into the translation model of the human-computer interaction model to extract its corresponding node text vector H v. Then, the node text vector H v The input data will be converted into textual form (for example, by converting a vector into textual form through a projection matrix). This textual input data will be input into a first interactive language model, such as a first LLM, in the human-computer interaction model along with the textual form question. The first LLM will then generate open-ended responses to generate response data corresponding to the form question, such as "The downstream forms using Form A are: Form C, Form P, and Form H."

[0125] It can be seen that through this example, human-computer question-answering interaction based on form TAG is effectively realized.

[0126] In another scenario example of human-computer interaction based on a knowledge graph corresponding to user information, the knowledge graph is implemented as a user TAG, each node in the TAG corresponds to a user entity, and its attribute text may include arbitrary attribute information used to describe the user.

[0127] Based on this, the human-computer interaction method may include: determining a user TAG corresponding to an input user information question, and obtaining node data corresponding to a node in the user TAG that matches the user information question through a graph model, wherein the node in the user TAG corresponds to a user entity, and the user text attribute of the node is used to describe the user's attribute information; based on the node data, generating input data that can be accepted by a first interactive language model, wherein the first interactive language model is a language model for performing open tasks; based on the input data and the user information question, generating a reply through the first interactive language model to generate reply data corresponding to the user information question.

[0128] In this example, a node in the TAG represents a user entity, and the edges between different nodes represent the relationships between different user entities. For each node, attribute text can be used to describe the attributes or characteristics of the user. Therefore, based on this TAG, the human-computer interaction model in the embodiment of this application can be used to conduct human-computer question-and-answer interactions on various user information.

[0129] When the human-computer interaction model trained in Example 1 is used to implement this example, as shown in Figure 4C , assuming the user information question is "What common interests does user A and his friends have?", the device hosting the human-computer interaction model first determines the corresponding user tag based on the user information question. In this example, the tag is set to TAG-2.

[0130] On this basis, the human-computer interaction model will process the nodes involved in the user information problem in TAG-2 (including the nodes corresponding to user A and the nodes that are friends with user A) through its graph model such as GraphSAGE to extract the node data corresponding to these nodes. In this example, it is represented as the node vector zv . Furthermore, the node vector z v It will be input into the translation model of the human-computer interaction model to extract its corresponding node text vector H υ . Then, the node text vector H υ The input data will be converted into textual form (for example, by converting a vector into textual form through a projection matrix). The textual input data will be input into a first interactive language model, such as a first LLM, in the human-computer interaction model together with the textual user information question. The first LLM will then perform open-ended response generation to generate response data corresponding to the user information question, such as "User A and his friends all like fishing."

[0131] It can be seen that through this example, human-computer question-answering interaction based on user TAG is effectively realized.

[0132] It should be noted that the scenario examples in Figures 4B and 4C above are all illustrative descriptions. Those skilled in the art can apply the solutions of the embodiments of the present application to various human-computer interaction scenarios based on graph data based on multiple embodiments of the present application and the solutions of the above examples, which are all within the scope of protection of the present application.

[0133] Example 3

[0134] 5 , a schematic structural diagram of an electronic device according to a fifth embodiment of the present application is shown. The specific embodiments of the present application do not limit the specific implementation of the electronic device.

[0135] As shown in FIG. 5 , the electronic device may include a processor 502 , a communications interface 504 , a memory 506 , and a communication bus 508 .

[0136] in:

[0137] The processor 502 , the communication interface 504 , and the memory 506 communicate with each other via a communication bus 508 .

[0138] The communication interface 504 is used to communicate with other electronic devices or servers.

[0139] The processor 502 is configured to execute the program 510 , and specifically may execute the relevant steps of any one of the above-mentioned method embodiments.

[0140] Specifically, the program 510 may include program codes, which include computer operation instructions.

[0141] Processor 502 may be a CPU, a Graphics Processing Unit (GPU), an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the smart device may be processors of the same type, such as one or more CPUs, or may be processors of different types, such as one or more CPUs and one or more ASICs.

[0142] The memory 506 is used to store the program 510. The memory 506 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0143] The program 510 may include multiple computer instructions. Specifically, the program 510 may enable the processor 502 to execute operations corresponding to the method described in any of the aforementioned method embodiments through the multiple computer instructions.

[0144] The specific implementation of each step in program 510 can refer to the corresponding description of the corresponding steps and units in the above-mentioned method embodiment, and has corresponding beneficial effects, which will not be repeated here. Those skilled in the art will clearly understand that for the convenience and brevity of description, the specific working process of the above-mentioned devices and modules can refer to the corresponding process description in the above-mentioned method embodiment, and will not be repeated here.

[0145] The present application also provides a computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any of the aforementioned method embodiments. The computer storage medium includes, but is not limited to, a compact disc read-only memory (CD-ROM), random access memory (RAM), a floppy disk, a hard disk, or a magneto-optical disk.

[0146] An embodiment of the present application also provides a computer program product, including computer instructions, which instruct a computing device to execute operations corresponding to any one of the above-mentioned multiple method embodiments.

[0147] After experimentation, the human-computer interaction model provided by the embodiment of the present application achieves better human-computer interaction performance, and the effect is better than the method of directly inputting the attribute text of the node itself and the attribute text of the neighboring nodes into the large language model LLM. This is because if the large language model LLM directly processes the original text containing the attribute text of the node itself and the attribute text of the neighboring nodes, it will be affected by a large amount of text noise and complexity. The human-computer interaction model provided by the embodiment of the present application can extract and summarize the attribute text of the node itself, the attribute text of the neighboring nodes, and the commonalities between the two through the translation model therein, which greatly reduces noise interference and reduces the complexity of the content contained in the text.

[0148] In addition, the scheme of the embodiment of the present application also improves the legal detection rate of data. This is because, in the method of inputting only the original attribute text of the node into the large language model LLM, because the information it contains is limited, it brings great reasoning challenges to the large language model LLM. In the method of inputting the attribute text of the node and the attribute text of its neighboring nodes into the large language model LLM, although the legal detection rate is improved, there are still problems with text noise and information redundancy. Through the scheme of the embodiment of the present application, the generator Producer is introduced in the training phase of the human-computer interaction model to concisely summarize the attribute text of the node itself, the attribute text of its neighboring nodes and the commonalities between the two, thereby reducing noise while providing rich content. Furthermore, the node representation output by the Translator translation graph model is used as a soft graph attribute prompt, which can contain more complex details than stacked text. In addition, in the scheme of the embodiment of the present application, the output of the Translator can be projected into a fixed-length TOKEN, which facilitates the large language model LLM's understanding of the graph node information, thereby improving the accuracy of the large language model's response.

[0149] In addition, it should be noted that the user-related information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to sample data used to train the model, data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0150] It should be pointed out that, according to the needs of implementation, the various components / steps described in the embodiments of the present application can be split into more components / steps, or two or more components / steps or partial operations of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of the present application.

[0151] The above-mentioned method according to the embodiment of the present application can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk or magneto-optical disk), or as computer code that is originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded via a network and will be stored in a local recording medium, so that the method described herein can be stored in such software processing on a recording medium using a general-purpose computer, a dedicated processor or programmable or dedicated hardware (such as an application-specific integrated circuit (ASIC) or a field programmable gate array (FPGA)). It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component (e.g., random access memory (RAM), read-only memory (ROM), flash memory, etc.) that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor or hardware, the method described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the method shown here, the execution of the code converts the general-purpose computer into a dedicated computer for executing the method shown here.

[0152] Those skilled in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for specific applications, but such implementation should not be considered to be beyond the scope of the embodiments of this application.

[0153] The above implementation methods are only used to illustrate the embodiments of the present application, and are not intended to limit the embodiments of the present application. Ordinary technicians in the relevant technical field can make various changes and modifications without departing from the spirit and scope of the embodiments of the present application. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of the present application, and the scope of patent protection of the embodiments of the present application should be defined by the claims.

Claims

1. A human - machine interaction method, comprising: Determine a knowledge graph corresponding to the input problem data, and obtain node data corresponding to the nodes in the knowledge graph that match the problem data through a graph model; Generate input data acceptable to a first interactive language model based on the node data, wherein the first interactive language model is a language model for performing open - ended tasks; Generate a reply corresponding to the problem data through the first interactive language model based on the input data and the problem data.

2. The method according to claim 1, wherein, The generating input data acceptable to the first interactive language model based on the node data includes: Obtain corresponding text features based on the node vectors corresponding to the node data; Map the text features to generate text input data acceptable to the first interactive language model.

3. The method according to claim 2, wherein The obtaining corresponding text features based on the node vectors corresponding to the node data includes: Perform cross - attention calculation on the node vectors corresponding to the node data to obtain corresponding text features.

4. The method according to any one of claims 1-3, wherein, The determining a knowledge graph corresponding to the input problem data, and obtaining node data corresponding to the nodes in the knowledge graph that match the problem data through a graph model includes: Determine a text attribute graph corresponding to the input problem data, and determine the nodes in the text attribute graph that match the problem data; Obtain the text attribute data of the matching nodes; Extract features from the text attribute data of the nodes through a graph model to obtain node vectors representing the node data.

5. A human - machine interaction model training method, comprising: Obtain the node attribute text of sample nodes in a knowledge graph sample, and generate node representations of the sample nodes through a pre - trained graph model; Generate node description text for describing the sample nodes through a generator based on the node attribute text; Construct modality - aligned data based on the node representations and the node description text; Use the modality - aligned data to perform a first - stage training on a translation model connecting the graph model and a pre - trained first interactive language model to obtain a translation model capable of converting the output of the graph model into input acceptable to the first interactive language model; wherein the first interactive language model is a language model for performing open - ended tasks.

6. The method according to claim 5, wherein, At least generating node description text for describing the sample nodes through a generator based on the node attribute text includes: Generate node information text for characterizing the features of the sample nodes themselves, neighbor information text for characterizing the features of the neighbor nodes of the sample nodes, and common information text for characterizing the commonalities between the sample nodes and the neighbor nodes through the generator based on the node attribute text of the sample nodes and the node attribute text of the neighbor nodes of the sample nodes; Generate node description text for describing the sample nodes based on the node information text, the neighbor information text, and the common information text.

7. The method according to claim 6, wherein, Based on the node attribute text of the sample node and the node attribute texts of the neighbor nodes of the sample node, a node information text for characterizing the characteristics of the sample node itself, a neighbor information text for characterizing the characteristics of the neighbor nodes of the sample node, and a common information text for characterizing the commonalities between the sample node and the neighbor nodes are generated by a generator, including: Based on the node attribute text of the sample node, node self-characteristic analysis and processing are performed through the second interactive language model in the generator to generate a node information text for characterizing the characteristics of the sample node itself; Based on the node attribute texts of the neighbor nodes of the sample node, neighbor node characteristic analysis and processing are performed through the second interactive language model to generate a neighbor information text for characterizing the characteristics of the neighbor nodes of the sample node; Based on the node information text and the neighbor information text, node commonality analysis and processing are performed through the second interactive language model to generate a common information text for characterizing the commonalities between the sample node and the neighbor nodes.

8. The method according to any one of claims 5-7, wherein, The first-stage training of the translation model connecting the graph model and the pre-trained first interactive language model using the modality-aligned data includes: Feature extraction is respectively performed on the node representations and node description texts in the modality-aligned data to obtain corresponding node feature vectors and node text vectors; Based on the attention processing results of the node feature vectors and the node text vectors, the first-stage training of the translation model is performed.

9. The method according to claim 8, wherein The training of the translation model based on the attention processing results of the node feature vectors and the node text vectors includes: Self-attention processing is performed on a preset learnable token vector and the text token vector corresponding to the node text vector to obtain corresponding learnable token attention features and node text attention features; Cross-attention processing is performed on the learnable token attention features and the node feature vectors to obtain node attention features; Based on the differences between the node attention features and the node text attention features, the first-stage training of the translation model is performed.

10. The method according to any one of claims 5-7, wherein, The method further includes: Taking the modality-aligned data as input, the modality-aligned data is processed by the translation model that has completed the first-stage training, and the processing results are converted into parameters that can be accepted by the first interactive language model; Through the first interactive language model, response generation is performed based on the parameters, and based on the generated responses, the second-stage training of the translation model is performed.

11. The method according to claim 10, wherein, The response generation based on the parameters through the first interactive language model includes: Taking the parameters as the prompt data of the first interactive language model and combining problem sample data as the input of the first interactive language model; Through the first interactive language model, response generation is performed based on the input.

12. A human-computer interaction method, including: Determine a form text attribute graph corresponding to the input form problem, and obtain node data corresponding to nodes in the form text attribute graph that match the form problem through a graph model, where nodes in the form text attribute graph correspond to form entities, and the form text attributes of the nodes are used to describe form structure information and / or form content information; Generate input data acceptable to a first interactive language model based on the node data, where the first interactive language model is a language model for performing open-ended tasks; Generate a response based on the input data and the form problem through the first interactive language model to generate response data corresponding to the form problem.

13. A human-computer interaction method, comprising: Determine a user text attribute graph corresponding to the input user information problem, and obtain node data corresponding to nodes in the user text attribute graph that match the user information problem through a graph model, where nodes in the user text attribute graph correspond to user entities, and the user text attributes of the nodes are used to describe user attribute information; Generate input data acceptable to a first interactive language model based on the node data, where the first interactive language model is a language model for performing open-ended tasks; Generate a response based on the input data and the user information problem through the first interactive language model to generate response data corresponding to the user information problem.

14. An electronic device, comprising: A processor, a memory, a communication interface, and a communication bus, where the processor, the memory, and the communication interface complete communication with each other through the communication bus; The memory is used to store at least one executable instruction, and the executable instruction causes the processor to perform operations corresponding to the method according to any one of claims 1-13.

Images