Human-computer interaction, method for training model thereof, and electronic device

Abstract

Embodiments of the present application provide a human-computer interaction method and model training method and an electronic device, wherein the human-computer interaction method comprises: determining a knowledge graph corresponding to input question data, and obtaining node data corresponding to a node in the knowledge graph that matches the question data through a graph model; generating input data that can be accepted by a first interactive language model based on the node data, wherein the first interactive language model is a language model used to perform an open task; and generating reply data corresponding to the question data through reply generation by the first interactive language model based on the input data and the question data. Through the embodiments of the present application, not only the accuracy of task processing can be ensured, but also the ability of the interactive language model to process open tasks can be fully utilized, so that the scheme has good openness and can be flexibly applied to various scenes and environments.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of artificial intelligence technology, and in particular to a human-computer interaction method, a training method of a human-computer interaction model, and an electronic device. BACKGROUND

[0002] In the field of artificial intelligence technology, GM (Graph Model) and interactive language models such as LLM (Large Language Model) that can perform open tasks are two relatively important models. Among them, the large language model refers to a deep learning model trained based on massive text data, which has strong open task processing capability and can realize functions such as generating natural language text, deeply understanding the meaning of the text, and processing various natural language tasks (such as text summarization, question answering, translation, etc.). The graph model is good at predefined tasks and basically does not have the ability to perform open tasks.

[0003] In order to enable the graph model to also utilize the capabilities of interactive language models such as LLM that have open task execution capabilities, so as to improve the task execution effect of the graph model, in one way, as shown in Figure 1 , first use LLM to process graph data, use its massive knowledge to enhance the text attributes of nodes, and then generate predictions through GM. Through this way, although GM can accurately predict predefined tasks, it still cannot handle open tasks.

[0004] Therefore, how to effectively combine interactive language models such as LLM and GM to fully utilize the ability of interactive language models to handle open tasks has become a problem to be solved. SUMMARY

[0005] Therefore, embodiments of the present application provide a human-computer interaction and its model training scheme to at least partially solve the above problems.

[0006] According to a first aspect of embodiments of the present application, a human-computer interaction method is provided, comprising: determining a knowledge graph corresponding to input question data, and obtaining node data corresponding to a node matched with the question data in the knowledge graph through a graph model; based on the node data, generating input data that can be accepted by a first interactive language model, wherein the first interactive language model is a language model for performing open tasks; based on the input data and the question data, generating a reply through the first interactive language model to generate reply data corresponding to the question data.

[0007] According to a second aspect of the embodiments of the present application, a human-computer interaction model training method is provided, including: obtaining node attribute text of a sample node in a knowledge graph sample, and generating node representation of the sample node through a pre-trained graph model; generating node description text for describing the sample node through a generator based on the node attribute text; constructing modal alignment data based on the node representation and the node description text; and performing first stage training on a translation model connected to the graph model and a pre-trained first interactive language model using the modal alignment data, to obtain a translation model capable of converting an output of the graph model into an input acceptable to the first interactive language model; wherein the first interactive language model is a language model for performing open tasks.

[0008] According to a third aspect of the embodiments of the present application, another human-computer interaction method is provided, including: determining a form text attribute graph corresponding to an input form question, and obtaining node data corresponding to a node matching the form question in the form text attribute graph through a graph model, wherein the nodes in the form text attribute graph correspond to form entities, and form text attributes of the nodes are used to describe form structure information and / or form content information; generating input data acceptable to a first interactive language model based on the node data, wherein the first interactive language model is a language model for performing open tasks; and generating a reply through the first interactive language model based on the input data and the form question, to generate reply data corresponding to the form question.

[0009] According to a fourth aspect of the embodiments of the present application, still another human-computer interaction method is provided, including: determining a user text attribute graph corresponding to an input user information question, and obtaining node data corresponding to a node matching the user information question in the user text attribute graph through a graph model, wherein the nodes in the user text attribute graph correspond to user entities, and user text attributes of the nodes are used to describe attribute information of the user; generating input data acceptable to a first interactive language model based on the node data, wherein the first interactive language model is a language model for performing open tasks; and generating a reply through the first interactive language model based on the input data and the user information question, to generate reply data corresponding to the user information question.

[0010] According to a fifth aspect of the embodiments of the present application, an electronic device is provided, including: a processor, a memory, a communication interface and a communication bus, the processor, the memory and the communication interface complete communication with each other through the communication bus; the memory is used to store at least one executable instruction, and the executable instruction causes the processor to perform operations corresponding to the method of the first aspect or the second aspect or the third aspect or the fourth aspect.

[0011] According to a fourth aspect of the embodiments of the present application, a computer storage medium is provided, and the computer storage medium has stored thereon a computer program, and the computer program is executed by a processor to implement the method according to the first aspect or the second aspect or the third aspect or the fourth aspect.

[0012] According to the scheme provided by the embodiments of the present application, in order to effectively combine and utilize the capabilities of the interactive language model and the graph model, after the knowledge graph corresponding to the question data is determined, the node data of the node in the knowledge graph that matches the question data is obtained by the graph model GM. Although the GM can accurately determine the node data, it lacks the ability to generate open-ended replies, and the node data cannot be accepted by the interactive language model (i.e., the first interactive language model) such as LLM, and there is a modal difference between the two. Therefore, in the embodiments of the present application, input data that can be accepted by the interactive language model (i.e., the first interactive language model) such as LLM is generated based on the determined node data, and the interactive language model (i.e., the first interactive language model) generates an open-ended reply based on the input data and the question data using its own open-ended task processing capability. Thus, the human-computer interaction scheme of the embodiments of the present application not only ensures the accuracy of task processing, but also fully utilizes the ability of the interactive language model to process open-ended tasks, so that the scheme has good openness and can be flexibly applied to various scenes and environments. BRIEF DESCRIPTION OF DRAWINGS

[0013] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments described in the embodiments of the present application, and other drawings can also be obtained by those skilled in the art based on these drawings.

[0014] Figure 1 A schematic diagram of a combination of an LLM and a GM in the related art;

[0015] Figure 2 A schematic diagram of an exemplary system suitable for the embodiments of the present application;

[0016] Figure 3A A step flowchart of a human-computer interaction model training method according to the first embodiment of the present application;

[0017] Figure 3B A structure diagram of a human-computer interaction model in the embodiment shown; Figure 3A

[0018] Figure 3C A structure diagram of a human-computer interaction model in the embodiment shown; Figure 3A ​A schematic diagram of one training process example in the illustrated embodiment;

[0019] Figure 4A A flowchart of steps of a human-computer interaction method according to Embodiment Two of the present application;

[0020] Figure 4B A flowchart of steps of a human-computer interaction method according to Embodiment Two of the present application; Figure 4A A schematic diagram of one scenario example in the illustrated embodiment;

[0021] Figure 4C A flowchart of steps of a human-computer interaction method according to Embodiment Two of the present application; Figure 4A A schematic diagram of another scenario example in the illustrated embodiment;

[0022] Figure 5 A schematic diagram of an electronic device according to Embodiment Three of the present application. DETAILED DESCRIPTION

[0023] In order to enable persons skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the embodiments of the present application, all other embodiments obtained by persons skilled in the art should belong to the scope of protection of the embodiments of the present application.

[0024] The specific implementation of the embodiments of the present application will be further described below with reference to the drawings in the embodiments of the present application.

[0025] Figure 2 An exemplary system to which the embodiments of the present application are applicable is shown. As shown, the system 100 can include a cloud server 102, a communication network 104, and / or one or more user devices 106, Figure 2 Figure 2 The example in the middle is a plurality of user devices.

[0026] ​The cloud server 102 can be any appropriate device for storing information, data, programs, and / or any other suitable type of content, including but not limited to a distributed storage system device, a server cluster, a computing cloud server cluster, etc. In some embodiments, the cloud server 102 can perform any appropriate function. For example, in some embodiments, the cloud server 102 can be used for question and answer interaction based on a graph model and an interactive language model. As an optional example, in some embodiments, upon receiving question data, the cloud server 102 can determine a corresponding knowledge graph, and extract node data of a node matching the question data from the knowledge graph through the graph model; then, based on the node data, generate input data that can be accepted by a first interactive language model; and further, based on the input data and the question data, the first interactive language model can be used to generate an open-ended reply. The first interactive language model is a language model used for performing an open-ended task. As another example, in some embodiments, the cloud server 102 can receive question data sent from the user device 106, and return the generated reply data to the user device 106. As another example, in some embodiments, the cloud server 102 can also train the models involved in the above question and answer interaction.

[0027] In some embodiments, the communication network 104 can be any appropriate combination of one or more wired and / or wireless networks. For example, the communication network 104 can include any one or more of the following: the Internet, an intranet, a wide area network (WAN), a local area network (LAN), a wireless network, a Digital Subscriber Line (DSL) network, a frame relay network, an Asynchronous Transfer Mode (ATM) network, a virtual private network (VPN), and / or any other suitable communication network. The user device 106 can connect to the communication network 104 through one or more communication links, such as the communication link 112, which can be linked to the cloud server 102 via one or more communication links, such as the communication link 114. The communication links can be any communication links suitable for communicating data among the user device 106 and the cloud server 102, such as network links, dial-up links, wireless links, hard-wired links, any other suitable communication links, or any suitable combination of such links.

[0028] The user device 106 can include any one or more user devices suitable for interacting with a user. In some embodiments, the user device 106 can receive question data in the form of text input by a user, or, receive question data in the form of speech and convert it into text form, and then send it to the cloud server 102. As an optional example, the user device 106 also receives reply data returned by the cloud server 102 according to the question data, and displays or plays the reply data to the user. In some embodiments, the user device 106 can include any suitable type of device. For example, in some embodiments, the user device 106 can include a mobile device, a tablet computer, a laptop computer, a desktop computer, and / or any other suitable type of user device.

[0029] Based on the above system, the present application describes the above human-computer interaction scheme through multiple embodiments. In order to facilitate the understanding of the scheme of the embodiments of the present application, first, the training process of the human-computer interaction model is described, and then the human-computer interaction method based on the trained human-computer interaction model is described.

[0030] Embodiment one

[0031] In the following, the human-computer interaction model of the present application is described. Figures 3A-3C The training method of the human-computer interaction model of the embodiments of the present application is described.

[0032] First, referring to Figure 3B , a structural schematic diagram of a human-computer interaction model of an embodiment of the present application is shown. It can be seen that the human-computer interaction model includes a graph model, a generator, a translation model and a first interactive language model. Figure 3B

[0033] Among them:

[0034] The graph model is used to process data in graph structure, such as processing knowledge graph data, etc., which can be implemented as any appropriate graph neural network model structure, including but not limited to GraphSAGE (Graph SAmple and aggreGatE) model, GNN (Graphic Neural Network), GCN (Graph Convolutional Networks) model, etc.

[0035] ​The generator is mainly used in the model training stage to construct node description text for describing the characteristics of the node itself, the characteristics of the node neighbors, and the commonalities of both, according to the node attribute text collected from the knowledge graph. The node description text can be used together with the node representation generated by the graph model to construct modal alignment data for translation model training, so that in the model inference stage, the human-computer interaction model can generate input acceptable to the first interactive language model based on the node representation output by the graph model through the translation model. It can be seen that the generator does not need to participate in processing in the model inference stage. In a feasible manner, the generator can be implemented based on the second interactive language model. Since the input of the generator is the node attribute text, it can be directly processed as input by the second interactive language model. Alternatively, the second interactive language model can be a language model that has been trained, is higher-order than the first interactive language model, and can perform open tasks, including but not limited to the LLM model.

[0036] The translation model is connected between the graph model and the first interactive language model, and is used for modal alignment of the two to convert the output of the graph model into input acceptable to the first interactive language model. Exemplarily, the translation model can be implemented based on an encoder structure.

[0037] The first interactive language model is used to perform open task processing, such as open reply generation, based on the output of the translation model (carrying information of the graph node). Exemplarily, the first interactive language model can also be implemented as an LLM model.

[0038] In the following, based on the above model structure, the training method of the human-computer interaction model of the embodiments of the present application is described.

[0039] Referring to Figure 3A , a step flowchart of a training method of a human-computer interaction model according to Embodiment One of the present application is shown.

[0040] The training method of the human-computer interaction model of the present embodiment includes the following steps:

[0041] Step S202: Obtain the node attribute text of the sample node in the knowledge graph sample, and generate the node representation of the sample node through the pre-trained graph model.

[0042] One of the purposes of the training of the human-computer interaction model of the present embodiment is to bridge the modal difference between the graph model and the first interactive language model. Therefore, for the graph model in the human-computer interaction model, a pre-trained graph model can be used, and the parameters of the graph model are frozen during the training, that is, the parameters of the graph model remain unchanged during the training of the human-computer interaction model.

[0043] In one possible implementation, the knowledge graph can be implemented in the form of a TAG (Text Attributed Graph). A TAG is a type of graph with text as node attributes. In addition to the structural information of the graph itself, the TAG also provides important text information of the node through the text attributes corresponding to the node. Therefore, the structural information of the graph, the text information, and the mutual relationship between the two need to be considered at the same time. However, because of this, the attribute information of the node can be directly collected and used.

[0044] In this embodiment, the TAG is represented as The graph model is represented as g θ (A, X) with A set of nodes of the TAG; A e {0, 1} N×N , representing the adjacency matrix of the TAG; for Each node υ in the TAG is associated with a sequential text feature, denoted as S υ . For g θ (A, X) representing the GNN, θ represents a set of learnable parameters, and X represents the node representation processed by a shallow method such as a BoW model (bag-of-words model) or a skip-gram model (skip-gram model). For example, in GraphSAGE, a fixed number of neighbor nodes around the target node υ are sampled, and the previous layer vector of the target node υ is concatenated with the aggregated neighbor node vector to obtain the local graph information of the target node υ as follows:

[0045]

[0046] where σ() represents an activation function, and W k represents a weight matrix. Then, the pre-trained graph model g θ* encodes the local graph information of the node to obtain the node representation of the target node υ, denoted as z υ , g θ* (A, X) υ .

[0047] The node attribute text of the sample node can be directly obtained through the attribute information of the node in the form of text.

[0048] Step S204: Based on the node attribute text, a generator is used to generate a node description text for describing the sample node.

[0049] In order for the subsequent first interactive language model to effectively understand the node representation output by the graph model, modal alignment data needs to be constructed, and the node description text used to describe the sample node is an important part of the modal alignment data.

[0050] For a certain node, although the node attribute text can represent the attributes of the node, it still cannot fully and objectively express the characteristics of the node. Therefore, in a feasible manner, based on the node attribute text of the sample node and the node attribute text of the neighbor nodes of the sample node, the generator can generate node information text for representing the characteristics of the sample node itself, neighbor information text for representing the characteristics of the neighbor nodes of the sample node, and common information text for representing the commonality between the sample node and the neighbor nodes; based on the node information text, the neighbor information text and the common information text, the generator can generate node description text for describing the sample node. Through the node description text, the characteristics of the node can be more fully and objectively represented.

[0051] In the foregoing manner, the node description text can be generated based on the node attribute text of the sample node and the node attribute text of the neighbor nodes of the sample node. In a feasible manner, the process of generating the node description text can be implemented as follows: based on the node attribute text of the sample node, the second interactive language model in the generator performs node self-characteristic analysis processing to generate node information text for representing the characteristics of the sample node itself; based on the node attribute text of the neighbor nodes of the sample node, the second interactive language model performs neighbor node characteristic analysis processing to generate neighbor information text for representing the characteristics of the neighbor nodes of the sample node; and based on the node information text and the neighbor information text, the second interactive language model performs node commonality analysis processing to generate common information text for representing the commonality between the sample node and the neighbor nodes. The second interactive language model itself has strong reasoning and open task execution capabilities, and on this basis, in a feasible manner, the second interactive language model can be used to construct high-quality node description text based on the CoT (Chain of Thought, thought chain) method. CoT generates output by triggering the step-by-step reasoning of the dialogue agent, and is an improved prompting strategy that can improve the performance of the second interactive language model such as LLM in complex reasoning tasks. In this way, the obtained node description text can be more accurate.

[0052] Taking the second interactive language model as an example, an exemplary generation process of the second LLM for constructing node description text along the three key dimensions is as follows:

[0053] Node information text: In TAG, each node has node attribute text, which is regarded as one of the characteristics of each node. It should be noted that in the present embodiment, the node attribute text is used to describe the attributes of each node in TAG. The node attribute text can be converted into a text vector form by, for example, a bag-of-words model or a word vector model. Based on this, the generator can use the LLM to perform feature summarization and analysis processing on the text vector of the node to obtain the description information of the node itself characteristics, which is called node information text, denoted as

[0054] Neighbor information text: A subset of neighboring nodes of the current sample node can be randomly extracted, and their representations can be aggregated to generate a neighbor information vector. Exemplarily, the node attribute texts of the sample node itself and the neighbor nodes can be fused together by concatenation splicing, and then the LLM is used to analyze the characteristics of the neighbor nodes to summarize and analyze the attributes of the neighbor nodes , to obtain the description information of the neighbor node characteristics, which is called neighbor information text, denoted as

[0055] Common information text: Considering that GraphSAGE is operated as a low-pass filter, in order to prompt the similarity between the sample node and its neighbor nodes for the purpose of smoothing, the present example also instructs the LLM to integrate and summarize the common information between the two. Exemplarily, the LLM can be used to perform node commonality analysis processing according to and to infer the commonality between the sample node υ and its neighbor nodes , to obtain the common information text, denoted as

[0056] After obtaining and , the high-quality node description text t υ , i.e.

[0057] Step S206: Constructing modal alignment data based on the node representation and the node description text of the sample node.

[0058] After obtaining the node representation z υ and the node description text t υ of the sample node, the modal alignment data can be constructed, where N P is a subset of nodes N.

[0059] ​​Step S208: using the modal alignment data, first-stage training is performed on the translation model connecting the graph model and the first pre-trained interactive language model to obtain a translation model capable of converting the output of the graph model into an input acceptable to the first interactive language model.

[0060] The first interactive language model is a language model for performing open tasks, including but not limited to LLM.

[0061] In this step, both the graph model and the first interactive language model are pre-trained models. Because of the modal difference between them, the first interactive language model cannot directly understand the node representation. Therefore, in this embodiment, the translation model between the graph model and the first interactive language model is trained based on the constructed modal alignment data, so that the trained translation model can not only convert the output of the graph model into an input acceptable to the first interactive language model, such as a text input or a TOKEN input, but also make the node representation of the node closer to its node description text, so as to realize accurate conversion of the node representation to the text representation.

[0062] To this end, in one feasible manner, the node representation and the node description text in the modal alignment data can be respectively subjected to feature extraction to obtain corresponding node feature vectors and node text vectors; and the translation model is subjected to first-stage training based on the attention processing results of the node feature vectors and the node text vectors. Through attention processing, more effective feature vectors can be obtained.

[0063] Optionally, the training of the translation model based on the attention processing results of the node feature vectors and the node text vectors can include: performing self-attention processing on a preset text token vector corresponding to a learnable token vector and a node text vector to obtain corresponding learnable token attention features and node text attention features; performing cross-attention processing on the learnable token attention features and the node feature vectors to obtain node attention features; and performing first-stage training on the translation model according to the difference between the node attention features and the node text attention features. In this way, more effective feature extraction can be performed, and the two parts of features can also be effectively fused to obtain feature vectors that are more comprehensive and objective.

[0064] In one specific example, for a pair (z υ , t υ ) in the modal alignment data, two encoders, denoted as f z (·) and f t (·), are used to extract language features for alignment. Among them, for the node description text t υ , a text encoder f t(·) (such as BERT) to extract features, that is, f t (t υ ), to obtain the corresponding node text vector T υ =f t (t υ ), where f t (·) can be implemented as an encoder containing 12 Transformer blocks. And for the node representation z... υ The encoder f can be used z (·) Perform feature extraction. z (·) can also be implemented as an encoder based on the Transformer architecture. In its implementation, this encoder can take M learnable token vectors as input; these M learnable token vectors are called query tokens, denoted as... via encoder f z The processing of (·) outputs M features. Furthermore, H υ =f z (Q, z) υ ).

[0065] To extract z υ In and t υ The most relevant information, in the specific implementation, is that Q is designed to use a self-attention layer for self-attention processing, and then connected to z through a cross-attention layer. υ Perform cross-attention processing to interact with z υ To interact and coordinate f t and f z The self-attention layer between t υ Interact with each other.

[0066] Based on the above process, the node attention feature H can be obtained. υ and node text attention features T υ Furthermore, H can be calculated based on a preset loss function, such as the cross-entropy loss function. υ and T υ The difference is the loss value. The translation model is trained in the first stage based on this difference until the termination condition of the first stage training is met, such as reaching a preset number of training times, or the loss value reaching a preset threshold, etc.

[0067] Through the first stage of training, the translation model can serve as an effective modal conversion bridge between the graph model and the interactive language model. In order to make the interaction between the translation model and the interactive language model more smooth, and to further improve the modal conversion accuracy of the translation model, in a feasible manner, the human-computer interaction model of the embodiments of the present application adopts a two-stage training method. Based on this, on the basis of completing the first stage of training, further, the modal alignment data can be taken as input, the modal alignment data is processed through the translation model completing the first stage of training, and the processing result is converted into a parameter that can be accepted by the first interactive language model; through the first interactive language model, the parameter is used to generate a reply, and based on the generated reply, the translation model is trained in the second stage.

[0068] In a specific implementation, when the reply is generated through the first interactive language model based on the parameter, the parameter can be taken as prompt data of the first interactive language model, combined with question sample data, as input of the first interactive language model; through the first interactive language model, the input is used to generate a reply. Then, based on the generated reply, the translation model is trained in the second stage. In the form of prompt data, the data output by the translation model is introduced into the interactive language model, and is used as input of the interactive language model together with the question sample data. Based on the generated reply, the modal alignment data input into the translation model is used to optimize the translation model again, i.e., the second stage of training. Thus, through the translation model, the node representation output by the graph model can be better aligned with the input vector acceptable by the first interactive language model, so that the first interactive language model can more accurately and effectively generate corresponding reply data.

[0069] In the following, the training process of the human-computer interaction model is exemplarily illustrated in the form of a specific example, as shown in Figure 3C . Figure 3C In the example shown in the example, the graph model is a pre-trained GraphSAGE model Figure 3C , which is schematically shown as a "frozen graph model", the first interactive language model is a pre-trained LLM model Figure 3C , which is schematically shown as a "frozen first LLM", the generator Figure 3C , which is schematically shown as a "Producer", is implemented based on a second interactive language model (specifically, an LLM model Figure 3C , which is schematically shown as a "second LLM", and the translation model Figure 3CThe example of the “Translator” illustrated in FIG. 1 is implemented based on an encoder structure, and the knowledge graph used for training is a TAG-form knowledge graph. In this example, the specific knowledge field of the knowledge graph is not limited, and can be set and selected by those skilled in the art according to actual needs.

[0070] The exemplary training process includes:

[0071] Process 1: Obtain a pre-trained graph model.

[0072] The pre-training of the graph model, specifically GraphSAGE in this example, can be implemented in a conventional manner, which will not be described in detail herein. The pre-trained graph model has the ability to perform a predefined task. In this example, the pre-trained GraphSAGE can process data of a graph structure, including but not limited to extracting features of nodes in a graph structure such as a knowledge graph. Taking a TAG as an example, GraphSAGE can extract node features of the TAG.

[0073] Exemplarily, GraphSAGE can sample a fixed number of neighbor nodes Then, the previous layer vector of the target node υ is concatenated with the aggregated neighbor node vector to obtain the local graph information of the target node υ as follows:

[0074]

[0075] where σ() represents an activation function, and W k represents a weight matrix. Then, the pre-trained graph model g θ* encodes the local graph information of the node to obtain the node representation of the target node υ, denoted as z υ , g θ* (A, X) υ .

[0076] It should be noted that, in the first stage training or the second stage training, the parameters of GraphSAGE remain unchanged, such as Figure 3C the “frozen graph model” illustrated in FIG. 1.

[0077] Process 2: Obtain a pre-trained first interactive language model.

[0078] In this example, the first interactive language model is specifically a first LLM. The first LLM is pre-trained on a wide range of text corpus to obtain a large amount of knowledge. In the first stage training, the parameters of the LLM remain unchanged, and in Figure 3CThe specific LLM is taken as an example of CHATGLM2-6B, which adopts a processing mode of autoregressive blank filling task, which involves a "destroy and rebuild" strategy, in which part of the original text is covered (destroyed) and then predicted (rebuilt). After extensive training on a large corpus, CHATGLM2-6B can obtain the ability to retain a large amount of knowledge and provide reasonable answers to queries.

[0079] Process 3: The processing process of the generator in the first stage of training.

[0080] In this example, the generator Producer is implemented by a second LLM, which is used to construct node description text from three dimensions of node information text, neighbor information text and common information text based on the CoT method when generating high-quality node description text. The second LLM is also a pre-trained LLM.

[0081] Specifically:

[0082] Node information text: In TAG, each node has node attribute text, which is considered as one of the characteristics of each node. In this example, node attribute text is used to describe the attributes of each node in TAG. The node attribute text can be converted into a text vector form by, for example, a bag-of-words model or a word vector model. Based on this, the generator can use the second LLM to summarize and analyze the text vector of the node to obtain the description information of the node's own characteristics, which is called node information text, denoted as

[0083] Neighbor information text: A subset of neighboring nodes of the current sample node can be randomly selected and their representations can be aggregated to generate a neighbor information vector. Exemplarily, the node attribute texts of the sample node itself and its neighbor nodes can be fused together by weighted summation or concatenation fusion, and then the second LLM is used for neighbor node feature analysis processing to summarize and analyze the attributes of the neighbor nodes, to obtain the description information of the neighbor node characteristics, which is called neighbor information text, denoted as

[0084] Common information text: Considering that GraphSAGE is run as a low-pass filter, in order to prompt the similarity between the current node and its neighbor nodes for the purpose of smoothing, the second LLM is guided to integrate and summarize the common information between the two. Exemplarily, the second LLM can be used to analyze the commonality between the current node υ and its neighbor nodes and to infer the commonality between the current node υ and its neighbor nodes , to obtain the common information text, denoted as ​

[0085] After obtaining and , a high-quality node description text t can be constructed by connecting and υ , that is

[0086] As can be seen from the above process, for the second LLM used to implement the generator, the input is the node attribute text of a certain node collected from the knowledge graph and the node attribute text of the neighbor nodes of the node, and the second LLM will output the node description text t υ after inference and analysis processing.

[0087] In the process of generating the above text, the second LLM will process based on prompt data such as the prompt data in Figure 3C to generate a corresponding reply based on which the node description text is determined. For the generation of the node information text, the exemplary prompt data is, for example, "Please summarize the information of the node", and the reply obtained by the second LLM based on the prompt data and the attribute text of the node can be used as the node information text, such as "The user's interest is ……"; for the generation of the neighbor information text, the exemplary prompt data is, for example, "Please summarize the information of the neighbor nodes", and the reply obtained by the second LLM based on the prompt data and the attribute text of the neighbor nodes of the node can be used as the neighbor information text, such as "The interest of the neighbor is ……"; for the common information text, the exemplary prompt data is, for example, "Please summarize their commonality", and the reply obtained by the second LLM based on the above node information text and the neighbor information text can be used as the common information text, such as "Their commonality is ……".

[0088] After obtaining the node representation z υ and the node description text t υ , the modal alignment data (z υ , t υ ) can be constructed to train the translation model using the modal alignment data.

[0089] Process 4: Processing process of the translation model in the first stage of training.

[0090] There is a modal difference between the trained graph model such as GraphSAGE and the first interactive language model such as the first LLM, and the first LLM cannot directly understand the node representation output by GraphSAGE, that is, the size of the node representation vector and the input token vector of the first LLM is different and has different feature spaces. In order to solve this difference, a translation model is introduced in the embodiment of the present application, Figure 3CThe Translator is intended to align the graph model and the interactive language model by converting the learned node representation into TOKENs.

[0091] While in a relatively simple way, one can apply a simple trainable projection matrix to convert z υ into TOKENs in the language feature space, aligning their dimensionality with that of the word vector space in the first LLM. But this simple conversion is hard to extract and translate the complex information contained in z υ into natural language, and is hard to generalize to unseen nodes.

[0092] To this end, in the Translator of the present example, for a pair (z υ , t υ ) in the modal alignment data, two encoders, denoted as f z (·) and f t (·) respectively, are used to extract language features for alignment. Here, for the node description text t υ , a text encoder f t (·) (e.g. BERT) is used to extract features, i.e. f t (t υ ), to obtain the corresponding node text vector T υ = f t (t υ ). Here, f t (·) can be implemented as an encoder containing 12 Transformer blocks. And for the node representation z υ , an encoder f z (·) is used for feature extraction. f z (·) can also be implemented as an encoder based on Transformer architecture. In a specific implementation, this encoder can take M learnable TOKEN vectors as input, which are called query TOKENs, Figure 3C denoted as Q in the figure. Via the processing of the encoder f υ (·), M features are output and H z = f υ (Q, z υ ).

[0093] To extract the most relevant information in z υ to t υ , in a specific implementation, Q is designed to be processed by a self-attention layer for self-attention, and then processed by a cross-attention layer with z υTo interact and coordinate f t and f z The self-attention layer between t υ Interact with each other.

[0094] Based on the above process, the node attention feature H can be obtained. υ and node text attention features T υ Furthermore, H can be calculated based on a preset loss function, such as the cross-entropy loss function. υ and T υ The difference is the loss value ( Figure 3C The first stage loss is indicated in the diagram. The translation model is trained in the first stage based on this difference until the first stage training termination condition is met, such as reaching a preset number of training iterations or the loss value reaching a preset threshold.

[0095] During the actual training, the node representation z output by the pre-trained graph model GraphSAGE is first processed. υ Freeze the process and train only the translation model (Translator). This is to align the H... υ and (T υ The [CLS]TOKEN vector (in the form of input acceptable to the first LLM) is first processed by maximizing H. υ and Mutual information to align H υ and For example, H can be calculated first. υ and The similarity between each token is calculated pairwise, and the highest similarity is selected as the similarity score between the two tokens. Then, the similarity of positive sample pairs is compared with the similarity of negative sample pairs. Secondly, the Translator is trained to generate descriptive text for nodes based on given representation text. Specifically, this can be achieved through f... z The query TOKEN in (·) is Extracting a given z υ Basic information, and then through sharing the self-attention layer ( Figure 3C The “SharedSelf-Attention” in fz(·) is seamlessly passed to the text TOKEN(·). Figure 3C (This is illustrated as "DescriptionTokens"). Therefore, the [CLS]TOKEN for the text generation task can be replaced with the [DEC]TOKEN. Furthermore, by optimizing based on Q and z... υ Generated text features H υ The feature T corresponding to the actual node description text υ Cross-entropy loss between t and Q, Q is forced to capture with t υRelated z υ More details on H. υ and T υ Matching is performed to learn finer-grained alignment. Then, each token can be... v，i ∈R v . and T υ [CLS]TOKEN The data are concatenated and fed into a binary classifier, where a matching score is calculated by averaging the logits of all queries. The Translator is then trained in the first stage based on this matching score.

[0096] This first stage of training enables the Translator to align the node representations of the graph model output with the text.

[0097] Process 5: The processing of the translation model in the second stage of training.

[0098] In the second stage of training, a linear layer can be used to process the output of the Translator, i.e., the TOKEN vector H. υ The projected vector is projected onto the same dimension as the word vectors of the first LLM. The projected vector can be viewed as a soft cue, analogous to human language instructions (…). Figure 4A The diagram illustrates a sequence of connections (e.g., "Please summarize the interests of users and their neighbors...") used as input to the first LLM. Then, through generative learning from the first LLM, the Translator parameters are fine-tuned using modal alignment data. Thus, the node representation z... υ This allows the word vectors to be aligned with the word vectors of the first pre-trained LLM.

[0099] The output of the first LLM is response data to human language instructions (also referred to as question data in this embodiment). This response data and the node description text output by the generator can be compared using a preset loss function to calculate the difference (i.e., the loss value) between the two. Figure 4B The loss value is shown as "second stage loss". The Translator is then fine-tuned based on this loss value to achieve parameter optimization.

[0100] The second stage of training enables the Translator to interface with graph models and interactive language models (such as GraphSAGE-first LLM).

[0101] As can be seen, this example uses a two-stage joint training paradigm for the lightweight Translator to gradually bridge the gap between graph models and interactive language models. In the first stage of training, the Translator is trained to generate node representations z from the graph model's output. υ Extracting from t υ Most relevant Hυ In the second stage training, generative learning is performed by connecting the output of the Translator to the frozen interactive language model to continue training the Translator so that its output can be understood by the interactive language model.

[0102] From the above, in the training process of the present example, based on the output of the pre-trained graph model, the generator first texturizes the target node, its neighbor nodes and the commonality of the two to construct modal alignment data, i.e., aligned (node representation and node description text) pairs. In the first stage training, the (node representation, node description text) pairs are used for alignment training. In order to make the node representation better conform to the text prompt and instruction, the Translator is bridged with the first LLM. Then, the (node representation and node description text) pairs are used to fine-tune the Translator in combination with the generative learning of the first LLM. Thus, the human-computer interaction model obtained after two-stage training can be generalized to the unseen node representation in the inference stage, and the open task is realized through the dialogue. Thus, through the training process of the human-computer interaction model of the present example, the graph model GraphSAGE and the first interactive language model, i.e., LLM, are modal aligned to make full use of the effective performance of the LLM on the open task.

[0103] When the human-computer interaction model trained by the first embodiment enters the inference stage, it can be applied to realize the corresponding human-computer interaction. Hereinafter, the human-computer interaction process is described through specific embodiments.

[0104] Embodiment Two

[0105] Referring to Figure 4C FIG. 4 shows a step flowchart of a human-computer interaction method according to the second embodiment of the present application.

[0106] The human-computer interaction method of the present embodiment includes the following steps:

[0107] Step S302: Determine the knowledge graph corresponding to the input question data, and obtain the node data corresponding to the node matching the question data in the knowledge graph through the graph model.

[0108] In this embodiment, the input question data can be question data input by a user through a user device, and the user device can be any appropriate user device, including but not limited to a smart interactive device such as a smart speaker, a mobile terminal, etc. In addition, different fields or industries or services or applications have different knowledge graphs, and for a certain question data, the knowledge graph to be used can be determined according to the field to which the question data belongs. However, a single question data can only involve part of the knowledge graph, such as a branch or a sub-branch of the knowledge graph, etc. Therefore, in this embodiment, the knowledge graph corresponding to the input question data can be a whole knowledge graph, or the part of the knowledge graph related to the question data in a whole knowledge graph. When determining the part of the knowledge graph related to the question data, methods such as traversal method or node similarity matching method can be used to determine the node matching the question data, and then determine the associated nodes associated with the node, and the part of the knowledge graph formed by the node and the associated nodes is taken as the knowledge graph corresponding to the question data.

[0109] When the knowledge graph corresponding to the question data is implemented in the form of TAG, this step can be implemented as follows: determining a text attribute graph corresponding to the input question data, and determining a node in the text attribute graph matching the question data; obtaining text attribute data of the matching node; and performing feature extraction on the text attribute data of the node by the graph model to obtain a node vector representing the node data. In this way, on the one hand, the node data in the TAG is more easily aligned with the interactive language model for conversion processing suitable for the interactive language model; on the other hand, the graph model is also convenient for feature extraction.

[0110] When the human-computer interaction model obtained by using the embodiment one is used to perform this step, the graph model of the human-computer interaction model can perform node feature extraction on the TAG to obtain a node vector representing the node data, such as z υ .

[0111] Step S304: generating input data acceptable to the first interactive language model based on the node data.

[0112] The first interactive language model is a language model for performing open tasks, including but not limited to LLM.

[0113] Specifically, the first interactive language model is mainly used for human-computer question and answer interaction based on graph data (such as knowledge graph), but as mentioned above, the node data output by the graph model cannot be directly processed by the first interactive language model, so it needs to be converted to generate input data, such as text data, which can be accepted by the first interactive language model.

[0114] Based on this, in a feasible manner, this step can be implemented as: obtaining corresponding text features based on the node vectors corresponding to the node data; and mapping the text features to generate text input data that can be accepted by the first interactive language model.

[0115] Among them, obtaining corresponding text features based on the node vectors corresponding to the node data can be implemented as: performing cross-attention calculation on the node vectors corresponding to the node data to obtain corresponding text features.

[0116] When using the human-computer interaction model obtained by training in embodiment one, this step can be implemented by the translation model therein, such as: through the cross-attention layer of the translation model, performing cross-attention calculation on the node vectors corresponding to the node data, i.e. z υ to obtain corresponding text features. And because the interactive language model can accept text data, the text features can be converted to obtain corresponding text data, i.e. text input data that can be accepted by the first interactive language model.

[0117] Step S306: generating a reply based on the input data and the question data through the first interactive language model to generate reply data corresponding to the question data.

[0118] Among them, the question data can be used as the human-computer interaction instruction of the first interactive language model, and the input data can be used as the graph node data corresponding to the human-computer interaction instruction, so that the first interactive language model performs open task processing based on the graph node data to generate corresponding reply data.

[0119] As can be seen, in order to effectively combine and utilize the capabilities of the interactive language model and the graph model, after determining the knowledge graph corresponding to the question data, the node data of the node in the knowledge graph that matches the question data is first extracted by the graph model GM in this embodiment. Although GM can accurately determine the node data, it lacks the ability of open reply generation, and the node data cannot be accepted by the interactive language model (i.e. the first interactive language model) such as LLM, and there is a modal difference between the two. Therefore, in this embodiment, input data that can be accepted by the interactive language model (i.e. the first interactive language model) such as LLM is generated based on the determined node data, and the interactive language model (i.e. the first interactive language model) generates an open reply based on the input data and the question data using its own open task processing capability. Thus, the human-computer interaction scheme of this embodiment not only ensures the accuracy of task processing, but also fully utilizes the ability of the interactive language model to process open tasks, so that the scheme has good openness and can be flexibly applied to various scenes and environments.

[0120] The following is an example of the above human-computer interaction process in the form of a specific scenario.

[0121] In one scenario example of human-computer interaction based on a form corresponding knowledge graph, the knowledge graph is implemented as a form TAG, each node in the TAG corresponds to a form, and the attribute text includes but is not limited to the metadata of the form (i.e. the structural information of the form) and the content data (i.e. the content information of the form).

[0122] Based on this, the human-computer interaction method can include: determining a form text attribute graph TAG corresponding to the input form question, and obtaining node data corresponding to the node in the form TAG that matches the form question through a graph model, wherein the node in the form TAG corresponds to a form entity, and the form text attribute of the node is used to describe the form structure information and / or the form content information; based on the node data, generating input data that can be accepted by a first interactive language model, wherein the first interactive language model is a language model for performing an open task; based on the input data and the form question, generating a reply through the first interactive language model to generate reply data corresponding to the form question.

[0123] Behind various human-computer interactions is a large amount of various data, and in many cases, these data are stored in the form of forms. Due to the correlation between the data, usually one form is associated with one or more other forms. Based on this, a graph structure can be used to represent and associate these forms, for example, TAG is used to represent and associate these forms in this example. One node in the TAG represents one form entity, and the edges between different nodes represent the association relationship between different form entities. For each node, attribute text can be used to record its structure information and / or content information. Thus, based on the TAG, various form data human-computer question and answer interactions can be performed through the human-computer interaction model in the embodiments of the present application.

[0124] When the human-computer interaction model trained by the embodiment one is used to implement this example, as shown in Figure 4B . Assuming that the form question is "Please help me check the downstream form of using form A", then first, the device where the human-computer interaction model is located will determine the form TAG corresponding to the form question based on the form question, and in this example, the TAG is set as TAG-1.

[0125] On this basis, the human-computer interaction model will process the nodes involved in the form question in TAG-1 through its graph model such as GraphSAGE to extract the corresponding node data, and in this example, the node vector z υ is extracted. υ Further, the node vector z υ. Further, the node text vector H υ The input data in text form will be input into the first interactive language model, such as the first LLM, in the human-computer interaction model together with the form question in text form, and the first LLM will generate an open reply to generate reply data corresponding to the form question, such as "The downstream forms of Form A are: Form C, Form P, and Form H."

[0126] As can be seen, through this example, the human-computer question and answer interaction based on the form TAG is effectively implemented.

[0127] In another scenario example of human-computer interaction based on a knowledge graph corresponding to user information, the knowledge graph is implemented as a user TAG, and each node in the TAG corresponds to a user entity, and the attribute text can include any attribute information used to describe the user.

[0128] Based on this, the human-computer interaction method can include: determining a user TAG corresponding to an input user information question, and obtaining node data corresponding to a node in the user TAG that matches the user information question through a graph model, wherein the nodes in the user TAG correspond to user entities, and the user text attributes of the nodes are used to describe the attribute information of the users; based on the node data, generating input data that can be accepted by a first interactive language model, wherein the first interactive language model is a language model used to perform an open task; based on the input data and the user information question, generating a reply through the first interactive language model to generate reply data corresponding to the user information question.

[0129] In this example, one node in the TAG represents a user entity, and the edges between different nodes represent the association relationship between different user entities. For each node, attribute text can be used to describe the attributes or characteristics of the user. Thus, based on the TAG, various human-computer question and answer interactions of user information can be performed through the human-computer interaction model in the embodiments of the present application.

[0130] When the human-computer interaction model trained using the embodiment one is used to implement this example, as shown in Figure 4C Assuming that the user information question is "What are the common interests of User A and his friends?", first, the device where the human-computer interaction model is located will determine the user TAG corresponding to the user information question based on the user information question. In this example, the TAG is set as TAG-2.

[0131] On this basis, the human-computer interaction model will process the nodes involved in the user information question in TAG-2 through its graph model such as GraphSAGE (including the node corresponding to user A and the node in the friend relationship with user A), to extract the node data corresponding to these nodes, which is represented as a node vector z in this example υ . Further, the node vector z υ will be input into the translation model of the human-computer interaction model to extract the corresponding node text vector H υ . In turn, the node text vector H υ will be converted into input data in the form of text (for example, by projecting the vector into text). The input data in the form of text will be input into the first interactive language model such as the first LLM in the human-computer interaction model together with the user information question in the form of text, and the first LLM will generate an open reply to generate reply data corresponding to the user information question, such as "user A and his friends all like fishing".

[0132] As can be seen, through this example, the human-computer question and answer interaction based on user TAG is effectively realized.

[0133] It should be noted that the above Figure 5 and Figure 5 scene examples are exemplary, and those skilled in the art can apply the schemes of the embodiments of the present application to various human-computer interaction scenarios based on graph data according to the schemes of the multiple embodiments of the present application and the above examples, which are all within the protection scope of the present application.

[0134] Embodiment Three

[0135] Referring to ​ , a structural schematic diagram of an electronic device according to an embodiment of the present application is shown, and the specific implementation of the electronic device is not limited by the embodiments of the present application.

[0136] As shown in ​ , the electronic device can include a processor 502, a communications interface 504, a memory 506, and a communications bus 508.

[0137] Among them:

[0138] The processor 502, the communications interface 504, and the memory 506 complete mutual communication through the communications bus 508.

[0139] The communications interface 504 is configured to communicate with other electronic devices or servers.

[0140] The processor 502 is configured to execute the program 510, and specifically, execute the steps of any of the methods described above.

[0141] Specifically, the program 510 can include program codes, which include computer operation instructions.

[0142] The processor 502 can be a CPU, or a GPU (Graphic Processing Unit), or an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the smart device can be the same type of processor, such as one or more CPUs; or can be different types of processors, such as one or more CPUs and one or more ASICs.

[0143] The memory 506 is configured to store the program 510. The memory 506 can include a high-speed RAM memory, and can further include a non-volatile memory, such as at least one disk memory.

[0144] The program 510 can include a plurality of computer instructions, and specifically, the program 510 can cause the processor 502 to perform the operations corresponding to the methods described in any of the above method embodiments.

[0145] The specific implementation of each step in the program 510 can refer to the corresponding description in the above method embodiments, and has corresponding beneficial effects, which will not be described here. Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described devices and modules can refer to the corresponding process description in the above method embodiments, which will not be described here.

[0146] The embodiments of the present application further provide a computer storage medium, which stores a computer program, and the program is executed by a processor to implement the method described in any of the above method embodiments. The computer storage medium includes, but is not limited to, a CD-ROM (Compact Disc Read-Only Memory), a RAM (Random Access Memory), a floppy disk, a hard disk, or a magneto-optical disk, etc.

[0147] The embodiments of the present application further provide a computer program product, which includes computer instructions, and the computer instructions instruct a computing device to perform the operations corresponding to any of the above method embodiments.

[0148] Experiments show that the human-computer interaction model provided in the embodiments of the present application achieves good human-computer interaction performance. Compared with the manner of directly inputting the attribute text of the node itself and the attribute text of the neighbor node into the large language model LLM, the effect is better. This is because the large language model LLM will be affected by a large amount of text noise and complexity if it directly processes the original text containing the attribute text of the node itself and the attribute text of the neighbor node. The human-computer interaction model provided in the embodiments of the present application can extract and summarize the attribute text of the node itself, the attribute text of the neighbor node, and the commonality of the two through the translation model therein, greatly reducing noise interference and the complexity of the content contained in the text.

[0149] In addition, the scheme of the embodiments of the present application also improves the legal detection rate of data. This is because in the manner of only inputting the original attribute text of the node into the large language model LLM, the information contained is limited, thus bringing great reasoning challenges to the large language model LLM. In the manner of inputting the attribute text of the node and the attribute text of its neighbor node into the large language model LLM, although the legal detection rate is improved, there are still problems of text noise and information redundancy. Through the scheme of the embodiments of the present application, in the training phase of the human-computer interaction model, the generator Producer is introduced to concisely summarize the attribute text of the node itself, the attribute text of its neighbor node, and the commonality of the two, providing rich content while reducing noise. Further, the node representation output by the Translator translation graph model is used as a soft graph attribute prompt, which can contain more complex details than stacked text. In addition, in the scheme of the embodiments of the present application, the output of the Translator can be projected into a fixed-length TOKEN, facilitating the understanding of the graph node information by the large language model LLM, and further improving the accuracy of the large language model reply.

[0150] In addition, it should be noted that the information related to the user (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to sample data for training the model, data for analysis, stored data, displayed data, etc.) involved in the embodiments of the present application are all information and data authorized by the user or authorized by all parties, and the collection, use, and processing of related data need to comply with relevant laws, regulations, and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose authorization or refusal.

[0151] It should be noted that according to the needs of implementation, each component / step described in the embodiments of the present application can be split into more components / steps, or two or more components / steps or part of the operations of the components / steps can be combined into new components / steps to achieve the purpose of the embodiments of the present application.

[0152] The methods according to the embodiments of the present application described above can be implemented in hardware, firmware, or software, or a combination thereof, and can be stored in a recording medium such as a CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk, or be downloaded by a network from a remote recording medium or non-transitory machine-readable medium originally stored in a local recording medium and then stored in a local recording medium, so that the methods described herein can be processed by such software processing using a general-purpose computer, a special-purpose processor, or programmable or special-purpose hardware such as an Application Specific Integrated Circuit (ASIC) or a Field Programmable Gate Array (FPGA). It can be understood that the computer, processor, microprocessor controller, or programmable hardware includes a storage component (for example, Random Access Memory (RAM), Read-Only Memory (ROM), flash memory, etc.) that can store or receive software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods described herein. Furthermore, when a general-purpose computer accesses code for implementing the methods shown herein, the execution of the code converts the general-purpose computer into a special-purpose computer for executing the methods shown herein.

[0153] Those skilled in the art can appreciate that the units and method steps of the examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether the functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. A skilled person can use different methods to implement the described functions for a specific application, but such implementation should not be considered beyond the scope of the embodiments of the present application.

[0154] The above embodiments are only used to illustrate but not limit the embodiments of the present application, and a person of ordinary skill in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of the present application, therefore all equivalent technical solutions belong to the scope of the embodiments of the present application, and the patent protection scope of the embodiments of the present application should be defined by the claims.

Claims

1. A human-computer interaction method, comprising: determining a knowledge graph corresponding to input question data, and obtaining node data corresponding to a node in the knowledge graph that matches the question data through a graph model; generating input data acceptable to a first interactive language model through a translation model based on the node data, wherein the first interactive language model is a language model for performing open tasks, and the translation model is trained to convert a node vector corresponding to the node data into text input data acceptable to the first interactive language model; generating a reply through the first interactive language model based on the input data and the question data to generate reply data corresponding to the question data.

2. The method of claim 1, wherein, The generating input data acceptable to a first interactive language model through a translation model based on the node data comprises: obtaining corresponding text features based on a node vector corresponding to the node data; mapping the text features to generate text input data acceptable to the first interactive language model.

3. The method of claim 2, wherein, The obtaining corresponding text features based on a node vector corresponding to the node data comprises: performing cross-attention calculation on the node vector corresponding to the node data to obtain corresponding text features.

4. The method according to any one of claims 1 to 3, wherein, The determining a knowledge graph corresponding to input question data, and obtaining node data corresponding to a node in the knowledge graph that matches the question data through a graph model comprises: determining a text attribute graph corresponding to input question data, and determining a node in the text attribute graph that matches the question data; obtaining text attribute data of the matching node; performing feature extraction on the text attribute data of the node through a graph model to obtain a node vector representing the node data.

5. A human-computer interaction model training method, comprising: obtaining node attribute text of a sample node in a knowledge graph sample, and generating node representation of the sample node through a pre-trained graph model; generating node description text for describing the sample node through a generator based on the node attribute text; constructing modal alignment data based on the node representation and the node description text; using the modal alignment data to perform first-stage training on a translation model connecting the graph model and a pre-trained first interactive language model to obtain a translation model capable of converting an output of the graph model into input acceptable to the first interactive language model; wherein the first interactive language model is a language model for performing open tasks.

6. The method of claim 5, wherein, The generating node description text for describing the sample node through a generator based on at least the node attribute text comprises: generating node information text for representing features of the sample node itself, neighbor information text for representing features of neighbor nodes of the sample node, and common information text for representing commonalities between the sample node and the neighbor nodes through a generator based on node attribute text of the sample node and node attribute text of neighbor nodes of the sample node. generate, based on the node attribute text of the sample node and the node attribute text of the neighbor node of the sample node, a node information text for representing the characteristics of the sample node itself, a neighbor information text for representing the characteristics of the neighbor node of the sample node, and a commonality information text for representing the commonality between the sample node and the neighbor node, by a generator, including:

7. The method of claim 6, wherein, generate, based on the node attribute text of the sample node, a node information text for representing the characteristics of the sample node itself, by a second interactive language model in the generator; generate, based on the node attribute text of the neighbor node of the sample node, a neighbor information text for representing the characteristics of the neighbor node of the sample node, by the second interactive language model; generate, based on the node information text and the neighbor information text, a commonality information text for representing the commonality between the sample node and the neighbor node, by the second interactive language model. The first-stage training of the translation model connecting the graph model and the first interactive language model pre-trained includes:

8. The method according to any one of claims 5-7, wherein, respectively extracting features of the node representation and the node description text in the modal alignment data to obtain corresponding node feature vectors and node text vectors; based on the attention processing results of the node feature vectors and the node text vectors, the first-stage training of the translation model. The training of the translation model based on the attention processing results of the node feature vectors and the node text vectors includes:

9. The method of claim 8, wherein, performing self-attention processing on the preset learnable token vector and the text token vector corresponding to the node text vector to obtain corresponding learnable token attention features and node text attention features; performing cross-attention processing on the learnable token attention features and the node feature vectors to obtain node attention features; based on the difference between the node attention features and the node text attention features, the first-stage training of the translation model. The method further includes:

10. The method of any one of claims 5-7, wherein, taking the modal alignment data as input, processing the modal alignment data by the translation model after the first-stage training, and converting the processing result into parameters acceptable by the first interactive language model; based on the parameters, generating a reply by the first interactive language model, and based on the generated reply, performing second-stage training of the translation model. The generation of a reply by the first interactive language model based on the parameters includes:

11. The method of claim 10, wherein, taking the parameters as prompt data of the first interactive language model, combining question sample data as input of the first interactive language model; generating a reply by the first interactive language model based on the input.

12. A human-computer interaction method, comprising: ​ determine a form text attribute graph corresponding to an input form question, and obtain node data corresponding to a node matching the form question in the form text attribute graph through a graph model, wherein the node in the form text attribute graph corresponds to a form entity, and a form text attribute of the node is used to describe form structure information and / or form content information; generate input data acceptable to a first interactive language model through a translation model based on the node data, wherein the first interactive language model is a language model used to perform an open task, and the translation model is trained to convert a node vector corresponding to the node data into text input data acceptable to the first interactive language model; generate a reply through the first interactive language model based on the input data and the form question to generate reply data corresponding to the form question.

13. A human-computer interaction method, comprising: determining a user text attribute graph corresponding to an input user information question, and obtaining node data corresponding to a node matching the user information question in the user text attribute graph through a graph model, wherein the node in the user text attribute graph corresponds to a user entity, and a user text attribute of the node is used to describe attribute information of the user; generate input data acceptable to a first interactive language model through a translation model based on the node data, wherein the first interactive language model is a language model used to perform an open task, and the translation model is trained to convert a node vector corresponding to the node data into text input data acceptable to the first interactive language model; generate a reply through the first interactive language model based on the input data and the user information question to generate reply data corresponding to the user information question.

14. An electronic device comprising: a processor, a memory, a communication interface, and a communication bus, the processor, the memory, and the communication interface complete communication with each other through the communication bus; the memory is used to store at least one executable instruction, and the executable instruction causes the processor to perform operations corresponding to the method in any one of claims 1-13.

Images