Spoken-to-written conversion method, device and equipment based on graph attention network

Abstract

This invention provides a method, apparatus, and device for spoken-to-written language conversion based on graph attention networks. The method includes: semantically encoding a spoken document to obtain a semantic representation of the spoken document; determining the initial representation of each node in the document structure graph of the spoken document based on the semantic representation, wherein the document structure graph includes document nodes, sentence nodes, and word segmentation nodes; performing message propagation on the initial representation of each node in the document structure graph based on an attention mechanism to obtain a structure graph representation of the document structure graph; and performing semantic decoding based on the structure graph representation to obtain the written document corresponding to the spoken document. The method, apparatus, and device provided by this invention, by constructing a document graph structure diagram, can obtain a more concise and readable written document, avoiding the omission of spoken terms crossing sentence boundaries during text conversion, and ensuring the effectiveness of document-level spoken text conversion to written text.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a method, apparatus, and device for converting spoken language to written language based on graph attention networks. Background Technology

[0002] Because people use language differently when speaking and writing, and because there may be syntactic and grammatical errors, lack of fluency, and noise during speech recording, the accessibility and readability of spoken text obtained from speech recognition can all be affected.

[0003] Therefore, converting spoken text into written text is crucial for reducing the difficulty of understanding the content. Current research on spoken-to-written text conversion typically involves sentence-by-sentence conversion. However, in reality, at the document level, spoken terms may cross sentence boundaries, and spoken texts are often lengthy and poorly organized. Sentence-by-sentence conversion cannot detect the influence between sentences and can only remove spoken terms, failing to achieve the restructuring and simplification of document-level spoken text, resulting in less than ideal conversion results. Summary of the Invention

[0004] This invention provides a spoken-to-written language conversion method, apparatus, and device based on graph attention networks, which solves the problem that the existing sentence-by-sentence spoken-to-written language conversion method is not suitable for document-level conversion scenarios and has unsatisfactory conversion results.

[0005] This invention provides a spoken-to-written language conversion method based on graph attention networks, comprising:

[0006] Semantic encoding is performed on the spoken language document to obtain its semantic representation;

[0007] Based on the semantic representation of the spoken language document, the initial representation of each node in the document structure graph of the spoken language document is determined. The document structure graph includes document nodes, statement nodes, and word segmentation nodes. The document nodes are connected to the statement nodes corresponding to the spoken language statements belonging to the spoken language document, and the statement nodes are connected to the word segmentation nodes corresponding to each word segmentation belonging to the spoken language statements.

[0008] Using an attention mechanism, message propagation is performed on the initial representations of each node in the document structure graph to obtain the structure graph representation of the document structure graph;

[0009] Semantic decoding is performed using the structure diagram to obtain the written document corresponding to the spoken document.

[0010] According to the present invention, a spoken-to-written language conversion method based on graph attention networks is provided, wherein the initial representation of each node in the document structure graph is propagated through message propagation based on an attention mechanism to obtain a structure graph representation of the document structure graph, including:

[0011] Based on the attention mechanism, message propagation is performed on the initial representations of nodes at the same level in the document structure graph to obtain the hierarchical representations of each node in the document structure graph;

[0012] Based on the attention mechanism, message propagation is performed on the hierarchical representation of each node in the document structure graph to obtain the structure graph representation of the document structure graph.

[0013] According to the present invention, a spoken-to-written language conversion method based on a graph attention network is provided. The method involves message propagation of the initial representations of nodes at the same level in the document structure graph based on the attention mechanism to obtain hierarchical representations of each node in the document structure graph, including:

[0014] Based on each word segmentation node in the document structure graph, a word-level fully connected graph is constructed;

[0015] Based on each statement node in the document structure graph, construct a sentence-level fully connected graph;

[0016] Based on the attention mechanism, message propagation is performed on the initial representation of each word segmentation node in the word-level fully connected graph to obtain the hierarchical representation of each word segmentation node;

[0017] Based on the attention mechanism, message propagation is performed on the initial representation of each statement node in the sentence-level fully connected graph to obtain the hierarchical representation of each statement node.

[0018] According to the present invention, a spoken-to-written language conversion method based on a graph attention network is provided, wherein semantic decoding based on the structural graph representation is performed to obtain the written document corresponding to the spoken document, comprising:

[0019] The semantic representation and the structural graph representation are fused to obtain a fused representation;

[0020] Semantic decoding is performed based on the fused representation to obtain the written document corresponding to the spoken document.

[0021] According to the present invention, a spoken-to-written language conversion method based on graph attention networks is provided, wherein the feature fusion of the semantic representation and the structural graph representation to obtain a fused representation includes:

[0022] Based on the semantic representation and the structural graph representation, a gating attention mechanism is calculated to obtain attention weights;

[0023] Based on the attention weights, the structure graph representation is enhanced to obtain an enhanced graph representation;

[0024] The semantic representation and the enhanced graph representation are fused to obtain the fused representation.

[0025] According to the present invention, a spoken-to-written language conversion method based on graph attention networks is provided, wherein determining the initial representation of each node in the document structure graph of the spoken document based on the semantic representation of the spoken document includes:

[0026] The semantic representation of each word segment in the spoken document is used as the initial representation of each word segment node in the document structure diagram;

[0027] Based on the semantic representation of word segmentation under each sentence in the spoken document, the initial representation of each sentence node in the document structure diagram is determined;

[0028] Based on the initial representation of each statement node in the document structure graph, the initial representation of the document node in the document structure graph is determined.

[0029] The present invention also provides a spoken-to-written language conversion device based on graph attention networks, comprising:

[0030] The encoding unit is used to perform semantic encoding on the spoken document to obtain the semantic representation of the spoken document;

[0031] The graph representation unit is used to determine the initial representation of each node in the document structure graph of the spoken language document based on the semantic representation of the spoken language document. The document structure graph includes document nodes, sentence nodes and word segmentation nodes. The document nodes are connected to the sentence nodes corresponding to the spoken language sentences belonging to the spoken language document, and the sentence nodes are connected to the word segmentation nodes corresponding to each word of the spoken language sentences.

[0032] The graph propagation unit is used to propagate messages to the initial representations of each node in the document structure graph based on an attention mechanism, so as to obtain the structure graph representation of the document structure graph.

[0033] The decoding unit is used to perform semantic decoding based on the structure graph representation to obtain the written document corresponding to the spoken document.

[0034] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the spoken-to-written language conversion method based on graph attention networks as described above.

[0035] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the spoken-to-written language conversion method based on graph attention networks as described above.

[0036] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the spoken-to-written language conversion method based on graph attention networks as described above.

[0037] This invention provides a spoken-to-written language conversion method, apparatus, and device based on graph attention networks. It performs semantic encoding on spoken documents to obtain a semantic representation of the spoken document. Using an attention mechanism, message propagation is performed on the initial representations of each node in the document structure graph to obtain a structure graph representation of the document structure graph. Semantic decoding is then performed using this structure graph representation to obtain a more concise and readable written document. This avoids the problem of missing spoken terms that cross sentence boundaries during text conversion, ensuring the effectiveness of document-level spoken-to-written language conversion. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0039] Figure 1 This is one of the flowcharts illustrating the spoken-to-written language conversion method based on graph attention networks provided by this invention.

[0040] Figure 2 This is a document structure diagram provided by the present invention;

[0041] Figure 3 This is the second flowchart of the spoken-to-written language conversion method based on graph attention networks provided by this invention;

[0042] Figure 4 This is a schematic diagram of the spoken-to-written language conversion device based on graph attention network provided by the present invention;

[0043] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0045] Current technologies for converting spoken to written text often work on a sentence-by-sentence basis, such as converting individual sentences in a spoken document. However, these methods fail to detect the influence between sentences, easily overlooking spoken terms that cross sentence boundaries. Furthermore, they can only delete spoken terms and cannot achieve document-level restructuring and simplification of spoken text, resulting in unsatisfactory conversion results.

[0046] To address the aforementioned problems, this invention provides a spoken-to-written language conversion method based on graph attention networks, enabling document-level conversion of spoken text to written text. Figure 1 This is one of the flowcharts illustrating the spoken-to-written language conversion method based on graph attention networks provided by this invention, such as... Figure 1 As shown, the method includes:

[0047] Step 110: Semantically encode the spoken document to obtain its semantic representation.

[0048] Here, "spoken document" refers to document-level spoken text that requires text conversion. Spoken documents can be converted documents generated by an automatic speech recognition system. Spoken documents typically include multiple conversational sentences. Examples include documents converted from voice recordings during meetings, documents converted in real-time by doctors during clinical diagnoses, and documents converted from customer inquiries using a call center's intelligent conversion function.

[0049] Here, semantic encoding of spoken documents can be achieved through the encoder of a language model, such as the encoder-decoder model Transformer. The semantic representation obtained through encoding can specifically be the semantic representation of each token under each sentence in the spoken document.

[0050] Step 120: Based on the semantic representation of the spoken document, determine the initial representation of each node in the document structure graph of the spoken document. The document structure graph includes document nodes, statement nodes, and word segmentation nodes. The document nodes are connected to the statement nodes corresponding to the spoken statements belonging to the spoken document, and the statement nodes are connected to the word segmentation nodes corresponding to each word of the spoken statement.

[0051] Specifically, a document structure diagram can be constructed based on the document structure of the spoken language document itself, namely the sentences contained in the spoken language document and the word segments contained in each sentence. This document structure diagram reflects the hierarchical structure of the spoken language document from word segments to sentences to the document as a whole. The document structure diagram can include three levels of nodes: word segmentation nodes, sentence nodes, and document nodes. Each spoken language document corresponds to one document node, and each document node includes multiple sentence nodes, and each sentence node includes multiple word segmentation nodes. There are connecting edges between the three levels of nodes; each sentence node is connected to its corresponding document node through an edge, and each word segmentation node is connected to its corresponding sentence node through an edge. For example, Figure 2 This is a document structure diagram provided by the present invention, such as... Figure 2 As shown, document nodes, statement nodes, and word segmentation nodes form a document structure diagram in the form of a tree diagram.

[0052] Furthermore, the initial representation of each node can be determined based on the semantic representation of the spoken document. For example, the semantic representation of each word in the spoken document can be used as the initial representation of each word node in the document structure graph. Alternatively, the semantic representation of each word in the spoken document can be concatenated with the position representation of each word node in the document structure graph as the initial representation of each word node. For another example, after obtaining the initial representation of each word node, a bottom-up average pooling strategy can be used to initialize the statement nodes and document nodes based on the initial representation of each word node.

[0053] Step 130: Based on the attention mechanism, perform message propagation on the initial representation of each node in the document structure graph to obtain the structure graph representation of the document structure graph.

[0054] Specifically, after obtaining the document structure graph, an attention mechanism can be applied for message propagation. During the propagation process, each node in the document structure graph can learn information from other nodes, thereby achieving information propagation between various pieces of information in the document structure graph.

[0055] Through message propagation, the node representation of each node in the document structure graph can incorporate information related to the document structure. The resulting structure graph representation is a semantic feature extracted based on the document structure. Compared with the feature representation of a single word or a single sentence directly obtained in related technologies, it is more adapted to the document structure and reflects richer document information.

[0056] Step 140: Perform semantic decoding based on the structure graph representation to obtain the written document corresponding to the spoken document.

[0057] Specifically, after obtaining the structural graph representation, semantic decoding can be performed based on the structural graph representation. Here, the structural graph representation can be directly applied for semantic decoding, or the structural graph representation and semantic representation can be concatenated before semantic decoding, or the graph representation and semantic representation can be fused through a gating attention mechanism before semantic decoding, thereby obtaining the document after the spoken document has been written, that is, the written document.

[0058] The method provided in this invention constructs a document graph structure diagram and performs message propagation within the document graph structure to obtain a graph representation that covers richer document information. Based on this, decoding can produce a more concise and readable written document, avoiding the problem of missing spoken terms that cross sentence boundaries during text conversion, and ensuring the effectiveness of document-level spoken text conversion to written form.

[0059] Based on the above embodiments, in step 130, message propagation based on the attention mechanism can be achieved through the following principle:

[0060] Message propagation is achieved by applying an attention mechanism. Specifically, masked attention is used to inject the document graph structure into the attention mechanism, forming a graph attention network (GAT) for message propagation. The GAT receives... As input, an initial representation of a set of nodes, and then output... This is a new representation of a set of nodes, also known as a structure graph representation. Graph attention networks operate according to the following equation:

[0061]

[0062] Where h is the initial representation of a set of nodes, and the document structure graph is denoted as... By using parameters h and Inputting a graph attention network, we obtain a new initial representation of a set of nodes using the above formula. .

[0063] Here, the attention mechanism used in this embodiment of the invention is Scaled Dot-Product Attention, which is applied in Transformer. In this embodiment, learnable embeddings are incorporated into the attention calculation to introduce edge information. Specifically, in the graph attention network, the calculation process is as follows:

[0064] First, the coefficient between node i and node j is calculated using the following equation. :

[0065]

[0066] The parameters in the equation include: the initial representation of node i. Initial representation of node j Learnable embedding vectors , That is, the edge between node i and node j.

[0067] Based on this, the attention coefficient between node i and node j can be calculated using the following equation. :

[0068]

[0069] The parameters in the equation include the coefficient between node i and node j. The coefficient between node i and node k , Parameters such as 'k' and 'i'. Here, node k is a node in the set of neighboring nodes of node i. Let k be the set of neighboring nodes of node i. .

[0070] Next, the attention head of node i is calculated from the attention coefficients and edges of nodes i and j. Specifically, attention head The following equation can be used to calculate:

[0071]

[0072] The parameters in the equation include the attention coefficient between node i and node j. Initial representation of a set of nodes j Learnable embedding vectors This refers to the edge between node i and node j. Here, node k is a node in the neighborhood node set of node i. Let k be the set of neighboring nodes of node i.

[0073] Then, by concatenating m attention points, the final output is obtained. The new initial representation of node i is calculated using the following equation. :

[0074]

[0075] Based on any of the above embodiments, step 130 includes:

[0076] Based on the attention mechanism, message propagation is performed on the initial representations of nodes at the same level in the document structure graph to obtain the hierarchical representations of each node in the document structure graph;

[0077] Based on the attention mechanism, message propagation is performed on the hierarchical representation of each node in the document structure graph to obtain the structure graph representation of the document structure graph.

[0078] Specifically, message propagation for a document structure graph can be divided into two phases:

[0079] Specifically, in the first stage, hierarchical graph attention message propagation can be achieved. Nodes at the same level are either word segmentation nodes or statement nodes at the same statement level. In this stage, to better model the interactions between nodes at the same level, including cross-statement interactions, message propagation can be performed first between nodes at the same level. Specifically, message propagation can be performed between word segmentation nodes to obtain a hierarchical representation of each word segmentation node, and message propagation can be performed between statement nodes to obtain a hierarchical representation of each statement node.

[0080] Specifically, in the second stage, message propagation between nodes at different levels in the document structure graph can be implemented. In this stage, to obtain overall document information, the information interaction between document structures needs to be modeled. Specifically, message propagation between document structures can be performed within the already constructed document structure graph. It can be understood that at this point in the document structure graph, the hierarchical representation of document nodes is the initial representation of document nodes, while the hierarchical representations of statement nodes and word segmentation nodes are obtained based on message propagation between nodes at the same level.

[0081] After completing the two-stage message propagation, a structural graph representation of the document structure graph is obtained. It can be understood that this structural graph representation encompasses the node representation after each node has undergone the two-stage message propagation.

[0082] Based on any of the above embodiments, in step 130, the step of performing message propagation on the initial representations of nodes at the same level in the document structure graph based on the attention mechanism to obtain the hierarchical representations of each node in the document structure graph includes:

[0083] Based on each word segmentation node in the document structure graph, a word-level fully connected graph is constructed;

[0084] Based on each statement node in the document structure graph, construct a sentence-level fully connected graph;

[0085] Based on the attention mechanism, message propagation is performed on the initial representation of each word segmentation node in the word-level fully connected graph to obtain the hierarchical representation of each word segmentation node;

[0086] Based on the attention mechanism, message propagation is performed on the initial representation of each statement node in the statement-level fully connected graph to obtain the hierarchical representation of each statement node.

[0087] Specifically, for message propagation among nodes at the same level, word segmentation nodes at the same level can be connected to construct a word-level fully connected graph, and message propagation among all word segmentation nodes can be achieved based on the word-level fully connected graph.

[0088] Assumption This is the initial set of representations for the word segmentation nodes. It is a word-level fully connected graph. In a word-level fully connected graph, the input... and The new initial representation set after message propagation by word-level nodes is calculated using the following equation. That is, the hierarchical representation of each word segmentation node:

[0089]

[0090] In addition, statement nodes at the same level can be connected to construct a sentence-level fully connected graph, and message propagation between all statement nodes can be achieved based on the sentence-level fully connected graph.

[0091] Assuming, This is the initial representation set for statement nodes. It is a sentence-level fully connected graph. Input and The new initial representation set after message propagation by sentence-level nodes is calculated using the following equation. That is, the hierarchical representation of each sentence-level node:

[0092]

[0093] Based on any of the above embodiments, in step 130, message propagation is performed on the hierarchical representation of each node in the document structure graph based on the attention mechanism.

[0094] Let the constructed document graph structure be denoted as... Let h be the initial representation of the document structure diagram. The new structure diagram representation of the document structure diagram is obtained by calculating the following equation. :

[0095]

[0096] Based on any of the above embodiments, step 140 includes:

[0097] The semantic representation and the structural graph representation are fused to obtain a fused representation;

[0098] Semantic decoding is performed based on the fused representation to obtain the written document corresponding to the spoken document.

[0099] Specifically, considering the structural graph representation of the document structure obtained after message propagation, although it incorporates the graph representation information of the document structure, some semantic information may be lost during message propagation. To ensure the effectiveness of spoken language conversion, the semantic representation and the graph representation of the document structure can be feature-fused, and semantic decoding can be performed based on the fused representation.

[0100] Here, feature fusion can be performed by concatenating features from the semantic representation and the structural graph representation of the document structure graph, or by performing a weighted summation of the semantic representation and the structural graph representation of the document structure graph. This embodiment of the invention does not specifically limit the specific method used.

[0101] Based on any of the above embodiments, in step 140, feature fusion is performed on the semantic representation and the structural graph representation of the document structure graph to obtain a fused representation, including:

[0102] Based on the semantic representation and the structural graph representation of the document structure graph, a gating attention mechanism is performed to calculate the attention weights;

[0103] Specifically, the graphical representation of the document structure diagram is denoted as... The semantic representation of a document is denoted as By input parameters And H, the gating attention coefficient is calculated using the following equation. :

[0104]

[0105] Input graph representation G and gating attention coefficients The following equation is used to calculate the result. , This is a new graphical representation of the document structure diagram, specifically a new graphical representation integrated in the form of gated attention. The equation is as follows:

[0106]

[0107] Based on the attention weights, the structural graph representation of the document structure graph is enhanced to obtain an enhanced graph representation;

[0108] Specifically, further representing the new diagram and semantic representation Integrate and input parameters and The following equation is used to obtain The enhanced graph representation is obtained by the new graph representation. The equation is as follows:

[0109]

[0110] The decoder receives the enhanced graph representation of the new graph representation. As input, it generates a text sequence after style conversion on the target side, and finally outputs written style text.

[0111] According to any of the above embodiments, the spoken-to-written language conversion method based on graph attention networks includes step 120 as follows:

[0112] The semantic representation of each word segment in the spoken document is used as the initial representation of each word segment node in the document structure diagram;

[0113] Based on the semantic representation of word segmentation under each sentence in the spoken document, the initial representation of each sentence node in the document structure diagram is determined;

[0114] Based on the initial representation of each statement node in the document structure graph, the initial representation of the document node in the document structure graph is determined.

[0115] Specifically, the document structure diagram reflects the hierarchical structure of the spoken document from word segmentation to sentences and then to the document as a whole. The document structure diagram can include three levels of nodes: word segmentation nodes, sentence nodes, and document nodes. Each spoken document corresponds to one document node, and each document node includes multiple sentence nodes, and each sentence node includes multiple word segmentation nodes.

[0116] Furthermore, the initial representation of each node can be determined based on the semantic representation of the spoken document. For example, the semantic representation of each word in the spoken document can be used as the initial representation of each word node in the document structure graph. Alternatively, the semantic representation of each word in the spoken document can be concatenated with the position representation of each word node in the document structure graph as the initial representation of each word node. For another example, after obtaining the initial representation of each word node, a bottom-up average pooling strategy can be used to initialize the statement nodes and document nodes based on the initial representation of each word node.

[0117] Based on any of the above embodiments, the initial representation of each node in the document structure diagram can be obtained in the following way:

[0118] Assumption Represents a node The level (token, sentence, document), Represents a node The set of neighboring nodes, where This represents the node embedding vector. By inputting the parameters, the initial representation of node i is calculated according to the following formula and output. :

[0119]

[0120] Based on any of the above embodiments Figure 3 This is the second flowchart illustrating the spoken-to-written language conversion method based on graph attention networks provided by this invention. Figure 3 As shown, the spoken-to-written language conversion method based on graph attention network may include the following steps: Step 210, input the spoken document.

[0121] Step 220: Semantically encode the spoken document input in step 210 using a Transformer encoder to obtain the semantic representation of the spoken document.

[0122] Step 230 involves initializing the semantic representation of the spoken document obtained in step 220 into a graph. Specifically, based on the semantic representation of the spoken document, the initial representation of each node in the document structure graph of the spoken document can be determined.

[0123] Step 240 involves message propagation of the initial representations of each node in the document structure graph based on an attention mechanism. This can be divided into two phases:

[0124] Specifically, in the first stage, hierarchical graph attention message propagation can be achieved. Nodes at the same level are either word segmentation nodes or statement nodes at the same statement level. In this stage, to better model the interactions between nodes at the same level, including cross-statement interactions, message propagation can be performed first between nodes at the same level. Specifically, message propagation can be performed between word segmentation nodes to obtain a hierarchical representation of each word segmentation node, and message propagation can be performed between statement nodes to obtain a hierarchical representation of each statement node.

[0125] Specifically, in the second stage, message propagation between nodes at different levels in the document structure graph can be implemented. In this stage, to obtain overall document information, the information interaction between document structures needs to be modeled. Specifically, message propagation between document structures can be performed within the already constructed document structure graph. It can be understood that at this point in the document structure graph, the hierarchical representation of document nodes is the initial representation of document nodes, while the hierarchical representations of statement nodes and word segmentation nodes are obtained based on message propagation between nodes at the same level.

[0126] After these two stages of message propagation, a structural diagram representation of the document structure diagram is obtained.

[0127] Step 250: Based on feature fusion of the semantic representation and the structural graph representation, a fused representation is obtained.

[0128] Specifically, considering that the structural graph representation obtained after message propagation, although it incorporates the graph representation information of the document structure, may lose some semantic information during message propagation, in order to ensure the effect of spoken language conversion, the semantic representation and the graph representation of the document structure can be feature fused, and semantic decoding can be performed based on the fused representation.

[0129] Here, feature fusion can be performed by concatenating features from the semantic representation and the structural graph representation of the document structure, or by performing a weighted summation of the semantic representation and the structural graph representation of the document structure. This embodiment of the invention does not specifically limit this.

[0130] Step 260: After semantic decoding by the decoder, the written document corresponding to the spoken document is obtained.

[0131] Based on any of the above embodiments Figure 4 This is a schematic diagram of the spoken-to-written language conversion device based on graph attention networks provided by the present invention, as shown below. Figure 4 As shown, the device includes:

[0132] Encoding unit 310 is used to perform semantic encoding on spoken language documents to obtain a semantic representation of the spoken language documents;

[0133] The graph representation unit 320 is used to determine the initial representation of each node in the document structure graph of the spoken document based on the semantic representation of the spoken document. The document structure graph includes document nodes, sentence nodes and word segmentation nodes. The document nodes are connected to the sentence nodes corresponding to the spoken sentences belonging to the spoken document, and the sentence nodes are connected to the word segmentation nodes corresponding to each word belonging to the spoken sentences.

[0134] Graph propagation unit 330 is used to propagate messages to the initial representations of each node in the document structure graph based on an attention mechanism, so as to obtain a structural graph representation of the document structure graph.

[0135] Decoding unit 340 is used to perform semantic decoding based on the structure graph representation to obtain the written document corresponding to the spoken document.

[0136] The apparatus provided in this embodiment of the invention constructs a document graph structure diagram and performs message propagation in the document graph structure to obtain a graph representation that covers richer document information. Based on this, decoding can produce a more concise and readable written document, avoiding the problem of missing spoken terms that cross sentence boundaries during text conversion, and ensuring the written conversion effect of document-level spoken text.

[0137] Based on any of the above embodiments, the graph propagation unit is used for:

[0138] Based on the attention mechanism, message propagation is performed on the initial representations of nodes at the same level in the document structure graph to obtain the hierarchical representations of each node in the document structure graph;

[0139] Based on the attention mechanism, message propagation is performed on the hierarchical representation of each node in the document structure graph to obtain the structure graph representation of the document structure graph.

[0140] Based on any of the above embodiments, the diagram representation unit is used for:

[0141] Based on each word segmentation node in the document structure graph, a word-level fully connected graph is constructed;

[0142] Based on each statement node in the document structure graph, construct a sentence-level fully connected graph;

[0143] Based on the attention mechanism, message propagation is performed on the initial representation of each word segmentation node in the word-level fully connected graph to obtain the hierarchical representation of each word segmentation node;

[0144] Based on the attention mechanism, message propagation is performed on the initial representation of each statement node in the sentence-level fully connected graph to obtain the hierarchical representation of each statement node.

[0145] Based on any of the above embodiments, the decoding unit is used for:

[0146] The semantic representation and the structural graph representation are fused to obtain a fused representation;

[0147] Semantic decoding is performed based on the fused representation to obtain the written document corresponding to the spoken document.

[0148] Based on any of the above embodiments, the diagram representation unit is further used for:

[0149] Based on the semantic representation and the structural graph representation, a gating attention mechanism is calculated to obtain attention weights;

[0150] Based on the attention weights, the structure graph representation is enhanced to obtain an enhanced graph representation;

[0151] The semantic representation and the enhanced graph representation are fused to obtain the fused representation.

[0152] Based on any of the above embodiments, the diagram representation unit is further used for:

[0153] The semantic representation of each word segment in the spoken document is used as the initial representation of each word segment node in the document structure diagram;

[0154] Based on the semantic representation of word segmentation under each sentence in the spoken document, the initial representation of each sentence node in the document structure diagram is determined;

[0155] Based on the initial representation of each statement node in the document structure graph, the initial representation of the document node in the document structure graph is determined.

[0156] Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5 As shown, the electronic device may include a processor 410, a communications interface 420, a memory 430, and a communication bus 440. The processor 410, communications interface 420, and memory 430 communicate with each other via the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute a spoken-to-written language conversion method based on a graph attention network. This method includes: semantically encoding the spoken document to obtain a semantic representation of the spoken document; determining the initial representation of each node in the document structure graph of the spoken document based on the semantic representation of the spoken document, where the document structure graph includes document nodes, sentence nodes, and word segmentation nodes; performing message propagation on the initial representation of each node in the document structure graph based on an attention mechanism to obtain a structural graph representation of the document structure graph; and performing semantic decoding based on the structural graph representation to obtain the written document corresponding to the spoken document.

[0157] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0158] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the spoken-to-written language conversion method based on graph attention networks provided by the above methods. The method includes: semantically encoding a spoken document to obtain a semantic representation of the spoken document; determining the initial representation of each node in the document structure graph of the spoken document based on the semantic representation of the spoken document, wherein the document structure graph includes document nodes, sentence nodes, and word segmentation nodes; performing message propagation on the initial representation of each node in the document structure graph based on an attention mechanism to obtain a structure graph representation of the document structure graph; and performing semantic decoding based on the structure graph representation to obtain the written document corresponding to the spoken document.

[0159] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the spoken-to-written language conversion method based on graph attention networks provided by the above methods. This method includes: semantically encoding a spoken document to obtain a semantic representation of the spoken document; determining the initial representation of each node in the document structure graph of the spoken document based on the semantic representation of the spoken document, wherein the document structure graph includes document nodes, sentence nodes, and word segmentation nodes; performing message propagation on the initial representation of each node in the document structure graph based on an attention mechanism to obtain a structural graph representation of the document structure graph; and performing semantic decoding based on the structural graph representation to obtain the written document corresponding to the spoken document.

[0160] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0161] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0162] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A spoken-to-written language conversion method based on graph attention networks, characterized in that, include: Semantic encoding is performed on the spoken language document to obtain its semantic representation; Based on the semantic representation of the spoken document, the initial representation of each node in the document structure graph of the spoken document is determined. The document structure graph includes document nodes, sentence nodes and word segmentation nodes. The document nodes are connected to the sentence nodes corresponding to the spoken sentences belonging to the spoken document, and the sentence nodes are connected to the word segmentation nodes corresponding to each word belonging to the spoken sentences. Based on the attention mechanism, message propagation is performed on the initial representation of each node in the document structure graph to obtain the structure graph representation of the document structure graph; Semantic decoding is performed based on the structural graph representation to obtain the written document corresponding to the spoken document; The process of using an attention mechanism to propagate messages through the initial representations of each node in the document structure graph to obtain a structural graph representation of the document structure graph includes: Based on the attention mechanism, message propagation is performed on the initial representations of nodes at the same level in the document structure graph to obtain the hierarchical representations of each node in the document structure graph; the hierarchical representations of each node include the initial representation of the document node, the hierarchical representation of the statement node, and the hierarchical representation of the word segmentation node. Based on the attention mechanism, message propagation is performed on the hierarchical representation of each node in the document structure graph to obtain the structure graph representation of the document structure graph; The step of performing message propagation on the initial representations of nodes at the same level in the document structure graph based on the attention mechanism to obtain hierarchical representations of each node in the document structure graph includes: Based on each word segmentation node in the document structure graph, a word-level fully connected graph is constructed; Based on each statement node in the document structure graph, construct a sentence-level fully connected graph; Based on the attention mechanism, message propagation is performed on the initial representation of each word segmentation node in the word-level fully connected graph to obtain the hierarchical representation of each word segmentation node; Based on the attention mechanism, message propagation is performed on the initial representation of each statement node in the sentence-level fully connected graph to obtain the hierarchical representation of each statement node.

2. The spoken-to-written language conversion method based on graph attention networks according to claim 1, characterized in that, The semantic decoding based on the structure graph representation to obtain the written document corresponding to the spoken document includes: The semantic representation and the structural graph representation are fused to obtain a fused representation; Semantic decoding is performed based on the fused representation to obtain the written document corresponding to the spoken document.

3. The spoken-to-written language conversion method based on graph attention networks according to claim 2, characterized in that, The feature fusion of the semantic representation and the structural graph representation to obtain the fused representation includes: Based on the semantic representation and the structural graph representation, a gating attention mechanism is calculated to obtain attention weights; Based on the attention weights, the structure graph representation is enhanced to obtain an enhanced graph representation; The semantic representation and the enhanced graph representation are fused to obtain the fused representation.

4. The spoken-to-written language conversion method based on graph attention networks according to any one of claims 1 to 3, characterized in that, The determination of the initial representation of each node in the document structure graph of the spoken document based on the semantic representation of the spoken document includes: The semantic representation of each word segment in the spoken document is used as the initial representation of each word segment node in the document structure diagram; Based on the semantic representation of word segmentation under each sentence in the spoken document, the initial representation of each sentence node in the document structure diagram is determined; Based on the initial representation of each statement node in the document structure graph, the initial representation of the document node in the document structure graph is determined.

5. A spoken-to-written language conversion device based on graph attention networks, characterized in that, include: The encoding unit is used to perform semantic encoding on the spoken document to obtain the semantic representation of the spoken document; The graph representation unit is used to determine the initial representation of each node in the document structure graph of the spoken language document based on the semantic representation of the spoken language document. The document structure graph includes document nodes, sentence nodes and word segmentation nodes. The document nodes are connected to the sentence nodes corresponding to the spoken language sentences belonging to the spoken language document, and the sentence nodes are connected to the word segmentation nodes corresponding to each word of the spoken language sentences. The graph propagation unit is used to propagate messages to the initial representations of each node in the document structure graph based on an attention mechanism, so as to obtain the structure graph representation of the document structure graph. A decoding unit is used to perform semantic decoding based on the structural graph representation to obtain the written document corresponding to the spoken document; The graph propagation unit is specifically used for: Based on the attention mechanism, message propagation is performed on the initial representations of nodes at the same level in the document structure graph to obtain the hierarchical representations of each node in the document structure graph; the hierarchical representations of each node include the initial representation of the document node, the hierarchical representation of the statement node, and the hierarchical representation of the word segmentation node. Based on the attention mechanism, message propagation is performed on the hierarchical representation of each node in the document structure graph to obtain the structure graph representation of the document structure graph; The graph propagation unit is also specifically used for: Based on each word segmentation node in the document structure graph, a word-level fully connected graph is constructed; Based on each statement node in the document structure graph, construct a sentence-level fully connected graph; Based on the attention mechanism, message propagation is performed on the initial representation of each word segmentation node in the word-level fully connected graph to obtain the hierarchical representation of each word segmentation node; Based on the attention mechanism, message propagation is performed on the initial representation of each statement node in the sentence-level fully connected graph to obtain the hierarchical representation of each statement node.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the spoken-to-written language conversion method based on graph attention networks as described in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the spoken-to-written language conversion method based on graph attention networks as described in any one of claims 1 to 4.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the spoken-to-written language conversion method based on graph attention networks as described in any one of claims 1 to 4.

Images