A rich descriptor framework using graph and structural neural encoders - Text generation

Abstract

Rich Descriptor Framework-Text Generation Using Graph and Structural Neural Encoders. Techniques for converting Resource Description Framework (RDF) data to natural language text using a bidirectional graph convolutional neural network ("BGCNN"). Some embodiments perform RDF-to-text generation by training a graph-augmented structural neural encoder consisting of (a) a bidirectional graph-based meta-path encoder, (b) a bidirectional graph convolutional network encoder, and (c) a separated attention mechanism that combines the encoder and decoder to convert RDF triplets into natural language descriptions.

Description

[Technical field]

[0001] The present invention relates generally to the field of transforming Rich Descriptor Framework (RDF) data into textual data using machine logic (eg, software). [Background technology]

[0002] The Wikipedia entry for "Convolutional Neural Network" (as of June 16, 2020) states, in part: "In deep learning, convolutional neural networks (CNNs, or ConvNets) are a class of deep neural networks most commonly applied to analyze visual images. They are also known as shift-invariant or space-invariant artificial neural networks (SIANNs) based on their shared weight architecture and transformation invariance properties. They have applications in natural language processing, where CNNs are a regularized version of multi-layer perceptrons."

[0003] Multilayer perceptron usually means a fully connected network, i.e., a network in which each neuron in one layer is connected to every neuron in the next layer. The "fully connected" nature of these networks makes them prone to overfitting the data. A typical approach to regularization involves adding some form of magnitude measure of the weights to the loss function. CNNs take a different approach to regularization: they exploit hierarchical patterns in the data and use smaller, more concise patterns to assemble more complex patterns. Thus, CNNs are extremely low in connectivity and complexity. The name "convolutional neural network" indicates that the network in question utilizes a mathematical operation called convolution. Convolution is a special kind of linear operation. Convolutional networks are simply neural networks that use convolution instead of the general-purpose matrix multiplication in at least one of their layers. Convolutional neural networks consist of input and output layers, as well as multiple hidden layers. The hidden layers of a CNN typically consist of a series of convolutional layers that convolve with multiplication or other dot products. The activation function is typically a RELU layer, followed by additional convolutions such as pooling layers, fully connected layers, and normalization layers; these layers are referred to as hidden layers because their inputs and outputs are masked by the activation function and the final convolution. The layer is commonly referred to as a convolution, but this is simply by convention. Mathematically, it is technically a sliding dot product or cross-correlation. This has significance in the index in the matrix in that it affects how the weights are determined at a particular index point. A convolution layer in a neural network should have the following attributes: A convolution kernel defined by width and height (hyper-parameters); The number of input and output channels (hyper-parameters); The depth of the convolution filter (input channels) should be equal to the number of channels (depth) of the input feature map.The vector of weights and biases is called a filter, which represents a particular feature of the input (e.g., a particular shape). A distinguishing feature of CNNs is that many neurons can share the same filter. This reduces the memory footprint because a single bias and a single vector of weights are used across all receptive fields sharing that filter, as opposed to each receptive field having its own bias and vector weights." (Footnotes omitted.)

[0004] The Wikipedia entry for "Resource Description Framework" (as of June 16, 2020) states, in part: "The Resource Description Framework (RDF) is a family of World Wide Web Consortium (W3C) specifications originally designed as a metadata data model. It has come to be used as a general way for conceptually describing or modeling information embodied in web resources, using a variety of syntax notations and data serialization formats. It is also used in knowledge management applications. The RDF data model is similar to traditional conceptual modeling approaches (such as entity-relationship or class diagrams). It is based on the concept of making statements about resources (especially web resources) in representations of the form subject-predicate-object, known as triples. The subject denotes the resource, the predicate denotes a quality or aspect of the resource, and expresses the relationship between the subject and the object. For example, one way to represent the concept "the sky has a blue color" in RDF is a triple: a subject denoting "sky", a predicate denoting "has a color", and an object denoting "blue". Thus, in contrast to the typical approach of the entity-attribute-value model in object-oriented design, i.e., entity (sky), attribute (color), and value (blue), RDF uses subjects instead of objects (or entities). RDF is an abstract model with several serialization formats (i.e., file formats), and thus the concrete encoding for resources or triples varies from format to format. A set of RDF statements essentially represents a labeled, directed multigraph. RDF puts information in a formal, machine-understandable way. The purpose of RDF is to provide an encoding and interpretation mechanism so that resources can be described in a way that particular software can understand them, in other words, so that software can access and use information that it could not otherwise use. " (Footnotes omitted).

[0005] The Wikipedia entry on "Bidirectional graph" (as of June 16, 2020) states, in part, as follows: "In the mathematical domain of graph theory, a bidirectional graph is a graph in which each edge is given an independent orientation (or direction, or arrow) at each end. There are thus three kinds of bidirectional edges: those with arrows pointing outward and toward the vertex at both ends, those with both arrows pointing inward, away from the vertex, and those with one arrow pointing away from the vertex to the other side, while the other arrow points away from the other side, toward its own vertex, in the same direction as the first. These three types of edges may be called outward edges, inward edges, and directed edges, respectively. A "directed" edge is the same as a regular directed edge in a directed graph, and thus a directed graph is a special kind of bidirectional graph. It may sometimes be desirable to also have edges with only one end (half edges), which have only one arrow. Edges with no ends (loose edges) have no arrows. Edges that are neither half edges nor loose edges may be called regular edges. A skew-symmetric graph is a double covering graph of a bidirectional graph. (Footnotes omitted.)

[0006] A note on terminology: As is currently customary in the art, the terms "bidirectional graph" and "bidirectional graph" mean the same thing and are used interchangeably. Summary of the Invention

[0007] According to an aspect of the invention, there is a method, computer program product, or system, or combination thereof, that performs the following operations (not necessarily in the following order): (i) training a bidirectional graph convolutional neural network (BGCNN) using a plurality of training datasets; (ii) receiving a Resource Description Framework (RDF) dataset including computer readable information corresponding to a plurality of RDF triples; (iii) creating, by machine logic, a bidirectional graph dataset including a bidirectional graph including all information of the plurality of RDF triples; and (iv) converting the bidirectional graph into natural language text using the BGCNN.

[0008] According to another aspect of the present invention, there is provided a computer implemented method (CIM) that includes training a bidirectional graph convolutional neural network (BGCNN) using a plurality of training datasets; receiving a Resource Description Framework (RDF) dataset including a computer readable representation corresponding to a plurality of RDF triples; creating, by machine logic, a bidirectional graph dataset including a bidirectional graph including all information of the plurality of RDF triples; and converting the bidirectional graph into natural language text using the BGCNN.

[0009] Preferably, the invention provides a computer implemented method, wherein said training comprises at least one of supervised learning or unsupervised learning or a combination thereof.

[0010] Preferably, the invention provides a computer-implemented method, wherein said converting of said bidirectional graph into natural language text comprises the sub-operation of encoding, by a meta-path encoder module of said BGCNN, a meta-path associated with said bidirectional graph.

[0011] Preferably, the invention provides a computer-implemented method, wherein said converting said bidirectional graph into natural language text comprises the sub-act of encoding said bidirectional graph by a Convolutional Network (CN) encoder portion of said BGCNN.

[0012] Preferably, the present invention provides a computer-implemented method, wherein converting the bidirectional graph into natural language text comprises the sub-operation of combining an encoder and a decoder with a separated attention mechanism part of BGCNN to convert the RDF dataset into natural language text.

[0013] Preferably, the present invention provides a computer-implemented method, wherein the BGCNN includes a bidirectional graph-based meta-path encoder, a graph convolutional network encoder input, and a bidirectional graph convolutional network encoder.

[0014] According to another aspect of the present invention, there is provided a Computer Program Product (CPP) comprising: a set of storage devices; and computer code collectively stored on the set of storage devices, the computer code comprising data and instructions to cause a set of processors to perform at least the following operations: training a Bidirectional Graph Convolutional Neural Network (BGCNN) using a plurality of training datasets; receiving a Resource Description Framework (RDF) dataset comprising computer readable information corresponding to a plurality of RDF triples; creating, by machine logic, a bidirectional graph dataset comprising a bidirectional graph including all information of the plurality of RDF triples; and converting the bidirectional graph into natural language text using the BGCNN.

[0015] Preferably, the present invention provides a computer program product, wherein said training comprises at least one of supervised learning or unsupervised learning or a combination thereof.

[0016] Preferably, the present invention provides a computer program product, wherein said converting of said bidirectional graph into natural language text comprises the sub-operation of encoding, by a meta-path encoder module of said BGCNN, a meta-path associated with said bidirectional graph.

[0017] Preferably, the present invention provides a computer program product, wherein said converting said bidirectional graph into natural language text comprises the sub-operation of encoding said bidirectional graph by a Convolutional Network (CN) encoder portion of said BGCNN.

[0018] Preferably, the present invention provides a computer program product, wherein said converting of said bidirectional graph into natural language text comprises the sub-operation of combining an encoder and a decoder with a separated attention mechanism part of said BGCNN to convert said RDF dataset into said natural language text.

[0019] Preferably, the present invention provides a computer program product, wherein the BGCNN includes a bidirectional graph-based meta-path encoder, a graph convolutional network encoder input, and a bidirectional graph convolutional network encoder.

[0020] According to another aspect of the present invention, there is provided a computer system (CS) comprising: a set of processors; a set of storage devices; and computer code collectively stored on the set of storage devices, the computer code comprising data and instructions to cause the set of processors to perform at least the following operations: training a bidirectional graph convolutional neural network (BGCNN) using a plurality of training datasets; receiving a Resource Description Framework (RDF) dataset comprising computer readable information corresponding to a plurality of RDF triples; creating, by machine logic, a bidirectional graph dataset comprising a bidirectional graph including all information of the plurality of RDF triples; and converting the bidirectional graph into natural language text using the BGCNN.

[0021] Preferably, the invention provides a computer system, wherein said training comprises at least one of supervised learning or unsupervised learning or a combination thereof.

[0022] Preferably, the invention provides a computer system, wherein said converting of said bidirectional graph into natural language text comprises the sub-operation of encoding, by a meta-path encoder module of said BGCNN, a meta-path associated with said bidirectional graph.

[0023] Preferably, the present invention provides a computer system, wherein said converting said bidirectional graph into natural language text comprises a sub-operation of encoding said bidirectional graph by a Convolutional Network (CN) encoder portion of said BGCNN.

[0024] Preferably, the present invention provides a computer system, wherein said converting of said bidirectional graph into natural language text comprises the sub-operation of combining an encoder and a decoder with a separated attention mechanism part of said BGCNN to convert said RDF dataset into said natural language text.

[0025] Preferably, the present invention provides a computer system, wherein the BGCNN includes a bidirectional graph-based meta-path encoder, a graph convolutional network encoder input, and a bidirectional graph convolutional network encoder. [Brief description of the drawings]

[0026] [Figure 1] FIG. 1 is a block diagram of a first embodiment of a system according to the present invention.

[0027] [Diagram 2] 4 is a flow chart illustrating a method of a first embodiment performed at least in part by a system of the first embodiment.

[0028] [Diagram 3] FIG. 2 is a block diagram showing the machine logic (eg, software) portion of the system of the first embodiment.

[0029] [Figure 4] FIG. 2 is a diagram showing a screenshot generated by the system of the first embodiment.

[0030] [Figure 5A] FIG. 2 is a block diagram of a second embodiment of a system according to the present invention.

[0031] [Figure 5B] FIG. 11 is a block diagram of a portion of a system according to a second embodiment.

[0032] [Figure 5C]FIG. 11 is a block diagram of another portion of the system of the second embodiment.

[0033] [Figure 5D] FIG. 11 is a block diagram of another portion of the system of the second embodiment.

[0034] [Figure 6] FIG. 10 is a diagram of a bidirectional graph generated by the system of the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0035] Some embodiments of the present invention are directed to using a bidirectional graph convolutional neural network ("BGCNN") to convert RDF data into natural language text. Some embodiments perform RDF-to-text generation by training a graph reinforced structural neural encoder that consists of (a) a bidirectional graph-based meta-path encoder, (b) a bidirectional graph convolutional network encoder, and (c) a separated attention mechanism that combines the encoder and decoder to convert RDF triplets into natural language descriptions.

[0036] This Detailed Description section is divided into the following subsections: (i) Hardware and Software Environment, (ii) Exemplary Embodiments, (iii) Further Comments or Embodiments or Combinations thereof, and (iv) Definitions. I. Hardware and Software Environment

[0037] The present invention may be a system, method, or computer program product, or combination thereof, integrated at any possible level of technical detail. The computer program product may include a computer readable storage medium having computer readable program instructions thereon for causing a processor to perform aspects of the present invention. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to perform aspects of the present invention.

[0038] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: portable computer diskettes, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), static random access memories (SRAMs), portable compact disk read-only memories (CD-ROMs), digital versatile disks (DVDs), memory sticks, floppy disks, mechanically encoded devices such as punch cards or ridge structures in grooves that record instructions, and any suitable combination of the foregoing. Computer-readable storage medium, as used herein, is not to be construed as a transitory signal per se, such as electric waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted through wires.

[0039] A "storage device" is hereby defined as anything created or adapted to store computer code in such a manner that the computer code may be accessed by a computer processor. A storage device typically includes a storage medium, which is the material in or on which the computer code data is stored. A single "storage device" may (i) have multiple separate parts spaced apart or distributed (e.g., a set of six solid-state storage devices each located in six laptop computers that collectively store a single computer program), or (ii) use multiple storage media (e.g., a set of computer code stored partially in a magnetic domain in the computer's non-volatile storage and partially in a set of semiconductor switches in the computer's volatile memory), or a combination thereof. The term "storage medium" should be interpreted to cover situations in which multiple different types of storage media are used.

[0040] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium into each computing / processing device, or may be downloaded to an external computer or storage device over a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may comprise copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and transfers the computer-readable program instructions to a computer-readable storage medium in the respective computing / processing device for storage.

[0041] The computer readable program instructions for carrying out the operations of the present invention may be source or object code written in any combination of one or more programming languages, including assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, or object oriented programming languages ​​such as Smalltalk, C++, and conventional procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may execute computer readable program instructions to personalize the electronic circuitry by utilizing state information of the computer readable program instructions to perform aspects of the invention.

[0042] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0043] These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine such that the instructions, executed by the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in one or more blocks of the flowchart or block diagram, or combination thereof. These computer readable program instructions may also be stored on a computer readable storage medium such that the instructions can instruct a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner such that the computer readable storage medium having the instructions stored thereon includes an article of manufacture including instructions that implement aspects of the functions / acts specified in one or more blocks of the flowchart or block diagram, or combination thereof.

[0044] The computer readable program instructions may also be loaded into a computer, other programmable data processing apparatus, or other device to cause the computer, other programmable apparatus, or other device to execute a series of operational steps to generate a computer-implemented process, such that the instructions executing on the computer, other programmable apparatus, or other device implement the function / operation specified in one or more blocks of the flowchart or block diagram or combination thereof.

[0045] The flowcharts and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions that includes one or more executable instructions that implement a specified logical function. In some alternative implementations, the functions described in the blocks may occur in a different order than that described in the figures. For example, two blocks shown in succession may in fact be executed substantially simultaneously, or the blocks may be executed in the reverse order depending on the functionality involved. It should also be noted that each block of the block diagram or flowchart diagram, or combination thereof, as well as combinations of blocks in the block diagram or flowchart diagram, or combination thereof, may be implemented by a dedicated hardware-based system that executes the specified functions or operations or executes a combination of dedicated hardware and computer instructions.

[0046] 1, networked computer system 100 is an embodiment of a hardware and software environment for use with various embodiments of the present invention. Networked computer system 100 includes a server subsystem 102 (sometimes referred to herein more simply as subsystem 102), client subsystems 104, 106, 108, 110, 112, and a communication network 114. Server subsystem 102 includes a server computer 200, a communication unit 202, a processor set 204, an input / output (I / O) interface set 206, a memory 208, persistent storage 210, a display 212, an external device 214, a random access memory (RAM) 230, a cache 232, and programs 300.

[0047] Subsystem 102 may be a laptop computer, a tablet computer, a netbook computer, a personal computer (PC), a desktop computer, a personal digital assistant (PDA), a smartphone, or any other type of computer (see the definition of "Computer" in the Definitions section below). Program 300 is a collection of machine-readable instructions or data, or a combination thereof, used to create, manage, and control certain software functions discussed in detail below in the Example Embodiments subsection of the Detailed Description of the Invention section.

[0048] The subsystems 102 may communicate with other computer subsystems via a communications network 114. The network 114 may be, for example, a local area network (LAN), a wide area network (WAN) such as the Internet, or a combination of the two, and may include wired, wireless, or fiber optic connections. In general, the network 114 may be any combination of connections and protocols that support communication between server and client subsystems.

[0049] Subsystem 102 is illustrated as a block diagram that includes a number of double arrows. These double arrows (not having separate reference numbers) represent a communications fabric that provides communication between various components of subsystem 102. This communications fabric may be implemented using any architecture designed to pass data or control information, or a combination thereof, between processors (e.g., microprocessors, communications and network processors, etc.), system memory, peripheral devices, and any other hardware components in a computer system. For example, the communications fabric may be implemented at least in part using one or more buses.

[0050] Memory 208 and persistent storage 210 are computer-readable storage media. In general, memory 208 may include any suitable volatile or non-volatile computer-readable storage media. It is further noted that, currently or in the near future, or a combination thereof, (i) external device 214 may be capable of providing some or all of the memory for subsystem 102, or (ii) a device external to subsystem 102 may be capable of providing memory for subsystem 102, or a combination thereof. Both memory 208 and persistent storage 210 (i) store data in a manner that is less transient than a signal in transit, and (ii) store data on a tangible medium (such as a magnetic or optical domain). In this embodiment, memory 208 is volatile storage, while persistent storage 210 provides non-volatile storage. The medium used by persistent storage 210 may also be removable. For example, a removable hard drive may be used for persistent storage 210. Other examples include optical and magnetic disks, thumb drives, and smart cards that are inserted into a drive for transfer to another computer-readable storage medium that is also part of persistent storage 210.

[0051] The communications unit 202 provides for communication with other data processing systems or devices external to the subsystem 102. In these examples, the communications unit 202 includes one or more network interface cards. The communications unit 202 may provide communication through the use of either or both physical and wireless communication links. Any software modules described herein may be downloaded to a persistent storage device (such as persistent storage 210) through a communications unit (such as communications unit 202).

[0052] The I / O interface set 206 allows input and output of data with other devices that may be locally connected in data communication with the server computer 200. For example, the I / O interface set 206 provides a connection to an external device set 214. The external device set 214 typically includes devices such as a keyboard, a keypad, a touch screen, or some other suitable input device, or a combination thereof. The external device set 214 may also include portable computer-readable storage media, such as thumb drives, portable optical or magnetic disks, and memory cards. Software and data used to implement embodiments of the present invention, such as the program 300, may be stored on such portable computer-readable storage media. The I / O interface set 206 also connects in data communication with a display 212. The display 212 is a display device that provides a mechanism for displaying data to a user, and may be, for example, a computer monitor or a smartphone display screen.

[0053] In this embodiment, the program 300 is typically stored in persistent storage 210 for access or execution by one or more computer processors of processor set 204 through one or more memories of memory 208, or a combination thereof. Those skilled in the art will appreciate that the program 300 may be stored in a more highly distributed manner during its operation or non-operational time, or a combination thereof. The program 300 may include both machine-readable and executable instructions or entity data (i.e., the type of data stored in a database). In this particular embodiment, persistent storage 210 includes a magnetic hard disk drive. To name a few possible variations, persistent storage 210 may include a solid-state hard drive, a semiconductor storage device, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, or any other computer-readable storage medium capable of storing program instructions or digital information.

[0054] The programs described herein are identified based on the applications for which they are implemented in specific embodiments of the invention. However, it should be understood that any specific program names herein are used merely for convenience, and thus the invention should not be limited to use in only any specific application identified or suggested by such names, or in any combination thereof.

[0055] The description of various embodiments of the present invention has been presented for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terms used in this specification are selected to best explain the principles of the embodiments, practical applications, or technical improvements to the art found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein. II. Exemplary Embodiments

[0056] As shown in Figure 1, networked computer system 100 is an environment in which an exemplary method in accordance with the present invention may be implemented. As shown in Figure 2, a flow chart 250 illustrates one exemplary method in accordance with the present invention. As shown in Figure 3, a program 300 performs or controls the performance of at least a portion of the operations of the method of flow chart 250. The method and associated software are now discussed throughout the following paragraphs with broad reference to the blocks of Figures 1, 2, and 3.

[0057] Processing begins at operation S255, where a receiving RDF data module ("module") 302 receives an RDF data set that is computer readable and includes information indicating a set of RDF triples (or "triplets"). As shown in area 402 of screenshot 400 of Figure 4, there are three triples in this example: (i) water contains an oxygen atom, (ii) water is a type of molecule, and (iii) water contains a hydrogen atom. The goal of the method flowchart 250 as a whole is to generate, by machine logic, natural language text that accurately communicates all of the information in these triples.

[0058] Processing continues to operation S260, where the create bidirectional graph module 304 creates a bidirectional graph based on the RDF dataset. This is illustrated by region 404 of screenshot 400 in Figure 4. More specifically, the single bidirectional graph in region 404 is created by combining the three separate triple graphs in region 402.

[0059] Processing proceeds to operation S265, where the training module 308 trains a bidirectional graph convolutional neural network (BGCNN) 320 using a training dataset (not shown separately). As shown in FIG. 3, the BGCNN 320 includes a meta-path encoder 322, a convolutional network encoder 324, and a separate attention mechanism 326. In the following subsections of the detailed description section, the BGCNN is also described, which includes (i) a bi-GMP (graph based meta-paths) encoder, (ii) a GCN (graph convolutional networks) encoder input, and (iii) a bi-GCN encoder.

[0060] Processing continues to operation S270, where the conversion module 310 uses the bidirectional graph CNN 320 to convert the bidirectional graph (shown in region 404 of screenshot 400) to natural language text (shown in region 406 of screenshot 400). More specifically, in this example, the conversion operation includes three (three) sub-operations as follows: (i) using the meta path encoder 322 to encode the meta path (this will be described in more detail in the following subsections of the description of the embodiment of the invention), (ii) using the convolutional network (CN) encoder 324 to encode the bidirectional graph (this will be described in more detail in the following subsections of the description of the embodiment of the invention), and (iii) using a separate attention mechanism 326 to combine the encoder and decoder to convert the RDF triples to natural language descriptions (this will be described in more detail in the following subsections of the description of the embodiment of the invention).

[0061] Processing continues to operation S275, where the output module 312 outputs the natural language text obtained in operation S265. In this example, the text is transmitted as a text message over the communications network 114 to the client subsystem 104 (in this example, the smartphone of the lead chemist at the Advanced Chemistry Laboratory). Alternatively or in addition, the natural language text can be transmitted as computer readable data to another computing device for storage or further processing, or a combination thereof. III. Further Comments or Embodiments or Combinations

[0062] Some embodiments of the present invention recognize that Natural Language Generation, Business Usage has applications in many areas such as: machine translation sequences, abstractive summarization sequences, image processing related fields such as image captioning, table related fields such as Wiki-infoboxes for biographies and sports report generation, graph related fields such as knowledge based QA and entity summarization, etc.

[0063] Some embodiments of the present invention address the following facts, potential problems, or potential areas of improvement, or a combination thereof, regarding the current state of the art: (i) one current conventional solution for converting RDF data to text data is called Graph-based Triple-Long Short Term Memory (GTR-LSTM1); (ii) GTR-LSTM1 encodes global information by maintaining a graph structure; (iii) GTR-LSTM1 cannot capture rich local complex structural information between entities and relationships; and (iv) another current conventional solution for converting RDF data to text data is called Modified Graph Convolutional Networks (GTR-LSTM1). Networks,DGCN,2,(v) Modified Graph Convolutional Networks (DGCN)2 perform better at capturing local structural information, (vi) Modified Graph Convolutional Networks (DGCN)2 easily overfit when GCN layers have a quantity greater than or equal to three, or (vii) Modified Graph Convolutional Networks (DGCN)2 fail to capture long-range dependency (global) information between RDF triples, or any combination thereof.

[0064] Some embodiments of the present invention may include one or more of the following operations, features, characteristics, or advantages, or combinations thereof: (i) performing RDF-to-text generation using a graph reinforced structural neural encoder; (ii) a combination of a graph encoder + a graph based meta path encoder; (iii) graph construction; (iv) a bidirectional graph based meta path encoder; (v) a bidirectional graph convolutional network encoder; or (vi) a combination of an encoder and a decoder; or combinations thereof.

[0065] Some embodiments of the present invention may include one or more of the following operations, features, properties, or advantages, or combinations thereof: (i) performing RDF-to-text generation by training a graph-reinforced structural neural encoder; (ii) including operations performed by a bidirectional graph-based meta-path encoder; (iii) including operations performed by a bidirectional graph convolutional network encoder; or (iv) combining an encoder and decoder with a decoupled attention mechanism to convert RDF triplets into corresponding natural language descriptions, respectively; or combinations thereof.

[0066] A method according to an embodiment of the present invention utilizes a computing device to generate a natural language description from resource description framework data, the method including the following operations (not necessarily in the following order): (i) receiving, by the computing device, resource description framework data, where the resource description framework data is represented as a directed graph, (ii) encoding, by the computing device, a bidirectional graph based on the resource description framework data, (iii) training, by the computing device, a bidirectional graph convolutional neural network, and (iv) utilizing, by the computing device, the bidirectional graph convolutional neural network to convert the resource description framework data into a natural language description.

[0067] Some embodiments of the present invention include one or more of the following operations, features, characteristics, or advantages, or combinations thereof, namely: (i) co-learning local and global structural information by combining two new graph-enhanced structured neural encoders (i.e., a bidirectional graph encoder and a bidirectional graph-based metapath encoder) for input triples through machine learning; (ii) the graph-based metapath encoder captures global structural information; (iii) faithfully generates natural language utterances (meanings) from subgraphs of a knowledge graph; (iv) uses much more advanced techniques by co-learning local and global structural information through combining two new graph-enhanced structured neural encoders; (v) directly performs RDF-to-text generation in a knowledge graph; or (vi) uses a deep learning-based method that can be used in any system, or a combination of one or more of the foregoing.

[0068] For the purposes of this document, "RDF-to-text generation" is defined herein by machine logic as generating descriptive text given a set of RDF triples, executed on a set of computers, where: (i) the input is computer code containing information representing a set of RDF triples; and (ii) the output is computer code representing descriptive natural language text.

[0069] An exemplary input set of triples is as follows. <Denmark, leaderName, Lars Lokke Rasmussen> <European University Association, headquarters, Brussels> <Social Sciences at the Aarhus University, Country, Denmark> <Denmark (Denmark), leaderTitle (Leader Title), Monarchy of Denmark (Kingdom of Denmark)> <Social Sciences at the Aarhus University (Business and Social Sciences Department of Aarhus University), affiliation (affiliation), European University Association (European University Association)> <Denmark (Denmark), religion (religion), Church of Denmark (Church of Denmark)> <Brussels (Brussels), capital of (capital of), Belgium (Belgium)>

[0070] An example output of the descriptive natural language text obtained as a result of processing the input described in the previous paragraph is as follows: "The Business and Social Sciences Department of Aarhus University in Denmark belongs to the European University Association. The headquarters of the European University Association is in Brussels. Brussels is the capital of Belgium. There is a monarchy in Denmark. Its religion is the Church of Denmark. Its leader is Lars Lokke Rasmussen."

[0071] The formulation of RDF-to-text generation is explained here. A set of RDF triples is mathematically: S = {t_1, t_(2), … t_n} represented as follows, where

Number

Number

[0072] The challenges in and motivation for RDF-to-text generation are described here. In some embodiments, better use is made of the structural information formed on a set of RDF triplets S by using a reasonable model with the following properties: (i) accurately predicting local entities and their attributes (within a triplet / local); (ii) fully covering the input information globally (across triplets / global). Some embodiments combine a graph reinforced structural neural encoder (specifically, a graph encoder and a graph-based meta-path encoder) to jointly learn local and global structural information. As shown in Figures 5A, 5B, 5C, and 5D, diagram 500 (including a meta-path input block 502, an RDF triplet block 504, a GCN (Graph Convolutional Network) encoder input 506, a bi-GMP encoder 512, a bi-GCN encoder 514, and a decoder 530) represents the framework of a combined graph reinforced structural neural encoder model.

[0073] The process of graph construction according to an embodiment of the present invention will now be described. Entity masking improves the generalization ability of the model. In the example, (i) "Bakewell Pudding" (FOOD-1) results in "Entity-1 FOOD" and (ii) "Bakewell Tart" (FOOD-2) results in "Entity-2 FOOD". "Relationships" are referred to as additional nodes. The received RDF triplets are converted into a graph data structure. In this example, a bi-GCN input of (FOOD-1, region, PLACE) results in the region associated with FOOD-1 and also the region associated with PLACE. In this example, a topological sort and single-source shortest path analysis are performed on the bi-GMP input, resulting in three results: (i) FOOD1→region→PLACE→leaderName→PERSON, (ii) FOOD-1→region→PLACE→county→COUNTY, (iii) FOOD-1→dishVariation→FOOD-2→ingredient→INGREDIENT. The graph jointly constructed by the bi-GCN and bi-GMP encoders is shown in graph 600 of FIG.

[0074] In some embodiments, the bidirectional graph-based meta-path encoder satisfies the following properties: (i) the input graph G1 is a set of meta-paths S_p={p_1, p_2, ...}, p_k=<w_(k,1),w_(k_2),…,w_(k,n_k)> (ii) a representation r_i of each token in each meta-path is computed according to the formula (see below); (iii) hidden state masking between different meta-paths; (iv) r_(i-1) is either an all-zero vector or the hidden state of the previous time step in the meta-path; (v) f(·) and g(·) are single LSTM (Long Short Term Memory) units; (vi) a copy mechanism is introduced; (vii) output: a set of representations R_1={r_1,r_2…}; or (viii) the graph embedding of graph G1 is given by formula (2) (see below), or has one or more combinations thereof.

[0075] Equations (1) and (2) (see above) are as follows: (Formula 1)

number

number

[0076] In some embodiments, the bidirectional graph-based convolutional network encoder has the following properties: (i) a vector representation at layer l (see Equation (3) below)

number

[0077] Equations (3) to (7) (see above) are as follows: (Formula 3)

number

number

number

number

number

number

[0078] Combining the encoder and decoder is now described: the combined graph embedding is represented by equation (8): (Formula 8)

number

number

number

number

number

number

[0079] The present invention: The subject matter described by the term "the present invention" should not be taken as an absolute indication that it is covered by either the claims as filed or any claims that may be issued after patent prosecution. The term "the present invention" is used to help the reader get a general sense that the disclosures herein are believed to be potentially new, but as indicated by the use of the term "the present invention", this understanding is hypothetical and provisional, and is subject to change during the course of patent prosecution as relevant information evolves, as the claims are potentially amended.

[0080] Embodiments: See the definition of "present invention" above. A similar caution applies to the term "embodiments."

[0081] "Or, or a combination thereof": Inclusive or, for example, A, B, "or" C, "or a combination thereof" means that at least one of A or B or C is true and applicable.

[0082] Including / include / includes: means "including but not necessarily limited to," unless expressly stated otherwise.

[0083] Module / Sub-Module: Any set of hardware, firmware, or software, or combination thereof, that is operable to perform a certain function, whether the module is (i) in a single local proximity, (ii) distributed over a wide area, (iii) in a single proximity within a larger software code, (iv) located within a single software code, (v) located within a single storage device, memory, or medium, (vi) mechanically connected, (vii) electrically connected, or (viii) connected in data communication, or any combination thereof.

[0084] Computer: Any device having significant data processing or machine-readable instruction reading capability, or a combination thereof, including, but not limited to, desktop computers, mainframe computers, laptop computers, field programmable gate array (FPGA) based devices, smart phones, personal digital assistants (PDAs), body worn or body inserted computers, embedded device style computers, application specific integrated circuit (ASIC) based devices.

Claims

1. training a bidirectional graph convolutional neural network (BGCNN) using a plurality of training datasets; receiving a Resource Description Framework (RDF) data set comprising a computer readable medium corresponding to a plurality of RDF triples; creating a bidirectional graph dataset comprising a bidirectional graph including information for all of the plurality of RDF triples; converting the bidirectional graph into natural language text using the BGCNN; A computer implemented method (CIM) comprising:

2. The CIM of claim 1 , wherein the training comprises at least one of supervised learning or unsupervised learning or a combination thereof.

3. The converting of the bidirectional graph into natural language text comprises: The CIM of claim 1 or 2, further comprising a sub-operation of encoding, by a meta-path encoder module of the BGCNN, a meta-path associated with the bidirectional graph.

4. The conversion of the bidirectional graph into natural language text comprises:

4. The CIM of claim 3, further comprising a sub-operation of encoding the bidirectional graph by a convolutional network (CN) encoder portion of the BGCNN.

5. The converting of the bidirectional graph into natural language text comprises:

5. The CIM of claim 4, further comprising the sub-operation of combining an encoder and a decoder with a separated attention mechanism portion of the BGCNN to convert the RDF dataset to the natural language text.

6. The BGCNN A bidirectional graph-based meta-path encoder; A graph convolutional network encoder input, Bidirectional graph convolutional network encoder and 6. The CIM of claim 1 , comprising:

7. training a bidirectional graph convolutional neural network (BGCNN) using a plurality of training datasets; receiving a Resource Description Framework (RDF) data set including a computer readable representation corresponding to a plurality of RDF triples; creating a bidirectional graph dataset including a bidirectional graph including information for all of the plurality of RDF triples; converting the bidirectional graph into natural language text using the BGCNN; A computer program for causing a processor set to execute the program.

8. The computer program product of claim 7 , wherein the training comprises at least one of supervised learning or unsupervised learning or a combination thereof.

9. The converting of the bidirectional graph into natural language text comprises:

9. The computer program product of claim 7 or 8, comprising a sub-operation of encoding, by a meta-path encoder module of the BGCNN, a meta-path associated with the bidirectional graph.

10. The converting of the bidirectional graph into natural language text comprises:

10. The computer program product of claim 9, comprising the sub-operation of encoding, by a convolutional network (CN) encoder portion of the BGCNN, the bidirectional graph.

11. The converting of the bidirectional graph into natural language text comprises:

11. The computer program product of claim 10, comprising the sub-operation of combining an encoder and a decoder with a separated attention mechanism portion of the BGCNN to convert the RDF dataset to the natural language text.

12. The BGCNN A bidirectional graph-based meta-path encoder; A graph convolutional network encoder input, Bidirectional graph convolutional network encoder and 12. A computer program according to any one of claims 7 to 11, comprising:

13. A processor set; A set of storage devices and computer code collectively stored on said set of storage devices, said computer code performing at least the following operations: training a bidirectional graph convolutional neural network (BGCNN) using a plurality of training datasets; receiving a Resource Description Framework (RDF) data set including a computer readable representation corresponding to a plurality of RDF triples; creating a bidirectional graph dataset including a bidirectional graph including information for all of the plurality of RDF triples; converting the bidirectional graph into natural language text using the BGCNN; and data and instructions for causing said processor set to execute said program. A computer system (CS).

14. The CS of claim 13 , wherein the training comprises at least one of supervised learning or unsupervised learning or a combination thereof.

15. The converting of the bidirectional graph into natural language text comprises: The CS of claim 13 or 14, comprising a sub-operation of encoding, by a meta-path encoder module of the BGCNN, a meta-path associated with the bidirectional graph.

16. The converting of the bidirectional graph into natural language text comprises: The CS of claim 15 , further comprising a sub-operation of encoding the bidirectional graph by a convolutional network (CN) encoder portion of the BGCNN.

17. The converting of the bidirectional graph into natural language text comprises:

17. The CS of claim 16, comprising the sub-operation of combining an encoder and a decoder with a separated attention mechanism portion of the BGCNN to convert the RDF dataset to the natural language text.

18. The BGCNN A bidirectional graph-based meta-path encoder; A graph convolutional network encoder input, Bidirectional graph convolutional network encoder and The CS of any one of claims 13 to 17, comprising:

Images