Method and system for enhancing language model performance through structural knowledge injection

Abstract

A method for enhancing language model performance via structural knowledge injection according to an embodiment of the present invention is a method for enhancing language model performance via structural knowledge injection in a computing system comprising memory and a processor, comprising the steps of: acquiring knowledge base data including a predetermined knowledge graph; generating linear structured data which is data structured in text format from the acquired knowledge base data; training a first language model based on the generated linear structured data; and providing a predetermined application service based on the trained first language model, wherein the step of generating the linear structured data includes generating first linear structured data which is data structured in text format from the knowledge graph based on multi-hop linearization.

Description

Technical Field

[0001] The present invention relates to a method for enhancing the performance of a language model through structural knowledge injection and a system thereof. More specifically, it relates to a method and a system for converting and structuring a predetermined knowledge graph based on multi-hop linearization and learning a language model based on the structured knowledge graph.

Background Art

[0002] Conventionally, a pipeline-based language model learning method has been used as a way to distill (integrate) external knowledge into a pre-trained language model.

[0003] The pipeline-based language model learning method is a traditional approach for performing natural language processing (NLP) tasks while sequentially going through the processing steps of various steps. Each step performs a specific operation (for example, tokenization, parsing, and / or named entity recognition, etc.) and transmits the result to the next step.

[0004] Such a conventional learning method was widely used in the initial natural language processing (NLP) systems, but has various problems as follows.

[0005] · Error Propagation: A small error occurring in the initial step of the pipeline may be propagated to subsequent steps and have a great impact on the performance of the entire system. For example, an error in the tokenization step may reduce the accuracy of parsing, which may greatly reduce the quality of the final result.

[0006] • Inter-module dependencies: Each step heavily depends on the output of the previous step, meaning the entire system is affected by the performance of each step. Furthermore, attempting to change or update one step can impact other modules in the entire pipeline, creating a limitation that makes maintenance difficult.

[0007] • Lack of flexibility: The pipeline-based approach follows a fixed processing flow, making it difficult to adapt to or optimize for new types of tasks and data. Meeting new requirements often necessitates redesigning the entire pipeline.

[0008] • Complexity and resource consumption: Because separate models and algorithms must be developed and optimized for each step, the overall complexity of the system can increase. This consumes a lot of time and resources during the development and learning process.

[0009] • Limitations of interaction: Each step in the pipeline operates largely independently and may not fully utilize the detailed information from previous steps. This can lead to limitations in the model's ability to fully understand the overall context and complex linguistic patterns.

[0010] Therefore, a new language model learning framework is needed that can resolve the problems mentioned above. [Overview of the Initiative] [Problems that the invention aims to solve]

[0011] One embodiment of the present invention was devised to solve the above-mentioned problems, and aims to provide a method and system for transforming and structuring a predetermined knowledge graph based on multi-hop linearization, and learning a language model based on the structured knowledge graph.

[0012] In this embodiment, the present invention aims to provide a method and system for learning a language model based on Masked Language Modeling.

[0013] However, the technical problems that the present invention and its embodiments aim to address are not limited to those described above, and other technical problems may exist. [Means for solving the problem]

[0014] A method for enhancing language model performance via structural knowledge injection according to an embodiment of the present invention is a method for enhancing language model performance via structural knowledge injection in a computing system comprising memory and a processor, comprising the steps of: acquiring knowledge base data including a predetermined knowledge graph; generating linear structured data which is data structured in text format from the acquired knowledge base data; training a first language model based on the generated linear structured data; and providing a predetermined application service based on the trained first language model, wherein the step of generating the linear structured data includes generating first linear structured data which is data structured in text format from the knowledge graph based on multi-hop linearization.

[0015] In other respects, the knowledge graph is graphic data that represents relationships between multiple entities based on nodes and edges, and includes at least one knowledge triple that represents the subject-predicate-object relationship of the data based on the nodes and edges.

[0016] In other aspects, the step of generating the first linear structured data includes the step of converting the subject-relation-object data into text format based on the multi-level linked knowledge triples in the knowledge graph.

[0017] In other aspects, a method for enhancing language model performance through structural knowledge injection according to an embodiment of the present invention further includes the step of acquiring the knowledge base data, which includes a predetermined table.

[0018] In other respects, the step of generating the linear structured data further includes the step of generating a second linear structured data, which is data that structures the table into text format based on a predetermined UnifiedSKG (Unified Structured Knowledge Grounding) and JSON (JavaScript® Object Notation).

[0019] In other respects, the step of training the first language model includes the steps of masking at least a portion of the text in the linearly structured data and predicting the masked text based on the remaining text in the linearly structured data.

[0020] In other respects, the step of masking at least a portion of the text in the linear structured data includes the steps of identifying the main text in the linear structured data and replacing the identified main text with a mask token.

[0021] In other respects, the step of training the first language model includes, if the knowledge base data includes the table, random masking at least a portion of the text in the second linear structured data, and predicting the randomly masked text based on the remaining text in the second linear structured data.

[0022] In other respects, the step of training the first language model includes the step of further training a pre-trained language model.

[0023] On the other hand, a language model performance enhancement system via structural knowledge injection according to an embodiment of the present invention comprises at least one memory and at least one processor that reads at least one application stored in the memory and performs a language model performance enhancement method via structural knowledge injection, wherein the processor structures knowledge base data including a predetermined knowledge graph into text format based on multi-hop linearization and salient span masking processes, trains a first language model based on the structured knowledge base data, and provides a predetermined application service based on the trained first language model. [Effects of the Invention]

[0024] A method and system for enhancing language model performance through structural knowledge injection according to one embodiment of the present invention has the effect of converting a predetermined knowledge graph into a clearer and easier-to-understand text format by transforming and structuring the knowledge graph based on multi-hop linearization, while simultaneously minimizing the loss of information that occurs in the process of reflecting the meaning of the knowledge graph in the text.

[0025] In addition, the method and system for enhancing the performance of a language model through structural knowledge injection according to an embodiment of the present invention can directly improve the learning efficiency and performance of the language model by learning the language model based on the structured knowledge graph and utilizing learning data with a clearer structure, and can significantly improve the processing capabilities of various tasks (such as question answering, inference, understanding, search, and / or recommendation, etc.) of the learned language model.

[0026] In addition, the method and system for enhancing the performance of a language model through structural knowledge injection according to an embodiment of the present invention perform the language model learning in a manner that enhances context understanding ability, inference ability, and learning efficiency by learning the language model based on Masked Language Modeling, can more effectively apply external knowledge, and can provide a distilled language model.

[0027] However, the effects that can be obtained in the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood from the following description.

Brief Description of Drawings

[0028] [Figure 1] An example of a block diagram of a computing system for realizing a service for providing a structural knowledge injection framework according to an embodiment of the present invention is shown. [Figure 2] An example of a block diagram of a computing device for realizing a service for providing a structural knowledge injection framework according to an embodiment of the present invention is shown. [Figure 3] An example of a block diagram in other aspects of a computing device for realizing a service for providing a structural knowledge injection framework according to an embodiment of the present invention is shown. [Figure 4] A flowchart for explaining a method for enhancing the performance of a language model through structural knowledge injection according to an embodiment of the present invention. [Figure 5]This is an example of a diagram used to explain a knowledge graph related to one embodiment of the present invention. [Figure 6] This is an example of a diagram illustrating the first linearly structured data relating to one embodiment of the present invention. [Figure 7] This is an example of a diagram illustrating the second linear structured data according to one embodiment of the present invention. [Modes for carrying out the invention]

[0029] The present invention can be modified in various ways and has various embodiments. Specific embodiments are illustrated in the drawings and described in detail in the detailed description. The effects and features of the present invention, and how to achieve them, will become clear when you refer to the embodiments described in detail below, along with the drawings. However, the present invention is not limited to the embodiments disclosed below and can be realized in various forms. In the following embodiments, terms such as "first," "second," etc., are used not in a restrictive sense but to distinguish one component from another. Also, singular expressions include plural expressions unless the context clearly indicates otherwise. Furthermore, terms such as "includes" or "has" mean that the features or components described in the specification exist, and do not preclude the possibility that one or more other features or components may be added. Also, in the drawings, for illustrative purposes, the size of components, etc., may be exaggerated or reduced. For example, the size and thickness of each component shown in the drawings are arbitrarily shown for illustrative purposes, and the present invention is not necessarily limited to what is shown.

[0030] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings. When describing with reference to the drawings, identical or corresponding components will be denoted by the same reference numerals, and redundant descriptions thereof will be omitted.

[0031] [Example system for implementing a structural knowledge injection framework service]

[0032] The following describes in detail, with reference to the attached diagrams, an exemplary system that provides a Structured Knowledge Injection (SKI) framework service, which transforms and structures a predetermined knowledge graph based on multi-hop linearization and learns a language model based on the structured knowledge graph.

[0033] Figure 1 shows an example block diagram of a computing system that implements a structural knowledge injection (SKI) framework provision service according to one embodiment of the present invention.

[0034] As shown in Figure 1, the computing system 1000 that realizes the structural knowledge injection (SKI) framework provision service of the present invention includes a user computing device 110, a server computing system 130, and a training computing system 150, the devices being able to communicate via a network 170.

[0035] The method for enhancing language model performance through structural knowledge injection according to an embodiment of the present invention is such that 1) the user computing device 110 can be implemented and provided locally, 2) the server computing system 130 that communicates with the user computing device 110 can be implemented and provided in the form of a web service, and 3) the user computing device 110 and the server computing system 130 can be implemented and provided in cooperation with each other.

[0036] In this embodiment, the user computing device 110 and / or the server computing system 130 can train the machine learning models 120 and / or 140 through interaction with a training computing system 150 which is communicatively connected via a network 170. The training computing system 150 may be separate from the server computing system 130 or may be part of the server computing system 130.

[0037] In this case, the artificial intelligence model (language model in an embodiment) can be trained in three ways: 1) by the user computing device 110 directly locally; 2) by the server computing system 130 and the user computing device 110 interacting with each other via the network 170; and 3) by a separate training computing system 150 using various training and learning techniques. Furthermore, the artificial intelligence model (language model in an embodiment) trained by the training computing system 150 can be transmitted to the user computing device 110 and / or the server computing system 130 via the network 170 for provision / update.

[0038] In some embodiments, the training computing system 150 may be part of the server computing system 130 or part of the user computing device 110.

[0039] The user computing device 110 may include smartphones, mobile phones, digital broadcasting devices, PDAs (personal digital assistants), PMPs (portable multimedia players), desktops, wearable devices, embedded computing devices, and / or tablet PCs, as well as all other types of computing devices.

[0040] Such a user computing device 110 comprises at least one processor 111 and memory 112. Here, the processor 111 can consist of at least one or more electrically connected processors from among a central processing unit (CPU), graphics processing unit (GPU), application-specific integrated circuits (ASICs), digital signal processors (DSSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, and / or other electrical units for performing functions.

[0041] The memory 112 may include one or more non-temporary / temporary computer-readable storage media and combinations thereof, such as RAM, ROM, EEPROM, EPROM, flash memory devices, and magnetic disks, and may include web storage of a server that performs memory storage functions over the internet. Such memory 112 can store data 113 and instruction words 114 necessary for the at least one processor 111 to perform functional operations such as training an artificial intelligence model (in an embodiment, a language model) or executing various application services through the artificial intelligence model (in an embodiment, a language model).

[0042] In one embodiment, the user computing device 110 can store at least one or more machine learning models 120.

[0043] Specifically, the machine learning model 120 can be various machine learning models such as multiple neural networks (e.g., deep neural networks) or other types of machine learning models including nonlinear and / or linear models, and can be composed of combinations thereof.

[0044] In this case, the neural network may include at least one of the following: feed-forward neural networks, cyclic neural networks (e.g., long-short-term memory cyclic neural networks), convolutional neural networks, and / or other forms of neural networks.

[0045] In one embodiment, the user computing device 110 receives at least one or more machine learning models 120 from the server computing system 130 via the network 170, stores them in the memory 112, and then executes the stored machine learning models 120 using the processor 111 to perform various language model-based application services and the like.

[0046] In another embodiment, the server computing system 130 includes at least one machine learning model 140 and operates via the machine learning model 140, and can interact with the user computing device 110 in a manner that communicates with the user computing device 110 and related data, thereby providing a Structural Knowledge Injection (SKI) framework provisioning service to the user.

[0047] For example, the user computing device 110 can provide a structured knowledge injection (SKI) framework service in which the server computing system 130 uses a machine learning model 140 via the web to provide output for user input.

[0048] Furthermore, an artificial intelligence model (a language model in an embodiment) can also be realized in a manner in which at least a portion of the machine learning models 120 and / or 140 is executed on the user computing device 110, and the remainder is executed on the server computing system 130.

[0049] Furthermore, the user computing device 110 may include at least one or more input components 121 that sense user input. For example, the user input component 121 may include a touch sensor (e.g., a touchscreen and / or touchpad, etc.) that senses touch from a user's input medium (e.g., a finger or stylus), an image sensor that senses user motion input, a microphone that senses user voice input, buttons, a mouse and / or a keyboard, etc. The user input component 121 may also include an interface and an external controller (e.g., a mouse and / or keyboard, etc.) if it receives input to an external controller via an interface.

[0050] The server computing system 130 comprises at least one processor 131 and memory 132. Here, the processor 131 can consist of at least one or more electrically connected processors from among a central processing unit (CPU), graphics processing unit (GPU), application-specific integrated circuits (ASICs), digital signal processors (DSSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, and / or other electrical units for executing functions.

[0051] The memory 132 may include one or more non-temporary / temporary computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, and magnetic disks, or combinations thereof. Such a memory 132 can store data 133 and instruction words 134 necessary for the processor 131 to perform functional operations, such as training an artificial intelligence model (in an embodiment, a language model) or executing various application services through the artificial intelligence model (in an embodiment, a language model).

[0052] In one embodiment, the server computing system 130 can be implemented by including at least one computing device. For example, the server computing system 130 can be implemented so that multiple computing devices operate in a sequential computing architecture, a parallel computing architecture, or a combination thereof. The server computing system 130 can also include multiple computing devices connected by a network 170.

[0053] Furthermore, the server computing system 130 can store at least one or more machine learning models 140. For example, the server computing system 130 may include neural networks and / or other multi-layer nonlinear models as machine learning models 140. Exemplary neural networks may include feedforward neural networks, deep neural networks, cyclic neural networks, and convolutional neural networks.

[0054] The training computing system 150 comprises at least one processor 151 and memory 152. Here, the processor 151 can consist of at least one or more electrically connected processors from among a central processing unit (CPU), graphics processing unit (GPU), application-specific integrated circuits (ASICs), digital signal processors (DSSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, and / or other electrical units for functional execution.

[0055] The memory 152 may include one or more non-temporary / temporary computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, and magnetic disks, or combinations thereof. Such a memory 152 can store data 153 and instruction words 154 necessary for the processor 151 to perform tasks such as training an artificial intelligence model (a language model in an embodiment).

[0056] For example, the training computing system 150 may include a model trainer 160 that trains machine learning models 120 and / or 140 stored in the user computing device 110 and / or server computing system 130 using various training or learning techniques, such as backward propagation of errors (according to the framework shown in Figure 3).

[0057] For example, such a model trainer 160 can perform backpropagation-based updates to one or more parameters of the machine learning models 120 and / or 140 based on a defined loss function.

[0058] In some implementations, error backpropagation may include truncated backpropagation through time. The model trainer 160 may perform several generalization techniques (e.g., weight reduction, dropout, and / or knowledge distillation) to improve the generalization capabilities of the machine learning models 120 and / or 140 being trained.

[0059] In particular, the model trainer 160 can train the machine learning models 120 and / or 140 based on a series of training data 161. Here, the training data 161 may include data of different forms, such as images, audio samples, and / or text. Examples of image types that can be used may include video frames, LiDAR point clouds, X-ray images, computed tomography scans, sec-second spectral images, and / or various other forms of images.

[0060] Such training data 161 can be provided by the user computing device 110 and / or the server computing system 130. When the training computing device trains the machine learning models 120 and / or 140 on specific data from the user computing device 110, the machine learning models 120 and / or 140 can be characterized into personalized models.

[0061] The model trainer 160 also includes computer logic that is utilized to provide the desired functionality.

[0062] Furthermore, the model trainer 160 can be implemented using hardware, firmware, and / or software that control a general-purpose processor. In one implementation, the model trainer 160 includes a program file stored in a storage device, which is loaded into memory 152 and can be executed by one or more processors 151. In another implementation, the model trainer 160 includes one or more sets of computer-executable data 153 and instruction words 154, etc., stored in a tangible computer-readable storage medium such as a RAM hard disk or an optical or magnetic medium.

[0063] Network 170 includes, but is not limited to, 3GPP® (3rd Generation Partnership Project) networks, LTE (Long Term Evolution) networks, WiMAX (World Interoperability for Microwave Access) networks, the Internet, LAN (Local Area Network), Wireless LAN (Wireless Local Area Network), WAN (Wide Area Network), PAN (Personal Area Network), Bluetooth® (registered trademark) networks, satellite broadcasting networks, analog broadcasting networks, and / or DMB (Digital Multimedia Broadcasting) networks.

[0064] In general, communication over network 170 can be conducted using any type of wired and / or wireless connection, via various communication protocols (e.g., TCP / IP, HTTP, SMTP, and / or FTP), encodings or formats (e.g., HTML and / or XML), and / or protective skimmers (e.g., VPN, secure HTTP, and / or SSL).

[0065] Figure 2 shows an example block diagram of a computing device that implements a structural knowledge injection (SKI) framework provisioning service according to one embodiment of the present invention.

[0066] As shown in Figure 2, the computing device 100 included in the user computing device 110, the server computing system 130, and the training computing system 150 includes multiple applications (e.g., Application 1 to Application N). Each application may include a machine learning library and one or more machine learning models. For example, applications may include image processing applications (e.g., Detection, Classification, and / or Segmentation), text messaging applications, email applications, writing applications, virtual keyboard applications, browser applications, and / or chatbot applications.

[0067] In one embodiment, the computing device 100 may include a model trainer 160 for training an artificial intelligence model (a language model in one embodiment), and by storing and operating the trained artificial intelligence model (a language model in one embodiment), it can provide output data based on predetermined input data (for example, predetermined question data in one embodiment).

[0068] Each application of the computing device 100 can communicate with several other components of the computing device 100, such as at least one sensor, a context manager, a device state component, and / or additional components. In one embodiment, each application can communicate with each device component using an API (e.g., a public API). In one embodiment, the API used by each application may be specific to that application.

[0069] Figure 3 shows an example of a block diagram of another aspect of a computing device 100 that implements a structural knowledge injection (SKI) framework provisioning service according to one embodiment of the present invention.

[0070] As shown in Figure 3, the computing device 200 includes multiple applications (e.g., Application 1 to Application N). Each application can communicate with the central intelligence layer. For example, applications may include an image processing application, a text messaging application, an email application, a writing application, a virtual keyboard application, and / or a browser application. In one embodiment, each application can communicate with the central intelligence layer (and the models stored therein) using an API (e.g., a common API across all applications).

[0071] The central intelligence layer can include multiple machine learning models. For example, as shown in Figure 3, at least a portion of each machine learning model can be provided to each application and managed by the central intelligence layer. In other implementations, two or more applications can share a single machine learning model. For example, in some implementations, the central intelligence layer can provide a single model to all applications. In some implementations, the central intelligence layer can be implemented within or separately from the operational structure of computing device 200.

[0072] The central intelligence layer can communicate with the central device data layer. The central device data layer can be a centralized data storage for the computing device 200. As shown in Figure 3, the central device data layer can communicate with several other components of the computing device 200, such as one or more sensors, context managers, device state components, and / or additional components. In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0073] The technologies described herein refer not only to servers, databases, software applications, and other computer infrastructure systems, but also to actions taken and information transmitted to or from said systems. The inherent flexibility of computer infrastructure systems will be recognized as allowing for a wide range of possible configurations, combinations, and divisions of work, as well as functionality between and from components. For example, the processes described herein can be implemented using a single device, or multiple devices or components operating as a single device or component or combination thereof. Databases and applications can be implemented in a single system or in a distributed system across multiple systems. Distributed components can operate sequentially or in parallel.

[0074] [Methods for enhancing language model performance through structural knowledge injection]

[0075] The following describes in detail how a computing system 1000 according to an embodiment of the present invention realizes a structured knowledge injection (SKI) framework service that transforms and structures a predetermined knowledge graph based on multi-hop linearization and learns a language model based on the structured knowledge graph.

[0076] The method for enhancing the performance of a language model through structural knowledge injection in a computing system 1000 according to an embodiment of the present invention can provide a language model learned based on structured training data (i.e., linearly structured data) according to an embodiment of the present invention, and can improve the performance and quality of various application services that utilize this model.

[0077] In this case, the method for enhancing the performance of a language model through structural knowledge injection of a computing system 1000 according to an embodiment of the present invention provides a language model learned based on a learning method (i.e., SSM, etc.) according to an embodiment of the present invention, and can further improve the task processing performance and quality of the language model.

[0078] The method for enhancing language model performance through structural knowledge injection according to an embodiment of the present invention will be described in more detail below with reference to the attached drawings.

[0079] Figure 4 is a flowchart illustrating a method for enhancing language model performance through structural knowledge injection according to one embodiment of the present invention.

[0080] As shown in Figure 4, the method for enhancing language model performance via structured knowledge injection according to an embodiment of the present invention may include the steps of: acquiring knowledge base data (S101); generating linear structured data based on the acquired knowledge base data (S103); training a language model based on the generated linear structured data (S105); and providing application services based on the trained language model (S107).

[0081] Specifically, the computing system 1000 according to the embodiment of the present invention can acquire knowledge base data. (S101)

[0082] Here, the knowledge base data according to the embodiment can mean data in various forms used for language model training.

[0083] In one embodiment, the knowledge base data may include a predetermined knowledge graph (KG), a table, and / or JSON data.

[0084] Figure 5 is an example of a diagram illustrating a knowledge graph related to one embodiment of the present invention.

[0085] In this context, as shown in Figure 5, a knowledge graph (KG) can be considered, for reference, as a graphical representation of data in which entities, concepts, and / or events, and the relationships between them, are represented by nodes and edges.

[0086] Such knowledge graphs (KGs) can clearly and intuitively represent complex information and relationships, enabling various inferences and analyses based on them.

[0087] Specifically, a knowledge graph (KG) can include nodes representing individuals, concepts, and / or events, edges representing relationships between nodes, and attributes that provide additional information about the nodes (e.g., properties or characteristics of an individual).

[0088] For example, the nodes of a given knowledge graph (KG) could be "Film Director Name" and "Hollywood," the edge could be "(Film Director Name, Area of ​​Activity: Hollywood)," and the attribute could be "Film Director Name: Date of Birth, Place of Birth."

[0089] Specifically, in this embodiment, the computing system 1000 can provide a user interface (hereinafter referred to as the knowledge base data input interface) that can input predetermined knowledge base data.

[0090] Furthermore, the computing system 1000 can acquire the aforementioned knowledge base data based on user input provided through the knowledge base data input interface.

[0091] Depending on the embodiment, the computing system 1000 can also acquire the aforementioned knowledge base data based on its interaction with a predetermined external server.

[0092] Furthermore, in this embodiment, the computing system 1000 can generate linearly structured data based on the acquired knowledge base data. (S103)

[0093] Here, the linearly structured data according to the embodiment can mean data obtained by converting predetermined knowledge base data into text format.

[0094] In other words, in this embodiment, linearly structured data can be data obtained by linearly transforming knowledge base data, which may include various data with heterogeneous structures, into text format.

[0095] In this embodiment, such linearly structured data may include first linearly structured data, which is data obtained by converting knowledge base data into text format based on a multi-hop linearization (MHL) method, and second linearly structured data, which is data obtained by converting knowledge base data into text format based on another method (in this embodiment, a predetermined UnifiedSKG and / or JSON method, etc.).

[0096] Specifically, as an embodiment, the computing system 1000 can generate first linear structured data, which is data obtained by converting the acquired knowledge base data into text format based on the multiple-hop linearization (MHL) method, when the acquired knowledge base data is knowledge graph (KG) data.

[0097] Here, for reference, multiple-hop linearization (MHL) can be defined as the process of transforming information linked through various steps in a complex knowledge graph (KG) or information structure into a linear form. This approach can be applied in various fields such as information retrieval, natural language processing (NLP), and / or recommendation systems, and is particularly useful in tasks that utilize knowledge graphs (KG).

[0098] In this case, the hop can be an element that represents the number of connections (trunks) or the distance traveled required to move from one node (vertex) to another node in a knowledge graph (KG) or network.

[0099] Furthermore, the term "multi-hop" refers to a situation where multiple connections are required between two nodes, which can mean that various steps must be taken to track or infer information.

[0100] Furthermore, linearization refers to the process of converting information or data into a linear form, that is, a sequential structure. In this process, data with complex relationships and structures can be reconstructed into a simpler and more accessible form (such as text format in the embodiment).

[0101] In one embodiment, through the multi-hop linearization (MHL) described above, the computing system 1000 can simplify and understand the complex relationships between various pieces of information across various steps contained in a given knowledge graph (KG), and generate first linear structured data in a form that is easy to use for various tasks and the computing system 1000.

[0102] More specifically, in an embodiment, the computing system 1000 can use a multiple-hop linearization (MHL) method to generate first linear structured data by converting at least one knowledge triple (KT) in knowledge graph (KG) data into text format.

[0103] For reference, a knowledge triple (KT) is a basic unit for representing information in a knowledge graph (KG), and is mainly composed of the form of "data subject - relationship between subject and object (predicate) - object that is the subject of the relationship (object)." These three elements can each represent a node and an edge in the knowledge graph (KG).

[0104] For example, the subject of a knowledge triple (KT) could be "Albert Einstein," the relation could be "place of birth," and the object could be "Germany."

[0105] A knowledge graph (KG) can be constructed based on such a collection of knowledge triples (KTs), and an interconnected network of large-scale information can be formed.

[0106] Specifically, in this embodiment, the computing system 1000 may set one of the multiple nodes included in the knowledge graph (KG) data as the principal node.

[0107] Furthermore, the computing system 1000 can link at least one knowledge triple (KT) associated with a configured principal node within a hop count range.

[0108] For example, computing system 1000 can obtain a first knowledge graph (KG) as shown below.

[0109] [First Knowledge Graph (KG)]

[0110] *Nodes: People (Alice, Bob), Cities (New York, Paris), Companies (Google)

[0111] *Edge: (Alice lives in New York), (Bob works at Google), (Google is located in New York), (Alice's friend is Bob), (New York is located in Paris)

[0112] In this example, the computing system 1000 can generate first linear structured data, which is obtained by linearizing the multi-hop information from "Alice" to "Paris" on the first knowledge graph (KG), as shown below.

[0113] [First Knowledge Graph (KG) - First Linear Structured Data]

[0114] 1. Alice → Lives in → New York

[0115] 2. New York → Located → Paris

[0116] In other words, in this example, the computing system 1000 can generate first linear structured data that clearly represents indirect relationships between "Alice" and "Paris" in the first knowledge graph (KG) in text form via multiple-hop linearization (MHL), as described above.

[0117] Thus, in this embodiment, the computing system 1000 can perform a knowledge structuring process that converts a given knowledge graph (KG) into a natural text format using a multiple-hop linearization (MHL) method.

[0118] Through this, the computing system 1000 minimizes the loss of information that occurs in the process of reflecting the meaning of the knowledge graph (KG) into text, compared to conventional methods (for example, pipeline transformation methods based on multi-stage processes such as "entity sensing and concatenation," "sub-graph representation," and "style injection between graph and text"), while simultaneously structuring the knowledge graph (KG) into a clearer and more understandable text format, and performing language model learning based on this.

[0119] Therefore, by utilizing training data with a clearer structure, the computing system 1000 can directly improve the learning efficiency and performance of the language model, and significantly enhance the ability of the trained language model to handle various tasks (e.g., question answering, reasoning, comprehension, retrieval, and / or recommendation).

[0120] On the other hand, in one embodiment, if the acquired knowledge base data is tabular data, the computing system 1000 can generate second linear structured data, which is data obtained by converting the tabular data into text format based on a predetermined UnifiedSKG and / or JSON format.

[0121] For reference, UnifiedSKG (Unified Structured Knowledge Grounding) can be defined as an integrated approach to performing natural language processing (NLP) tasks by utilizing various structured knowledge sources (e.g., tables, knowledge graphs (KG), and / or lists). This approach enables more accurate and consistent processing of natural language tasks through the understanding and utilization of various structured knowledge.

[0122] For reference, JSON (JavaScript® Object Notation) can be defined as a lightweight text-based data exchange format used to store or transmit data.

[0123] Specifically, in this embodiment, the computing system 1000 can generate second linear structured data by converting table data into text format based on a predetermined UnifiedSKG and / or JSON format.

[0124] As described above, in the embodiment, the computing system 1000 can convert and structure knowledge base data (such as knowledge graphs (KG) and / or table data in the embodiment) that may include various types of heterogeneous data into a simple and clear text format.

[0125] This enables the computing system 1000 to perform language model learning based on structured knowledge injection from a text-based foundation, thereby enhancing deep learning learning performance, i.e., language model knowledge enhancement.

[0126] In addition, in this embodiment, the computing system 1000 can be trained to learn a language model based on the generated linear structured data. (S105)

[0127] Specifically, in this embodiment, the computing system 1000 can be trained on a predetermined language model using the linearly structured data generated as described above.

[0128] In other words, the computing system 1000 can train a predetermined language model using first linear structured data, which is data structured into text format via a multi-hop linearization (MHL) method of knowledge graph (KG) data, and / or second linear structured data, which is data structured into text format via a predetermined UnifiedSKG or the like.

[0129] Here, the language model according to the embodiment (hereinafter referred to as the first language model) may include a pre-trained language model (PLM).

[0130] This allows the computing system 1000 to perform additional training on the first language model based on linearly structured data.

[0131] In this embodiment, the computing system 1000 can perform first language model training on a linearly structured data base based on the Masked Language Modeling (MLM) method.

[0132] For reference, Masked Language Modeling (MLM) is one of the pre-training methods used in the field of natural language processing (NLP), and is particularly widely used for training transformer-based models (e.g., BERT (Bidirectional Encoder Representations from Transformers)).

[0133] Such masked language modeling (MLM) can operate as a principle that develops the ability to understand the bidirectional context of text by randomly masking some words or tokens within a given text and having the model predict the masked words or tokens based on the context of the remaining words or tokens.

[0134] Through this process, the model can achieve improved performance in better understanding the context in which words and tokens are used within text. A more detailed explanation of this is provided by the previously disclosed explanation.

[0135] More specifically, in an embodiment, the computing system 1000 can train a first language model on a linearly structured data base based on a salient span masking (SSM) scheme, which is an extended form of the masked language modeling (MLM) learning strategy.

[0136] For reference, Salient Span Masking (SSM) is a learning method that, during pre-training, does not randomly mask arbitrary words or tokens in the text. Instead, it identifies semantically important parts of the text, replaces (i.e., masks) these identified important parts with [MASK] tokens, and predicts the masked span based on the surrounding context. This method develops contextual understanding of the text and the ability to infer key information.

[0137] Figure 6 is an example of a diagram illustrating the first linear structured data according to one embodiment of the present invention.

[0138] For example, as shown in Figure 6, the computing system 1000 can generate the first linear structured data using the second knowledge graph (KG) in Figure 6 as follows.

[0139] [Second Knowledge Graph (KG) - First Linear Structured Data]

[0140] 1(one hop).the yearling starring actors Gregory peck

[0141] 2(two hop).the yearling starring actors Gregory peck act in the gunfighter

[0142] 3(Three hop).the yearling starring actors Gregory peck act in the gunfighter has tag henry king

[0143] In this example, the computing system 1000 can apply the above-described first linear structured data to the learning of a first language model via the salient span masking (SSM) method.

[0144] In this case, by performing salient span masking (SSM), the first linear structured data may be such that, for example, in the case of one-hop and two-hop linearization, the first object is masked, and in the case of three-hop linearization, the first and last objects are masked.

[0145] [Second Knowledge Graph (KG) - First Linear Structured Data - Salient Span Masking (SSM) Applied]

[0146] 1(one hop).the yearling starred actors [MASK]

[0147] 2(two hop).the yearling starred actors [MASK] act in the gunfighter

[0148] 3(Three hop).the yearling starred actors [MASK] act in the gunfighter has tag [MASK]

[0149] Figure 7 is an example of a diagram illustrating the second linear structured data according to one embodiment of the present invention.

[0150] As another example, as shown in Figure 7, the computing system 1000 can generate the second linear structured data using the first table in Figure 7 as follows.

[0151] [Table 1 - Linear Structured Data 2 (UnifiedSKG Style)]

[0152] Renaissance (band) col: Year | Title | Char Position | Comment row:1971 | Illusion | - | 1976(UK)

[0153] [Table 1 - Linear Structured Data (JSON Style)]

[0154] { “PAGE_NAME”:“Renaissance(band)”, "Year": "1971" “Title”: “Illusion”, “Char Position”: “”, "Comment": "1976 (UK)" }

[0155] In this example, the computing system 1000 can apply the above-described second linear structured data to the first language model training via the salient span masking (SSM) method.

[0156] In this embodiment, the computing system 1000 can also perform random masking in the case of second linear structured data.

[0157] [Table 1 - Linear Structured Data (UnifiedSKG Style) - Salient Span Masking (SSM) Applied]

[0158] Renaissance (band) col:Year | Title | Char Position | Comment row:[MASK] | Illusion | - | 1976(UK)

[0159] [Table 1 - Linear Structured Data (JSON Style) - Salient Span Masking (SSM) Applied]

[0160] { “PAGE_NAME”:“Renaissance(band)”, “Year”: “[MASK]”, “Title”: “Illusion”, “Char Position”: “”, "Comment": "1976 (UK)" }

[0161] Thus, in this embodiment, the computing system 1000 can train a language model using linearized knowledge as described above, based on a learning method on a mask language modeling (MLM) platform.

[0162] In other words, the computing system 1000 can perform language model learning based on linearly structured data according to the embodiment of the present invention using a training method that enhances contextual understanding, reasoning ability, and learning efficiency.

[0163] Through this, the computing system 1000 can realize and provide a language model (in one embodiment, a pre-trained language model (PLM)) that more effectively applies and distills predetermined external knowledge.

[0164] In addition, in this embodiment, the computing system 1000 can provide application services based on the learned language model. (S107)

[0165] In other words, in this embodiment, the computing system 1000 can provide various application services based on a language model learned based on the linearly structured data generated as described above.

[0166] As an embodiment, the computing system 1000 can provide services such as a chatbot (virtual secretary) service, an automatic translation service, a text generation and summarization service, an educational and learning assistant service, an information retrieval and recommendation service, and / or a code generation and analysis tool service based on the learned language model.

[0167] Thus, in this embodiment, the computing system 1000 can easily support various application services based on a language model with improved learning and task processing performance, and can effectively enhance their performance and quality.

[0168] The following describes a method and system for enhancing language model performance through structural knowledge injection according to one embodiment of the present invention. By transforming and structuring a predetermined knowledge graph (KG) based on multiple-hop linearization (MHL), the method converts the knowledge graph (KG) into a clearer and easier-to-understand text format, while simultaneously minimizing the loss of information that occurs in the process of reflecting the meaning of the knowledge graph (KG) into text.

[0169] Furthermore, the method and system for enhancing language model performance through structural knowledge injection according to one embodiment of the present invention can directly improve the learning efficiency and performance of a language model by learning the language model based on the structured knowledge graph (KG), thereby utilizing learning data with a clearer structure, and can significantly improve the ability of the learned language model to process various tasks (e.g., question answering, reasoning, comprehension, retrieval, and / or recommendation).

[0170] Furthermore, the method and system for enhancing language model performance through structural knowledge injection according to one embodiment of the present invention has the effect of providing a language model that improves contextual understanding, reasoning ability, and learning efficiency by learning the language model based on masked language modeling (MLM), thereby more effectively applying external knowledge and providing a distilled language model.

[0171] On the other hand, embodiments of the present invention described above can be implemented in the form of program instructions that can be executed through various computer components and recorded on a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc., individually or in combination. The program instructions recorded on the computer-readable recording medium may be specifically designed and configured for the present invention or may be publicly known and available to those skilled in the field of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROMs, RAMs, and flash memory. Examples of program instructions include not only machine code, such as that produced by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. Hardware devices can be modified into one or more software modules to perform the processing according to the present invention, and vice versa.

[0172] The specific embodiments described in this invention are merely examples and do not limit the scope of the invention in any way. For the sake of brevity, descriptions of conventional electronic configurations, control systems, software, and other functional aspects of such systems may be omitted. Furthermore, connections such as lines or connecting members between components shown in the drawings are illustrative representations of functional and / or physical or circuit connections and may be substituted or represented as various additional functional, physical, or circuit connections in actual devices. In addition, components that are not necessarily required for the application of this invention may not be required unless specifically mentioned, such as "essential" or "important."

[0173] Furthermore, while the detailed description of the present invention has been provided with reference to preferred embodiments, a person skilled in the art or with ordinary knowledge in the art will understand that the present invention can be modified and altered in various ways without departing from the spirit and technical scope of the invention as described in the claims below. Therefore, the technical scope of the present invention should not be limited to what is described in the detailed description of the specification, but should be determined by the claims. Industrial Applicability

[0174] This invention relates to a method and system for enhancing the performance of language models through structural knowledge injection, and is applicable to the artificial intelligence industry, thus possessing industrial applicability.

Claims

1. A method for enhancing the performance of a language model through structural knowledge injection in a computing system comprising memory and a processor, The steps include: acquiring knowledge base data that includes a predetermined knowledge graph; The steps include generating linearly structured data, which is data structured in text format from the acquired knowledge base data, The steps include training a first language model based on the generated linear structured data, The steps include providing a predetermined application service based on the learned first language model, Includes, The step of generating the linearly structured data is: A method for enhancing the performance of a language model through structural knowledge injection, which includes the step of generating first linear structured data, which is data structured in text format from the knowledge graph, based on multi-hop linearization.

2. The aforementioned knowledge graph is, Graphic data that represents the relationships between multiple entities based on nodes and edges. A method for enhancing the performance of a language model via structural knowledge injection according to claim 1, comprising at least one knowledge triple which is data representing a Subject-Predate-Object relationship based on the nodes and edges.

3. The step of generating the first linear structured data is: A method for enhancing the performance of a language model via structural knowledge injection according to claim 2, comprising the step of converting the subject-relationship-object data into text format based on the multi-level linked knowledge triples in the knowledge graph.

4. A method for enhancing the performance of a language model via structural knowledge injection according to claim 1, further comprising the step of acquiring the knowledge base data including a predetermined table.

5. The step of generating the linearly structured data is: A method for enhancing language model performance via structural knowledge injection according to claim 4, further comprising the step of generating a second linear structured data which is data that structures the table into text format based on a predetermined UnifiedSKG (Unified Structured Knowledge Grounding) and JSON (JavaScript® Object Notation).

6. The step of training the first language model is: The steps include masking at least a portion of the text within the linearly structured data, The steps include predicting the masked text based on the remaining text in the linearly structured data, A method for enhancing the performance of a language model through structural knowledge injection as described in claim 1, including the above.

7. The step of masking at least a portion of the text in the linearly structured data is: The steps include identifying the main text within the linearly structured data, The steps include replacing the identified main text with a mask token, A method for enhancing the performance of a language model through structural knowledge injection as described in claim 6, including the above.

8. The step of training the first language model is: If the knowledge base data includes the table, the steps include random masking at least a portion of the text in the second linear structured data, The steps include predicting the randomly masked text based on the remaining text in the second linearly structured data, A method for enhancing the performance of a language model through structural knowledge injection as described in claim 5, including the above.

9. The step of training the first language model is: A method for enhancing the performance of a language model via structural knowledge injection according to claim 1, comprising the step of further training a pre-trained language model.

10. At least one memory, A processor that reads at least one application stored in the memory and performs a method of enhancing language model performance through structural knowledge injection, Equipped with, The aforementioned processor, Based on the Multi-hop Linearization and Salient Span Masking processes, knowledge base data including a predetermined Knowledge Graph is structured into text format. A first language model is trained based on the aforementioned structured knowledge base data. A language model performance enhancement system via structural knowledge injection that provides predetermined application services based on the learned first language model.

Images