An analysis framework supporting natural language structure computation

Abstract

The application provides an analysis framework supporting natural language structure calculation, comprising three functional modules of a grid, a data table and a finite state automaton, and designs a series of API to cooperate with a Lua script language to complete writing of a language structure calculation script, including a general control script and a finite state automaton script; the language structure calculation script is executed through an executable program, so that language structure calculation is realized. The application takes symbolic calculation as a guide, receives input and outputs results, and has better controllability, expandability and interpretability; in the calculation process, various parameter calculation models can be dispatched, so that the capacity of the parameter models can be fully brought into play; finally, the framework can also introduce explicit knowledge, so that deep and fine language structure calculation is possible under the guidance of the explicit knowledge.

Description

TECHNICAL FIELD

[0001] The present application relates to a natural language processing method, in particular to an analysis framework supporting natural language structure calculation. BACKGROUND

[0002] Language structure calculation is the process of structuring natural language, that is, analyzing natural language into language structure, which is the essence of natural language understanding. Language structure is the embodiment of the regularity of natural language in form, content and usage. This regularity is manifested in form as grammatical structure, in content as semantic structure, and in usage as pragmatic structure.

[0003] There are two strategies for language structure calculation in the prior art: end-to-end strategy and cascade strategy. The end-to-end strategy refers to inputting raw data and directly outputting the final result, and the mainstream method is based on deep learning. It is often data-driven, sets an optimization function for the target, uses a parameterized vector to represent the language unit, and constructs a model through parameter learning. For example, when using the end-to-end strategy for Chinese syntax analysis tasks, the labeled syntax information corpus is usually used as training data, and a suitable model is designed for parameter learning.

[0004] The end-to-end strategy is a commonly used method in recent years, and the effect of almost all natural language processing tasks, whether it is the ontology task of NLP or the landing application, has been greatly improved. However, this strategy has encountered the following bottlenecks in its development:

[0005] 1) Lack of ability to use explicit knowledge. The knowledge of deep learning models comes entirely from data, such as domain knowledge, grammatical knowledge, semantic knowledge in labeled data, and world knowledge in large-scale pre-trained language models. However, external non-data knowledge, such as expert knowledge and offline knowledge, cannot be embedded into the model in a suitable way, and the model cannot guide the solution to complex problems based on knowledge.

[0006] 2) Lack of reasoning and analysis ability. Deep learning models are black box parameter systems, and their internal structure is not a human-like concept system. They cannot be connected to world knowledge externally, and they cannot reason and analyze.

[0007] 3) Lack of explainability and controllability. Due to the black box characteristics of end-to-end calculation, the internal structure of deep learning models is completely parameterized, which has the problem of poor explainability and controllability.

[0008] Cascade strategy refers to decomposing a problem into two or more successive sub-tasks, each of which is completed independently, and the input of the subsequent sub-task is the output of the previous task. For example, when using cascade strategy to perform Chinese syntax analysis task, it is often divided into two stages: one is Chinese word segmentation, and the other task is syntax structure analysis, wherein the output of the Chinese word segmentation task is taken as the input of the syntax structure analysis.

[0009] The problem of cascade strategy is mainly error propagation, and due to the "probability multiplication" between cascade models, the accuracy and recall rate of the overall model are not ideal. SUMMARY

[0010] The application provides an analysis framework supporting natural language structure calculation, which is used for solving the problem of language structure calculation, and the technical scheme is as follows:

[0011] An analysis framework supporting natural language structure calculation, characterized by comprising:

[0012] Grid: a data structure used for storing language structure, serving as a calculation platform for carrying various language structures;

[0013] Data table: used for encapsulating unary knowledge and binary knowledge of symbol types, serving as an encapsulating component;

[0014] Finite state automaton: used for representing calculation control components of language context, cooperating with script language to complete the control task of code function structure,

[0015] API system, cooperating with Lua script language to complete the writing of language structure calculation script around the grid, data table and finite state automaton;

[0016] Executable program of language structure calculation script: having indexing function, capable of realizing indexing of data table and finite state automaton; having execution function, capable of executing language structure calculation script and realizing language structure calculation; having execution function of total control script, writing the index file path of data table and finite state automaton and the IP address and interface of external service in the configuration file, and transmitting the configuration file into the executable program.

[0017] The grid comprises the following internal variables: text to be analyzed, grid attribute, grid unit, and relationship between grid units;

[0018] The grid unit comprises the following internal variables: stored language unit, grid unit attribute, feature and score of the grid unit attribute;

[0019] The relationship between grid units comprises the following internal variables: attribute of the relationship between grid units, feature and score of the relationship between grid units.

[0020] The language structure is represented as a directed graph with attribute information, including language units, relations, and attributes, and is loaded into the grid in a carrying manner: the language units are carried by the grid units, and each language unit corresponds to a grid unit; the relations between the language units are carried by the relations between the grid units; the attributes of the language units are carried by the attributes of the grid units; and the attributes of the relations between the language units are carried by the attributes of the relations between the grid units.

[0021] The language structure is represented as G=(U,R), wherein:

[0022] 1) U is the node of the graph, which is a finite set of language units, denoted as: U={tu,au}, tu∈Lex, Lex is a set of language unit strings, and au is an attribute of the language unit, which is a set of key-value pairs, denoted as: au={K=V}, K is an attribute name, and V is an attribute value;

[0023] 2) R is a finite set of edges connecting two different nodes in U, that is, a set of relations between two language units; since the language units are often not dual, it is a directed edge, denoted as: R={u i ,u j ,r,ar}, wherein: u i ∈U, u j ∈U, r∈RT, RT is a set of relation types. ar is an attribute of the language unit relation, which is a set of key-value pairs, denoted as: ar={K=V}, K is an attribute name, and V is an attribute value.

[0024] The API includes: (1) an API for calling external services and importing service results into the grid, thereby realizing the scheduling of the parameter calculation model; (2) an API for controlling internal variables in the grid structure, including adding, testing, and obtaining internal variables, thereby realizing the controllability of the language structure; (3) an API for applying a data table, that is, realizing the interaction between the data table and the grid, thereby introducing display knowledge; and (4) an API for applying a finite state automaton, that is, realizing the interaction between the finite state automaton and the grid, thereby expanding the functions of the framework.

[0025] The data table is defined as a set of triples, that is: TableName={<Item,Attribute,Condition>}, wherein:

[0026] TableName: table name;

[0027] Item: data item, which is a string in the form of Word or a key-value expression KV; when it is Word, it corresponds to a language unit, and when it is KV, it corresponds to all grid units in the grid that satisfy the KV.

[0028] Attribute: the attribute of Item, is a set of key-value pairs, that is, {K=V}, which is brought into the attribute of the grid cell or the relationship attribute between cells;

[0029] Condition: the limiting condition of applying Item, is a set of key-value expressions, that is, {KV}, when the grid cell corresponding to the current Item satisfies one of the key-value expressions, the next operation is continued, including adding grid cells, attributes or establishing relationships.

[0030] The data table is divided into two categories, one is a description type data table, and the other is a relational type data table;

[0031] The description type data table describes an independent language unit, gives the form of the language unit and the attribute knowledge of the language unit, which is used to determine the language unit and set the language unit attribute;

[0032] The relational type data table describes two language units, one is the center language unit, and the other is the language unit having a certain relationship with the center language unit, encapsulates the binary relationship with multiple data tables, and is designed in the form of one master table and multiple slave tables, the master table stores the list of center language units, and the slave table stores the list of language units forming a certain relationship with the center language unit.

[0033] The finite state automaton is defined as a set of four tuples, that is: FSAName={<Enter, Path, Operation, Exit>}, wherein: FSAName: the name of the finite state automaton, globally unique name; Enter: entry node, a finite state automaton has a unique entrance; Path: path corresponding to the finite state automaton, representing context information in structure calculation; Operation: operation node, that is, the action to be performed when the context corresponding to Path is successfully tested; Exit: exit node, a finite state automaton has a unique exit.

[0034] One script can correspond to one or more finite state automata, and the script of each finite state automaton includes FSA name, parameter item, control item, and a function library shared by multiple finite state automata;

[0035] The FSA name: different finite state automata are distinguished by FSA name;

[0036] The parameter item: configuration related to the current FSA running;

[0037] The control item: describes different context situations and their corresponding operations, which consists of Context and Operation, Context is used to describe the condition part of FSA path, which consists of multiple Items; Operation describes the operation under the corresponding condition;

[0038] The function library: defines the function library name in the form of "NameSpace Name", and defines the function body in the form of Lua script, where "NameSpace" is a reserved word.

[0039] Further, the API for calling external services and importing service results into the grid realizes the application of parameter models, including:

[0040] (1) CallService (Sentence, ServiceName): API for calling services, the called service is a parameterized model for structure analysis, which structures the text to be analyzed, and returns the data of the initial language structure; The first parameter is the input to be passed to the service, and the second parameter is the service name, which needs to be configured in the configuration file, and the return value is the language structure returned by the service;

[0041] (2) AddStructure (Sentence_JSon): inject the language structure in JSon format into the grid.

[0042] The API for controlling internal variables of the grid structure realizes the addition, acquisition and testing of internal variables of the grid structure, improves the controllability and interpretability of language structure calculation, including: (1) internal variable addition class API; (2) internal variable acquisition class API; (3) internal variable test class API.

[0043] The API for applying data table is divided into three categories: function class, acquisition class and test class, the function class API realizes the interaction between the grid and the data table, so that the framework has the ability to apply explicit knowledge, including:

[0044] 1) Segment (TableName)

[0045] The function of Segment is to segment the text in the grid based on the data table and add attributes;

[0046] 2) SetLexicon (TableName)

[0047] The function of SetLexicon is to add the attributes of data items in the data table to the grid cells, providing application attribute information for the grid cells;

[0048] 3) Relate(TableName)

[0049] The function of Relate is to import the relational type data table by calling the main table name, the relation class name and the relation name;

[0050] When importing the relational type data table into the grid by using the Relate function, the data items in the data table correspond to the grid cells, and the relations formed by the data items in the main table and the data items in the sub-table correspond to the cell relations in the grid. The function implemented by the Relate function is decomposed into the following steps:

[0051] ①Import the main table data items and their attributes that meet the Limit condition into the grid; and add the "ST-Unit" attribute to the grid, whose value is TableName, indicating that there are grid cells in the grid that are derived from the data table TableName;

[0052] ②Obtain all the sub-tables corresponding to the main table data items through the Coll attribute;

[0053] ③Add the sub-table data items that meet the Limit condition to the grid;

[0054] ④Add the grid cell relations, U-type attributes and R-type attributes to the grid cells in the binary data table with TableName as the main table;

[0055] ⑤Add the attributes of the sub-table data items to the attributes of the grid cell relations;

[0056] ⑥Add the "URoot", "URootTableName", "RRoot", "RRootTableName", "ST-Relation" attributes to the grid. For each relation <HeadUnit, SubUnit, Relation> between the grid cells successfully added to the grid, the attribute values of the above attributes are: HeadUnit, HeadUnit, Relation, Relation, TableName.

[0057] 4) Str = GetPrefix(TableName, String)

[0058] GetPrefix judges whether a string is prefixed with a data item in a data table. If so, it returns the longest matching string;

[0059] 5) Str = GetSuffix(TableName, String)

[0060] GetSuffix judges whether a string is suffixed with a data item in a data table, and returns the longest matching string if so.

[0061] The API of the application finite state automaton realizes the interaction between the grid and the finite state automaton, so that the framework has the ability to efficiently recognize and process context; the API includes RunFSA, GetFSANode and GetParam:

[0062] (1) RunFSA(FSAName(,Param))

[0063] The function of the API is to execute a finite state automaton, that is, to complete the matching of Context and the grid in the FSA script and execute the corresponding Operation, wherein FSAName is the name of the finite state automaton to be executed, and Param is the parameter to be passed, which can be called by the API GetParam in the finite state automaton script;

[0064] (2) No=GetFSANode(-1) or No1,No2=GetFSANode("$Tag")

[0065] Executing a finite state automaton is to complete the matching of Context and the grid in the finite state automaton script, if the matching is successful, that is, the connection from Enter to Exit node is completed, at this time one or more paths realize the connection; in the script of Operation, the grid unit corresponding to the Node in Context can be accessed, therefore the attribute nodes on the FSA path are numbered in turn, and the grid unit is accessed by referring to the node number.

[0066] (3) Str=GetParam(Key)

[0067] The parameter value passed in by the RunFSA function is read in the FSA.

[0068] The analysis framework supporting natural language structure calculation is dominated by symbolic calculation, which receives input and outputs results, and has better controllability, scalability and interpretability; in the calculation process, various parameter calculation models can be scheduled, so that the ability of the parameter model can be fully utilized; finally, the framework can also introduce explicit knowledge, which makes deep and fine language structure calculation possible under the guidance of explicit knowledge. BRIEF DESCRIPTION OF DRAWINGS

[0069] Figure 1 is a schematic diagram of the analysis framework supporting natural language structure calculation;

[0070] Figure 2 The language structure is represented as a directed graph with attribute information;

[0071] Figure 3 It is a finite state transition diagram compiled from a finite state automaton script. Detailed Implementation

[0072] like Figure 1 As shown, the analysis framework supporting natural language structure computation includes three functional modules: a grid, a data table, and a finite state automaton. A series of APIs (Application Programming Interfaces) are designed around these three functional modules to work with the Lua scripting language to write language structure computation scripts, including a control script and a finite state automaton script. Finally, this framework designs an executable program to execute the language structure computation scripts, thereby realizing language structure computation.

[0073] 1. Grid: A computational platform that supports various language structures. The complexity of natural language lies in the fact that a one-dimensional sequence of symbols corresponds to a two-dimensional formal and conceptual structure. Therefore, during computation, the computational structure must accommodate various ambiguities, including ambiguities at the boundaries of language units and ambiguities in language concepts. A grid can simultaneously accommodate analytical structures generated by different levels, different algorithms, and even different systems, enabling them to both collaborate and be independently distinguishable, collectively supporting the generation of complex target structures.

[0074] 2. Data Tables: Components for representing and processing knowledge. This framework uses a knowledge-driven expert system as the central control to complete complex language structure analysis, including deep semantic analysis. Therefore, it needs to formally represent knowledge and support rapid computation of massive amounts of knowledge. In this framework, data tables are used to meet these requirements.

[0075] 3. Finite State Automata: A computational control unit used to represent the context of a language. Natural languages ​​have complex contexts, requiring a concise, efficient, expressive, and powerful control unit. In scripting languages, logical control statements are typically used to handle branching and conquering processes based on different conditions. This framework employs finite state automata in conjunction with a scripting language to complete the control task of the code's functional structure.

[0076] The API system of this framework has the following design: (1) This framework is designed with APIs that call external services and import the service results into the grid, thereby realizing the scheduling of parameter calculation models; (2) This framework is designed with APIs that control the internal variables of the grid structure, including the addition, testing and acquisition of internal variables, thereby realizing the controllability of the language structure; (3) This framework is designed with APIs that apply data tables, that is, realize the interaction between data tables and the grid, thereby introducing explicit knowledge; (4) This framework is designed with APIs that apply finite state automata, that is, realize the interaction between finite state automata and the grid, thereby expanding the functionality of this framework.

[0077] The executable program of the language structure calculation script has the following design in this framework: (1) The executable program in this framework has an indexing function, which can index the data table and finite state automaton, so that they can be used efficiently during execution; (2) The executable program in this framework has an execution function, which can execute the language structure calculation script and realize the language structure calculation; (3) The executable program in this framework is the execution of the master control script. In order to realize the calling of the data table, finite state automaton and external services in the master control script, this framework has designed a configuration file. The specific method is as follows: write the index file path of the data table and finite state automaton as well as the IP address and interface of the external service in the configuration file, and pass the configuration file to the executable program.

[0078] like Figure 1 As shown below, the three components and API system in this framework will be described in detail.

[0079] 1. Grid

[0080] As a data structure, the grid is used to store language structures. The following explains how the grid stores language structures, including: the definition of the grid as a data structure, the formal definition of language structures, and the correspondence between the grid and language structures.

[0081] 1.1 Definition of Mesh

[0082] A grid, as a data structure, includes the following internal variables: the text to be analyzed, grid attributes, grid cells, and relationships between grid cells. Each grid cell is also a data structure, including the following internal variables: the stored language units, grid cell attributes, and the characteristics and scores of those attributes. The relationships between grid cells are also a data structure, including the following internal variables: the attributes of the relationships between grid cells, and the characteristics and scores of those relationships. The specific definitions of these variables are as follows:

[0083] 1) Text to be analyzed: This refers to the analysis object in the current analysis process. Its variable value is usually a sentence, a paragraph, or a chapter.

[0084] 2) Grid attributes: These are the attributes of the current grid, consisting of a set of one or more key-value pairs in the form of "K=V". They are used to store information related to other internal variables in the grid and log information during analysis. "K" is the attribute name and "V" is the attribute value. The attribute value can be one or a set of multiple values.

[0085] 3) Grid cell: The grid cell is the basic element of the grid. A grid usually contains multiple grid cells arranged in a matrix. Each grid cell has a unique cell number, which consists of its column number c and its row number r (c, r).

[0086] 4) Grid cell attributes: Grid cell attributes describe the properties of the grid cell. Attributes include attribute name and attribute value, which are stored in key-value pairs, i.e., Key = Value (abbreviated as K = V). Attribute values ​​can be a single value or multiple unequal attribute values, i.e., a set of attribute values, which can be abbreviated as "K = [v1, v2]" using the logical symbol "[]".

[0087] Based on the different content described, attributes can be divided into property attributes and relational attributes. Property attributes describe the properties of the grid cell itself, such as the type and location of the grid cell; relational attributes describe the relationship between the current grid cell and other grid cells.

[0088] 5) Features and scores of grid cell attributes: These are the features of the attribute value. They are a set of one or more key-value pairs in the form of "Feature=Score", which are used to store the features and scores of the current attribute value. "Feature" is the feature description and "Score" is the score of the feature.

[0089] 6) Relationships between mesh cells: The partial order binary relation between two mesh cells is of the form:<HeadUnit,SubUnit,Relation> This means that "the HeadUnit points to the SubUnit via the Relation relation".

[0090] 7) Attributes of relationships between grid cells: These are the attributes of the relationships between the grid cells. They are a set of one or more key-value pairs in the form of "K=V", used to store information related to the relationships between the current grid cells. "K" is the attribute name and "V" is the attribute value. The attribute value can be one or a set of multiple values.

[0091] 8) Features and scores of relationships between grid cells: These are the features of relationships between grid cells, consisting of a set of one or more key-value pairs in the form of "Feature = Score". They are used to store the features and scores of relationships between current grid cells, where "Feature" is the feature description and "Score" is the score of that feature.

[0092] 1.2 Formal Definition of Language Structure

[0093] This framework treats language structure from the perspectives of units, relations, and attributes, representing language structure as a directed graph with attribute information, such as... Figure 2 As shown. Figure 2 In the diagram, nodes u1 and u2 represent language units. A directed edge r, starting from node u2 and ending at node u1, represents the relationship between two language units. Key-value pairs of the form K=V can be used to represent the attributes of language units and the attributes of the relationship between units on both nodes and edges.

[0094] The language structure is a directed graph, which, according to the definition of a graph, is a pair of tuples, formally defined as: G = (U, R), where:

[0095] 1) U is a node in the graph, which is a finite set of language units, denoted as: U = {tu, au}, where: tu ∈ Lex, Lex is the set of language unit tokens, and au is the attribute of the language unit, which is a set of key-value pairs, denoted as: au = {K = V}, where K is the attribute name and V is the attribute value.

[0096] 2) R is a finite set of edges connecting two distinct nodes in U, that is, the set of relationships between two language units. Since language units are often not dual, it is a directed edge. It is denoted as: R = {u i ,u j ,r,ar},here:u i ∈U, u j Let r ∈ U, r ∈ RT, where RT is the set of relation types. ar is an attribute of a language unit relation, a set of key-value pairs, denoted as: ar = {K = V}, where K is the attribute name and V is the attribute value.

[0097] 1.3 Correspondence between grids and language structures

[0098] Any language structure, including grammatical, semantic, and pragmatic structures, can be formally represented and loaded into a grid using the concept of "language units, relations, and attributes." This allows the grid to support multi-source and multi-type language structures. The loading mechanism is as follows: language units are carried by grid units, with each language unit corresponding to one grid unit; relationships between language units are carried by relationships between grid units; attributes of language units are carried by attributes of grid units; and attributes of relationships between language units are carried by attributes of relationships between grid units.

[0099] 2. Data Table

[0100] Data tables are encapsulation components of explicit knowledge, which includes unary and binary knowledge. Unary knowledge is descriptive, describing the attributes of language units in terms of pronunciation, part of speech, and translation. Binary knowledge is relational, describing the types of relationships between language units.

[0101] In this framework, data tables are used to encapsulate unary and binary knowledge of symbol types, providing an important source of information for constructing grid cells, establishing relationships, and setting attributes during language structure calculations.

[0102] 2.1 Formal Definition of Data Tables

[0103] A data table can be defined as a set of triples, i.e.: TableName = {<Item,Attribute,Condition>},in:

[0104] TableName: Table name;

[0105] Item: A data item, which can be a string (Word) or a key-value expression (KV). When it is a Word, it corresponds to a language unit, and in grid computing, it corresponds to a grid cell; when it is a KV, the data item corresponds to all grid cells within the grid that satisfy KV as true.

[0106] Attribute: The property of an Item is a collection of key-value pairs, i.e., {K=V}. Usually, {K=V} will be passed into the properties of grid cells or the relationship properties between cells.

[0107] Condition: The condition for applying the Item is a set of key-value expressions, i.e., {KV}. If the grid cell corresponding to the current Item satisfies one of the key-value expressions in {KV}, then the next operation can be continued, including adding grid cells, attributes, or establishing relationships.

[0108] It should be noted that for key-value expressions (KV), this framework defines them as follows: A key-value pair K = V used to describe an attribute. In computational scenarios, the attribute can also be tested, returning a logical "true" or "false" value, indicating whether the computed object has a key-value pair "K = V". This "K = V" used for computation is called a key-value expression, denoted as "KV". Furthermore, logical operations involving multiple key-value expressions are also called key-value expressions. Logical operators include: AND "&"; NOT "!"; OR "[]".

[0109] 2.2 Data Table Format Definition

[0110] This framework contains one or more data tables, stored in one or more files. The format of the data tables is defined as follows:

[0111] Table TableName

[0112] #Global{K=V}Limit=[{KV}

[0113] Word{K=V}Limit=[{KV}]

[0114] KV{K=V}Limit=[{KV}]

[0115] in:

[0116] Line 1: Table is a reserved word, and TableName is the name of the data table. It is globally unique and is written after the reserved word "Table" in the first line. When applying the application, the data table is referenced by the data table name.

[0117] Line 2: "#Global" is a reserved word, followed by a key-value pair or a key-value expression enclosed in Limit. This entry is optional; if present, it indicates that the following content is shared by all entries in this table.

[0118] Line 3: Item is a string (Word) usage example, {K=V} is a set of key-value pairs, "Limit" is a reserved word indicating the limiting conditions for applying the word, and one or more key-value expressions can be written in the logical OR "[]".

[0119] Line 4: Item is a usage example of key-value expression (KV), {K=V} is a set of key-value pairs, "Limit" is a reserved word, indicating the limiting condition for applying this term, and one or more key-value expressions can be written in the logical OR "[]".

[0120] 2.3 Types of Data Tables

[0121] The data tables are divided into two categories: one is a descriptive data table similar to a dictionary, and the other is a relational data table designed to facilitate the construction of relationships between two language units.

[0122] Descriptive data tables describe individual language units, providing information such as the form and attributes of the language units. This information is mainly used to identify language units and set their attributes.

[0123] Relational tables describe two types of language units: a head language unit and language units that have a certain relationship with the head language unit. Unlike semantic fields, the binary relationship here is generally a partial order relationship, which can be syntactic or semantic. Relational tables typically encapsulate binary relationships using multiple tables, usually designed as a master table with multiple slave tables. The master table stores the list of head language units, and the slave tables store the lists of language units that form a certain relationship with the head language unit.

[0124] In the main table, you need to use attributes to specify the subordinate tables that can form a specific relationship with the current data item. Specifically, you need to give the relationship name in the Coll attribute of the main table, and specify the name of the subordinate table that can form a specific relationship with the current data item through the attribute in the form of "Coll-relation name=[subordinate table name1 subordinate table name2]".

[0125] 3. Finite automata

[0126] As a control component, finite state automata can efficiently express and execute the context in language structure analysis. They are written using finite state automata script grammars, which logically provide a clearer description of the context conditions. In the application phase, the finite state automata described by the script grammar is transformed into an internal node connection graph using compilation tools. The state of the grid cell is tested using key-value expressions (KV) within each node; if the key-value expression is true, the process moves to the next node.

[0127] In this invention, one or more finite state automata can be designed as needed. Each finite state automaton is independent of the others and is called by the upper-level script to realize a preset function.

[0128] 3.1 Definition of Finite State Automata

[0129] In this invention, a finite state automaton is defined as a set of four-tuples, i.e.: FSAName = {<Enter,Path,Operation,Exit>},in:

[0130] FSAName: The name of the finite state automaton, globally unique. Enter: The entry node; a finite state automaton has a unique entry point. Path: The path corresponding to the finite state automaton; during structural computation, it represents context information. It is an ordered sequence of multiple attribute test nodes; in this invention, each attribute test node stores a key-value expression (KV), which is used for attribute testing of the mesh cells. Operation: The operation node, i.e., the action to be executed when all contexts corresponding to the Path are successfully tested. Exit: The exit node; a finite state automaton has a unique exit point.

[0131] 3.2 Grammar for Writing Finite State Automata

[0132] In this invention, one script can correspond to one or more finite state automata. Each finite state automaton's script includes three parts: FSA name, parameter items, and control items. In addition, there is a function library shared by multiple finite state automata. The contents of each part will be described in detail below with reference to the script.

[0133] 1.FSAFSAName

[0134] 2. #Entry EntryNode = [KV]

[0135] 3.#Parameter Order=Yes MaxLen=Yes Nearby=Yes Bound=Clause

[0136] 4. #Include CodeLib 5.

[0138] 6. Context1

[0139] 7.{

[0140] 8. Operation 1

[0141] 9.} 10. 11....

[0144] 12. Item1 Item2 SubName

[0145] 13.{

[0146] 14. Operation

[0147] 15.} 16.

[0149] 17.sub SubName 18.(

[0151] 19. Context 20.) twenty one.

[0154] 22. Namespace CodeLib

[0155] 23. function FuncName1() twenty four....

[0157] 25.end 26.

[0159] 27. function FuncName2() 28....

[0161] 29.end

[0162] 1)FSA name

[0163] The name of a finite state automaton is user-defined and written after the reserved word "FSA". Different finite state automata are distinguished by their FSA names, meaning that the name of a finite state automaton is globally unique, as shown in line 1 of the script.

[0164] 2) Parameter Items

[0165] The parameter items configure the current FSA operation, such as lines 2 to 4 in the script.

[0166] The conditions that the pre-aligned nodes of the current finite state automaton must satisfy are specified in the form of "#Entry EntryNode=[KV]", where "#Entry" is a reserved word, EntryNode can be customized, and KV is a key-value expression related to the attributes of the pre-aligned node;

[0167] The matching condition of the current finite state automaton is specified in the form of "#Parameter K=V", where "#Parameter" is a reserved word, and K=V mainly includes the following four types:

[0168] Order = Yes / No: Sets whether the node order in the finite state automaton is consistent with the word order. The default is Yes.

[0169] MaxLen = Yes / No: Sets the operating mode of the finite state automaton. If "Yes" is selected, only the operation corresponding to the longest matching path is executed; if "No" is selected, the operation corresponding to all matching paths is executed. The default is No.

[0170] Nearby = Yes / No: Sets whether two adjacent nodes in a finite state automaton are required to be adjacent in the grid. The default is No.

[0171] Bound = Sent / Clause / Group / Chunk: Sets the matching range of the finite state automaton, the default is Sent.

[0172] The function library to be called by the current FSA is declared in the form of "#Include ModuleName", where "#Include" is a reserved word and ModuleName is the name of the function library, which can be a function library defined in the current file or a function library defined in other files.

[0173] 3) Control Items

[0174] Control items are a major component of FSA, describing different contexts and their corresponding operations. They consist of two parts: Context and Operation, as shown in lines 6-15 of the script. Context describes the conditions of the FSA path and is composed of multiple Items; Operation describes the actions taken under those conditions.

[0175] When a Context is too complex or a certain part is highly reusable, it can be encapsulated in the form of "sub SubName()", where SubContext encapsulates that part, and then the name of SubContext "SubName" can be called within the Context, such as 12-20 in the script. SubContexts can be nested. When there is only one level of SubContext, its position in the script is not limited, and it can be placed before or after the main context. When a SubContext calls another SubContext, the callee should be placed first.

[0176] 4) Function library

[0177] Define the function library name in the form of "NameSpace Name" and define the function body in Lua script form. "NameSpace" is a reserved word, as shown in lines 22-29 of the script. When using the function library in the current FSA, it needs to be declared in the parameter item "#Include", as shown in line 3 of the script, and then the function in the function library is called in the Operation.

[0178] The API system in this framework will be introduced next.

[0179] 1. An API that calls external services and imports the service results into the mesh. This type of API implements the application of the parametric model in this framework, including:

[0180] 1.1 CallService(Sentence, ServiceName): This function calls the API of a service, typically a parametric model for structural analysis. Its main function is to structure the text to be analyzed and return the initial language structure data. The first parameter is the input to be passed to the service, and the second parameter is the service name, which needs to be configured in the configuration file. The return value is the language structure returned by the service.

[0181] 1.2 AddStructure(Sentence_JSon): Injects a JSon-formatted language structure into the mesh. During injection, it checks if the text content matches the one in the mesh. If they match, the structure is overlaid in the mesh; otherwise, a new analysis mesh is created.

[0182] The format definition of Sentence_JSon is as follows:

[0183] {"Type" = "","Units":[string / Tree],"POS":[string],"Groups":[{"HeadID":int,"Group":[{"Role":string,"SubID":int}]}]}, field descriptions are as follows:

[0184] Key names Type, Units, POS, Groups, HeadID, Group, Role, and SubID are reserved words.

[0185] 1) Type:

[0186] When "Sent" is selected, it indicates a sentence; Units represents unannotated text.

[0187] When it is "Word", it indicates word segmentation, and Units is a sequence of words.

[0188] When it is "Chunk", it indicates a chunk, and Units is a sequence of chunks.

[0189] When set to "Tree", it indicates a tree structure; Units represents a tree structure in bracket form.

[0190] The default value is "Chunk". The content of Type determines the cell type when Units are imported into the mesh.

[0191] 2) Units:

[0192] Based on the Type content, it can be a sentence, a word sequence, a tree structure, or a chunk sequence. When the Type is a chunk, each Unit in Units can be a chunk or a word sequence that makes up a chunk, such as: "Word / (KV KV)Word / (KV KV)...". The Words in a Unit are connected to form the content of the current chunk.

[0193] 3) POS:

[0194] The POS sequence corresponds to the attribute information of Units. When a language unit in the Units sequence does not need to be added to the grid, the corresponding element in the POS sequence can be set to "None".

[0195] 4) Groups:

[0196] This represents dependency structure information; for the representation of non-dependency structures, this item may not be included.

[0197] 5) HeadID:

[0198] The dependent node information is represented by the index (starting from 0) of the corresponding unit in Units.

[0199] 6) Group:

[0200] Represents dependent node information

[0201] 7) Role:

[0202] Role of Dependent Nodes

[0203] 8) SubID:

[0204] The value is the index (starting from 0) of the unit corresponding to the dependent node in Units.

[0205] The effect produced by this API is:

[0206] 1) Add language units from Units as grid units to the grid; 2) Add type attributes to the attributes of all grid units based on the value of Type; 3) Add language unit property information from POS to the attributes of the corresponding grid units; 4) Add the relationship information between language units from Groups to the attributes of grid units, the relationships between grid units, and the grid attributes.

[0207] 2. APIs for controlling internal variables of the mesh structure

[0208] This type of API enables the addition, retrieval, and testing of variables within the grid structure, improving the controllability and interpretability of language structure computation. Specifically, it includes the following APIs:

[0209] 2.1 Adding Internal Variables (Class API)

[0210]

[0211] 2.2 API for retrieving internal variables

[0212]

[0213] 2.3 Test Class API for Internal Variables

[0214]

[0215] 3. Application Data Table API

[0216] APIs for data table applications can be divided into three categories: functional APIs, retrieval APIs, and test APIs. See the table below:

[0217]

[0218] The functional APIs enable interaction between the grid and the data table, giving the framework the ability to apply explicit knowledge. Therefore, the functional APIs will be the focus of the following introduction.

[0219] 1) Segment(TableName)

[0220] The function of Segment is to segment text in a grid based on a data table and add attributes.

[0221] Its specific functions are:

[0222] ① In the current grid, perform a full split from left to right based on the data items in the data table.

[0223] ② Add the attributes from the data table to the corresponding grid cell attributes.

[0224] ③ Add "Type" and "ST" attributes to each segmented grid cell, with "Type" set to "Word" and "ST" set to "TableName".

[0225] 2) SetLexicon(TableName)

[0226] The `SetLexicon` function adds attributes of data items from a data table to grid cells, providing application attribute information for the grid cells. During grid computation, after calling the data table via `SetLexicon`, when a new grid cell is generated, it automatically checks whether the Word in the current grid cell exists in the data table, or whether the grid cell satisfies the key-value expression in the data table. If it does, the attributes are imported, that is, the attributes under the corresponding `SetLexicon` data item are added to the grid cell's attributes.

[0227] 3) Relate(TableName)

[0228] The Relate function imports relational data tables by calling the main table name, relation class name, and relation name.

[0229] When importing a relational data table into a grid using the Relate function, the data items in the data table correspond to grid cells, and the relationship between the data items in the main table and the data items in the secondary table corresponds to the cell relationship in the grid.

[0230] The functionality achieved by the Relate function can be broken down into the following steps:

[0231] ① Import the main table data items and their attributes that meet the Limit condition into the grid; and add the "ST-Unit" attribute to the grid with the value TableName, indicating that there are grid cells in the grid whose source is the data table TableName.

[0232] ② Use the Coll property to retrieve all child tables corresponding to data items in the main table.

[0233] ③ Add the data items from the sub-table that meet the Limit condition to the grid;

[0234] ④ Add grid cell relationships, U-type attributes, and R-type attributes to the grid cells. In the binary metadata table with TableName as the main table, record the relationships between each grid cell successfully added to the grid.<HeadUnit,SubUnit,Relation> HeadUnit and SubUnit have the following properties:

[0235]

[0236]

[0237] In addition to adding the above relational attributes to the grid cells, additional relational attributes with relational sources will be added. That is, ST (TableName in this case) will be used after Head / Sub of the binary relational attributes shown in the table above to distinguish the relational source. For example, USub can be represented as USubTableName, and USub-Relation can be represented as USubTableName-Relation, as shown below.

[0238] Attribute Extension USub = SubUnit USubTableName = SubUnit USub-Relation = SubUnit USubTableName-Relation = SubUnit RSub = Relation RSubTableName = Relation UHead = HeadUnit UHeadTableName = HeadUnit UHead-Relation = HeadUnit UHeadTableName-Relation = HeadUnit RHead = Relation RHeadTableName = Relation

[0239] ⑤ Add the attributes of the data items from the table to the attributes of the grid cell relationship;

[0240] ⑥ Add the attributes "URoot", "URootTableName", "RRoot", "RRootTableName", and "ST-Relation" to the grid to define the relationships between each grid cell successfully added to the grid.<HeadUnit,SubUnit,Relation> The attribute values ​​for the above attributes are: HeadUnit, HeadUnit, Relation, Relation, and TableName.

[0241] 4)Str=GetPrefix(TableName,String)

[0242] GetPrefix determines whether a string has a prefix of a data item in a data table. If so, it returns the longest matching string.

[0243] 5)Str=GetSuffix(TableName,String)

[0244] GetSuffix determines whether a string has a suffix from a data item in a table. If so, it returns the longest matching string.

[0245] 4. Applying the API of finite state automata

[0246] This API class enables interaction between the grid and finite state automata, allowing the framework to efficiently identify and process context. This API class includes RunFSA, GetFSANode, and GetParam, which are described in detail below.

[0247] 4.1 RunFSA(FSAName(,Param))

[0248] The function of this API is to execute a finite state automaton, that is, to match the Context and grid in the FSA script and execute the corresponding Operation. Here, FSAName is the name of the finite state automaton to be executed, and Param is the parameter to be passed. This parameter can be called in the finite state automaton script through the API GetParam.

[0249] Before execution, the finite state automaton script needs to be compiled into a finite state transition diagram for computation; this process is called grammar compilation. During compilation, each control item in the FSA script corresponds to one or more paths in the FSA. A control item contains a Context and an Operation. Each Node in the Context corresponds to an attribute test node in the FSA path, and the Operation corresponds to an operation node in the FSA path.

[0250] The following script:

[0251] FSA Example

[0252] #Include Lib

[0253] #ParameterNearby=Yes MaxLen=No Order=Yes

[0254] [(K=V&K=V+String)]

[0255] (K=V|K=V?String Unit:K=V)]

[0256] {Process1()}

[0257] [(K=VTab_C)

[0258] SubName]

[0259] {Process2()}

[0260] sub SubName

[0261] ([Tab_ATab_B])

[0262] NameSpace Lib

[0263] functionProcess1()

[0264] print("process1")

[0265] end

[0266] functionProcess2()

[0267] print("process2")

[0268] end

[0269] Compile into a finite state transition diagram as follows Figure 3 As shown.

[0270] After being compiled into a finite state transition graph, the nodes on the graph include entry nodes, exit nodes, attribute test nodes, and operation nodes. For example... Figure 3 As shown, there is an Enter entry node and an Exit exit node. The node before Exit that contains Process() is the Operation node, and the other nodes are attribute test nodes. The key-value expressions in the attribute test nodes are used to determine whether the corresponding mesh cells meet the conditions.

[0271] FSA paths define a context; each path represents a set of context constraints. When these constraints are satisfied, the corresponding operation node's content is executed. For example... Figure 3 As shown, in FSA, there are multiple possible paths from Enter to Exit, corresponding to multiple sets of context constraints.

[0272] When running RunFSA, the following steps are performed:

[0273] ① Obtain the corresponding pre-aligned unit based on the #Entry parameter in the FSA script.

[0274] ② Obtain the #Parameter parameter from the FSA script.

[0275] ③ Perform bidirectional matching starting from the pre-aligned node based on the parameters, and obtain the successfully matched path.

[0276] ④ Select which path to execute the operation based on the MaxLen parameter set in the script.

[0277] 4.2No=GetFSANode(-1) or No1,No2=GetFSANode("$Tag")

[0278] Executing a finite state automaton involves matching the Context with the mesh in the finite state automaton script. If the match is successful, the connection from the Enter node to the Exit node is established, and at this point, one or more paths can achieve this connection. In the Operation script, the mesh cells corresponding to the Nodes in the Context can be accessed. Therefore, the attribute nodes on the FSA path are numbered sequentially, and the mesh cells are accessed by referencing the node numbers. There are two numbering strategies:

[0279] 1. From left to right, the numbers are 0, 1, 2, ..., n-1;

[0280] 2. From right to left, they are -1, -2, ..., -n.

[0281] When a set of cells satisfying the complete FSA path is found in the mesh, the FSA path and the corresponding mesh cells for the nodes in the path are determined. At this point, the mesh cells can be accessed by referencing the node numbers. The specific function of this API is to obtain the FSA path number. When the parameter is -1, it indicates that the number of nodes in the current path is obtained; when the parameter is "$Tag", it indicates that the path number corresponding to the node with the "Tag" tag in the Context is obtained. No1 and No2 are the starting and ending path numbers of the Tag, respectively.

[0282] 4.3Str = GetParam(Key)

[0283] In FSA, the parameter value passed to the RunFSA function is read. For example, when RunFSA(FSAName,"Key=Value") is executed, GetParam("Key") can be run in the Operation of the FSAName script of the finite state automaton to get the return value "Value".

[0284] In one embodiment, when using this framework to perform language structure calculation, the following steps are included:

[0285] S1: Analyze the problem and determine the parameter model and explicit knowledge that contain implicit knowledge based on the problem;

[0286] S2: Based on the analysis results of step one, train the required parameter model and collect the required explicit knowledge;

[0287] S3: Expose the parameter model as a service and configure it in the configuration file of this framework, that is, write the IP address and port number of the service into the configuration file; optionally, perform batch preprocessing of the text to be analyzed through the parameter model, and use the initial language structure obtained after batch processing as the input of this framework;

[0288] S4: Enclose the collected explicit knowledge in a data table file;

[0289] S5: Write the master control script and the finite state automaton script using the Lua scripting language and the APIs provided by this framework;

[0290] S6: Index the data table file and the finite state automaton file using the indexing function of the executable program;

[0291] S7: Write the index paths of the data table file and the finite state automaton file to the configuration file;

[0292] S8: Execute the language structure calculation script using the execution function of the executable program. In this step, the master control script and the configuration file need to be passed in.

[0293] The following takes phrase recognition as an example to illustrate the specific implementation of this framework.

[0294] S1: Here, phrase recognition is specifically carried out to recognize scores, ordinal numbers, and game times in sports news texts and determine the required parameter models and explicit knowledge.

[0295] 1) Observe these three types of phrases. Among them, scores are in the forms of "3:2", "3-2", "3 to 2", "three to two"; ordinal numbers are in the forms of "the 1st", "the 23rd"; game times are in the forms of "the 34th minute of the first half", "the 34th minute of the first half", "34 minutes of the first half", "the 34th minute", "34 minutes".

[0296] 2) Determine the required parameter model. The target recognition phrases often exist within a chunk and do not cross chunks. Therefore, a chunk sequence annotation model needs to be introduced as an auxiliary constraint condition for phrase recognition.

[0297] 3) Determine the required explicit knowledge. In the recognition of score phrases, its important features are numbers and characters with the meaning of "ratio". Among them, numbers include Chinese numbers and Arabic numbers, and characters with the meaning of "ratio" include "ratio", "-", ":". Therefore, corresponding attribute descriptions need to be given to these words and characters; in the recognition of ordinal number phrases, its important features are numbers and the prefix "the" indicating order. Therefore, corresponding attribute descriptions need to be given to these words and characters; in the recognition of game time phrases, its important features are the time words "first half", "second half", "minute" in the game scenario and numbers. Therefore, corresponding attribute descriptions need to be given to these words and characters.

[0298] S2: Train the required parameter models and collect the required explicit knowledge according to the analysis results of S1. Specifically:

[0299] 1) Train the chunk sequence annotation model, including preparing training data, designing a training model, and training the model. Among them, the training data is chunk dependency annotation data, and the model is a Bert+CRF sequence annotation model.

[0300] 2) Collect the required explicit knowledge, including characters with the meaning of "ratio" such as "than", "-", ":", the prefix "the" indicating order, time words "first half", "second half", "minute", as well as Chinese numbers and Arabic numbers.

[0301] S3: Open the parameter model as a service and configure it in the configuration file of this framework, that is, write the IP address and port number of the service into the configuration file. Specifically:

[0302] 1) Open the chunk sequence annotation model as a service;

[0303] 2) Configure it in the configuration file "config.txt", and the configuration format is as follows:

[0304] Server:{"name":"chunk","IP":"127.0.0.0","Port":8080}

[0305] Among them, "Server", "name", "IP", and "Port" are all reserved words. "Server" indicates that this line is the configuration item of an external service, and "name", "IP", and "Port" respectively represent the name, IP address, and port number of the external service.

[0306] S4: Package the collected explicit knowledge in the data table file "merge.tab".

[0307] 1) Package the characters with the meaning of "ratio" such as "than", "-", ":", the prefix "the" indicating order, time words "first half", "second half", "minute" in a descriptive data table named "Merge_Dict". Among them, these language units with different meanings are used as individual data items, and their specific meanings are used as the attributes of the corresponding data items and are represented in the form of "K = V";

[0308] 2) Package all numbers, including Chinese numbers and Arabic numbers, in a descriptive data table named "Num_List". The specific content of the final data table file Merge.tab is as follows:

[0309] Table Merge_Dict

[0310] -Entry=Score

[0311] :Entry=Score

[0312] Entry = Score

[0313] Entry=Order

[0314] First Half Tag = Time

[0315] Second Half Tag = Time

[0316] Minute Entry = Minute

[0317] Table Num_List 0 1 2 3 4 5 6 7 8 9

[0328] zero one two three Four five six seven eight Nine ten

[0339] S5: Use the Lua scripting language and the APIs provided by this framework to write the overall control script and finite state automaton script.

[0340] 1) A finite state automaton script implements the feature description of the phrases for target recognition, and the further operations after satisfying the features. The phrase recognition task involves feature descriptions of score, ordinal number, and match time phrases, as well as further operations after satisfying the features. The feature and operation descriptions of these three types of phrases are represented by a set of Context-Operation pairs, and written in a finite state automaton script named Merge. Finally, this script is stored in the finite state automaton file Merge.fsa. The finite state automaton script named Merge includes parameter settings, pre-alignment node settings, and the writing of the three sets of Context-Operation pairs.

[0341] 1.1) Parameter Settings. The parameter settings determine the matching method and execution method. In this task, the matching method is set to: adjacent matching, ordered matching, and all matched nodes must be in the same block, i.e., Order = Yes, Nearby = Yes, Bound = Clause; the execution method is set to: only perform the maximum length matching, i.e., MaxLen = Yes.

[0342] 1.2) Pre-aligned node settings. Pre-aligned node settings determine the starting point of the match. For the recognition of scores, ordinal numbers, and match time phrases, the pre-aligned nodes are: words with score meaning, prefixes indicating order, and words indicating minutes, respectively. In the pre-aligned node parameter settings, key-value expressions are used, namely: EntryScore = [Entry = Score], EntryOrder = [Entry = Order], EntryTime = [Entry = Minute].

[0343] 1.3) Writing the Context-Operation pair for the score phrase. The identification features of the score phrase can be described as: one or more numbers, a word representing the score, and one or more numbers appearing consecutively. The corresponding Context is written as: +Num_List EntryScore+Num_List; Subsequent operations include: merging all matched nodes from beginning to end and adding them into a grid cell, and adding the attribute "Tag=MatchScore" to this grid cell.

[0344] 1.4) Writing Context-Operation pairs for ordinal numbers. The identification features of ordinal numbers can be described as: a prefix indicating order, one or more numbers appearing consecutively, and the corresponding Context is written as: EntryOrder+Num_List; subsequent operations include: merging all matched nodes from beginning to end and adding them into a grid cell, and adding the attribute "Tag=MatchOrder" to this grid cell.

[0345] 1.5) Writing of the Context-Operation pair for the game time phrase. The recognition features of the game time phrase can be described as follows: game time words regarding the first and second halves, "of", "the", one or more numbers, and the word indicating minutes appear consecutively. Among them, the game time words regarding the first and second halves can be present or absent, "of" and "the" can appear simultaneously, can appear only one of them, or can both be absent. The corresponding Context is written as:?Tag=Time?[of the]+Num_ListEntryTime; The subsequent operations include: merging all the matched nodes from start to end and adding them as a grid cell, and adding the attribute "Tag=MatchTime" to this grid cell.

[0346] 2) The overall control script Merge.lua realizes the control of the analysis process. In the example task, the specific process it describes is as follows:

[0347] 2.1) Use the API SetText to input the text to be analyzed into the grid. At this time, each character in the text to be analyzed corresponds to a grid cell, and the grid cell has the unit number attribute (Unit), type attribute (Type), starting column number (From) and ending column number (To) attribute, unit content attribute (Word), core content attribute (HeadWord), and clause number attribute (ClauseID). For example, when the input text to be analyzed is "In the 38th minute of the second half, Li Ming scored the 1st goal and successfully equalized the score to 1-1.", 30 grid cells are formed in the first row of the grid. The grid cell number corresponding to the first character "下" is "(0,1)", indicating that its grid position is the 0th column of the 1st row, and it has the following attributes: Unit=(0,1), Type=Char, Char=HZ, From=0, To=0, Word=下, HeadWord=下, ClauseID=0.

[0348] 2.2) Use the API CallService to call the chunk sequence annotation service to analyze the text to be analyzed and store the result in the variable ChunkRet. When the input text to be analyzed is "In the 38th minute of the second half, Li Ming scored the 1st goal and successfully equalized the score to 1-1.", the returned result is: "{"Type":"Chunk","Units":["下半场的38分钟",",","李明","攻入","第1个球",",","成功","将比分","扳平至","1-1","。"],

[0349] "POS":["NULL-MOD","w","NP-SBJ","VP-PRD","NP-OBJ","w","NULL-MOD","NULL-MOD","VP-PRD","NP-OBJ","w"],"ST":"Chunk"}"

[0350] 2.3) Use the API AddStructure to import the results returned by the chunk sequence labeling service into the grid. When importing the results from 2.2) into the grid, chunks in Units will be added to the grid as grid cells. Grid cells "," and "." already exist in the grid and will not be added again. The label property sequence corresponding to the chunk sequence, i.e., the content in "POS", will be added to the corresponding grid cell as the property attribute (POS). The content in "Type" and "ST" will be added to all corresponding grid cells as the type attribute (Type) and source attribute (ST). Other attributes, including the unit number attribute (Unit), starting column number (From), ending column number (To), word content attribute (Word), headword content attribute (HeadWord), and clause ID attribute (ClauseID), are added to the attributes of the newly added grid cells. For example, the grid cell corresponding to "Li Ming" has the following attributes: POS=NP-SBJ, Type=Chunk, ST=Chunk, Unit=(10,1), From=9, To=10, Word=Li Ming, HeadWord=Li Ming, ClauseID=1.

[0351] 2.4) The API Segment application table Merge_Dict is used to perform maximum-length word segmentation from left to right on the current text to be analyzed. When a data item in the Merge_Dict table exists in the current grid structure and does not span grid cells, the data item is segmented to form a new grid cell, and the corresponding attributes from the data table are added to the segmented grid cell. In the current example, after applying the Merge_Dict table, "second half", "minute", "number", and "-" are all segmented to form new grid cells, and the attributes Tag=Time, Entry=Minute, Entry=Order, and Entry=Score are added respectively.

[0352] 2.5) Use the API RunFSA to apply the finite state automaton Merge to merge the phrases identified by the target and add corresponding attributes to them. In this step, "38 minutes of the second half", "1st", and "1-1" are respectively matched with the Context in the finite state automaton Merge: "?Tag=Time?[the 38th minute of the second half", "1st", and "1-1" respectively.

[0353] +Num_List EntryTime", "EntryOrder+Num_List", "+Num_List EntryScore

[0354] "+Num_List" is added to the grid as a new grid unit during the Operation phase, and the attributes "Tag=MatchTime", "Tag=MatchOrder", and "Tag=MatchScore" are added respectively.

[0355] 2.6) Use the GetUnits API to retrieve grid cells, obtain grid cells with the attributes “Tag=MatchTime”, “Tag=MatchOrder”, and “Tag=MatchScore”, and output them. The output results are: “38 minutes of the second half”, “1”, and “1-1”.

[0356] S6: Use the indexing function of the executable program to index the data table file and the finite state automaton file;

[0357] 1) Index the data table file Merge.tab. Each data table file corresponds to a set of index data, generated by gpf.exe. The indexing command is: gpf.exe -table Merge.tab. / idx / , where:

[0358] gpf.exe: Indexing tool;

[0359] -table: Specifies the function type as an indexed data table;

[0360] Merge.tab: The filename of the data table, located in the same directory as gpf.exe;

[0361] . / idx / : Path where index data is stored.

[0362] After the data table file is indexed, one file corresponds to two index files. The index file name is the concatenation of the data table file name and "table", with the suffixes "idx" and "dat" respectively. The above indexing results in the following two files:

[0363] . / idx / Mergetable.idx

[0364] . / idx / Mergetable.dat

[0365] 2) Index the finite state automaton file Merge.fsa. Each finite state automaton file corresponds to a set of index data, generated by fsa.exe. The indexing command is:

[0366] fsa.exe -fsaMerge.fsa. / idx /

[0367] fsa.exe: Finite state automaton indexing tool;

[0368] -fsa: Used to specify the function type as an indexed finite state automaton script;

[0369] Merge.fsa: The filename of the finite state automaton script;

[0370] . / idx / : Path where index data is stored.

[0371] After the finite state automaton script file is indexed, one file corresponds to two index files. The index file name is the concatenation of the finite state automaton script file name and "fsa", with the suffix "idx".

[0372] The above example, after indexing "dat", will result in the following two files.

[0373] . / idx / Mergefsa.idx

[0374] . / idx / Mergefsa.dat

[0375] S7: Write the index paths of the data table file and the finite state automaton file into the configuration file. Apply the data table Merge.tab and the finite state automaton Merge.fsa, both written in JSON format in the configuration file, as follows:

[0376] Table:{"Path":". / idx / ","Data":["Mergetable"]}

[0377] FSA:{"Path":". / idx / ","Data":["Mergefsa"]}

[0378] The first line is the configuration of the data table. "Table", "Path", and "Data" are all reserved words. "Table" indicates that this line is a configuration item for the data table. "Path" and "Data" represent the path and file name of the data table index file, respectively. When there are multiple file names under data, the index file with the later position is called first.

[0379] The second line is the configuration of the finite state automaton script. "FSA", "Path", and "Data" are all reserved words. "FSA" indicates that this line is a configuration item for the finite state automaton script. "Path" and "Data" represent the path and filename of the finite state automaton script index file, respectively. When there are multiple filenames under data, the index file with the later position is called first.

[0380] S8: Use the executable program's execution function to execute the language structure calculation script. In this step, the master control script and configuration file need to be passed in.

[0381] The command to run the central control script locally is as follows:

[0382] gpf.exe-luaMerge.lua config.txt

[0383] gpf.exe: Run the tool;

[0384] -lua: Specifies the function type as running Lua scripts;

[0385] Merge.lua: The filename of the Lua script;

[0386] config.txt: The configuration file for local operation.

[0387] After executing the above indexing and execution commands in the command prompt, the following content will be output in the window: "38 minutes of the second half", "1st", and "1-1", thus realizing the recognition of the target phrase.

[0388] The analytical framework for language structure computation involved in this invention has the following characteristics:

[0389] 1) Knowledge and data collaboration: Compared with the integrated modeling method that combines knowledge and data, the collaborative method focuses on using knowledge and data to complete different tasks separately;

[0390] 2) Construct an expert system based on knowledge-based symbolic computation, which serves as the central control center for computation, scheduling other models to jointly complete the overall task;

[0391] 3) Decompose complex tasks into multiple sub-tasks and send each sub-task to the model for computation to fully leverage the capabilities of deep learning models;

[0392] 4) This framework supports language structure calculation using a multi-source feature-based ambiguity resolution strategy. That is, the expert system, which acts as the central control center, collects multi-source features and inputs them into the decision model. The decision model can use parameter calculation methods such as machine learning to make decisions using the features and return the decision results to the central control center, which then completes the overall task output.

[0393] Applying this framework allows for better utilization of knowledge, and the system's process control aligns with human cognitive processes. During implementation, different strategies can be employed depending on the complexity of the problem. For simple tasks, the expert system can complete the task independently and provide the final result; for complex tasks, the expert system can serve as a feature generation component, using parameter calculations to make decisions based on the features. For example, deep semantic analysis, with knowledge as the central control, invokes word meaning resolution and decision models, collaborating with these models through structural analysis, word meaning resolution, and relation resolution to address the aforementioned problems.

[0394] This framework can be used for research in lexical analysis, syntactic analysis, joint lexical-syntactic analysis, and semantic analysis. It can also be used for the practical development of natural language understanding applications. The applications of this framework in language research can be categorized as follows:

[0395] 1) Lexical Analysis: Compared to Indo-European languages, Chinese words lack formal boundaries and morphological markers. The morphology of Chinese words is presented through the regularity of character combinations. This framework can be applied to dynamically identify words and their internal structural types. Examples include: reduplicated word identification, separable word identification, and prefix / minus word identification.

[0396] 2) Syntactic Analysis: By leveraging phrase combination features, this framework identifies common phrases and their internal structures, such as time phrase identification and proper noun identification. By collaborating with a parametric computation model and incorporating structural knowledge, the accuracy of syntactic analysis is improved.

[0397] 3) Combined lexical and syntactic analysis: The internal structure of Chinese words is related to their external grammatical functions. The internal structure of verbs and their external context are analyzed in an integrated manner.

[0398] 4) Semantic analysis: Using a combination method and an intermediate structure, the framework's knowledge computation is used to collect features that support semantic analysis. These features are then fed into the decision model to complete semantic disambiguation.

[0399] The application of this framework in the practical development of natural language understanding applications can be divided into the following aspects:

[0400] 1) This framework can also be used as a deep semantic parsing tool. It can be customized according to the domain to solve fine-grained application problems and achieve semantic analysis and other tasks that meet the needs of the scenario.

[0401] 2) This framework can also be used for knowledge acquisition, such as acquiring various types of vocabulary collocation data; outputting the entity and event structure of sentences, thereby constructing domain knowledge graphs, etc.

[0402] This framework can be used for data pre-labeling. Based on the original input data, attribute labels or relational labels of words injected according to the analysis structure are used to pre-label the data.

Claims

1. An analysis device supporting natural language structure computation, characterized in that, include: Grid: A data structure used to store language structures, serving as a computing platform to support various language structures; Data table: Used to encapsulate unary and binary knowledge of symbol types, serving as an encapsulation component; Finite state automata: computational control units used to represent language context, working in conjunction with scripting languages ​​to control the functional structure of code. The API system revolves around grids, data tables, and finite state automata, and uses the Lua scripting language to write language-structured computational scripts. The executable program for the language structure calculation script has the following features: it has an indexing function, enabling it to index data tables and finite state automata; it has an execution function, enabling it to execute the language structure calculation script and perform language structure calculations; it has the function to execute the overall control script, writing the index file paths of data tables and finite state automata, as well as the IP addresses and interfaces of external services, into the configuration file and passing the configuration file to the executable program. The language structure is represented as G = (U, R), where: 1) U is a node in the graph, which is a finite set of language units, denoted as: U={tu,au}, tu∈Lex, Lex is the set of language unit strings, and au is the attribute of the language unit, which is a set of key-value pairs, denoted as: au={K=V}, K is the attribute name, and V is the attribute value; 2) R is a finite set of edges connecting two distinct nodes in U, that is, the set of relationships between two language units; since language units are often not dual, it is a directed edge, denoted as: R = {u i , u j ,r,ar},here:u i ∈U, u j ∈U, r∈RT, RT is the set of relation types; ar is the attribute of the language unit relation, which is the set of key-value pairs, denoted as: ar={K=V}, where K is the attribute name and V is the attribute value; The APIs include: (1) APIs that call external services and import service results into the grid, thereby realizing the scheduling of parameter calculation models; (2) APIs that control the internal variables of the grid structure, including the addition, testing and acquisition of internal variables, thereby realizing the controllability of the language structure; (3) APIs that apply data tables, that is, realize the interaction between data tables and the grid, thereby introducing explicit knowledge; (4) APIs that apply finite state automata, that is, realize the interaction between finite state automata and the grid, thereby expanding the functionality of this framework. The API for the application data table is divided into three categories: functional, retrieval, and test. The functional API enables interaction between the grid and the data table, giving the framework the ability to apply explicit knowledge, including: Segment(TableName) The function of Segment is to segment text in a grid based on a data table and add attributes; TableName represents the table name; SetLexicon(TableName) The SetLexicon function adds the attributes of data items in the data table to the grid cells, providing application attribute information to the grid cells; Relate(TableName) The Relate function imports relational data tables by calling the main table name, relation class name, and relation name. Str=GetPrefix(TableName, String) GetPrefix determines whether a string has a prefix of a data item in a data table. If so, it returns the longest matching string. Str=GetSuffix(TableName, String) GetSuffix determines whether a string has a suffix from a data item in a data table. If so, it returns the longest matching string. When importing a relational data table into a grid using the Relate function, the data items in the data table correspond to grid cells, and the relationships between the data items in the main table and the data items in the secondary table correspond to the cell relationships in the grid. The functionality achieved by the Relate function can be broken down into the following steps: ① Import the main table data items and their attributes that meet the Limit condition into the grid; and add the "ST-Unit" attribute to the grid with the value TableName; indicating that there are grid cells in the grid whose source is the data table TableName; ② Use the Coll property to retrieve all sub-tables corresponding to data items in the main table; ③ Add the data items from the sub-table that meet the Limit condition to the grid; ④ Add the grid cell relationships, U-type attributes, and R-type attributes to the grid cells; ⑤ Add the attributes of the data items from the table to the attributes of the grid cell relationship; ⑥ Add the attributes "URoot", "URootTableName", "RRoot", "RRootTableName", and "ST-Relation" to the grid to define the relationships between each grid cell successfully added to the grid.<HeadUnit, SubUnit,Relation> The attribute values ​​for the above attributes are: HeadUnit, HeadUnit, Relation, Relation, and TableName.

2. The analysis device supporting natural language structure calculation according to claim 1, characterized in that: The grid includes the following internal variables: the text to be analyzed, grid attributes, grid cells, and the relationships between grid cells; The grid cell includes the following internal variables: the stored language units, grid cell attributes, and the features and scores of the grid cell attributes; The relationships between the grid cells include the following internal variables: the attributes of the relationships between the grid cells, the characteristics of the relationships between the grid cells, and the scores.

3. The analysis device supporting natural language structure calculation according to claim 2, characterized in that: The language structure is represented as a directed graph with attribute information, including language units, relations, and attributes. It is loaded into the grid in the following way: language units are carried by grid units, and each language unit corresponds to one grid unit; the relations between language units are carried by the relations between grid units; the attributes of language units are carried by the attributes of grid units; and the attributes of the relations between language units are carried by the attributes of the relations between grid units.

4. The analysis device supporting natural language structure calculation according to claim 1, characterized in that: The data table is defined as a set of triples, i.e.: TableName={<Item, Attribute, Condition>},in: TableName: Table name; Item: A data item is either a string (Word) or a key-value expression (KV). When it is a Word, it corresponds to a language unit, which in grid computing corresponds to a grid cell. When it is a KV, the data item corresponds to all grid cells within the grid that satisfy the condition that KV is true. Attribute: The property of an Item is a collection of key-value pairs, i.e., {K=V}. {K=V} will be passed into the properties of grid cells or the relationship properties between cells; Condition: The condition for applying the Item is a set of key-value expressions, i.e., {KV}. If the grid cell corresponding to the current Item satisfies one of the key-value expressions in {KV}, then the next operation can be continued, including adding grid cells, attributes, or establishing relationships.

5. The analysis device supporting natural language structure calculation according to claim 1, characterized in that: The data tables are divided into two categories: descriptive data tables and relational data tables. The descriptive data table describes individual language units, providing the form and attribute knowledge of the language units. This knowledge is used to identify language units and set their attributes. Relational data tables describe two language units: a head language unit and another language unit that has a certain relationship with the head language unit. The binary relationship is encapsulated in multiple data tables, designed as a master table and multiple slave tables. The master table stores the list of head language units, and the slave tables store the list of language units that form a certain relationship with the head language units.

6. The analysis device supporting natural language structure calculation according to claim 1, characterized in that: The finite state automaton is defined as a set of four-tuples, i.e.: FSAName = {<Enter, Path, Operation,Exit>} where: FSAName: The name of the finite state automaton, globally unique; Enter: The entry node, a finite state automaton has a unique entry point; Path: The path corresponding to the finite state automaton, representing context information during structural computation; Operation: The operation node, i.e., the action to be executed when all contexts corresponding to the Path are successfully tested; Exit: The exit node, a finite state automaton has a unique exit point.

7. The analysis device supporting natural language structure calculation according to claim 1, characterized in that: A script can correspond to one or more finite state automata. Each script for a finite state automaton includes the FSA name, parameter items, control items, and a function library shared by multiple finite state automata. The FSA name is used to distinguish different finite state automata. The parameter item: configuration of the current FSA operation status; The control item describes different context situations and their corresponding operations, consisting of two parts: Context and Operation. Context describes the condition part of the FSA path and is composed of multiple Items; Operation describes the operation under the corresponding condition. The function library is defined with a name in the form of "NameSpace Name" and a function body in the form of Lua script, where "NameSpace" is a reserved word.

8. The analysis device supporting natural language structure calculation according to claim 1, characterized in that: The API that calls external services and imports the service results into the grid enables the application of the parametric model, including: (1) CallService(Sentence, ServiceName): Calls the API of the service. The service called is a parameterized model for structural analysis, which structures the text to be analyzed and returns the initial language structure data. The first parameter is the input to be passed to the service, and the second parameter is the service name, which needs to be configured in the configuration file. The return value is the language structure returned by the service. (2) AddStructure(Sentence_JSon): Injects the language structure in JSon format into the grid.

9. The analysis device supporting natural language structure calculation according to claim 1, characterized in that: The API for controlling the internal variables of the grid structure realizes the addition, acquisition and testing of the internal variables of the grid structure, which improves the controllability and interpretability of language structure calculation. It includes: (1) API for adding internal variables; (2) API for acquiring internal variables; and (3) API for testing internal variables.

10. The analysis device supporting natural language structure calculation according to claim 1, characterized in that: The API for the application of finite state automata enables interaction between the grid and the finite state automaton, thereby giving the framework the ability to efficiently identify and process context. This API includes RunFSA, GetFSANode, and GetParam. (1) RunFSA(FSAName(, Param)) The function of this API is to execute a finite state automaton, that is, to match the Context and grid in the FSA script and execute the corresponding Operation. Here, FSAName is the name of the finite state automaton to be executed, and Param is the parameter to be passed. This parameter can be called in the finite state automaton script through the API GetParam. (2) No=GetFSANode(-1) or No1,No2=GetFSANode("$Tag") Executing a finite state automaton involves matching the Context and the grid in the finite state automaton script. If the match is successful, the connection from the Enter node to the Exit node is completed, and one or more paths are connected. In the Operation script, the grid cells corresponding to the Nodes in the Context can be accessed. Therefore, the attribute nodes on the FSA path are numbered sequentially, and the grid cells are accessed by referencing the node numbers. (3) Str = GetParam(Key) Read the parameter values ​​passed to the RunFSA function in FSA.

Images