Financial Behavior Judicial Composite Knowledge Extraction Method and System Based on Transfer Learning

Abstract

The present invention discloses a method and system for extracting judicial composite knowledge of financial behavior based on transfer learning. Entities + relationships + events are extracted from the original data of judicial knowledge of financial behavior using behavioral semantic information as trigger words to obtain a model for extracting judicial knowledge of financial behavior. The legal information of financial judicial events is input into the model for extracting judicial knowledge of financial behavior, and a judicial composite knowledge graph of financial behavior corresponding to the factual determination information is output. The entity extraction, relationship extraction, and event extraction in the model for extracting knowledge of financial judicial events are integrated to establish a unified task model for extracting judicial knowledge of financial behavior, and pre-training of the unified task model for extracting judicial knowledge of financial behavior is carried out. Compared with the prior art, the present invention realizes unified modeling of entities + relationships + events for extracting composite knowledge in the field of judicial knowledge of financial behavior.

Description

Technical Field

[0001] The present invention relates to the technical fields of artificial intelligence and knowledge graphs, and particularly to a method and system for extracting compound knowledge of financial behaviors in the judicial field. Background Art

[0002] Knowledge extraction is an important part of natural language processing technology and an important basis for constructing domain knowledge graphs and other upper-layer applications. The scope of knowledge extraction covers entity extraction, relation extraction, and event extraction. Among them, entity extraction is to identify words (entities) with specific meanings in the text, mainly including personal names, place names, organization names, proper nouns, etc., and mark the words to be identified in the text sequence. Relation extraction is to find the relationships existing between the subject and the object in unstructured or semi-structured data and represent them as entity-relation triples, that is, (subject, relation, object). Event extraction is to identify and extract event information from the text describing event information and present it in a structured form, including the time of occurrence, place, participating roles, and related actions or state changes, and output a multi-tuple (trigger word, argument 1, argument 2, argument 3,...).

[0003] There are various documents in the field of financial behavior in the judiciary, such as judgment documents of relevant charges, prosecution opinions, indictments, confessions, etc. Upper-layer applications in the financial judiciary field, such as retrieval and fact determination, often require fine-grained and accurate knowledge extraction of these documents to construct features. However, in actual scenarios, the available labeled corpus for specific types of documents is extremely scarce, and it is difficult to build extraction models for them respectively and there are cost problems.

[0004] Chinese Invention Patent 201910145396.1, "A Method and System for Extracting Relationships in the Judicial Field Based on Neural Networks". This invention constructs a special dataset for the judicial field in an open neural network relationship and forms a feature set of charges in the judicial field. The core algorithm includes the following steps:

[0005] a. Construct a BERT text vector representation through word embedding and position embedding;

[0006] b. Construct a keyword-based text vector representation by means of TF-IDF plus word vectors;

[0007] c. Use vector concatenation and neural network methods to construct a relationship extraction model for the judicial field.

[0008] Chinese Invention Patent 202110349911.5, "A Method for Extracting Events in the Legal Field Based on a Pre-trained Model and a Convolutional Neural Network Algorithm". This invention patent performs text processing on the basis of an open legal corpus and constructs an extraction algorithm. The core model is as follows:

[0009] a. Cluster similar words for the word vectors of high-frequency words and key nouns, and manually screen keywords to construct templates for event extraction, namely trigger words and arguments;

[0010] b. Manually annotate the seed event corpus, obtain seed legal events from semi-structured legal text data using rules or template methods, and automatically annotate new corpus data using remote supervision learning and add it to the legal event knowledge base IE;

[0011] c. A legal event classification system based on the NEZHA pre-trained language model + DMCNN neural network model.

[0012] Chinese invention patent 202110693377.X, "A method for extracting main relationships in multiple relationships for legal texts". This invention patent constructs a relationship extraction application based on legal regulations corpus, and the core model is as follows:

[0013] a. Model entities and relationships according to business scenarios, expressing synonymous relationships, antonymous relationships, contract role relationships, etc.;

[0014] b. Manually annotate the corpus in the BIO form;

[0015] c. Model the extraction problem as a sequence labeling problem, use the encoding and decoding structure of bert-bilstm-crf to implement entity extraction, and use rules, etc. to generate relationships, and then generate a knowledge graph in the legal field.

[0016] Chinese invention patent 202110693377.X, "A method for constructing a dynamic legal event graph for the legal field", constructs the following algorithm model based on legal document data:

[0017] a. Define basic legal events, such as event names, legal scenarios, legal intentions, etc.;

[0018] b. Define relationships between dynamic events, such as causal relationships, temporal relationships, reversal relationships, conditional relationships, and superior-subordinate event relationships;

[0019] c. Use the bi-lstm-crf model to first extract entities, and then use rules and probability models to generate basic events and relationships between events.

[0020] The above-mentioned prior invention patents have the following problems for the upper-layer application requirements in the financial judicial field:

[0021] 1. It fails to use a unified model to process entity, relationship, and event extraction tasks in the same framework. Each task requires separate modeling, and there are high requirements for business and annotation.

[0022] 2. It has a high dependence on the annotation of specific types of corpora, such as indictment and opinion of prosecution, etc. In real-world scenarios, the quantity of corpora of the same type is often limited and difficult to obtain.

[0023] 3. It can only identify entities and relatively simple relationships, and cannot accurately correspond elements with the relevant persons of the behavior. It cannot meet the requirements in scenarios with multiple persons or the same person involved in multiple plots and multiple crimes, and has insufficient assistance for intelligent fact-finding.

[0024] 4. The corpus and method target broad legal fields and lack pertinence in the definition level of events and relationships for the financial behavior field, making it difficult to achieve high precision and high practicality. Summary of the Invention

[0025] To solve the technical problems mentioned in the above background, the present invention proposes a method and system for extracting composite knowledge of financial behavior based on transfer learning, which realizes the unified modeling of entities + relationships + events for extracting composite knowledge in the field of judicial knowledge of financial behavior.

[0026] The present invention is realized by the following technical solutions:

[0027] A method for extracting judicial composite knowledge of financial behavior based on transfer learning, the method comprising the following steps:

[0028] Step 1: Extract entities + relationships + events with behavior-type semantic information as trigger words according to the original data of judicial knowledge of financial behavior, and obtain a model for extracting judicial knowledge of financial behavior;

[0029] Step 2: Input the legal information of financial judicial events into the model for extracting judicial knowledge of financial behavior, and output a composite knowledge graph of financial behavior corresponding to the fact-finding information;

[0030] Step 3: Integrate entity extraction, relationship extraction and event extraction in the model for extracting knowledge of financial behavior judicial events, and establish a unified task model for extracting knowledge of financial behavior judicature;

[0031] Step 4: Perform pre-training on the unified task model for extracting knowledge of financial behavior judicature.

[0032] A system for extracting judicial composite knowledge of financial behavior based on transfer learning, the system comprising a knowledge extraction module, a composite knowledge graph generation module, a unified task model integration module for extracting knowledge and a pre-training module; wherein:

[0033] The knowledge extraction module extracts entities + relationships + events with behavior-type semantic information as trigger words according to the original data of judicial knowledge of financial behavior, and obtains a model for extracting judicial knowledge of financial behavior;

[0034] The composite knowledge graph generation module inputs the legal information of financial judicial events into the financial behavior judicial knowledge extraction model, and outputs a financial behavior judicial composite knowledge graph corresponding to the factual determination information;

[0035] The unified knowledge extraction task model integration module integrates entity extraction, relationship extraction, and event extraction in the financial behavior judicial event knowledge extraction model to establish a financial behavior judicial unified knowledge extraction task model:

[0036] The pre-training module performs pre-training on the financial behavior judicial unified knowledge extraction task model.

[0037] Compared with the prior art, the present invention has achieved the following beneficial technical effects:

[0038] (1) Aiming at the problems of few financial behavior judicial knowledge annotation corpora and difficult training, the model design innovatively applies a pre-training + domain pre-training + fine-tuning composite training architecture, and through the modeling of financial judicial domain knowledge, realizes the goal of few-shot learning and solves the problem of corpus sparsity in the training of the financial behavior judicial knowledge extraction model;

[0039] (2) At present, there is no alternative solution for the composite knowledge extraction (entity + relationship + event) in the field of financial behavior judicial knowledge.

[0040] (3) Compared with traditional knowledge extraction methods, it has the characteristics of supporting more knowledge types to be extracted and being simple for incremental development for different applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is a flowchart of a financial behavior judicial composite knowledge extraction method based on transfer learning according to the present invention;

[0042] Figure 2 is a flowchart of pre-training of a financial behavior judicial unified knowledge extraction task model;

[0043] Figure 3 is a schematic diagram of the fine-tuning process for downstream tasks;

[0044] Figure 4 is a module diagram of a financial behavior judicial composite knowledge extraction system based on transfer learning according to the present invention;

[0045] Figure 5 is an example diagram of the extraction results of entities + relationships + events in financial behavior judicature using behavior-based semantic information as trigger words;

[0046] Figure 6 is an example diagram of data annotation and preprocessing. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0047] The technical solutions in the present invention will be clearly and completely described below with reference to the accompanying drawings. Based on the embodiments in the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present invention.

[0048] As Figure 1 shown, it is a flowchart of a method for extracting judicial composite knowledge of financial behavior based on transfer learning in the present invention. The process specifically includes the following steps:

[0049] Step 1: Extract entities + relationships + events with behavioral semantic information as trigger words from the original financial behavior judicial knowledge data to obtain a financial behavior judicial knowledge extraction model; the specific processing is as follows:

[0050] First, the given financial judicial legal text also needs to be preprocessed, and the specific operations are as follows:

[0051] Filter out HTML tags, tables, and pictures in the original financial judicial original text;

[0052] Use the Simhash algorithm to perform deduplication on the financial judicial original text;

[0053] Perform structured processing on the document, filter out irrelevant charge documents, execute document structuring, cut the judgment documents into different paragraphs, and retain the fields of "found after hearing" and "held by this court" for system use;

[0054] After that, perform entity + relationship + event extraction on the financial judicial original data obtained after preprocessing. The event information constructed according to the four elements of criminal law includes subjective elements, objective elements, subject elements, and object elements, and output a multi-tuple of (trigger word - behavior, subjective element - argument n1, objective element - argument n2, subject element - argument n3, object element - argument n4). This multi-tuple is the financial behavior judicial event knowledge extraction model;

[0055] Since phenomena such as one person committing multiple crimes or multiple people committing multiple crimes often occur in the texts related to financial behavior, if separate entity extraction modeling is carried out, the elements cannot be effectively associated with multiple behavioral persons, and the relationship extraction mode can only process two arguments. Therefore, it is most appropriate to use entity + relationship + event to model financial behavior problems;

[0056] Step 2: Input the financial judicial event legal information into the financial behavior judicial knowledge extraction model, output a financial behavior judicial composite knowledge graph corresponding to the fact-finding information, and apply the financial behavior judicial composite knowledge graph to scenarios such as fact-finding;

[0057] Specifically, the financial behavior judicial event knowledge extraction model will take the behavior - type semantic information trigger word as the core and extract fields associated with this behavior - type semantic information, including at least fields such as the subject of the behavior - related person, the behavior amount involved in the behavior, the result, and the object, etc., and output a financial behavior judicial composite knowledge graph corresponding to the factual determination information; the behavior - type semantic information is obtained through pre - annotation.

[0058] Step 3: Integrate entity extraction, relationship extraction, and event extraction in the financial behavior judicial event knowledge extraction model to establish a unified financial behavior judicial knowledge extraction task model:

[0059] Use a unified machine reading comprehension framework to integrate entity extraction, relationship extraction, and event extraction in the financial behavior judicial event knowledge extraction model into a unified machine reading comprehension task. The specific method is to add a prompt field, that is, a question - description field, at the end of the input text, and inform the model in text form at the input end what SPAN needs to be output in the original text and what type of restrictions the SPAN has.

[0060] The prompt field follows the following design rules:

[0061] For entity extraction problems, the prompt is designed as: original text + {entity type}, telling the model to extract entities of a specific type in the original text.

[0062] For relationship extraction problems, the prompt suffix is designed to be carried out in two steps in sequence:

[0063] Given the relationship type, output all subject entities: original text + {entity type} + "subject" + {subject type}, telling the model what kind of subjects need to be extracted in the original text and what the type restrictions of the subjects are.

[0064] Given the subject entity, output all object entities: original text + "object" + {object type} + [o]{subject type}[a]{subject original text}[i]{subject start position}.

[0065] For event extraction problems, the prompt suffix design pattern is carried out in three steps in sequence:

[0066] Given the event type, extract all event trigger words: directly extract in the text segment: original text + {event type}

[0067] Given the event type and argument type restrictions, extract all argument texts: original text + {event type}-{argument type}

[0068] In the case of multiple events of the same type in the text, given a specific argument, extract its belonging trigger word: original text + {event type}[o]{argument type}[a]{argument text}[i]{start position}.

[0069] Step 4: Pre-train the financial behavior judicial unified knowledge extraction task model and fine-tune the downstream tasks;

[0070] In order to achieve the goal of small sample learning, the system encoder is implemented using a pre-trained Transformer model, such as bert-base or nezha-large, and the decoder is implemented using a global pointer. Figure 2 The figure below is a flowchart of the pre-training of the unified knowledge extraction task model for financial behavior justice. It includes the following 4 steps:

[0071] Step 4.1, build a pre-training extraction model (NeZha-large+globalpointer (Transformer+global pointe)) model;

[0072] Step 4.2, unified task conversion of general pre-training corpus: In the domain pre-training stage, the system collects a variety of corpora from different fields, sources, and tasks, and adds prompt suffixes and converts them into domain pre-training task corpora in the manner described in step 4.3. The domain corpora cover both legal professional and general categories.

[0073] Step 4.3, extracting legal information from PKU Law’s own data. The data mainly includes judgment documents and laws and regulations. Judgment documents mainly include plaintiffs, defendants, lawyers, litigation agents, compensation amounts, charges, sentences, etc.; laws and regulations include subjects, objects, assumptions, behavior patterns, legal consequences, etc.

[0074] like Figure 3 The following is a schematic diagram of the downstream task fine-tuning process. This process uses the GloablPointer decoder to improve decoding accuracy. The model structure and input and output flows are as follows:

[0075] Input the original text + prompt suffix into the model, which is equivalent to searching the target field in the original text under the given question and answer type restrictions, decomposing it into word vectors and position vectors, and inputting them into the multi-head attention weight network;

[0076] Semantic encoding is performed. In the encoding stage, the original text is concatenated with the prompt suffix using a unified MRC framework to form a phased input;

[0077] Decoding is performed. The gloabl pointer method is used in the decoding stage. The text SPAN is considered as an element of a matrix. If the (starting position in the matrix is 1, the SPAN represented by (starting position, ending position) in the original text is considered to be the extraction target.

[0078] like Figure 4As shown in the figure, it is a module diagram of a financial behavior judicial composite knowledge extraction system based on transfer learning according to the present invention. The system includes a knowledge extraction module 1, a composite knowledge graph generation module 2, a unified knowledge extraction task model integration module 3, and a pre-training module 4; wherein:

[0079] The knowledge extraction module 1 extracts entities + relationships + events with behavioral semantic information as trigger words based on the original financial behavior judicial knowledge data, and obtains a financial behavior judicial knowledge extraction model;

[0080] The composite knowledge graph generation module 2 inputs the legal information of financial judicial events into the financial behavior judicial knowledge extraction model, and outputs a financial behavior judicial composite knowledge graph corresponding to the fact-finding information;

[0081] The unified knowledge extraction task model integration module 3 integrates entity extraction, relationship extraction, and event extraction in the financial behavior judicial event knowledge extraction model to establish a financial behavior judicial unified knowledge extraction task model:

[0082] The pre-training module 4 performs pre-training on the financial behavior judicial unified knowledge extraction task model.

[0083] In summary, compared with the traditional single-type knowledge extraction model, a financial behavior judicial composite knowledge extraction method and system based on transfer learning according to the present invention innovatively proposes a unified extraction architecture for financial behavior judicial composite knowledge, realizing unified modeling of three types of knowledge: entities, relationships, and events.

[0084] The present invention first proposes a knowledge extraction model based on transfer learning technology, makes full use of the information contained in cross-domain and cross-project corpora, uses a framework based on unified machine reading comprehension, and realizes effective entity extraction, relationship extraction, and event extraction with only a small amount of labeled corpus training, achieving the transfer learning goal based on small samples, and obtaining good results in actual projects.

[0085] As shown in Table 1, it is an example of the extraction results of entities + relationships + events with behavioral semantic information as trigger words in financial behavior judicature.

[0086]

[0087]

[0088] This example is a text sample of a financial judicial event taking the crime of illegally absorbing public deposits as an example. The text of the financial judicial event constructed according to the four-element system of criminal law includes four elements: subjective element, objective element, subject element, and object element.

[0089] ①Objective elements: Based on the trigger word - act, extract the argument - result. For example, based on the trigger word, i.e., the semantic information of the act category "illegally absorbing public deposits", extract the argument - result - "disrupting the financial order" from the three pieces of text, such as "illegally absorbing funds in the forms of substitute planting (breeding), leased planting (breeding), joint planting (breeding), etc.", "without the true content of selling goods or providing services or not mainly aiming at selling goods or providing services, illegally absorbing funds in the forms of commodity repurchase, consignment sale, etc.", and "illegally absorbing funds in the forms of online lending, investment in shares, virtual currency trading, etc.", specifically including "the amount is more than 1 million yuan", "the number of targets is more than 150 people", and "the direct economic loss caused to depositors is more than 500,000 yuan";

[0090] ②Subject elements, extract the argument - natural person "natural persons who have reached the age of 16" and the argument - entity "entities, including financial institutions";

[0091] ③Objective elements, extract the argument - act object, indicating the source of the absorbed deposits, such as "the public", "deposits"

[0092] ④Subjective elements, extract the argument - culpability "intent" and the argument - purpose "for the purpose of illegal possession", etc.

[0093] As Figure 6 shown, it is an example of a judicial composite knowledge graph for financial acts. Among them, the pre - defined trigger words of the semantic information of the act category, such as "real - estate - style illegal absorption", and the given legal text of the financial judicial event, such as "publicly promoting investment in projects such as a certain area and a certain etc. can guarantee capital and return interest, and signing loan contracts with more than 100 investors.", extract the fields associated with this semantic information of the act category, such as: the main body of the act - related person - "the defendant so - and - so", the amount of the act involved - "more than 20 million yuan", the result - "disrupting the financial order", the object - "public deposits".

[0094] The general - domain information extraction corpus involved in the present invention mainly comes from news, academic literature, etc., and includes time, place, person, position, and relevant relationships and events.

Claims

1. A judicial composite knowledge extraction method for financial behavior based on transfer learning, characterized in that, The method includes the following steps: Step 1: Extract entities + relationships + events with behavioral semantic information as trigger words based on the original financial behavior judicial knowledge data to obtain a financial behavior judicial knowledge extraction model; the judicial knowledge extraction model includes extracting entities + relationships + events from the original financial judicial data, and according to the four elements of subjective elements, objective elements, subject elements, and object elements, outputting a multi-tuple of (trigger word - behavior, subjective element - argument n1, objective element - argument n2, subject element - argument n3, object element - argument n4), and this multi-tuple is the financial behavior judicial event knowledge extraction model; Step 2: Input the legal information of the financial judicial event into the financial behavior judicial knowledge extraction model, and output a financial behavior judicial composite knowledge graph corresponding to the fact-finding information; the financial behavior judicial composite knowledge graph further includes Taking the trigger word of the behavioral semantic information as the core, extracting fields associated with this behavioral semantic information, including at least relevant person subjects, the amount of the behavior involved in the behavior, results, and object fields, and outputting a financial behavior judicial composite knowledge graph corresponding to the fact-finding information; Step 3: Integrate entity extraction, relationship extraction, and event extraction in the financial behavior judicial event knowledge extraction model to establish a financial behavior judicial unified knowledge extraction task model; the financial behavior judicial unified knowledge extraction task model further includes integrating entity extraction, relationship extraction, and event extraction in the financial behavior judicial event knowledge extraction model into a unified machine reading comprehension task using a unified machine reading comprehension framework, and adding a prompt field at the end of the input text; among them, the prompt field follows the following design rules: For entity extraction problems, the prompt is designed as: original text + {entity type}, telling the model to extract entities of a specific type in the original text; For relationship extraction problems, the prompt suffix is designed to be carried out in two steps: Given the relationship type, output all subject entities: original text + {entity type} + "subject" + {subject type}, telling the model what kind of subjects need to be extracted in the original text and what the type limit of the subjects is; Given the subject entity, output all object entities: original text + "object" + {object type} + [o]{subject type}[a]{original subject text}[i]{starting position of the subject}; For event extraction problems, the prompt suffix design pattern is executed in three steps: Given the event type, extract all event trigger words: directly extract in the text segment: original text + {event type}; Given the event type and argument type limit, extract all argument texts: original text + {event type} - {argument type}; In the case of multiple events of the same type in the text, given a specific argument, extract the trigger word to which it belongs: original text + {event type} [o] {argument type} [a] {argument text} [i] {starting position}; Step 4: Perform pre-training on the financial behavior judicial unified knowledge extraction task model.

2. The financial behavior judicial composite knowledge extraction method based on transfer learning according to claim 1, wherein Model pre-training: It includes performing pre-training on the financial behavior judicial unified knowledge extraction task model and fine-tuning downstream tasks.

3. A judicial composite knowledge extraction system for financial behavior based on transfer learning, characterized in that, The system includes a knowledge extraction module, a composite knowledge graph generation module, a unified knowledge extraction task model integration module, and a pre-training module; among them: The knowledge extraction module extracts entities + relationships + events with behavioral semantic information as trigger words based on the original financial behavior judicial knowledge data, and obtains a financial behavior judicial knowledge extraction model; the financial behavior judicial knowledge extraction model includes extracting entities + relationships + events from the original financial judicial data, and according to the four elements of subjective elements, objective elements, subject elements and object elements, outputs a multi-tuple (trigger word - behavior, subjective element - argument n1, objective element - argument n2, subject element - argument n3, object element - argument n4), and this multi-tuple is the financial behavior judicial event knowledge extraction model; The composite knowledge graph generation module inputs the legal information of the financial judicial event into the financial behavior judicial knowledge extraction model, and outputs a financial behavior judicial composite knowledge graph corresponding to the fact-finding information; the financial behavior judicial composite knowledge graph further includes fields related to this behavioral semantic information triggered by the behavioral semantic information trigger word, at least including relevant person subjects, the amount of behavior involved in the behavior, results and object fields, and outputs a financial behavior judicial composite knowledge graph corresponding to the fact-finding information; The unified knowledge extraction task model integration module integrates the entity extraction, relationship extraction and event extraction in the financial behavior judicial event knowledge extraction model to establish a financial behavior judicial unified knowledge extraction task model. The financial behavior judicial unified knowledge extraction task model further includes integrating the entity extraction, relationship extraction and event extraction in the financial behavior judicial event knowledge extraction model into a unified machine reading comprehension task using a unified machine reading comprehension framework, and adding a prompt field at the end of the input text; among them, the prompt field follows the following design rules: For entity extraction problems, the prompt is designed as: original text + {entity type}, telling the model to extract entities of a specific type in the original text; For relationship extraction problems, the prompt suffix is designed to be carried out in two steps in sequence: Given the relationship type, output all subject entities: original text + {entity type} + "subject" + {subject type}, telling the model what kind of subjects need to be extracted in the original text and what the type limit of the subjects is; Given the subject entity, output all object entities: original text + "object" + {object type} + [o]{subject type}[a]{original subject text}[i]{starting position of the subject}; For event extraction problems, the prompt suffix design pattern is executed in three steps in sequence: Given the event type, extract all event trigger words: directly extract in the text segment: original text + {event type}; Given the event type and argument type limit, extract all argument texts: original text + {event type} - {argument type}; In the case of multiple events of the same type in the text, given a specific argument, extract the trigger word to which it belongs: original text + {event type} [o] {argument type} [a] {argument text} [i] {starting position}; The pre-training module performs pre-training on the financial behavior judicial unified knowledge extraction task model.

Images