Electronic file case element cognition method and system based on Transform framework

Through the multimodal and multi-task network model based on the Transformer framework, combined with text and visual encoder, the problem of low efficiency of multimodal and multi-task data processing in the cognition of electronic file case elements is solved, and high accuracy and high The identification of efficiency is suitable for deployment in multiple scenarios.

CN119007230AActive Publication Date: 2024-11-22SHANGHAI JIAOTONG UNIV

Patent Information

Application Number
CN202410998710.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-24
Publication Date
2024-11-22
Estimated Expiration
2044-07-24

AI Technical Summary

Technical Problem

In the understanding of electronic file case elements, it is difficult to effectively process multimodal and multitasking data in the prior art, resulting in low data management efficiency, low classification accuracy, large computing resources, and long delay.

Method used

Using a multimodal, multitasking network model based on the Transformer framework, combining text encoder and visual encoder, features are extracted through the Swin-Transformer architecture, and fine-tuned using a pre-trained BART model to build end-to-end electronic file case element cognition System, realizes robust identification and efficient analysis of multimodal data.

Benefits of technology

It improves the accuracy and robustness of identification of electronic file case elements, realizes parallel computing, reduces computing resource requirements, improves processing efficiency, and supports analysis of multiple data forms and case categories, which is suitable for deployment in different scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119007230A_ABST
    Figure CN119007230A_ABST
Patent Text Reader

Abstract

The invention provides an electronic file case element cognition method and system based on a Transform framework. The method comprises the steps that S1, simulation generation is conducted on multi-mode electronic file data; s2, building an electronic file case element cognitive network model; s3, determining a multi-task learning framework, setting a loss function used for optimizing network parameters, and outputting and using the loss function of each task and an optimization target for different tasks; s4, pre-training the electronic file case element cognitive network model; s5, performing fine tuning training by using the pre-trained model; and S6, exporting the electronic file case element cognitive network model after fine tuning training, and directly deploying the model into a service or integrating the model into a system for use. According to the invention, different key case elements can be extracted according to different types of data forms and causes, and the method has good robustness and high recognition accuracy.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision image processing and natural language processing text analysis, and in particular to a method and system for recognizing elements of electronic case files based on a Transformer framework. Background Art

[0002] With the rapid development of the economy and society, the number of electronic files is also increasing. In the process of sorting and collecting electronic files of different categories and causes, the realization of multi-modal and multi-task case element recognition methods for electronic files can improve the data management efficiency of electronic files, enhance the accuracy of document classification and summary review of electronic files, and reduce labor costs.

[0003] Multimodal and multitask element recognition of electronic files can help improve the efficiency of courts and other departments in organizing and classifying electronic files, and can reduce labor costs. Therefore, multimodal and multitask element recognition methods have become a hot topic in recent years. Among them, processing electronic file documents in image form and completing element recognition of electronic files is an important part of it. An important part of the multimodal element recognition network model is processing electronic file documents in image form. For electronic file data in image format, traditional OCR technology is generally used to extract text information in the image, and after integration, the text information is recognized and extracted.

[0004] Based on the traditional OCR technology, the common practice is to use OCR technology to extract text from the input electronic file data in the form of pictures. The text information extraction process includes preprocessing, text box annotation, text extraction, text correction and other steps. The output electronic file text stream data can be obtained from the input electronic file data in the form of pictures. The method of element recognition for text stream data mainly uses the knowledge graph method. According to the existing knowledge graph construction process, it includes: information extraction, knowledge fusion, knowledge processing, etc. As the main component of the knowledge graph, information extraction extracts structured information such as the relationship between entities from the data source, including the extraction of entities, relationships, and events.

[0005] Among the existing electronic case file element recognition methods, the patent with the authorization announcement number CN116011428B uses the reverse strategy to iteratively update the civil case information extraction model and determine the best reverse correction prediction topology structure to extract civil case information. This is significantly different from the electronic case file element recognition network model of the multi-task, multi-modal Transformer constructed by the present invention.

[0006] The patent with the authorization announcement number CN117496542B discloses a document information extraction method, device, electronic device and storage medium, in which the method uses the encoded feature vectors of text and vocabulary to construct a graph, and then uses a graph neural network to extract document information. This is significantly different from the electronic case file element cognitive network model of the multi-task, multi-modal Transformer constructed by the present invention.

[0007] The paper "Case Element Identification Method Based on Machine Reading Comprehension" published in "Computer Engineering and Design" in 2023 proposed a case element identification method based on machine reading comprehension, which uses case element label information to construct questions, refers to the type of case elements to be identified, and concatenates the questions and texts into the RoBERTa model for encoding; provides supplementary information for the interaction between context and questions from two directions through a bidirectional attention mechanism; and predicts whether the case elements are present through a fully connected network. This is significantly different from the multi-task, multi-modal Transformer electronic case element cognitive network model constructed by the present invention. Summary of the invention

[0008] In view of the defects in the prior art, the purpose of the present invention is to provide a method and system for recognizing elements of electronic case files based on the Transformer framework.

[0009] The electronic case file element recognition method based on the Transformer framework provided by the present invention includes:

[0010] Step S1: Use the electronic case file template for the entire trial business process of the court to simulate and generate multimodal electronic case file data, and the generated format includes electronic case file text, electronic case file scanned image, and electronic case file catalog structure tree;

[0011] Step S2: Building an electronic case file element cognitive network model based on a multimodal, multitask Transformer framework, which consists of two parts: a multimodal encoder based on the Transformer framework for extracting electronic case file features, and a multitask decoder based on the Transformer framework for completing downstream tasks;

[0012] The multimodal encoder includes a text encoder based on the Transformer architecture, which encodes the formatted text information in the multimodal input, and a visual encoder based on the Swin-Transformer architecture, which learns to recognize and extract the text content on the picture according to reading habits for feature encoding processing;

[0013] The multi-task decoder uses a pre-trained multi-language BART model for natural language processing, fine-tunes the understanding of the multi-language BART model on the encoded text based on the decoder, and uses decoders with different fine-tuning directions according to different tasks to output case elements and downstream task information in the decoded electronic file data;

[0014] Step S3: Determine a multi-task learning framework, set a loss function for optimizing network parameters, and use the loss function and optimization target of each task for different task outputs;

[0015] Step S4: using the generated multimodal electronic case file data, pre-training the electronic case file case element cognitive network model based on the multimodal, multi-task Transformer framework, training the model for multimodal text extraction capabilities, and training the model into a multimodal language model;

[0016] Step S5: Use the pre-trained model for fine-tuning training, construct multi-task prompt words for multi-task learning, and understand multi-tasks as multiple different JSON format data prediction problems for different downstream tasks. The training goal is to map each document to a database consistent with the feature extraction data format or to perform integrity matching and review on JSON format data;

[0017] Step S6: Export the fine-tuned and trained electronic case element cognitive network model based on the multimodal, multi-task Transformer framework, and directly deploy it as a service or integrate it into the system for use.

[0018] Preferably, in step S1, the electronic file generated by simulating the electronic file template for the entire court trial business process has a list of extracted element attributes that have been labeled, and is generated using templates according to a variety of different document categories and different categories of guilt. The same category of document data contains text data and document image data of different degrees of regularity; the simulated electronic file data set samples are divided into training set and test set in a ratio of 9:1.

[0019] Preferably, step S2 comprises:

[0020] Step S2.1: The Swin-Transformer-based visual encoder converts the input document information x into a set of embedding vectors. The encoder contains multiple Swin-Transformer blocks, each of which contains a windowed self-attention mechanism module, a shifted windowed multi-head self-attention mechanism, and a dimensionality reduction module.

[0021] The self-attention mechanism in the encoder takes the same input sequence, calculates the dot product between the query Q and the index K in the input, then scales it to control the gradient size, applies the Softmax function to get the attention weights, and finally multiplies these weights by the answer value V to get the output. The expression is:

[0022]

[0023] A feedforward network is also used in the encoder, expressed as:

[0024] FFN(X)=Relu(x·W1+b1)·W2+b2

[0025] Among them, W1 and W2 represent the learned weight matrices; b1 and b2 are bias vectors; Relu() is the activation function;

[0026] The calculation process of the residual connection in the encoder is to add the output of the feedforward network to the input, and perform layer normalization before adding. The expression is:

[0027] Output=LayerNorm(x+FFN(LayerNorm(x)))

[0028] Among them, LayerNorm() is the normalization function;

[0029] Step S2.2: For the embedding vector {z} obtained by the encoder, the decoder of the text data generates a token sequence (y i )*m i =1, where y i ∈R v is the one-hot vector of the ith token, v is the size of the token vocabulary, and m is a hyperparameter;

[0030] The decoder also uses the self-attention mechanism, which represents the self-attention calculation inside the decoder and is used to model its own sequence. The expression is:

[0031]

[0032] Where query Q′ is the output of the decoder, key K′ and value V′ are the output of the encoder;

[0033] The decoder also uses a feed-forward network and residual connections to pass information between layers.

[0034] Preferably, in step S5, a multi-task prompt word is constructed to perform multi-task learning. For the input document information x, a prompt function f is used. prompt (), which helps the model recall the information x' it learned during pre-training. The expression is:

[0035] x'=f prompt (x)

[0036] The function is a piece of natural language containing two empty positions, one for filling in the input and the other for generating the answer text.

[0037] Preferably, in step S5, the comprehensive performance is evaluated using the field-level F1 score and the accuracy based on the tree edit distance;

[0038] The field-level F1 score method checks that if the extracted field information is not in the true value, the field extraction is judged to have failed, otherwise the extraction is successful;

[0039] The tree-based edit distance method is used for any document represented as a tree. The edit distance refers to the minimum number of edit operations required to convert two strings from one to another. Edit operations include replacing, inserting or deleting a character. The calculation formula of the tree-based edit distance T is:

[0040]

[0041] Among them, gt, pr and φ represent the true value, prediction and empty tree respectively; EditDist() is the edit distance function; T a ,T b For two strings.

[0042] The electronic case file element recognition system based on the Transformer framework provided by the present invention includes:

[0043] Module M1: Use the electronic case file template that accompanies the entire trial business process of the court to simulate and generate multimodal electronic case file data. The generated formats include electronic case file text, electronic case file scanned images, and electronic case file catalog structure tree;

[0044] Module M2: Build an electronic case file element cognitive network model based on a multimodal, multi-task Transformer framework, which consists of two parts: a multimodal encoder based on the Transformer framework for extracting electronic case file features, and a multi-task decoder based on the Transformer framework for completing downstream tasks;

[0045] The multimodal encoder includes a text encoder based on the Transformer architecture, which encodes the formatted text information in the multimodal input, and a visual encoder based on the Swin-Transformer architecture, which learns to recognize and extract the text content on the picture according to reading habits for feature encoding processing;

[0046] The multi-task decoder uses a pre-trained multi-language BART model for natural language processing, fine-tunes the understanding of the multi-language BART model on the encoded text based on the decoder, and uses decoders with different fine-tuning directions according to different tasks to output case elements and downstream task information in the decoded electronic file data;

[0047] Module M3: Determine the multi-task learning framework, set the loss function for optimizing network parameters, and use the loss function and optimization objectives of each task for different task outputs;

[0048] Module M4: Use the generated multimodal electronic case file data to pre-train the electronic case file case element cognitive network model based on the multimodal, multi-task Transformer framework, train the model's multimodal text extraction capabilities, and train the model into a multimodal language model;

[0049] Module M5: Use the pre-trained model for fine-tuning training, build multi-task prompt words for multi-task learning, and understand multi-task as multiple different JSON format data prediction problems for different downstream tasks. The training goal is to map each document to a database consistent with the feature extraction data format or to perform integrity matching and review on JSON format data;

[0050] Module M6: Export the fine-tuned and trained electronic case element cognitive network model based on the multimodal and multi-task Transformer framework, and directly deploy it as a service or integrate it into the system for use.

[0051] Preferably, in the module M1, the electronic file generated by simulating the electronic file template for the entire trial process of the court has a list of extracted element attributes that have been labeled, and is generated using templates according to a variety of different document categories and different categories of guilt. The same category of document data contains text data and document image data of different degrees of regularity; the simulated electronic file data set samples are divided into training set and test set in a ratio of 9:1.

[0052] Preferably, the module M2 comprises:

[0053] Module M2.1: The Swin-Transformer-based visual encoder converts the input document information x into a set of embedding vectors. The encoder contains multiple Swin-Transformer blocks, which contain windowed self-attention mechanism modules, shifted windowed multi-head self-attention mechanisms, and dimensionality reduction modules.

[0054] The self-attention mechanism in the encoder takes the same input sequence, calculates the dot product between the query Q and the index K in the input, then scales it to control the gradient size, applies the Softmax function to get the attention weights, and finally multiplies these weights by the answer value V to get the output. The expression is:

[0055]

[0056] A feedforward network is also used in the encoder, expressed as:

[0057] FFN(X)=Relu(x·W1+b1)·W2+b2

[0058] Among them, W1 and W2 represent the learned weight matrices; b1 and b2 are bias vectors; Relu() is the activation function;

[0059] The calculation process of the residual connection in the encoder is to add the output of the feedforward network to the input, and perform layer normalization before adding. The expression is:

[0060] Output=LayerNorm(x+FFN(LayerNorm(x)))

[0061] Among them, LayerNorm)) is the normalization function;

[0062] Module M2.2: For the embedding vector {z} obtained by the encoder, the decoder of the text data generates a token sequence (y i )*m i =1, where y i ∈R v is the one-hot vector of the ith token, v is the size of the token vocabulary, and m is a hyperparameter;

[0063] The decoder also uses the self-attention mechanism, which represents the self-attention calculation inside the decoder and is used to model its own sequence. The expression is:

[0064]

[0065] Where query Q′ is the output of the decoder, key K′ and value V′ are the output of the encoder;

[0066] The decoder also uses a feed-forward network and residual connections to pass information between layers.

[0067] Preferably, in the module M5, a multi-task prompt word is constructed to perform multi-task learning. For the input document information x, a prompt function f is used. prompt (), which helps the model recall the information x' it learned during pre-training. The expression is:

[0068] x'=f prompt (x)

[0069] The function is a piece of natural language containing two empty positions, one for filling in the input and the other for generating the answer text.

[0070] Preferably, in the module M5, the comprehensive performance is evaluated using the field-level F1 score and the accuracy based on the tree edit distance;

[0071] The field-level F1 score method checks that if the extracted field information is not in the true value, the field extraction is judged to have failed, otherwise the extraction is successful;

[0072] The tree-based edit distance method is used for any document represented as a tree. The edit distance refers to the minimum number of edit operations required to convert two strings from one to another. Edit operations include replacing, inserting or deleting a character. The calculation formula of the tree-based edit distance T is:

[0073]

[0074] Among them, gt, pr and φ represent the true value, prediction and empty tree respectively; EditDist() is the edit distance function; T a ,T b For two strings.

[0075] Compared with the prior art, the present invention has the following beneficial effects:

[0076] (1) Based on deep learning technology, the present invention constructs an end-to-end electronic case file case element cognitive network model based on multimodal and multi-task Transformer, which can extract different key case elements according to different types of data forms and case causes, and has good robustness and high recognition accuracy. At the same time, this method can realize parallel computing to a certain extent and thus improve processing efficiency, requiring less computing resources in the reasoning stage and having lower latency;

[0077] (2) The model proposed in the present invention can accurately extract case elements of input multi-modal and multi-task electronic files, extract different key information of case elements according to different causes of action, and widely support the cause of action categories in various electronic files. The model has good analysis and recognition capabilities for multi-modal and multi-task data of electronic files and can cope with various complex scenarios. At the same time, the implementation method of the model is extremely easy to train and deploy;

[0078] (3) The model proposed in the present invention can simultaneously support the data input of electronic files in the form of text, pictures, etc., while ensuring that the key elements of the input data are retained, it can intelligently extract and identify important information, and integrate related information according to the internal logic of different cases. It can be widely deployed in different scenarios and devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0080] Figure 1 This is the overall framework diagram of the electronic case file element cognitive network model based on multi-modal and multi-task Transformer of the present invention;

[0081] Figure 2 This is a schematic diagram of the electronic file data stream processing framework based on deep learning proposed by the present invention;

[0082] Figure 3 This is a schematic diagram of the framework of the electronic case file element cognitive network model based on deep learning proposed by the present invention;

[0083] Figure 4 This is the Swin-Transformer block network structure diagram proposed by the present invention;

[0084] Figure 5 This is a network structure diagram of the text encoder module proposed in the present invention;

[0085] Figure 6 This is the network structure diagram of the task decoder module proposed in the present invention. DETAILED DESCRIPTION

[0086] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0087] Example 1

[0088] The present invention provides a method for recognizing case elements of electronic files based on Transformer, comprising:

[0089] Step S1: Use the electronic case file template of the court trial full business process to simulate the generation of electronic case files, including text format, image format, and directory structure tree, etc.

[0090] According to the electronic case file templates for the entire business process of court trials under different causes of action, simulation generation is performed based on the existing document data information of existing cases. The different attributes in the electronic case files are generated after correlation matching, ensuring that some attributes remain relevant according to the characteristics of the electronic case files, and that the attributes in the electronic case files meet the actual conditions and legal restrictions. For example, the sentencing of crimes in certain causes of action is within the scope of legal provisions, and the start time is earlier than the end time. The use of simulated electronic case files can ensure that the information of electronic documents has actual reliability while protecting the privacy of electronic case files. The simulated electronic case file data includes text and images, among which the image-based electronic case file data undergoes certain data enhancement operations such as slight blurring and rotation. The simulated electronic case file data is divided into training set and test set in a ratio of 9:1.

[0091] Step S2: Build an electronic case file element cognitive network model based on multimodal and multi-task Transformer, which includes four parts:

[0092] Model input, based on the Transformer, uses a teacher reinforcement scheme, a model training strategy that uses true values ​​as input instead of the output of the model at the previous time step. The model generates a token sequence given a prompt during the test phase, adding new special tokens as prompts for the case element recognition extraction task in our experiment.

[0093] Encoder, the Swin-Transformer-based visual encoder uses multiple dimension reduction modules and the window multi-head self-attention mechanism in the Swin-Transformer block to encode the document image. The visual encoder takes the input document image x∈R H×W×C Converted into a set of embedding vectors {z i |z i ∈R d ,1≤i≤n}, where n is the feature map size or the number of image blocks, and d is the dimension of the encoder's latent vector. The self-attention mechanism is used directly in the text encoder. The inputs all come from the same sequence. The dot product between the query Q and the index K in the input is calculated, and then scaled to control the gradient size. The softmax function is applied to obtain the attention weights, and finally these weights are multiplied by the answer value V to obtain the output. The encoder uses a feedforward network and a residual connection. The calculation process of the residual connection is to add the output of the feedforward network to the input, and both layers are normalized before adding.

[0094] Decoder: For {z} obtained by the encoder, the decoder of the text data will generate a token sequence (y i )*m i =1, where y i ∈R vis the unique hot vector of the ith token, v is the size of the token vocabulary, and m is a hyperparameter. The decoder also uses the self-attention mechanism, which means the self-attention calculation inside the decoder, which is the same as the self-attention in the encoder, but is used in the decoder to model its own sequence. In addition, the encoder-decoder attention mechanism is also implemented in the encoder, which means the attention calculation between the decoder and the encoder. Like the encoder, the decoder also uses a feedforward network and residual connections to pass information between layers. The decoder model weights are initialized using the weights of the publicly available pre-trained BART model.

[0095] Output transformation, the output token sequence is converted to the expected structured format. The JSON format is adopted, and two special tokens are added to indicate the start and end of the field to be extracted. If the structure of the output token sequence is wrong, we simply treat the field as missing. This method is implemented through regular expressions.

[0096] Step S3: Determine the multi-task learning framework and set the loss function for optimizing network parameters, including the following steps:

[0097] In pre-training, the model is trained to read all the text in the image in reading order. The goal is to minimize the cross entropy loss of next token prediction by jointly conditioning on the image and previous context.

[0098] In fine-tuning, the decoder is trained to generate a token sequence that can be converted into JSON representing the required output information, that is, the model is trained to map the electronic case file data into a structured information form consistent with the electronic case file database model. The loss function is the cross entropy loss of the decoder to generate the token sequence

[0099] Step S4: Using the training set in the multimodal electronic case data set generated in step S1, the electronic case case element cognitive network model based on multimodal and multi-task Transformer is trained to have multimodal text extraction capabilities. At this stage, the model is trained as a multimodal language model. The PyTorch deep learning framework is used to build and train the model;

[0100] Step S5: Use the model pre-trained in step S4 for fine-tuning training. For different downstream tasks, the multi-task is understood as multiple different JSON format data prediction problems. The training goal is to map each document to a database consistent with the feature extraction data format or to perform integrity matching and review on the JSON format data. Train a multi-modal, multi-task electronic case element recognition model. The comprehensive performance evaluation indicators used include field-level F1 scores and accuracy based on tree edit distance, including the following steps:

[0101] Step S5.1: The F1 score method checks whether the extracted field information is in the true value. If one character is missed, the score also considers that the field extraction fails, otherwise the extraction is successful.

[0102] Step S5.2: The tree-based edit distance method can be used for any document represented as a tree. The edit distance refers to the minimum number of edit operations required to convert two strings from one to another. Edit operations include replacing, inserting or deleting a character. The tree-based edit distance method can be used for any document represented as a tree. The edit distance refers to the minimum number of edit operations required to convert two strings from one to another. Edit operations include replacing, inserting or deleting a character. The tree-based edit distance d calculation formula is:

[0103]

[0104] Among them, gt, pr and Represent the true value, prediction and empty tree respectively, where the TED formula is as follows:

[0105]

[0106] Step S6: Export the electronic case file case element cognitive network model based on multimodal, multi-task Transformer that has been fine-tuned and trained in step S5, and use it by directly deploying it as a service or integrating it into other systems.

[0107] Example 2

[0108] The present invention also provides an electronic case file element recognition system based on the Transformer framework. The electronic case file element recognition system based on the Transformer framework can be implemented by executing the process steps of the electronic case file element recognition method based on the Transformer framework, that is, a person skilled in the art can understand the electronic case file element recognition method based on the Transformer framework as a preferred implementation of the electronic case file element recognition system based on the Transformer framework.

[0109] like Figure 1 to Figure 6 According to the electronic case file element recognition system based on the Transformer framework provided by the present invention, it includes:

[0110] Module M1: Use the electronic case file template that accompanies the entire trial business process of the court to simulate and generate multimodal electronic case file data. The generated formats include electronic case file text, electronic case file scanned images, and electronic case file catalog structure tree;

[0111] Module M2: Build an electronic case file element cognitive network model based on a multimodal, multi-task Transformer framework, which consists of two parts: a multimodal encoder based on the Transformer framework for extracting electronic case file features, and a multi-task decoder based on the Transformer framework for completing downstream tasks;

[0112] The multimodal encoder includes a text encoder based on the Transformer architecture, which encodes the formatted text information in the multimodal input, and a visual encoder based on the Swin-Transformer architecture, which learns to recognize and extract the text content on the picture according to reading habits for feature encoding processing;

[0113] The multi-task decoder uses a pre-trained multi-language BART model for natural language processing, fine-tunes the understanding of the multi-language BART model on the encoded text based on the decoder, and uses decoders with different fine-tuning directions according to different tasks to output case elements and downstream task information in the decoded electronic file data;

[0114] Module M3: Determine the multi-task learning framework, set the loss function for optimizing network parameters, and use the loss function and optimization objectives of each task for different task outputs;

[0115] Module M4: Use the generated multimodal electronic case file data to pre-train the electronic case file case element cognitive network model based on the multimodal, multi-task Transformer framework, train the model's multimodal text extraction capabilities, and train the model into a multimodal language model;

[0116] Module M5: Use the pre-trained model for fine-tuning training, build multi-task prompt words for multi-task learning, and understand multi-task as multiple different JSON format data prediction problems for different downstream tasks. The training goal is to map each document to a database consistent with the feature extraction data format or to perform integrity matching and review on JSON format data;

[0117] Module M6: Export the fine-tuned and trained electronic case element cognitive network model based on the multimodal and multi-task Transformer framework, and directly deploy it as a service or integrate it into the system for use.

[0118] In the module M1, the electronic file generated by the electronic file template that accompanies the entire trial process of the court has a list of extracted element attributes that have been labeled, and is generated using templates according to a variety of different document categories and different categories of guilt. The same category of document data contains text data and document image data of different degrees of regularity; the simulated electronic file data set samples are divided into training set and test set in a ratio of 9:1.

[0119] The module M2 comprises:

[0120] Module M2.1: The Swin-Transformer-based visual encoder converts the input document information x into a set of embedding vectors. The encoder contains multiple Swin-Transformer blocks, which contain windowed self-attention mechanism modules, shifted windowed multi-head self-attention mechanisms, and dimensionality reduction modules.

[0121] The self-attention mechanism in the encoder takes the same input sequence, calculates the dot product between the query Q and the index K in the input, then scales it to control the gradient size, applies the Softmax function to get the attention weights, and finally multiplies these weights by the answer value V to get the output. The expression is:

[0122]

[0123] A feedforward network is also used in the encoder, expressed as:

[0124] FFN(X)=Relu(x·W1+b1)·W2+b2

[0125] Among them, W1 and W2 represent the learned weight matrices; b1 and b2 are bias vectors; Relu() is the activation function;

[0126] The calculation process of the residual connection in the encoder is to add the output of the feedforward network to the input, and perform layer normalization before adding. The expression is:

[0127] Output=LayerNorm(x+FFN(LayerNorm(x)))

[0128] Among them, LayerNorm({is the normalization function;

[0129] Module M2.2: For the embedding vector {z{ obtained by the encoder, the decoder of the text data generates a token sequence (y i )*m i =1, where y i ∈R v is the one-hot vector of the ith token, v is the size of the token vocabulary, and m is a hyperparameter;

[0130] The decoder also uses the self-attention mechanism, which represents the self-attention calculation inside the decoder and is used to model its own sequence. The expression is:

[0131]

[0132] Where query Q′ is the output of the decoder, key K′ and value V′ are the output of the encoder;

[0133] The decoder also uses a feed-forward network and residual connections to pass information between layers.

[0134] In the module M5, a multi-task prompt word is constructed for multi-task learning. For the input document information x, a prompt function f is used prompt (), which helps the model recall the information x' it learned during pre-training. The expression is:

[0135] x'=f prompt (x)

[0136] The function is a piece of natural language containing two empty positions, one for filling in the input and the other for generating the answer text.

[0137] In the module M5, the comprehensive performance is evaluated using the field-level F1 score and the accuracy based on the tree edit distance;

[0138] The field-level F1 score method checks that if the extracted field information is not in the true value, the field extraction is judged to have failed, otherwise the extraction is successful;

[0139] The tree-based edit distance method is used for any document represented as a tree. The edit distance refers to the minimum number of edit operations required to convert two strings from one to another. Edit operations include replacing, inserting or deleting a character. The calculation formula of the tree-based edit distance T is:

[0140]

[0141] Among them, gt, pr and φ represent the true value, prediction and empty tree respectively; EditDist() is the edit distance function; T a ,T b For two strings.

[0142] Those skilled in the art know that, in addition to implementing the system, device and its various modules provided by the present invention in a purely computer-readable program code, it is entirely possible to implement the same program in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers and embedded microcontrollers by logically programming the method steps. Therefore, the system, device and its various modules provided by the present invention can be considered as a hardware component, and the modules included therein for implementing various programs can also be considered as structures within the hardware component; the modules for implementing various functions can also be considered as both software programs for implementing the method and structures within the hardware component.

[0143] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A method for recognizing elements of electronic case files based on the Transformer framework, characterized in that: include: Step S1: Use the electronic case file template for the entire trial business process of the court to simulate and generate multimodal electronic case file data, and the generated format includes electronic case file text, electronic case file scanned image, and electronic case file catalog structure tree; Step S2: Building an electronic case file element cognitive network model based on a multimodal, multitask Transformer framework, which consists of two parts: a multimodal encoder based on the Transformer framework for extracting electronic case file features, and a multitask decoder based on the Transformer framework for completing downstream tasks; The multimodal encoder includes a text encoder based on the Transformer architecture, which encodes the formatted text information in the multimodal input, and a visual encoder based on the Swin-Transformer architecture, which learns to recognize and extract the text content on the picture according to reading habits for feature encoding processing; The multi-task decoder uses a pre-trained multi-language BART model for natural language processing, fine-tunes the understanding of the multi-language BART model on the encoded text based on the decoder, and uses decoders with different fine-tuning directions according to different tasks to output case elements and downstream task information in the decoded electronic file data; Step S3: Determine a multi-task learning framework, set a loss function for optimizing network parameters, and use the loss function and optimization target of each task for different task outputs; Step S4: using the generated multimodal electronic case file data, pre-training the electronic case file case element cognitive network model based on the multimodal, multi-task Transformer framework, training the model for multimodal text extraction capabilities, and training the model into a multimodal language model; Step S5: Use the pre-trained model for fine-tuning training, construct multi-task prompt words for multi-task learning, and understand multi-tasks as multiple different JSON format data prediction problems for different downstream tasks. The training goal is to map each document to a database consistent with the feature extraction data format or to perform integrity matching and review on JSON format data; Step S6: Export the fine-tuned and trained electronic case case element cognitive network model based on the multimodal, multi-task Transformer framework, and directly deploy it as a service or integrate it into the system for use.

2. The electronic case file element recognition method based on the Transformer framework according to claim 1 is characterized in that: In step S1, the electronic file generated by simulating the electronic file template for the entire trial process of the court has a list of extracted element attributes that have been labeled, and is generated using templates according to a variety of different document categories and different categories of guilt. The same category of document data contains text data and document image data of different degrees of regularity; the simulated electronic file data set samples are divided into a training set and a test set in a ratio of 9:

1.

3. The electronic case file element recognition method based on the Transformer framework according to claim 1 is characterized in that: The step S2 comprises: Step S2.1: The Swin-Transformer-based visual encoder converts the input document information x into a set of embedding vectors. The encoder contains multiple Swin-Transformer blocks, each of which contains a windowed self-attention mechanism module, a shifted windowed multi-head self-attention mechanism, and a dimensionality reduction module. The self-attention mechanism in the encoder takes the same input sequence, calculates the dot product between the query Q and the index K in the input, then scales it to control the gradient size, applies the Softmax function to get the attention weights, and finally multiplies these weights by the answer value V to get the output. The expression is: A feedforward network is also used in the encoder, expressed as: FFN(X)=Relu(x·W1+b1)·W2+b2 Among them, W1 and W2 represent the learned weight matrices; b1 and b2 are bias vectors; Relu() is the activation function; The calculation process of the residual connection in the encoder is to add the output of the feedforward network to the input, and perform layer normalization before adding. The expression is: Output=LayerNorm(x+FFN(LayerNorm(x))) Among them, LayerNorm() is the normalization function; Step S2.2: For the embedding vector {z} obtained by the encoder, the decoder of the text data generates a token sequence (y i )*m i =1, where y i ∈R v is the one-hot vector of the ith token, v is the size of the token vocabulary, and m is a hyperparameter; The decoder also uses the self-attention mechanism, which represents the self-attention calculation inside the decoder and is used to model its own sequence. The expression is: Where query Q′ is the output of the decoder, key K′ and value V′ are the output of the encoder; The decoder also uses a feed-forward network and residual connections to pass information between layers.

4. The electronic case file element recognition method based on the Transformer framework according to claim 1 is characterized in that: In step S5, a multi-task prompt word is constructed to perform multi-task learning. For the input document information x, a prompt function f is used prompt (), which helps the model recall the information x' it learned during pre-training. The expression is: x’=f prompt (x) The function is a piece of natural language containing two empty positions, one for filling in the input position and the other for generating the answer text.

5. The electronic case file element recognition method based on the Transformer framework according to claim 1 is characterized in that: In step S5, the comprehensive performance is evaluated using the field-level F1 score and the accuracy based on the tree edit distance; The field-level F1 score method checks that if the extracted field information is not in the true value, the field extraction is judged to have failed, otherwise the extraction is successful; The tree-based edit distance method is used for any document represented as a tree. The edit distance refers to the minimum number of edit operations required to convert two strings from one to another. Edit operations include replacing, inserting or deleting a character. The calculation formula of the tree-based edit distance T is: Among them, gt, pr and φ represent the true value, prediction and empty tree respectively; EditDist() is the edit distance function; T a ,T b For two strings.

6. An electronic case file element recognition system based on the Transformer framework, characterized in that: include: Module M1: Use the electronic case file template that accompanies the entire trial business process of the court to simulate and generate multimodal electronic case file data. The generated formats include electronic case file text, electronic case file scanned images, and electronic case file catalog structure tree; Module M2: Build an electronic case file element cognitive network model based on a multimodal, multi-task Transformer framework, which consists of two parts: a multimodal encoder based on the Transformer framework for extracting electronic case file features, and a multi-task decoder based on the Transformer framework for completing downstream tasks; The multimodal encoder includes a text encoder based on the Transformer architecture, which encodes the formatted text information in the multimodal input, and a visual encoder based on the Swin-Transformer architecture, which learns to recognize and extract the text content on the picture according to reading habits for feature encoding processing; The multi-task decoder uses a pre-trained multi-language BART model for natural language processing, fine-tunes the understanding of the multi-language BART model on the encoded text based on the decoder, and uses decoders with different fine-tuning directions according to different tasks to output case elements and downstream task information in the decoded electronic file data; Module M3: Determine the multi-task learning framework, set the loss function for optimizing network parameters, and use the loss function and optimization objectives of each task for different task outputs; Module M4: Use the generated multimodal electronic case file data to pre-train the electronic case file case element cognitive network model based on the multimodal, multi-task Transformer framework, train the model's multimodal text extraction capabilities, and train the model into a multimodal language model; Module M5: Use the pre-trained model for fine-tuning training, build multi-task prompt words for multi-task learning, and understand multi-task as multiple different JSON format data prediction problems for different downstream tasks. The training goal is to map each document to a database consistent with the feature extraction data format or to perform integrity matching and review on JSON format data; Module M6: Export the fine-tuned and trained electronic case element cognitive network model based on the multimodal and multi-task Transformer framework, and directly deploy it as a service or integrate it into the system for use.

7. The electronic case file element recognition system based on the Transformer framework according to claim 6 is characterized in that: In the module M1, the electronic file generated by the electronic file template that accompanies the entire trial process of the court has a list of extracted element attributes that have been labeled, and is generated using templates according to a variety of different document categories and different categories of guilt. The same category of document data contains text data and document image data of different degrees of regularity; the simulated electronic file data set samples are divided into training set and test set in a ratio of 9:

1.

8. The electronic case file element recognition system based on the Transformer framework according to claim 6 is characterized in that: The module M2 comprises: Module M2.1: The Swin-Transformer-based visual encoder converts the input document information x into a set of embedding vectors. The encoder contains multiple Swin-Transformer blocks, which contain windowed self-attention mechanism modules, shifted windowed multi-head self-attention mechanisms, and dimensionality reduction modules. The self-attention mechanism in the encoder takes the same input sequence, calculates the dot product between the query Q and the index K in the input, then scales it to control the gradient size, applies the Softmax function to get the attention weights, and finally multiplies these weights by the answer value V to get the output. The expression is: A feedforward network is also used in the encoder, expressed as: FFN(X)=Relu(x·W1+b1)·W2+b2 Among them, W1 and W2 represent the learned weight matrices; b1 and b2 are bias vectors; Relu() is the activation function; The calculation process of the residual connection in the encoder is to add the output of the feedforward network to the input, and perform layer normalization before adding. The expression is: Output=LayerNorm(x+FFN(LayerNorm(x))) Among them, LayerNorm() is the normalization function; Module M2.2: For the embedding vector {z} obtained by the encoder, the decoder of the text data generates a token sequence (y i )*m i =1, where y i ∈R v is the one-hot vector of the ith token, v is the size of the token vocabulary, and m is a hyperparameter; The decoder also uses the self-attention mechanism, which represents the self-attention calculation inside the decoder and is used to model its own sequence. The expression is: Where query Q′ is the output of the decoder, key K′ and value V′ are the output of the encoder; The decoder also uses a feed-forward network and residual connections to pass information between layers.

9. The electronic case file element recognition system based on the Transformer framework according to claim 6 is characterized in that: In the module M5, a multi-task prompt word is constructed for multi-task learning. For the input document information x, a prompt function f is used prompt (), which helps the model recall the information x' it learned during pre-training. The expression is: x’=f prompt (x) The function is a piece of natural language containing two empty positions, one for filling in the input position and the other for generating the answer text.

10. The electronic case file element recognition system based on the Transformer framework according to claim 6 is characterized in that: In the module M5, the comprehensive performance is evaluated using the field-level F1 score and the accuracy based on the tree edit distance; The field-level F1 score method checks that if the extracted field information is not in the true value, the field extraction is judged to have failed, otherwise the extraction is successful; The tree-based edit distance method is used for any document represented as a tree. The edit distance refers to the minimum number of edit operations required to convert two strings from one to another. Edit operations include replacing, inserting or deleting a character. The calculation formula of the tree-based edit distance T is: Among them, gt, pr and φ represent the true value, prediction and empty tree respectively; EditDist() is the edit distance function; T a ,T b For two strings.

Citation Information

Patent Citations

  • Methods, devices, media, and equipment for information extraction and judgment prediction in civil cases

    CN116011428B

  • Document information extraction method, device, electronic device and storage medium

    CN117496542B

  • Court electronic file oriented case information automatic extraction method

    CN112417880A

  • Judicial domain depth event extraction method fusing multi-task and multi-label learning

    CN114580428A

  • Document processor and document processing method

    JP1999250041A

Cited By

  • Metadata intelligent extraction-based rapid electronic file composition method and device

    CN121166616A

  • A method and device for quickly grouping electronic files based on metadata intelligent extraction

    CN121166616B

  • Document cataloguing method and related device

    CN121564486A

  • Case cause model training and extraction analysis method and system

    CN121581042A