A Method and System for Recognizing Case Elements in Electronic Case Files Based on the Transformer Framework

Abstract

This invention provides a method and system for recognizing case elements in electronic case files based on the Transformer framework, comprising: Step S1: Simulating and generating multimodal electronic case file data; Step S2: Constructing an electronic case file case element recognition network model; Step S3: Determining a multi-task learning framework, setting a loss function for optimizing network parameters, and using the loss function and optimization objective for different task outputs; Step S4: Pre-training the electronic case file case element recognition network model; Step S5: Fine-tuning the pre-trained model; Step S6: Exporting the fine-tuned electronic case file case element recognition network model, which can be directly deployed as a service or integrated into a system. This invention can extract different key case elements according to different data formats and case causes, exhibiting good robustness and high recognition accuracy.

Description

Technical Field

[0001] This invention relates to the fields of computer vision image processing and natural language processing text analysis technology, specifically to a method and system for recognizing electronic case file elements based on the Transformer framework. Background Technology

[0002] With the rapid development of the economy and society, the number of electronic case files is also constantly increasing. In the process of organizing and collecting electronic case files of different categories and causes of action, implementing a multimodal and multi-task method for recognizing case elements in electronic case files can improve the efficiency of data management, enhance the accuracy of document classification and summarization review, and reduce labor costs.

[0003] Multimodal and multi-task element recognition of electronic case files helps improve the efficiency of courts and other departments in organizing and classifying electronic case files, and can reduce labor costs. Therefore, multimodal and multi-task element recognition methods have become a research hotspot in recent years, among which methods for processing image-based electronic case file documents to complete element recognition of electronic case files are an important part. An important link in the multimodal element recognition network model is processing image-based electronic case file documents. For image-formatted electronic case file data, traditional OCR technology is generally used to extract text information from the image, and then the text information is integrated for element recognition extraction.

[0004] The method of extracting text information based on traditional OCR technology and then performing element recognition extraction commonly involves using OCR technology to extract text from input image-based electronic case file data. The text extraction process includes preprocessing, text box annotation, text extraction, and text correction, resulting in output electronic case file text stream data from input image-formatted electronic case file data. For element recognition of text stream data, the primary method is to use knowledge graphs. The construction process of existing knowledge graphs includes information extraction, knowledge fusion, and knowledge processing. Information extraction, as a major component of knowledge graphs, extracts structured information such as entities and relationships between entities from the data source, including the extraction of entities, relationships, and events.

[0005] Among existing methods for recognizing case elements in electronic case files, the patent with authorization announcement number CN116011428B uses a reverse evolution strategy to iteratively update the civil case information extraction model and determine the optimal reverse evolution correction prediction topology structure to extract civil case information. This is significantly different from the multi-task, multi-modal Transformer electronic case file case element recognition network model constructed in this invention.

[0006] Patent CN117496542B discloses a method, apparatus, electronic device, and storage medium for extracting document information. This method uses encoded feature vectors of text and vocabulary to construct a graph, and then utilizes a graph neural network to extract document information. This is significantly different from the multi-task, multi-modal Transformer-based electronic case file element recognition network model constructed in this invention.

[0007] The 2023 paper "A Case Element Recognition Method Based on Machine Reading Comprehension," published in *Computer Engineering and Design*, proposed a case element recognition method based on machine reading comprehension. This method uses case element label information to construct a question, indicating the type of case element to be identified. The question is then concatenated with the text and input into a RoBERTa model for encoding. A bidirectional attention mechanism provides supplementary information on the interaction between the context and the question from two directions. Finally, a fully connected network predicts whether the case element is present. This method differs significantly from the multi-task, multi-modal Transformer electronic case file case element recognition network model constructed in this invention. Summary of the Invention

[0008] In view of the deficiencies in the prior art, the purpose of this invention is to provide a method and system for recognizing electronic case file elements based on the Transformer framework.

[0009] The method for recognizing electronic case file elements based on the Transformer framework provided by the present invention includes:

[0010] Step S1: Use the court trial full business process electronic case file template to simulate and generate multimodal electronic case file data. The generated format includes electronic case file text, electronic case file scanned image, and electronic case file categorization structure tree.

[0011] Step S2: Build a cognitive network model of electronic case file elements based on the multimodal, multi-task Transformer framework. It consists of two parts: a multimodal encoder based on the Transformer framework for extracting features from electronic case files, and a multi-task decoder based on the Transformer framework for completing downstream tasks.

[0012] The multimodal encoder includes a text encoder based on the Transformer architecture, which encodes formatted text information in the multimodal input, and a visual encoder based on the Swin-Transformer architecture, which learns to recognize and extract text content from images according to reading habits and performs feature encoding.

[0013] The multi-task decoder uses a pre-trained multilingual BART model for natural language processing. Based on this decoder, the understanding of the encoded text of the multilingual BART model is fine-tuned. Depending on the task, the decoder with different fine-tuning directions outputs case elements and downstream task information in the decoded electronic case file data.

[0014] Step S3: Determine the multi-task learning framework, set the loss function for optimizing network parameters, and use the loss function and optimization objective of each task for different task outputs;

[0015] Step S4: Using the generated multimodal electronic case file data, pre-train the electronic case file case element cognitive network model based on the multimodal, multi-task Transformer framework, train the model's ability to extract multimodal text, and train the model into a multimodal language model.

[0016] Step S5: Fine-tune the pre-trained model, construct multi-task prompts for multi-task learning, and treat the multi-task as multiple different JSON format data prediction problems for different downstream tasks. The training objective is to map each document to a database consistent with the feature extraction data format or to perform integrity matching and evaluation on the JSON format data.

[0017] Step S6: Export the fine-tuned and trained electronic case file case element cognitive network model based on the multimodal, multi-task Transformer framework, and use it by directly deploying it as a service or integrating it into the system.

[0018] Preferably, in step S1, the electronic case file generated by the template of the electronic case file accompanying the court trial process has a list of extracted element attributes that has been labeled, and is generated using templates according to various document categories and different crime categories. Under the same category of document data, there are text data and document image data with different degrees of regularity; the sample of the simulated electronic case file dataset is divided into training set and test set in a 9:1 ratio.

[0019] Preferably, step S2 includes:

[0020] Step S2.1: The Swin-Transformer-based visual encoder converts the input document information x into a set of embedding vectors. The encoder contains multiple Swin-Transformer blocks, which include a windowed self-attention mechanism module, a shifted window multi-head self-attention mechanism, and a dimensionality reduction module.

[0021] The self-attention mechanism in the encoder takes inputs from the same sequence. It calculates the dot product between the query Q and the index K in the input, then scales the gradient to control the magnitude. A softmax function is applied to obtain the attention weights, and finally, these weights are multiplied by the response value V to obtain the output. The expression is as follows:

[0022]

[0023] The encoder also uses a feedforward network, expressed as:

[0024] FFN(X)=Relu(x·W1+b1)·W2+b2

[0025] Where W1 and W2 represent the learned weight matrices; b1 and b2 are bias vectors; and ReLU() is the activation function.

[0026] The calculation process for residual connections in the encoder involves adding the output of the feedforward network to the input, and performing layer normalization before addition. The expression is as follows:

[0027] Output=LayerNorm(x+FFN(LayerNorm(x)))

[0028] LayerNorm() is the normalization function;

[0029] Step S2.2: For the embedding vector {z} obtained by the encoder, the decoder of the text data generates a token sequence (y). i )*m i =1, where y i ∈R v is the one-hot vector of the i-th token, v is the size of the token vocabulary, and m is a hyperparameter;

[0030] The decoder also uses a self-attention mechanism, which represents the self-attention computation within the decoder, used for modeling its own sequence. The expression is:

[0031]

[0032] Where query Q′ is the output of the decoder, and key K′ and value V′ are the outputs of the encoder;

[0033] The decoder also uses feedforward networks and residual connections to pass information between layers.

[0034] Preferably, in step S5, multi-task prompt words are constructed for multi-task learning. For the input document information x, a prompt function f is used. prompt (), helps the model recall the information x' it learned during pre-training, expressed as:

[0035] x'=f prompt (x)

[0036] This function is a natural language string containing two empty positions: one for filling in the input and the other for generating the answer text.

[0037] Preferably, in step S5, a comprehensive performance evaluation is performed using field-level F1 scores and tree-edit distance-based accuracy;

[0038] The field-level F1 score method checks if the extracted field information is not in the true value; if so, the field extraction is considered to have failed, otherwise it is considered to have succeeded.

[0039] The tree-based edit distance method is applicable to any document represented as a tree. Edit distance refers to the minimum number of editing operations required to transform one string into another. Editing operations include replacing, inserting, or deleting a character. The formula for calculating the tree-based edit distance T is:

[0040]

[0041] Where gt, pr, and φ represent the ground truth, prediction, and empty tree, respectively; EditDist() is the edit distance function; T a ,T b It consists of two strings.

[0042] The electronic case file element recognition system based on the Transformer framework provided by the present invention includes:

[0043] Module M1: Uses the electronic case file template accompanying the entire business process of court trial to simulate and generate multimodal electronic case file data. The generated formats include electronic case file text, electronic case file scanned images, and electronic case file categorization structure tree.

[0044] Module M2: Constructs a cognitive network model for electronic case file elements based on a multimodal, multi-task Transformer framework. It consists of two parts: a multimodal encoder based on the Transformer framework for extracting features from electronic case files, and a multi-task decoder based on the Transformer framework for completing downstream tasks.

[0045] The multimodal encoder includes a text encoder based on the Transformer architecture, which encodes formatted text information in the multimodal input, and a visual encoder based on the Swin-Transformer architecture, which learns to recognize and extract text content from images according to reading habits and performs feature encoding.

[0046] The multi-task decoder uses a pre-trained multilingual BART model for natural language processing. Based on this decoder, the understanding of the encoded text of the multilingual BART model is fine-tuned. Depending on the task, the decoder with different fine-tuning directions outputs case elements and downstream task information in the decoded electronic case file data.

[0047] Module M3: Defines the multi-task learning framework, sets the loss function for optimizing network parameters, and uses the loss function and optimization objective for different task outputs.

[0048] Module M4: Using the generated multimodal electronic case file data, pre-train the electronic case file case element cognitive network model based on the multimodal, multi-task Transformer framework, train the model's ability to extract multimodal text, and train the model into a multimodal language model;

[0049] Module M5: Uses the pre-trained model for fine-tuning training, constructs multi-task prompts for multi-task learning, and treats different downstream tasks as multiple JSON format data prediction problems. The training objective is to map each document to a database consistent with the feature extraction data format or to perform integrity matching and evaluation on the JSON format data.

[0050] Module M6: Exports a finely tuned, trained electronic case file element cognitive network model based on a multimodal, multi-task Transformer framework, which can be deployed directly as a service or integrated into a system for use.

[0051] Preferably, in module M1, the electronic case file generated by the template of the electronic case file accompanying the court trial process has a list of extracted element attributes that has been labeled, and is generated using templates according to various document categories and different crime categories. Under the same category of document data, there are text data and document image data with different degrees of regularity; the simulated electronic case file dataset sample is divided into training set and test set in a 9:1 ratio.

[0052] Preferably, the module M2 includes:

[0053] Module M2.1: The Swin-Transformer-based visual encoder transforms the input document information x into a set of embedding vectors. This encoder contains multiple Swin-Transformer blocks, which include a windowed self-attention mechanism module, a shifted window multi-head self-attention mechanism, and a dimensionality reduction module.

[0054] The self-attention mechanism in the encoder takes inputs from the same sequence. It calculates the dot product between the query Q and the index K in the input, then scales the gradient to control the magnitude. A softmax function is applied to obtain the attention weights, and finally, these weights are multiplied by the response value V to obtain the output. The expression is as follows:

[0055]

[0056] The encoder also uses a feedforward network, expressed as:

[0057] FFN(X)=Relu(x·W1+b1)·W2+b2

[0058] Where W1 and W2 represent the learned weight matrices; b1 and b2 are bias vectors; and ReLU() is the activation function.

[0059] The calculation process for residual connections in the encoder involves adding the output of the feedforward network to the input, and performing layer normalization before addition. The expression is as follows:

[0060] Output=LayerNorm(x+FFN(LayerNorm(x)))

[0061] Where LayerNorm is the normalization function;

[0062] Module M2.2: For the embedding vector {z} obtained by the encoder, the decoder of the text data generates a token sequence (y). i )*m i =1, where y i ∈R v is the one-hot vector of the i-th token, v is the size of the token vocabulary, and m is a hyperparameter;

[0063] The decoder also uses a self-attention mechanism, which represents the self-attention computation within the decoder, used for modeling its own sequence. The expression is:

[0064]

[0065] Where query Q′ is the output of the decoder, and key K′ and value V′ are the outputs of the encoder;

[0066] The decoder also uses feedforward networks and residual connections to pass information between layers.

[0067] Preferably, in module M5, multi-task prompt words are constructed for multi-task learning. For the input document information x, a prompt function f is used. prompt (), helps the model recall the information x' it learned during pre-training, expressed as:

[0068] x'=f prompt (x)

[0069] This function is a natural language string containing two empty positions: one for filling in the input and the other for generating the answer text.

[0070] Preferably, in module M5, the overall performance is evaluated using field-level F1 scores and tree-edit distance-based accuracy;

[0071] The field-level F1 score method checks if the extracted field information is not in the true value; if so, the field extraction is considered to have failed, otherwise it is considered to have succeeded.

[0072] The tree-based edit distance method is applicable to any document represented as a tree. Edit distance refers to the minimum number of editing operations required to transform one string into another. Editing operations include replacing, inserting, or deleting a character. The formula for calculating the tree-based edit distance T is:

[0073]

[0074] Where gt, pr, and φ represent the ground truth, prediction, and empty tree, respectively; EditDist() is the edit distance function; T a ,T b It consists of two strings.

[0075] Compared with the prior art, the present invention has the following beneficial effects:

[0076] (1) Based on deep learning technology, this invention constructs an end-to-end electronic case file case element cognitive network model based on multimodal and multi-task Transformer, which can extract different key case elements according to different types of data formats and case causes, and has good robustness and high recognition accuracy. At the same time, this method can achieve parallel computing to a certain extent, thereby improving processing efficiency, requiring less computing resources in the reasoning stage, and having lower latency.

[0077] (2) The model proposed in this invention can accurately extract case elements from multimodal and multi-task electronic case files, extract key information of different case elements according to different causes of action, and widely support case categories in various electronic case files. The model has good analytical and identification capabilities for multimodal and multi-task data in electronic case files and can cope with various complex scenarios. At the same time, the implementation method of the model is very easy to train and deploy.

[0078] (3) The model proposed in this invention can simultaneously support the input of electronic case files containing text, images and other forms. While ensuring that the key element information in the input data is retained, it can intelligently extract and identify important information and integrate related information according to the internal logic of different case types. It can be widely deployed in different scenarios and devices. Attached Figure Description

[0079] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0080] Figure 1 This is a diagram illustrating the overall framework of the electronic case file case element cognitive network model based on multimodal, multi-task Transformer of this invention.

[0081] Figure 2 This is a schematic diagram of the deep learning-based electronic case file data stream processing framework proposed in this invention.

[0082] Figure 3 This is a schematic diagram of the cognitive network model framework for electronic case file elements based on deep learning proposed in this invention.

[0083] Figure 4 This is a diagram of the Swin-Transformer block network structure proposed in this invention;

[0084] Figure 5 This is a network structure diagram of the text encoder module proposed in this invention;

[0085] Figure 6 This is a network structure diagram of the task decoder module proposed in this invention. Detailed Implementation

[0086] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0087] Example 1

[0088] This invention provides a method for recognizing case elements in electronic case files based on Transformer, including:

[0089] Step S1: Use the court's full-process electronic case file template to simulate the generation of electronic case files. The generated formats include text format, image format, and a category structure tree, etc.

[0090] Based on templates for electronic case files accompanying the entire trial process under different causes of action, and using existing case documents, simulated data generation is performed. Different attributes within the electronic case file are matched for relevance, ensuring that some attributes maintain correlation according to the characteristics of the electronic case file. Furthermore, the attributes in the electronic case file conform to actual conditions and legal restrictions; for example, the sentencing for certain crimes is within the legally prescribed range, and the start time is earlier than the end time. Using simulated electronic case files ensures the actual reliability of the information in the electronic documents while protecting the privacy of the electronic case file. The simulated electronic case file data includes text and images, with image data undergoing data augmentation operations such as slight blurring and rotation. The simulated electronic case file data is divided into training and test sets at a 9:1 ratio.

[0091] Step S2: Construct a cognitive network model for electronic case file elements based on a multimodal, multi-task Transformer, comprising four parts:

[0092] For model input, a teacher reinforcement scheme was used based on Transformer, a model training strategy that uses real values ​​as input instead of the model's output at the previous time step. During the testing phase, the model generates a token sequence given a cue, adding new special tokens as cue for the case element recognition extraction task in our experiment.

[0093] The encoder, a Swin-Transformer-based visual encoder, encodes document images using multiple dimensionality reduction modules and a windowed multi-head self-attention mechanism within the Swin-Transformer block. The visual encoder takes the input document image x∈R... H×W×C Convert to a set of embedding vectors {z i |z i ∈R d , 1 ≤ i ≤ n, where n is the feature map size or the number of image patches, and d is the dimension of the encoder's latent vector. The text encoder directly uses a self-attention mechanism. The inputs all come from the same sequence. The attention weights are obtained by calculating the dot product between the query Q and the index K in the input, then scaling to control the gradient magnitude, applying the softmax function, and finally multiplying these weights by the response value V to obtain the output. The encoder uses a feedforward network connected to a residual network. The residual connection is calculated by adding the output of the feedforward network to the input, and layer normalization is performed before addition.

[0094] For the {z} obtained from the encoder, the decoder of the text data will generate a token sequence (y). i )*m i =1, where y i ∈R vis the one-hot vector of the i-th token, v is the size of the token vocabulary, and m is a hyperparameter. The decoder also uses a self-attention mechanism, representing the self-attention computation within the decoder, which is the same as the self-attention in the encoder, but used in the decoder for modeling its own sequence. Furthermore, the encoder implements an encoder-decoder attention mechanism, representing the attention computation between the decoder and encoder. Similar to the encoder, the decoder also uses feedforward networks and residual connections to pass information between layers. The decoder model weights are initialized using the weights of a publicly available pre-trained BART model.

[0095] The output token sequence is transformed into the desired structured format. JSON format is used, with two special tokens added to indicate the start and end of the field to be extracted. If the output token sequence has an incorrect structure, the field is simply treated as missing; this is achieved using regular expressions.

[0096] Step S3: Determine the multi-task learning framework and set the loss function for optimizing network parameters, including the following steps:

[0097] In pre-training, the model is trained to read all the text in an image in reading order. The goal is to minimize the cross-entropy loss for predicting the next token by co-conditioning the image and the preceding context.

[0098] In the fine-tuning, the decoder is trained to generate token sequences that can be converted into JSON representing the desired output information. This means training the model to map electronic case file data into a structured information format consistent with the database schema of electronic case file case elements. The loss function is the cross-entropy loss of the decoder for generating the token sequences.

[0099] Step S4: Using the training set from the multimodal electronic case file dataset generated in Step S1, train the multimodal, multi-task Transformer-based electronic case file case element cognitive network model to improve its multimodal text extraction capabilities. At this stage, the model is trained as a multimodal language model. The PyTorch deep learning framework is used to build and train the model.

[0100] Step S5: Fine-tune the model pre-trained in Step S4. For different downstream tasks, treat the multi-task approach as multiple prediction problems using different JSON format data. The training objective is to map each document to a database consistent with the data format extracted from the features, or to perform integrity matching and evaluation on the JSON format data. Train a multimodal, multi-task electronic case file feature recognition model. The comprehensive performance evaluation metrics used include field-level F1 scores and tree-edit distance-based accuracy, including the following steps.

[0101] Step S5.1: The F1 score method checks whether the extracted field information is in the true value. If a character is missed, the score also considers the field extraction to have failed; otherwise, the extraction is considered successful.

[0102] Step S5.2: The tree-based edit distance method can be used for any document represented as a tree. Edit distance refers to the minimum number of editing operations required to transform one string into another. Editing operations include replacing, inserting, or deleting a character. The tree-based edit distance d is calculated as follows:

[0103]

[0104] Among them, gt, pr and These represent the truth value, prediction, and empty tree, respectively, where the TED formula is as follows:

[0105]

[0106] Step S6: Export the electronic case file case element cognitive network model based on multimodal, multi-task Transformer, which was fine-tuned and trained in Step S5, and use it by directly deploying it as a service or integrating it into other systems.

[0107] Example 2

[0108] The present invention also provides an electronic case file element recognition system based on the Transformer framework. The electronic case file element recognition system based on the Transformer framework can be implemented by executing the process steps of the electronic case file element recognition method based on the Transformer framework. That is, those skilled in the art can understand the electronic case file element recognition method based on the Transformer framework as a preferred embodiment of the electronic case file element recognition system based on the Transformer framework.

[0109] like Figures 1-6 The electronic case file element recognition system based on the Transformer framework provided by the present invention includes:

[0110] Module M1: Uses the electronic case file template accompanying the entire business process of court trial to simulate and generate multimodal electronic case file data. The generated formats include electronic case file text, electronic case file scanned images, and electronic case file categorization structure tree.

[0111] Module M2: Constructs a cognitive network model for electronic case file elements based on a multimodal, multi-task Transformer framework. It consists of two parts: a multimodal encoder based on the Transformer framework for extracting features from electronic case files, and a multi-task decoder based on the Transformer framework for completing downstream tasks.

[0112] The multimodal encoder includes a text encoder based on the Transformer architecture, which encodes formatted text information in the multimodal input, and a visual encoder based on the Swin-Transformer architecture, which learns to recognize and extract text content from images according to reading habits and performs feature encoding.

[0113] The multi-task decoder uses a pre-trained multilingual BART model for natural language processing. Based on this decoder, the understanding of the encoded text of the multilingual BART model is fine-tuned. Depending on the task, the decoder with different fine-tuning directions outputs case elements and downstream task information in the decoded electronic case file data.

[0114] Module M3: Defines the multi-task learning framework, sets the loss function for optimizing network parameters, and uses the loss function and optimization objective for different task outputs.

[0115] Module M4: Using the generated multimodal electronic case file data, pre-train the electronic case file case element cognitive network model based on the multimodal, multi-task Transformer framework, train the model's ability to extract multimodal text, and train the model into a multimodal language model;

[0116] Module M5: Uses the pre-trained model for fine-tuning training, constructs multi-task prompts for multi-task learning, and treats different downstream tasks as multiple JSON format data prediction problems. The training objective is to map each document to a database consistent with the feature extraction data format or to perform integrity matching and evaluation on the JSON format data.

[0117] Module M6: Exports a finely tuned, trained electronic case file element cognitive network model based on a multimodal, multi-task Transformer framework, which can be deployed directly as a service or integrated into a system for use.

[0118] In module M1, the electronic case file generated by the template of the court trial process has a list of extracted element attributes that has been labeled. It is generated using templates according to various document categories and different crime categories. Under the same category of document data, there are text data and document image data with different degrees of regularity. The simulated electronic case file dataset samples are divided into training set and test set in a 9:1 ratio.

[0119] The module M2 includes:

[0120] Module M2.1: The Swin-Transformer-based visual encoder transforms the input document information x into a set of embedding vectors. This encoder contains multiple Swin-Transformer blocks, which include a windowed self-attention mechanism module, a shifted window multi-head self-attention mechanism, and a dimensionality reduction module.

[0121] The self-attention mechanism in the encoder takes inputs from the same sequence. It calculates the dot product between the query Q and the index K in the input, then scales the gradient to control the magnitude. A softmax function is applied to obtain the attention weights, and finally, these weights are multiplied by the response value V to obtain the output. The expression is as follows:

[0122]

[0123] The encoder also uses a feedforward network, expressed as:

[0124] FFN(X)=Relu(x·W1+b1)·W2+b2

[0125] Where W1 and W2 represent the learned weight matrices; b1 and b2 are bias vectors; and ReLU() is the activation function.

[0126] The calculation process for residual connections in the encoder involves adding the output of the feedforward network to the input, and performing layer normalization before addition. The expression is as follows:

[0127] Output=LayerNorm(x+FFN(LayerNorm(x)))

[0128] Where LayerNorm({ is the normalization function;

[0129] Module M2.2: For the embedding vector {z{} obtained by the encoder, the decoder of the text data generates a token sequence (y{z}). i )*m i =1, where y i ∈R v is the one-hot vector of the i-th token, v is the size of the token vocabulary, and m is a hyperparameter;

[0130] The decoder also uses a self-attention mechanism, which represents the self-attention computation within the decoder, used for modeling its own sequence. The expression is:

[0131]

[0132] Where query Q′ is the output of the decoder, and key K′ and value V′ are the outputs of the encoder;

[0133] The decoder also uses feedforward networks and residual connections to pass information between layers.

[0134] In module M5, multi-task prompt words are constructed for multi-task learning. For the input document information x, a prompt function f is used. prompt (), helps the model recall the information x' it learned during pre-training, expressed as:

[0135] x'=f prompt (x)

[0136] This function is a natural language string containing two empty positions: one for filling in the input and the other for generating the answer text.

[0137] In module M5, the overall performance is evaluated using field-level F1 scores and tree-edit distance-based accuracy.

[0138] The field-level F1 score method checks if the extracted field information is not in the true value; if so, the field extraction is considered to have failed, otherwise it is considered to have succeeded.

[0139] The tree-based edit distance method is applicable to any document represented as a tree. Edit distance refers to the minimum number of editing operations required to transform one string into another. Editing operations include replacing, inserting, or deleting a character. The formula for calculating the tree-based edit distance T is:

[0140]

[0141] Where gt, pr, and φ represent the ground truth, prediction, and empty tree, respectively; EditDist() is the edit distance function; T a ,T b It consists of two strings.

[0142] Those skilled in the art will understand that, in addition to implementing the system, apparatus, and their modules provided by this invention in purely computer-readable program code, the same program can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system, apparatus, and their modules provided by this invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; alternatively, modules for implementing various functions can be considered both software programs implementing the method and structures within the hardware component.

[0143] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A method for recognizing elements in electronic case files based on the Transformer framework, characterized in that, include: Step S1: Use the court trial full business process electronic case file template to simulate and generate multimodal electronic case file data. The generated format includes electronic case file text, electronic case file scanned image, and electronic case file categorization structure tree. Step S2: Build a cognitive network model of electronic case file elements based on the multimodal, multi-task Transformer framework. It consists of two parts: a multimodal encoder based on the Transformer framework for extracting features from electronic case files, and a multi-task decoder based on the Transformer framework for completing downstream tasks. The multimodal encoder includes a text encoder based on the Transformer architecture, which encodes formatted text information in the multimodal input, and a visual encoder based on the Swin-Transformer architecture, which learns to recognize and extract text content from images according to reading habits and performs feature encoding. The multi-task decoder uses a pre-trained multilingual BART model for natural language processing. Based on this decoder, the understanding of the encoded text of the multilingual BART model is fine-tuned. Depending on the task, the decoder with different fine-tuning directions outputs case elements and downstream task information in the decoded electronic case file data. Step S3: Determine the multi-task learning framework, set the loss function for optimizing network parameters, and use the loss function and optimization objective of each task for different task outputs; Step S4: Using the generated multimodal electronic case file data, pre-train the electronic case file case element cognitive network model based on the multimodal, multi-task Transformer framework, train the model's ability to extract multimodal text, and train the model into a multimodal language model. Step S5: Fine-tune the pre-trained model, construct multi-task prompts for multi-task learning, and treat the multi-task as multiple different JSON format data prediction problems for different downstream tasks. The training objective is to map each document to a database consistent with the feature extraction data format or to perform integrity matching and evaluation on the JSON format data. Step S6: Export the fine-tuned and trained electronic case file case element cognitive network model based on the multimodal, multi-task Transformer framework, and use it by directly deploying it as a service or integrating it into the system.

2. The method for recognizing electronic case file elements based on the Transformer framework according to claim 1, characterized in that, In step S1, the electronic case file generated by the template of the court trial process has a list of extracted element attributes that has been labeled, and is generated using templates according to various document categories and different crime categories. Under the same category of document data, there are text data and document image data with different degrees of regularity; the simulated electronic case file dataset sample is divided into training set and test set in a 9:1 ratio.

3. The method for recognizing electronic case file elements based on the Transformer framework according to claim 1, characterized in that, Step S2 includes: Step S2.1: The Swin-Transformer-based visual encoder converts the input document information x into a set of embedding vectors. The encoder contains multiple Swin-Transformer blocks, which include a windowed self-attention mechanism module, a shifted window multi-head self-attention mechanism, and a dimensionality reduction module. The self-attention mechanism in the encoder takes inputs from the same sequence. It calculates the dot product between the query Q and the index K in the input, then scales the gradient to control the magnitude. A softmax function is applied to obtain the attention weights, and finally, these weights are multiplied by the response value V to obtain the output. The expression is as follows: The encoder also uses a feedforward network, expressed as: FFN(X)=Relu(x·W1+b1)·W2+b2 Where W1 and W2 represent the learned weight matrices; b1 and b2 are bias vectors; and ReLU() is the activation function. The calculation process for residual connections in the encoder involves adding the output of the feedforward network to the input, and performing layer normalization before addition. The expression is as follows: Output=LayerNorm(x+FFN(LayerNorm(x))) LayerNorm() is the normalization function; Step S2.2: For the embedding vector {z} obtained by the encoder, the decoder of the text data generates a token sequence (y). i )*m i =1, where y i ∈R v is the one-hot vector of the i-th token, v is the size of the token vocabulary, and m is a hyperparameter; The decoder also uses a self-attention mechanism, which represents the self-attention computation within the decoder, used for modeling its own sequence. The expression is: Where query Q′ is the output of the decoder, and key K′ and value V′ are the outputs of the encoder; The decoder also uses feedforward networks and residual connections to pass information between layers.

4. The method for recognizing electronic case file elements based on the Transformer framework according to claim 1, characterized in that, In step S5, multi-task prompt words are constructed for multi-task learning. For the input document information x, a prompt function f is used. prompt (), helps the model recall the information x' it learned during pre-training, expressed as: x’=f prompt (x) This function is a natural language string containing two empty positions: one for filling in the input and the other for generating the answer text.

5. The method for recognizing electronic case file elements based on the Transformer framework according to claim 1, characterized in that, In step S5, a comprehensive performance evaluation is performed using field-level F1 scores and tree-edit distance-based accuracy. The field-level F1 score method checks if the extracted field information is not in the true value; if so, the field extraction is considered to have failed, otherwise it is considered to have succeeded. The tree-based edit distance method is applicable to any document represented as a tree. Edit distance refers to the minimum number of editing operations required to transform one string into another. Editing operations include replacing, inserting, or deleting a character. The formula for calculating the tree-based edit distance T is: Where gt, pr, and φ represent the ground truth, prediction, and empty tree, respectively; EditDist() is the edit distance function; T a ,T b It consists of two strings.

6. A case element recognition system for electronic case files based on the Transformer framework, characterized in that, include: Module M1: Uses the electronic case file template accompanying the entire business process of court trial to simulate and generate multimodal electronic case file data. The generated formats include electronic case file text, electronic case file scanned images, and electronic case file categorization structure tree. Module M2: Constructs a cognitive network model for electronic case file elements based on a multimodal, multi-task Transformer framework. It consists of two parts: a multimodal encoder based on the Transformer framework for extracting features from electronic case files, and a multi-task decoder based on the Transformer framework for completing downstream tasks. The multimodal encoder includes a text encoder based on the Transformer architecture, which encodes formatted text information in the multimodal input, and a visual encoder based on the Swin-Transformer architecture, which learns to recognize and extract text content from images according to reading habits and performs feature encoding. The multi-task decoder uses a pre-trained multilingual BART model for natural language processing. Based on this decoder, the understanding of the encoded text of the multilingual BART model is fine-tuned. Depending on the task, the decoder with different fine-tuning directions outputs case elements and downstream task information in the decoded electronic case file data. Module M3: Defines the multi-task learning framework, sets the loss function for optimizing network parameters, and uses the loss function and optimization objective for different task outputs. Module M4: Using the generated multimodal electronic case file data, pre-train the electronic case file case element cognitive network model based on the multimodal, multi-task Transformer framework, train the model's ability to extract multimodal text, and train the model into a multimodal language model; Module M5: Uses the pre-trained model for fine-tuning training, constructs multi-task prompts for multi-task learning, and treats different downstream tasks as multiple JSON format data prediction problems. The training objective is to map each document to a database consistent with the feature extraction data format or to perform integrity matching and evaluation on the JSON format data. Module M6: Exports a finely tuned, trained electronic case file element cognitive network model based on a multimodal, multi-task Transformer framework, which can be deployed directly as a service or integrated into a system for use.

7. The electronic case file element recognition system based on the Transformer framework according to claim 6, characterized in that, In module M1, the electronic case file generated by the template of the court trial process has a list of extracted element attributes that has been labeled. It is generated using templates according to various document categories and different crime categories. Under the same category of document data, there are text data and document image data with different degrees of regularity. The simulated electronic case file dataset samples are divided into training set and test set in a 9:1 ratio.

8. The electronic case file element recognition system based on the Transformer framework according to claim 6, characterized in that, The module M2 includes: Module M2.1: The Swin-Transformer-based visual encoder transforms the input document information x into a set of embedding vectors. This encoder contains multiple Swin-Transformer blocks, which include a windowed self-attention mechanism module, a shifted window multi-head self-attention mechanism, and a dimensionality reduction module. The self-attention mechanism in the encoder takes inputs from the same sequence. It calculates the dot product between the query Q and the index K in the input, then scales the gradient to control the magnitude. A softmax function is applied to obtain the attention weights, and finally, these weights are multiplied by the response value V to obtain the output. The expression is as follows: The encoder also uses a feedforward network, expressed as: FFN(X)=Relu(x·W1+b1)·W2+b2 Where W1 and W2 represent the learned weight matrices; b1 and b2 are bias vectors; and ReLU() is the activation function. The calculation process for residual connections in the encoder involves adding the output of the feedforward network to the input, and performing layer normalization before addition. The expression is as follows: Output=LayerNorm(x+FFN(LayerNorm(x))) LayerNorm() is the normalization function; Module M2.2: For the embedding vector {z} obtained by the encoder, the decoder of the text data generates a token sequence (y). i )*m i =1, where y i ∈R v is the one-hot vector of the i-th token, v is the size of the token vocabulary, and m is a hyperparameter; The decoder also uses a self-attention mechanism, which represents the self-attention computation within the decoder, used for modeling its own sequence. The expression is: Where query Q′ is the output of the decoder, and key K′ and value V′ are the outputs of the encoder; The decoder also uses feedforward networks and residual connections to pass information between layers.

9. The electronic case file element recognition system based on the Transformer framework according to claim 6, characterized in that, In module M5, multi-task prompt words are constructed for multi-task learning. For the input document information x, a prompt function f is used. prompt (), helps the model recall the information x' it learned during pre-training, expressed as: x’=f prompt (x) This function is a natural language string containing two empty positions: one for filling in the input and the other for generating the answer text.

10. The electronic case file element recognition system based on the Transformer framework according to claim 6, characterized in that, In module M5, the overall performance is evaluated using field-level F1 scores and tree-edit distance-based accuracy. The field-level F1 score method checks if the extracted field information is not in the true value; if so, the field extraction is considered to have failed, otherwise it is considered to have succeeded. The tree-based edit distance method is applicable to any document represented as a tree. Edit distance refers to the minimum number of editing operations required to transform one string into another. Editing operations include replacing, inserting, or deleting a character. The formula for calculating the tree-based edit distance T is: Where gt, pr, and φ represent the ground truth, prediction, and empty tree, respectively; EditDist() is the edit distance function; T a ,T b It consists of two strings.

Images