Case information structuring and extracting method and system based on cross-media knowledge graph

Abstract

The invention provides a case information structuring and extracting method and system based on a cross-media knowledge graph. The method comprises the following steps of S1, obtaining an electronic file text, image and video and audio structured data set; s2, constructing a cross-media domain knowledge graph of multi-source heterogeneous full-amount data oriented to the electronic file data; and S3, introducing a data-driven deep learning target detection model and a knowledge-driven cross-media knowledge graph method, and carrying out structured processing and knowledge extraction on the multimedia electronic file data. According to the method, the target detection technology based on deep learning is applied, and technologies such as a rule base, a concept base, a fact base and a question bank are introduced, so that the specific field characteristics of the electronic file are solved, the problem of automatic dynamic construction of the cross-media knowledge graph of the electronic file is solved, and the case structured information in the multimedia data of the electronic file is systematically extracted.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of wisdom justice, in particular to a case information structuring and extraction method and system based on a cross-media knowledge graph. BACKGROUND

[0002] With the rapid development of information technology and the digital transformation of the legal system, electronic files are increasingly used in the judicial field. Since most of the case elements of the currently used element-based trial system still remain in the stage of partial automatic extraction and manual input, the ability of automatically extracting full elements of multimedia electronic files for supporting element-based trials in the whole chain has not been formed, and there is a lack of visualized organization and presentation of case elements.

[0003] In recent years, with the development of computer vision technology and the explosive growth of new technology applications, it is urgent to convert massive unstructured data into structured information, thereby providing efficient information management and decision support for the fields of law, justice, insurance and the like, and meeting the requirements of diversity and security of actual applications. Therefore, it is urgent to design a case information structuring and extraction method based on a cross-media knowledge graph. SUMMARY

[0004] The application provides a case information structuring and extraction method and system based on a cross-media knowledge graph. The method applies target detection technology based on deep learning, and introduces a rule base, a concept base, a fact base and a topic base to solve the problem of automatic and dynamic construction of a cross-media knowledge graph of electronic files, so as to extract structured information of a case from multimedia data of electronic files in a systematic way. The method adopts NLP technologies such as syntax analysis, entity concept recognition, concept mapping, ambiguity resolution, synonym expansion, event element extraction and relationship extraction, and computer vision technologies such as OCR recognition, target detection and event detection based on deep learning to realize knowledge extraction of full data of massive multi-source heterogeneous electronic files. The method studies the structured expression and dynamic construction of a cross-media knowledge graph, forms a large-scale cross-media knowledge pedigree composed of an ontology base, a concept base, a fact base and a rule base, studies a knowledge backtracking mechanism, provides an effective way of supervising and optimizing the automatic knowledge acquisition process, makes knowledge form an iterative cycle in the system, and improves the precision, self-adaptability and expandability of the knowledge base.

[0005] The technical scheme of the application is as follows: a case information structuring and extraction method based on a cross-media knowledge graph, comprising the following steps:

[0006] Step S1: obtaining structured data sets of electronic file texts, images and videos; wherein, the structured data sets of electronic file texts, images and videos with a scale of up to ten thousand can be self-built, and the multimedia electronic file data is preprocessed;

[0007] Step S2, constructing a cross-media field knowledge graph of multi-source heterogeneous full data oriented to large-scale electronic dossier data;

[0008] Step S3, introducing a data-driven deep learning target detection model and a knowledge-driven cross-media knowledge graph technology, thereby structuring and knowledge extracting the multimedia electronic dossier data;

[0009] Further, the step S1 specifically comprises:

[0010] Self-built electronic dossier text, image, video and audio structured data set with a scale of ten thousand levels, and preprocessed the multimedia electronic dossier data.

[0011] Further, the preprocessing of the multimedia electronic dossier data specifically comprises:

[0012] For electronic dossier multimedia data, including images, texts, videos and audios, images are converted to texts through OCR recognition technology and image object positioning through target detection technology; for videos, video shot segmentation and party behavior recognition are needed; for audios, different accents in the court trial process are identified through speech recognition technology.

[0013] Further, the step S2 specifically comprises:

[0014] Step S21, using an entity relationship extraction NLP method based on a pre-trained model, comprehensively capturing the complex relationship between entities and relationships through the deep semantic representation learned by the pre-trained model;

[0015] Step S22, using an OCR recognition technology based on multi-decoder joint learning, extracting text region images from the original image through the coordinate information provided by the text positioning module, then recognizing long sequences, short sequences and single characters based on CTC, Attention and ACE encoding and decoding, training by using the multi-decoder joint learning method to realize parameter sharing, and finally outputting the text string and text character position information contained in the text region image.

[0016] Further, the step S21 specifically comprises:

[0017] Step S211, first, joint learning is adopted to directly extract entity relationship triplets through an end-to-end model, realizing effective integration of entity and relationship information;

[0018] Step S212, the model adopts the first extraction rule to process entity boundaries, entity types and relationship matching, these rules combine the principles of sequence labeling and relationship matching, making the triplet extraction process more targeted and adaptive, effectively handling different entity combinations.

[0019] Step S213, finally combined with the pre-training model BERT and sequence labeling model Bilstm-CRF, using the deep semantic representation ability of BERT and the sequence labeling ability of Bilstm-CRF, realizes the end-to-end process from input sentence to triple extraction.

[0020] Further, the step S22 specifically comprises:

[0021] Step S221, first using CTPN for Chinese detection;

[0022] Step S222, then using CRNN (Convolutional Recurrent Neural Network) for text recognition. Utilize CNN feature extraction layer and BLSTM sequence feature extraction layer, and simultaneously use the model of end-to-end joint training, CTPN uses BLSTM model and CTC transcription component to process the relationship between the front and back texts in the character image;

[0023] Step S223, in the multi-decoder coding stage, three different decoder structures are adopted: CTC, Attention and ACE.

[0024] Further, the step S221 specifically comprises: the CTPN method is not directly connected to the full connection layer for classification or regression after the RPN, but is connected to the full connection layer through an LSTM (Long Short-Term Memory, long short-term memory network) again;

[0025] Further, the step S3 specifically comprises

[0026] Step S31, unstructured text knowledge extraction includes the structure and knowledge extraction of evidence information such as interrogation records, identification reports, and text witness testimony obtained by voice content recognition, confession and defense, and text contracts obtained by OCR recognition;

[0027] Step S32, the establishment of structured knowledge extraction mainly uses four kinds of data: business standards, including case structure specification and case classification specification; text data, including interrogation records and identification reports; audio data, including witness testimony and confession and defense; image / video data, including evidence books and videos.

[0028] Further, the step S31 specifically comprises:

[0029] Step S311, first, the case electronic file data is represented by vectors; then, the context representation of the attribute sequence corresponding to the case elements is captured by using the recurrent neural network based on the bidirectional common attention mechanism.

[0030] Step S312, then the declarative judicial knowledge is compiled into FOL rules, the rules in symbolic knowledge are converted into continuous real numbers using a mapping function, that is, the Boolean operation of FOL is converted into probability logic.

[0031] Step S313, the prediction probability distribution of the deep neural network is reweighted through the first-order logic rules in symbolic knowledge, and finally the prediction output will be determined by the deep neural network learning module and the symbolic knowledge module

[0032] Further, the step S4, the above-mentioned related technology is used for dataset test, through the content of 230 original electronic files and corresponding case element annotation files, including 17779 JPG picture files, 10 PDF files and 10 DOC files, 200 hours of original video files, more than 4000 audio files and 15 hours of Chinese voice data, the text type electronic file structure and extraction method is carried out 3 times, the audio type electronic file structure and extraction method is carried out 3 times, and the video type electronic file structure and extraction method is carried out 3 times.

[0033] According to another aspect of the present application, the present application also provides a case information structure and extraction system, comprising:

[0034] The acquisition module is used for acquiring electronic file text, image, video and audio structured data sets.

[0035] The construction module is used for constructing a cross-media domain knowledge graph of multi-source heterogeneous full data for electronic file data.

[0036] The structured extraction module is used for introducing a data-driven deep learning target detection model and a knowledge-driven cross-media knowledge graph method to structure and extract knowledge from multimedia electronic file data. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 A case information structure and extraction method flow chart based on a cross-media knowledge graph according to the present application;

[0038] Figure 2 A joint learning model diagram of the present application;

[0039] Figure 3 A flow chart of multi-decoder joint learning of the embodiment of the present application;

[0040] Figure 4 A pre-training language model structure diagram of the present application;

[0041] Figure 5The application is a case information structuring and extraction system based on a cross-media knowledge graph. DETAILED DESCRIPTION

[0042] The technical solutions in the embodiments of the application will be clearly and completely described below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only some of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments of the application, all other embodiments obtained by a person of ordinary skill in the art without creative effort are within the protection scope of the application.

[0043] According to the embodiments of the application, a case information structuring and extraction method based on a cross-media knowledge graph is provided, as shown in the figure. Figure 1 To solve the problem of understanding the content of electronic case files multimedia data, a cross-media domain knowledge graph for large-scale electronic case files is constructed. By constructing the cross-media domain knowledge graph, not only can the text data and multimedia data be associated, but also the case information in the electronic case files can be more comprehensively understood. For example, by associating the case text information with video and image data, more accurate identification of involved items and restoration of key events can be achieved. Traditional knowledge graphs mainly focus on general domain knowledge, while the domain knowledge of electronic case files has certain professionalism and particularity. Based on the above domain professional knowledge and particularity of electronic case files, by constructing a domain knowledge graph for electronic case files, the demand for structuring of electronic case file data can be better met, active extraction and organization of case information can be achieved, which will help to improve the management efficiency and data analysis capability of electronic case file data in practice.

[0044] Step S1, obtaining electronic case file text, image, and video and audio structured data sets; wherein a self-built electronic case file text, image, and video and audio structured data set with a scale of up to ten thousand levels can be obtained, and the multimedia electronic case file data is preprocessed;

[0045] Step S2, constructing a cross-media domain knowledge graph for multi-source heterogeneous full data of electronic case file data;

[0046] Step S3, introducing a data-driven deep learning target detection model and a knowledge-driven cross-media knowledge graph technology, so as to structure and extract knowledge from the multimedia electronic case file data;

[0047] Step S4, testing the data set used by the method and the method itself in terms of functionality and performance requirements.

[0048] Further, the step S1 acquires electronic file text, image, audio and video structured data set, wherein the electronic file includes 230 original electronic file documents and corresponding case element annotation file contents, wherein 17779 JPG picture files, 10 PDF files, 10 DOC files, 200 hours of original video files, more than 4000 audio files and 15 hours of Chinese audio data are included. Further, the above-mentioned text, image, audio and video data can be preprocessed; specifically, for the electronic file multimedia data, including image, text, video and audio, the image can be converted into text through OCR identification technology and the image object can be quickly positioned through target detection technology; for the video, video shot segmentation and party behavior identification are needed; for the audio, different accents in the court trial process need to be identified through speech recognition technology.

[0049] The step S2 constructs a cross-media field knowledge graph of multi-source heterogeneous full data facing large-scale electronic file data, and the specific process is as follows:

[0050] In step S21, an entity relationship extraction NLP technology based on a pre-training model is used, through deep semantic representation learned by the pre-training model, complex connections between entities and relationships can be more comprehensively captured, so that the accuracy and generalization ability of relationship extraction are improved, and the specific process includes:

[0051] In step S211, joint learning is first adopted, and an end-to-end model is used to directly extract entity relationship triples, so that the effective integration of entity and relationship information is realized, and the end-to-end model learning process is as shown in the figure. Figure 2 The method introduces a bit body label strategy, and the relationship information is integrated into the label of the entity, so that the relationship between entities is determined at the same time of entity identification. The bit body label strategy refers to the strategy of integrating relationship information into the label, when the label of the entity is determined, the relationship between entities is determined. The label includes three parts of contents: B-X represents the beginning of the element belonging to the X type and the element in the beginning of the segment, I-X represents the element belonging to the X type and the element in the middle position of the segment, and O represents not belonging to any type.

[0052] Wherein, "B-X" represents that the segment where the element is located belongs to X type and the element is in the beginning of the segment, "I-X" represents that the segment where the element is located belongs to X type and the element is in the middle position of the segment, and "O" represents not belonging to any type.

[0053] For example, the three labels of BIO are as follows when the present application represents X as a noun phrase (Noun Phrase, NP):

[0054] (1) B-NP: beginning of the noun phrase

[0055] (2) I-NP: middle of the noun phrase

[0056] (3) O: Not a noun phrase

[0057] This strategy makes entity relation extraction more accurate and consistent, avoiding the information loss ignored in traditional methods.

[0058] Step S212, secondly, the model adopts the first extraction rule to process entity boundary, entity type and relationship matching, and the first extraction rule specifically includes:

[0059] (1) When determining the entity boundary, the input sequence corresponding to the label subsequence satisfying the regular expression "BI*" is the recognized entity, that is, when determining the entity boundary, the relationship information and head-tail entity markers in the label are ignored, only the entity position is concerned, single-word entity is the label "B", single-word label "I" will not be recognized as an entity, and the position of the multi-word entity B label is the starting position of the entity;

[0060] (2) When determining the entity type, the entity type of a single-word entity can be uniquely determined by a separate label, while for a multi-word entity, if the type recognized for each word is the same, the entity can be determined as that type; if there is inconsistency in the type recognized for each word, the type recognized for the first word is specified to determine the entity type, and the inconsistency in the type does not affect the recognition of the entity boundary, which is still determined according to the previous rule;

[0061] (3) When performing relationship matching, the direction principle should be followed, that is, the entity with "1" in the label, i.e. the head entity in the triple, can only match the tail entity of the corresponding relationship in the nearest direction, and the entity with "2" in the label, i.e. the tail entity in the triple, can only match the head entity of the corresponding relationship in the nearest direction, and it is also specified that each entity can only match entities with the same relationship, and if a matching entity cannot be found during the search process, the entity is also discarded;

[0062] (4) When performing relationship matching, the greedy principle should be followed, that is, the entity matches the pattern in the forward or backward direction, and each entity tries to match as many entities as possible to complete the combination, and if an entity involving other relationships is encountered, the matching is terminated.

[0063] These rules are aimed at the relationship extraction of electronic case files, image and video structured data sets, combining the principles of sequence labeling and relationship matching, making the electronic case file triple extraction process more targeted and adaptive, and effectively handling different entity combinations.

[0064] Step S213, finally combined with the pre-training model (such as BERT) and the sequence labeling model (such as Bilstm-CRF), the deep semantic representation ability of BERT and the sequence labeling ability of Bilstm-CRF are used to realize the end-to-end process from the input sentence to the triple extraction. The overall architecture of the BERT model is the encoder part of the transformer, and the model parameters are updated by gradient back propagation. Compared with other neural network structures, BERT contains more semantic knowledge and can solve the problem of polysemy, has stronger semantic representation ability, and is easier to identify entity relationships in text when applied to experiments. The input of the encoder is the original sentence, which is converted into an input vector accepted by BERT. The model is fine-tuned on BERT to update the model parameters to adapt to the new parameters of the entity relationship extraction task, and the model will focus more on the task to be processed. Given an input sentence S of length n, each word after conversion will consist of three parts, namely (input_ids, attetion_masks, segment_ids), which represent the index of the input word in the dictionary, the masking mark of the input word, and the sentence index to which the input word belongs. At the same time, add “[CLS]” at the beginning of the sentence and “[SEP]” at the end of the sentence. By using the powerful semantic representation ability of the encoder, the model can capture the relationship between the corresponding words and form a hidden layer vector containing feature information.

[0065] In the decoder stage, the model adopts the Bilstm-CRF (Bidirectional long short-termmemory-Conditional Random Field) architecture, as shown in Figure 3 Bidirectional LSTM is a combination structure of forward propagation LSTM and backward propagation LSTM, which can be used to model the context information of the input sequence. When the neural network processes the input at time t, the forward propagation LSTM can output the previous hidden layer vector from left to right and the previous hidden layer vector from right to left. After combining the above two hidden layer vectors and the current input through the neural network, the decoded neural network vector contains global and local features, enriching the overall feature information.

[0066] At the same time, in the generation of the label sequence, each label depends not only on the semantic information of the word, but also on the label sequence of the context. Therefore, the application splices a CRF layer after the bidirectional LSTM network to make the model consider the transition probability information of the label in the final label distribution calculation. The CRF layer can capture the dependency between labels during model calculation. The hidden layer vector h output by the decoding layer is used as the input of this layer, and the optimal label path searched by the Viterbi method is used as the output to realize sequence labeling.

[0067] Step S22, using the OCR recognition technology based on multi-decoder joint learning, the text region image is extracted from the original image by the coordinate information provided by the text positioning module, and then based on the coding and decoding of CTC (Connectionist Temporal Classification), Attention and ACE (Automatic Content Extraction), long sequence, short sequence and single character are recognized respectively, the method of multi-decoder joint learning is used for training, parameter sharing is realized, and finally the text string contained in the text region image and the text character position information are output, the specific process includes:

[0068] Step S221, first, CTPN (Convolutional Text Proposal Network) is used for Chinese detection. The CTPN method is good at detecting characters arranged horizontally in natural scenes. It is also based on Faster R-CNN, but the RPN region is modified. Because the width is not considered when judging positive and negative samples, the anchor is not selected with different widths. Only a fixed width is needed, and then a few appropriate heights are selected according to the specific task. Moreover, after RPN, a LSTM (Long Short-Term Memory) is not directly connected to a full connection layer for classification or regression, but a full connection layer is connected again.

[0069] Step S222, then CRNN (Convolutional Recurrent Neural Network) is used for text recognition. The model can be trained end-to-end simultaneously by using CNN feature extraction layer and BLSTM sequence feature extraction layer. CTPN uses BLSTM model and CTC transcription component, which can handle the relationship between the front and back of the character image, thereby effectively improving the accuracy of text recognition, and greatly improving the robustness of the whole model.

[0070] Step S223, in the multi-decoder coding and decoding stage, three different decoder structures are adopted: CTC, Attention and ACE. For example Figure 4The three decoders are respectively focused on different types of text sequences, i.e., long sequences, short sequences, and single characters. The CTC (Connectionist Temporal Classification) decoder is suitable for processing problems with variable-length output sequences, such as recognizing long sentences. The Attention decoder performs better when processing short sequences, as it can focus on the importance of different regions in the input image. The ACE (Attn-CTC-Edit) decoder focuses on single character recognition and is suitable for character-level tasks. Through joint training, the decoders can correct each other's errors and improve the overall recognition performance.

[0071] Step S23, using the Center and Scale Prediction target detection technology, only key point prediction is performed (abstracting the semantics of the image and detecting semantic feature points), and a heatmap is generated by a fully convolutional network, the peak point of the heatmap is the center point of the target, and the peak point can also predict the width and height information of the target, finally improving the accuracy of the detection result, the specific process includes:

[0072] Step S231, first, the CSP method extracts features of the input image through a fully convolutional network, and extracts semantic information of the image layer by layer.

[0073] Step S232, secondly, the peak point in the heatmap is interpreted as the center point of the target. By detecting the peak in the heatmap, the CSP method can locate the accurate position of the target, thereby realizing accurate detection of the target. The loss function of the neural network is as follows:

[0074] L=λ c L center +λ s L scale +λ o L offset ,

[0075] Where λc, λs and λo are the weights of the center classification, scale regression and offset regression loss respectively, L center is the position prediction loss function, L scale is the scale prediction loss function, L offset is the offset prediction branch loss function, and L is the final total loss function.

[0076] The position prediction loss function is as follows:

[0077]

[0078] Where K is the number of targets in the image, the input image size is HxW, the size of the finally connected feature map is H / rxW / r, and r is the down-sampling ratio, The weights of easy-to-classify samples are reduced, so that the training process can pay more attention to difficult-to-classify samples.p ij ∈[0,1] is a network estimated probability indicating whether there is a center of an object in position (i,j), y ij =1 represents a positive sample, α ij and γ are focus hyperparameters, the hyperparameter β controls the penalty term, K is the number of targets in the image, is the center coordinate, width and height of the kth target, G(.) is a 2D Gaussian mask. M ij is a Gaussian mask, the farther the distance, the larger the M ij , the smaller the loss function, these negative samples will be ignored.

[0079] The size prediction loss function is as follows:

[0080]

[0081] Where K is the number of targets in the image, s k and t k represent the network prediction result and the true value of the positive sample, if the masks overlap, select the maximum value of the overlapping area, where SmoothL1 is a commonly used loss function, usually used in regression tasks, its advantage over mean square error (MSE) loss function is that the penalty for outliers (such as too large or too small outliers) is smaller, so that the model is more robust.

[0082] Further, the step S3 adopts three relatively complete knowledge extractions: (1) structured knowledge extraction; (2) semi-structured knowledge extraction; (3) unstructured knowledge extraction. Through the above three relatively complete knowledge extractions, the knowledge base construction of large-scale multi-source heterogeneous data is realized, including the semantic knowledge extraction of massive video data, image data, audio data and text data. Specifically, it includes:

[0083] Step S31, unstructured text knowledge extraction, mainly including the structured and knowledge extraction of evidence information such as interrogation records, identification reports, and text witness statements, confessions and defenses obtained through voice content recognition, and text contracts obtained through OCR recognition. The specific process includes:

[0084] Step S311, first, the case electronic file data is represented by vectors; then the context representation of the attribute sequence corresponding to the case elements is captured by using a recurrent neural network based on a bidirectional common attention mechanism.

[0085] Step S312, then the declarative judicial knowledge is compiled into FOL (First-Order Logic) rules, and the rules in symbolic knowledge are converted into continuous real numbers using a mapping function, that is, the Boolean operation of FOL is converted into probabilistic logic.

[0086] Step S313, the prediction probability distribution of the deep neural network is reweighted through the first-order logic rules in the symbolic knowledge, and the final prediction output will be determined by the deep neural network learning module and the symbolic knowledge module.

[0087] Step S32, structured knowledge extraction is mainly established by four kinds of data: business standards, such as case structure specifications and case classification specifications; text data, such as interrogation records and identification reports; audio data, such as witness testimony and confession defense; image / video data, such as material evidence and trial video. The structured knowledge extraction is established by the four kinds of data to form a large-scale Chinese knowledge pedigree composed of ontology library, concept library, fact library and rule library. By innovatively introducing the BERT model, the case information input is changed into the format of upper and lower sentences. Due to the existence of multiple events in the input text, the argument corresponding to the event role may not exist or multiple arguments may exist, so instead of using the traditional fixed threshold, a dynamic threshold is used, and only the argument with a probability higher than the threshold can be correctly identified. The case information extraction process in this algorithm includes:

[0088] Step S321, the original case information input is changed to the format of upper and lower sentences suitable for BERT (Bidirectional Encoder Representations from Transformers) input, the upper sentence is the template question, and the lower sentence is the original input, that is, “[CLS] <question>[SEP] <sentence>[SEP]”. Wherein "[CLS]" is a special representation word used by BERT for classification, and "[SEP]" is the end mark of the sequence.

[0089] Step S322, model construction is performed. It is mainly divided into two modules: trigger word extraction module and event argument extraction module. Due to its own special properties, legal texts can define event types through trigger words. The model first generates a template through the trigger word question, converts the input sequence into an upper and lower sentence format suitable for BERT input, and adds special marks "[CLS]" and "[SEP]" to complete the upper and lower sentence cutting; then the model will transform the input into a representation vector containing semantic information through BERT, and through the parameter update process of back propagation to make the whole model more suitable for the question and answer task of trigger word extraction; finally, through the long short-term memory neural network (LSTM) for feature abstraction, the head position and tail position of the trigger word sequence are identified to obtain the trigger word and complete the event detection. Assuming that the input is E, its calculation process is shown in the following formula.

[0090] A = bert tr (E)

[0091] P tr = softmax(LSTM(A))

[0092] Wherein, A is the representation vector after the BERT model is transformed, P tr is the trigger word after feature abstraction by the long short-term memory neural network. The bert tr function is the input function of the BERT model based on Transformers.

[0093] Step S323, a dynamic threshold strategy is used when extracting candidate arguments.

[0094] Further, the step S4 uses the above related technology to test the data set, by annotating the contents of 230 original electronic files and corresponding case element files, including 17779 JPG picture files, 10 PDF files and 10 DOC files, 200 hours of original video files, more than 4000 audio files and 15 hours of Chinese audio data, respectively. 3 times of text type electronic file structuring and extraction method, 3 times of audio type electronic file structuring and extraction method and 3 times of video type electronic file structuring and extraction method are performed.

[0095] The case information structuring and extraction method based on the cross-media knowledge graph of the embodiment of the application realizes case information structuring and extraction on a cross-media field knowledge graph of massive multi-source heterogeneous data of electronic files by constructing a multimedia electronic file dataset based on the professionalism and characteristics of the field of judicial electronic files, and provides a more efficient and intelligent data processing solution for the judicial industry.

[0096] According to another aspect of the application, referring to Figure 5 The application further provides a case information structuring and extraction system, comprising:

[0097] The acquisition module 51 is configured to acquire structured data sets of electronic file texts, images, and audio and video.

[0098] The construction module 52 is configured to construct a cross-media field knowledge graph of multi-source heterogeneous full data of electronic file data.

[0099] The structured extraction module 53 is configured to introduce a data-driven deep learning target detection model and a knowledge-driven cross-media knowledge graph method to perform structured processing and knowledge extraction on the multimedia electronic file data.

[0100] Although the above describes the specific embodiments of the application in detail, so as to facilitate the understanding of the application by those skilled in the art, it should be clear that the application is not limited to the scope of the specific embodiments, and for those skilled in the art, all kinds of changes within the spirit and scope of the application defined and determined by the appended claims are obvious, and all kinds of applications utilizing the concept of the application are included in the protection.< / sentence> < / question>

Claims

1. A method for structuring and extracting case information based on cross-media knowledge graphs, characterized in that, Includes the following steps: Step S1: Obtain the structured dataset of electronic case file text, images, and audio / video. Step S2: Construct a cross-media domain knowledge graph for multi-source heterogeneous full-volume electronic case file data; Step S3 introduces a data-driven deep learning object detection model and a knowledge-driven cross-media knowledge graph method to perform structured processing and knowledge extraction on multimedia electronic case file data.

2. The method for structuring and extracting case information based on cross-media knowledge graphs according to claim 1, characterized in that, Step S1 specifically includes: We have built a structured dataset of tens of thousands of electronic case file texts, images, and audio / video data, and preprocessed the multimedia electronic case file data.

3. The method for structuring and extracting case information based on cross-media knowledge graphs according to claim 2, characterized in that, The preprocessing of multimedia electronic case file data specifically includes: For electronic case file multimedia data, including images, text, video, and audio, images are converted into text using OCR recognition technology and object detection technology is used to quickly locate image objects; for video, video shot segmentation and identification of party behavior are required; for audio, speech recognition technology is needed to identify different accents during the court hearing.

4. The method for structuring and extracting case information based on cross-media knowledge graphs according to claim 1, characterized in that, Step S2 specifically includes: Step S21: Use the entity relation extraction NLP method based on pre-trained model to fully capture the complex relationship between entities and relations through the deep semantic representation learned by the pre-trained model. Step S22: Using OCR recognition technology based on multi-decoder joint learning, the text region image is extracted from the original image through the coordinate information provided by the text localization module. Then, based on the encoding and decoding of CTC, Attention and ACE, long sequences, short sequences and single characters are recognized respectively. The multi-decoder joint learning method is used for training to achieve parameter sharing. Finally, the text string and text character position information contained in the text region image are output.

5. The method for structuring and extracting case information based on cross-media knowledge graphs according to claim 1, characterized in that, Step S21 specifically includes: Step S211: First, joint learning is adopted to directly extract entity relation triples through an end-to-end model, thereby achieving effective integration of entity and relation information. In step S212, the model uses the first extraction rule to handle entity boundaries, entity types and relation matching. These rules combine the principles of sequence labeling and relation matching, making the triple extraction process more targeted and adaptive, and effectively handling different combinations of entities. Step S213 finally combines the pre-trained model BERT and the sequence labeling model Bilstm-CRF, utilizing BERT's deep semantic representation capabilities and Bilstm-CRF's sequence labeling capabilities to achieve an end-to-end process from input sentence to triple extraction.

6. The method for structuring and extracting case information based on cross-media knowledge graphs according to claim 4, characterized in that, Step S22 specifically includes: Step S221: First, use CTPN to perform Chinese character detection; Step S222: Then, text recognition is performed using CRNN (Convolutional Recurrent Neural Network). A model jointly trained end-to-end is used, employing a CNN feature extraction layer followed by a BLSTM sequence feature extraction layer. CTPN uses a BLSTM model and a CTC transcription component to learn the contextual relationships within character images. In step S223, during the multi-decoder encoding and decoding stage, three different decoder structures were adopted: CTC, Attention, and ACE.

7. The method for structuring and extracting case information based on cross-media knowledge graphs according to claim 6, characterized in that, The specific steps of S221 include: the CTPN method does not directly connect a fully connected layer after RPN for classification or regression, but instead uses an LSTM (Long Short-Term Memory) network followed by a fully connected layer.

8. The method for structuring and extracting case information based on cross-media knowledge graphs according to claim 1, characterized in that, Step S3 specifically includes Step S31, unstructured text knowledge extraction includes the structuring and knowledge extraction of evidence information such as interrogation transcripts, expert reports, and text witness testimonies, statements and defenses obtained through speech content recognition, and text contracts obtained through OCR recognition. Step S32 establishes structured knowledge extraction, which mainly uses four types of data: business standards, including case structure specifications and case classification specifications; text data, including interrogation transcripts and expert reports; audio data, including witness testimonies and statements; and image / video data, including physical evidence and interrogation videos.

9. The method for structuring and extracting case information based on cross-media knowledge graphs according to claim 8, characterized in that, Step S31 specifically includes: Step S311: First, the electronic case file data is vectorized; then, a recurrent neural network based on a bidirectional common attention mechanism is used to capture the contextual representation of the attribute sequence corresponding to the case elements. Step S312, then the declarative judicial knowledge is compiled into FOL rules, and the rules in the symbolic knowledge are converted into continuous real numbers using a mapping function, that is, the Boolean operation of FOL is converted into probabilistic logic. Step S313: The prediction probability distribution of the deep neural network is reweighted using the first-order logic rules in symbolic knowledge. The final prediction output will be jointly determined by the deep neural network learning module and the symbolic knowledge module.

10. A system for structuring and extracting case information based on cross-media knowledge graphs, characterized in that, include: The acquisition module is used to acquire structured datasets of electronic case files, including text, images, and audio / video files. The module is used to build a cross-media domain knowledge graph for multi-source heterogeneous full-volume electronic case file data; The structured extraction module is used to introduce data-driven deep learning object detection models and knowledge-driven cross-media knowledge graph methods to perform structured processing and knowledge extraction on multimedia electronic case file data.

Images