Judicial scene-oriented multi-modal data fusion method and system

Abstract

The invention discloses a judicial scene-oriented multi-modal data fusion method and system. The method comprises the following steps: 1) collecting multi-modal data in a judicial application scene and converting the multi-modal data into data in a uniform format; 2) mapping the data to a unified semantic space to realize cross-modal alignment; then associating the multi-modal data to obtain the text description of the same case and the corresponding image evidence as the multi-modal features of the corresponding case; 3) constructing a knowledge graph of a judicial application scene based on the multi-modal features; driving multi-modal data fusion based on the knowledge graph and constructing a multi-modal evidence chain of each entity; 4) quantifying the integrity of the knowledge graph and the reliability of the multi-modal evidence chain according to a preset index, marking abnormal nodes in the knowledge graph according to a quantification result, and adjusting the abnormal nodes; 5) generating a structured knowledge graph according to the knowledge graph and creating a dynamic desensitization report; edges in the structured knowledge graph represent relationships between legal entities, and each relationship is bound with a multi-modal evidence chain.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of big data, and relates to a multi-modal data fusion method and system for a judicial scene. BACKGROUND

[0002] In the field of justice, case processing involves multi-source heterogeneous data, including structured data (such as case registration forms, legal provision databases), semi-structured data (such as XML format judgment books), and unstructured data (such as legal document texts, volume scanning images, court trial audio and video recordings, etc.). The existing technology faces the following significant challenges when processing such multi-modal data:

[0003] 1. Data heterogeneity leads to low processing efficiency: Traditional systems usually use independent modules to process different modal data, for example, structured data relies on rule engines, text data uses natural language processing technology, and images and audio and video are parsed through special algorithms. However, the lack of a unified semantic representation framework leads to data format incompatibility, difficulty in field standardization, and difficulty in effectively integrating cross-modal information. For example, entities mentioned in legal documents (such as parties, evidence materials) are difficult to automatically associate with signatures in images and key segments in audio and video, and need to rely on manual comparison, which is low in efficiency and prone to errors.

[0004] 2. Missing of cross-modal evidence chain: The fact determination of a judicial case needs to rely on the logical consistency verification of multi-modal evidence. In the existing technology, the spatio-temporal alignment capability of text description, image evidence and audio and video recording is insufficient, and it is difficult to build a complete cross-modal evidence chain. For example, the timestamp in the court trial recording cannot be automatically matched with the legal fact description in the document, which may lead to timeline conflicts or evidence islands, affecting the rigor of case analysis.

[0005] 3. Information silos and security compliance issues: The data sharing mechanism between judicial agencies is not perfect, different systems use independent data standards and interfaces, and there are technical barriers to cross-agency collaboration. At the same time, the desensitization process of sensitive information (such as the privacy of parties) in data sharing involves a lack of dynamic and automated means, making it difficult to balance the compliance requirements of data utilization and privacy protection.

[0006] The above problems seriously restrict the efficiency and accuracy of judicial case analysis. Although existing research attempts to improve data processing capabilities through knowledge graphs or single-modal deep learning models, there are still significant defects in multi-modal alignment, dynamic quality assessment, and cross-agency secure sharing. The present application aims to address these technical pain points and proposes a systematic solution. SUMMARY

[0007] In view of the problems existing in the prior art, the purpose of the present application is to provide a multi-modal data fusion method and system for a judicial scene.

[0008] The technical scheme of the present application is:

[0009] A multi-modal data fusion method for judicial scenarios, the steps of which include:

[0010] 1) Collecting multi-modal data in judicial application scenarios and converting them into data in a unified format;

[0011] 2) Mapping the unified format data obtained in step 1) to a unified semantic space to achieve cross-modal alignment; then correlating the aligned multi-modal data to obtain the text description and corresponding image evidence of the same case as the multi-modal features of the corresponding case;

[0012] 3) Based on the multi-modal features of each case after alignment, constructing a knowledge graph of the judicial application scenario; then driving multi-modal data fusion based on the knowledge graph and constructing a multi-modal evidence chain for each entity;

[0013] 4) Quantifying the completeness of the knowledge graph and the reliability of the multi-modal evidence chain according to the preset indicators, marking and adjusting the abnormal nodes in the knowledge graph according to the quantification results;

[0014] 5) Generating a structured knowledge graph from the adjusted knowledge graph and creating a dynamic desensitization report; the nodes in the structured knowledge graph contain legal entities and their attributes, and the edges in the structured knowledge graph represent the relationships between legal entities,

[0015] Each relationship is bound to a multi-modal evidence chain.

[0016] Further, in step 2), the alignment effect is optimized by a contrast learning strategy, the method of which is:

[0017] 21) Automatically associating the text description and corresponding image evidence in the same case to obtain the text-image pair of the case;

[0018] 22) Using a pre-trained Legal-BERT model to process the legal text passages in the text-image pair, outputting a D-dimensional text embedding vector E t ;

[0019] 23) Using a ViT model to extract the global features E i of the image evidence in the text-image pair and embedding them into a vector space with a dimension of D to obtain a D-dimensional image embedding vector E i ;

[0020] 24) According to the text embedding vector E t and image embedding vector E iThe cosine similarity S(E) of the text-image pair is calculated. t E i If S(E) t E i )≥R th If the text-image pair is found to be associated, it is determined to be an associated text-image pair; otherwise, it is a non-associated text-image pair. th The preset threshold;

[0021] 25) Based on the calculated loss value The Legal-BERT model is optimized to make the similarity of associated text-image pairs higher than that of unassociated text-image pairs; where σ is the temperature coefficient, N is the set of text-image pairs, and S(E t′ E i′ Let E be the text embedding vector corresponding to the non-associated text-image pair (t′, i′). t′ Image embedding vector E i′ The calculated cosine similarity.

[0022] Furthermore, the method for driving multimodal data fusion based on knowledge graphs and constructing a multimodal evidence chain for each entity is as follows:

[0023] 31) Extract legal entities from the multimodal features of each case, and attach a multimodal source tag to each entity;

[0024] 32) Utilize graph convolutional network algorithm models to identify potential relationships between entities, and combine them with predefined relationship types in the legal knowledge base to generate initial relationships between entities;

[0025] 33) Perform cross-modal verification on each of the initial relationships. If the verification passes, calculate the evidence chain confidence of the initial relationship to obtain a set of legal relationships. Each verified initial relationship in the set of legal relationships is accompanied by an associated multimodal evidence index, and high-confidence initial relationships are automatically selected and injected into the knowledge graph based on the confidence threshold. For initial relationships with evidence chain confidence lower than the confidence threshold or with conflicts, manual review is performed before injection into the knowledge graph.

[0026] 34) Construct a multimodal evidence chain for each entity based on the knowledge graph obtained in step 33).

[0027] Furthermore, the method for calculating the confidence level of the evidence chain for this initial relationship is as follows: First, calculate the timestamp t in the court hearing video. v With the time description in the document paragraph t t The error between them is used to determine the multimodal evidence for each initial relationship; then through Calculate the initial relation r i Evidence Chain Confidencei ); wherein, S(r i , ·) is the initial relation r i and the similarity score of the multi-modal evidence, θ is the similarity threshold, is the jth text data in the multi-modal evidence, is the kth image data in the multi-modal evidence, is the mth audio data in the multi-modal evidence.

[0028] Further, the timestamp t v in the court trial video and the time description t t in the document paragraph are calculated, and the error Δ(t v , t t ) is calculated, if Δ(t v , t t ) < ∈, ∈ is a predetermined defined threshold, then it is determined that the court trial video and the document paragraph are the same event evidence; the similarity score S(r, e) of the relation and the evidence is calculated S(r, e) = α·S text-image (r, e) + β·S text-av (r, e) + γ·S time (r, e) - σ·ConflictPenalty(r, e), if S(r, e) is greater than the similarity threshold θ, then the entity behavior in the text paragraph r has a dynamic association with the signature area in the image evidence and the key segment in the audio / video e; wherein, S text-image is the text-image semantic cosine similarity, S text-av (r, e) is the text-audio / video content consistency, S time (r, e) is the quantization error of the text description time and the audio / video timestamp, ConflictPenalty(r, e) is the contradiction penalty between different modal evidences, α, β, γ, σ are weight coefficients, and satisfy α + β + γ = 0.9.

[0029] Further, τ is the attenuation coefficient, t text,r is the text detection timestamp, t av,e is the audio / video timestamp.

[0030] Further, the multi-modal data includes structured data, semi-structured data and unstructured data; first, the multi-modal data is preprocessed; wherein for structured data, field standardization is performed on structured data through regular expressions and rule engines, and missing fields are completed based on a legal knowledge base; for unstructured text data, a segmented semantic parsing model is used to segment the text data by paragraphs, and a semantic tag is labeled for each paragraph; for semi-structured image, audio and video data, feature extraction is performed on image data through a multi-scale convolutional network, and audio data is converted into timestamped text using speech recognition technology; then, the preprocessed data is mapped to a unified semantic space to realize cross-modal alignment, and unified format data is obtained.

[0031] A multi-modal data fusion system for a judicial scene, characterized in that it comprises a multi-modal judicial data preprocessing and feature representation module, a judicial knowledge graph driven multi-modal data fusion module, a dynamic quality evaluation and adaptive optimization module, and a structured output module.

[0032] The multi-modal judicial data preprocessing and feature representation module is used to collect multi-modal data in a judicial application scene and convert it into data in a unified format.

[0033] The judicial knowledge graph driven multi-modal data fusion module is used to map the obtained data in a unified format to a unified semantic space to realize cross-modal alignment; then, the aligned multi-modal data is associated to obtain the text description and corresponding image evidence of the same case as the multi-modal features of the corresponding case; then, based on the multi-modal features of each case after alignment, a knowledge graph of the judicial application scene is constructed; then, the multi-modal data fusion is driven based on the knowledge graph and the multi-modal evidence chain of each entity is constructed.

[0034] The dynamic quality evaluation and adaptive optimization module quantifies the integrity of the knowledge graph and the reliability of the multi-modal evidence chain according to a preset index, and marks and adjusts the abnormal nodes in the knowledge graph according to the quantification result.

[0035] The structured output module is used to generate a structured knowledge graph and create a dynamic desensitization report according to the adjusted knowledge graph; the nodes in the structured knowledge graph include legal entities and their attributes, and the edges in the structured knowledge graph represent the relationships between legal entities, and each relationship is bound to a multi-modal evidence chain.

[0036] A server, characterized in that it comprises a memory and a processor, the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing the above method.

[0037] A computer readable storage medium, wherein a computer program is stored, and the computer program is executed by a processor to implement the method.

[0038] Advantages of the present application are as follows:

[0039] The present application significantly improves the efficiency and accuracy of judicial data processing through multi-modal data preprocessing, cross-modal semantic alignment, judicial knowledge graph driven fusion and dynamic optimization mechanism. The present application solves the problem of data heterogeneity by mapping to a unified semantic space, realizes the standardized integration of structured, text, image and audio / video evidence; the cross-modal evidence chain constructed based on the spatiotemporal consistency algorithm enhances the logical verification ability of legal facts, ensures the rigor of case analysis; the dynamic quality evaluation and self-adaptive parameter optimization mechanism can detect and repair multi-modal conflicts in real time, improve the credibility of the fusion results; at the same time, the cross-agency collaborative interface and dynamic desensitization function break down the information silos, support efficient collaboration of multiple agencies, and ultimately provide transparent, traceable and full-dimensional evidence support for judicial decision-making, greatly reducing the labor cost and misjudgment risk. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 is a multi-modal data fusion method processing schematic diagram for the judicial application scenario of the present application.

[0041] Figure 2 is a multi-modal data fusion system framework schematic diagram for the judicial application scenario. DETAILED DESCRIPTION

[0042] The present application will be further described in detail below with reference to the accompanying drawings, and the examples are only used to explain the present application, and not to limit the scope of the present application.

[0043] The present application provides a multi-modal data fusion method and system for the judicial scene, which realizes multi-modal data preprocessing through the normalization processing of structured data, semantic segmentation of unstructured text, and feature extraction of images and audio / video, maps different modal data to a unified semantic space after preprocessing to realize cross-modal alignment, and realizes the conversion of all data into a unified format; based on the aligned multi-modal features, a knowledge graph for the judicial application scenario is constructed, multi-modal fusion driven by the judicial knowledge graph is realized, and the text, image and audio / video evidence chain is associated through the spatiotemporal consistency algorithm; dynamic quality evaluation indicators and self-adaptive parameter optimization mechanism are designed to improve the multi-modal data fusion effect through closed-loop iteration; finally, the visual knowledge graph, cross-agency collaborative data interface and dynamic desensitization report are output, solving the problems of judicial data heterogeneity, information silos and security compliance, realizing the efficiency improvement of case analysis and cross-agency data security sharing.

[0044] The processing flow of the present application is as follows Figure 1As shown, the steps include:

[0045] Step one, multi-modal data input and preprocessing, collect multi-modal data in judicial application scenarios, according to the different types of data, respectively using regular expression, segmented semantic analysis model, etc. All data are converted into a unified format. First, collect multi-modal data in the judicial scene, including structured data, semi-structured data and unstructured data. For structured data, field standardization is performed through regular expressions and rule engines, and missing fields are automatically completed based on the legal knowledge base. For unstructured text data, segmented semantic analysis model is used to divide the text data into "fact determination", "legal basis" and other types according to paragraphs, and semantic tags are labeled for each paragraph. For image and audio / video data, key features are extracted through multi-scale convolutional networks, such as signature and seal areas in the file image, or court recording is converted into timestamped text using speech recognition technology. After preprocessing, all data of different types are converted into a unified format, laying a foundation for subsequent feature extraction and fusion.

[0046] In view of the characteristics of complex data types, heterogeneous formats in the judicial scene(including structured data, semi-structured text, image / audio / video unstructured data), the hierarchical preprocessing mechanism is optimized:

[0047] For structured data, regular expressions and rule engines are used for field standardization processing, and a unified data model is defined:

[0048] D struct ={(f i ,v i )|f i ∈F legal ,v i ∈ValueDomain(f i )} (1)

[0049] Where F legal is a set of legal data fields, and ValueDomain(f i ) is a field value domain constraint.

[0050] Then use regular expression to match field name f i , check whether the field value v i conforms to the pre-defined value domain ValueDomain(f i ), if the field is missing, automatically complete according to the context, then output the standardized structured data set D struct , and each field conforms to the relevant legal norms.

[0051] For unstructured text data, a segmented semantic parsing model is used to capture the context-dependent relationship of text sequences through a bidirectional long short-term memory network, identify the potential start and end positions of the paragraph, and then globally optimize the label sequence output by the bidirectional long short-term memory network in the conditional random field layer to constrain the rationality and continuity of the label. Finally, the original text is divided into semantically complete paragraphs, and each paragraph is labeled with a type label, and the labeled paragraph set Segment(T) is output, and each paragraph contains content and semantic labels, thereby realizing fine-grained structural parsing of legal documents:

[0052]

[0053] wherein T is the original text, s j is a semantic paragraph, t j is a paragraph type label.

[0054] For image / audio / video evidence data, a multi-scale feature extraction network based on attention mechanism is constructed to identify the overall layout at a low resolution level, and then focus on the detail area at a high resolution level, and dynamically weight the features of different scales through the attention mechanism to highlight the key information of legal effectiveness. The multi-modal feature set F image is output, which has semantic description and spatio-temporal direction:

[0055]

[0056] wherein I is the input image, a k is the attention weight of the kth scale, ConvBlock k is the convolution module corresponding to the scale.

[0057] For audio / video evidence, speech recognition technology is used to convert the court trial recording into a timestamped text record, and speaker voiceprint features are extracted simultaneously to distinguish judges, plaintiffs, defendants and other roles. At the same time, key frames are extracted from the video stream, and the timestamp, role identity, text content and visual features are bound into structured metadata to ensure the spatio-temporal consistency of audio / video evidence and document description:

[0058]

[0059] wherein M av is the output audio structured metadata set, t v is the timestamp, marking the start time of the voice segment, SpeakerID is the speaker identity label, and Transcript is the text content of the voice transcription.

[0060] Step two, cross-modal feature representation and alignment. After pre-processing, different modal data is mapped to a unified semantic space to achieve cross-modal data alignment, and the alignment effect is optimized through a contrast learning strategy. By retrieving some keywords (such as plaintiff's name, defendant's name, or case number, or judge's name, etc.) identified in the text, the automatic association of text description and corresponding image evidence in the same case is realized as the multi-modal features of the corresponding case. At the same time, the cosine similarity of the text-image pair is calculated, and the model parameters are dynamically adjusted to pull the distance of related features and push away irrelevant features. Finally, different modal data forms a closely related vector distribution in the shared semantic space, thereby constructing the logical consistency of the cross-modal evidence chain and providing multi-dimensional fusion support for the underlying support of the judicial knowledge graph.

[0061] The pre-trained Legal-BERT model is used to process the legal text segment, and the output semantic vector has a dimension of D:

[0062] E t = Legal-BERT(s j ) s j ∈Segment(T) (5)

[0063] The Vision Transformer (ViT) model algorithm is used to extract the global features E i of the image, and the text embedding is mapped to the same vector space with a dimension of D:

[0064] E i = Proj(ViT(I i )) (6)

[0065] Then the cosine similarity of the text-image pair is calculated, and the associated data is screened

[0066]

[0067] If S(t,i)≥R th , it is determined as an associated pair, where R th is a pre-set threshold.

[0068]

[0069] where j is the jth input text segment, I i is the ith input image, E t is the tth text embedding vector mapped to a vector with a dimension of D by Segment(T), and E i is the ith image embedding vector mapped to a vector with a dimension of D by F imagewhere S(·) is the cosine similarity function, σ is the temperature coefficient, N is the set of randomly sampled non-correlated text-image pairs, and S(E t′ ,E i′ ) is the cosine similarity function of randomly sampled non-correlated text-image pairs, and L align is the loss function of the Legal-BERT model, which makes the similarity of the correlated pairs significantly higher than that of the non-correlated pairs.

[0070] Step three, use the knowledge graph to drive the fusion of multi-modal data in the judicial field, and based on the aligned multi-modal features, construct the multi-modal evidence chain of the judicial application scene. Receive the standardized data processed in step two, and then perform legal entity extraction: structured data directly extracts predefined fields as entities; text data parses the context semantics of the paragraph through a pre-trained legal domain language model to identify potential entities; image / audio / video data provides cross-modal verification, the signature area features in the image are matched with the names of the parties mentioned in the text; the speaker identity of the audio / video is associated with the description of the law in the text; finally, output the legal entity set, each entity is attached with a multi-modal source label.

[0071] The processing process of entity relationship extraction is based on pre-processed structured data, text semantic vectors and multi-modal evidence. First, the graph convolution network (GCN) algorithm model is used to identify the potential association between entities from the text semantics, and the initial relationship hypothesis is generated in combination with the relationship types predefined by the legal knowledge base.

[0072] Subsequently, the initial relationship hypothesis in the text is verified cross-modally — if the "borrowing" relationship in the text is spatiotemporally aligned with the image contract signature and the audio recognition segment, as shown in equation (9), then the evidence chain association strength (confidence) of the initial relationship is calculated, as shown in equations (10) and (11); finally, the weighted legal relationship set is output, each relationship is attached with the index of the associated multi-modal evidence, and according to the confidence threshold, high-confidence relationships are injected into the knowledge graph, and low-confidence or conflicting relationships are pushed to the artificial review queue, ensuring the rigor of legal fact analysis and the integrity of the evidence chain.

[0073] Finally, based on the fusion results, a multi-modal evidence chain with spatio-temporal consistency is generated to ensure the traceability of judicial facts. The above-mentioned successfully associated multi-modal evidence (such as text summary, signature image index, audio-video key segment) is integrated, the time description in the text (such as "signed on May 1, 2023") is extracted, the audio-video timestamp and image metadata (such as the scanning date) are calculated, the time error is calculated (if within the threshold, it is considered as the same event), and the related evidence is bound to form a structured evidence chain unit (for example: "loan signing event chain" contains: text paragraph ID, image region coordinates, audio-video start and end timestamp), which is mounted to the graph as an independent node or relationship attribute. For example, the relationship edge "Zhang San-loan-Li Si" binds the "loan signing evidence chain", supporting one-key retrieval of full-dimensional evidence when the case is traced back. The integrity and logical consistency of the evidence chain are verified in real time by the graph (such as avoiding conflict citation of the same law). Finally, the knowledge graph output contains entities, relationships, and bound multi-modal evidence chains, forming a complete case spatio-temporal logical view.

[0074] The timestamp t v in the courtroom video t , calculate its error:

[0075] Δ(t v ,t t ) =‖TS-Encoder(t v )-TS-Encoder(t t )‖ 2 (9)

[0076] If Δ(t v ,t t )<∈, ∈ is a predefined threshold, then it is determined as the same event evidence.

[0077] Where TS-Encoder is the encoding of the video timestamp t v , and TS-Encoder(t t ) is the encoding of the time description t t in the document paragraph.

[0078] The strength (confidence) of each legal fact associated evidence associated multi-modal evidence chain is as follows:

[0079]

[0080] Where S(r i ,·) is the similarity score of the initial relationship and multi-modal evidence, and θ is the similarity threshold. r i is the i-th multi-modal-based relationship, is the j-th input text data, is the k-th image data, For the mth audio data, if S(r i ,·)≥θ, it indicates that the multi-modal data is a strong correlation evidence chain.

[0081] The similarity score formula of the application is used to quantify the correlation strength of legal relationship and multi-modal evidence, which integrates text, image, audio and video, and time consistency, and the formula is defined as follows:

[0082] S(r,e)=α·S text-image (r,e)+β·S text-av (r,e)+γ·S time (r,e)-σ·ConflictPenalty(r,e) (11)

[0083] Wherein, r is text data, e is other data, S text-image is text-image semantic cosine similarity, S text-av (r,e) is text-audio and video content consistency, and the calculation formula is as follows:

[0084]

[0085] Wherein ‖E t ‖ is the modulus of the text data vector, and ‖E av ‖ is the modulus of the audio data vector. S time (r,e) is the quantization error of text description time and audio and video timestamp, and the calculation formula is as follows:

[0086]

[0087] τ is the attenuation coefficient. t text,r is the text detection timestamp, t av,e is the audio and video timestamp.

[0088] ConflictPenalty(r,e) is the conflict penalty between different modal evidence,

[0089]

[0090] α,β,γ,σ are weight coefficients, and satisfy α+β+γ=0.9.

[0091] Step four, dynamic quality assessment and parameter optimization, real-time monitoring of the results of multi-modal data fusion, and optimizing the fusion results through dynamic quality assessment and feedback mechanism. First, the completeness of the knowledge graph and the reliability of the evidence chain are quantitatively evaluated in real time through pre-set multi-dimensional indicators. Specifically including: 1) graph integrity index, statistics of key entity (such as the party, evidence goods) coverage and relationship edge (such as lending, infringement) logical completeness; 2) evidence chain reliability index, calculation of multi-modal evidence spatio-temporal consistency (such as the error value of the description time of the document and the time stamp of the audio and video) and semantic matching degree (such as the cosine similarity of text-image features). When the quantitative results exceed the pre-set threshold (for example, the same law is cited in conflict in different evidence chains, or the time stamp error exceeds the allowed range), it is determined that the quality of the multi-modal data fusion result is low, and the subsequent processing process is triggered.

[0092] Then when the quality of multi-modal data fusion is detected to be low, the entities or evidence involved in the low-quality fusion are highlighted in the knowledge graph, and then the low-quality subgraph is frozen and a to-be-audited task queue is generated, which is manually intervened to check the original evidence (such as comparing the scanned copy of the volume with the trial video). After the audit is passed, the graph is updated according to the correction results—if the detection result is true, delete the invalid relationship or supplement the missing evidence; if it is a false alarm, remove the abnormal mark. Finally, output a logically consistent version of the knowledge graph, and ensure the spatio-temporal closure of the evidence chain.

[0093] Finally, the credibility of different data sources (i.e. pre-processed multi-modal input channels such as structured fields, text semantic vectors, image / audio / video features) is adapted. The system dynamically calculates the credibility of each modality based on historical fusion performance: 1) statistics of conflict frequency, record the alignment error rate of each modality in historical tasks (such as the image signature recognition error rate of a certain court reaching 20%); 2) dynamic adjustment of weight coefficient, automatically reduce the contribution weight of low-quality modality in the fusion formula according to the error rate (for example, the weight of the frequently error-prone image modality α is reduced from 0.4 to 0.1), while improving the weight of high-credibility modality (such as accurate structured data). And update the modality error rate library after each conflict resolution, continuously calibrate the weight coefficient, so as to improve the robustness of the method provided by the application.

[0094] Step five, output and application, the system outputs the structured knowledge graph, the cross-agency collaborative sharing data interface and the dynamic de-sensitization report. The knowledge graph displays the case entities, legal logical relationships and multi-modal evidence chains in a visual form, supporting the judge to quickly locate the key information. The cross-agency collaborative sharing data interface is encapsulated through the API data interface or the data interface based on the federated learning framework, allowing cross-agency sharing of multi-modal fusion data without leaking the original data, and breaking the information silos. The dynamic de-sensitization report automatically processes the de-sensitized information according to user demands, such as the grassroots court submitting a case summary to the higher court through the report generated by the system while protecting the privacy of the parties. This process not only improves the efficiency of judicial data processing, but also strictly meets the data security and compliance requirements.

[0095] The application also provides a multi-modal data fusion system for a judicial scenario, mainly composed of an input, a multi-modal judicial data preprocessing and feature representation module, a multi-modal data fusion module driven by a judicial knowledge graph, a dynamic quality evaluation and self-adaptive optimization module, and a structured output module, as shown in Figure 2 .

[0096] The multi-modal judicial data preprocessing and feature representation module, for structured data, the system checks the field legality through regular expressions and legal knowledge base. Unstructured text is segmented and annotated by a segmented semantic parsing model to identify "fact determination" and "legal basis" paragraphs. A pre-trained Legal-BERT model is used to process legal text paragraphs, outputting a semantic vector with dimension D. Image data is extracted through a multi-scale feature extraction network based on an attention mechanism to identify the overall layout at a low resolution level, and then focus on signature and seal details at a high resolution level. The multi-scale features are dynamically weighted through an attention mechanism to generate a D-dimensional image vector. Audio and video data are converted into timestamped text through speech recognition, and the speaker's identity is distinguished through voiceprint recognition. Key frames are extracted from the video to form structured metadata that binds timestamps, roles, text, and visual features.

[0097] The normalized multi-modal data is mapped to a unified semantic space to achieve cross-modal correlation. After preprocessing, different modal data is mapped to a unified semantic space to achieve cross-modal data alignment, and the alignment effect is optimized through a contrast learning strategy. The text description and corresponding image evidence in the same case are automatically associated, and the cosine similarity of the text-image pair is calculated to dynamically adjust the model parameters to close the distance between related features and push away irrelevant features. Finally, different modal data forms a closely related vector distribution in the shared semantic space, outputting standardized structured data, text semantic vectors, image / audio / video feature vectors and cross-modal correlation labels, providing multi-dimensional and highly consistent input for knowledge graph construction.

[0098] The judicial knowledge graph driven multi-modal data fusion module is based on pre-processed multi-modal features, and realizes deep fusion of multi-dimensional data through legal entity and relationship extraction, evidence chain construction and dynamic optimization. First, the module extracts legal entities from text semantic vectors, structured fields and image / audio / video features, and improves entity credibility through cross-modal verification, outputs a legal entity set, and each entity is attached with a cross-modal evidence label. Then based on semantic similarity and feature matching technology, the entity behavior in the text paragraph is dynamically associated with the signature area in the image, the key segment in the audio / video, and the logical integrity of the evidence chain is verified through a cross-modal confidence weighting model; if the multi-modal evidence is highly consistent, a strong correlation evidence chain is automatically generated and injected into the knowledge graph, if a conflict is detected, an abnormal node is marked and an artificial review process is triggered; finally, the multi-modal evidence chain is constructed, and the text legal fact description is accurately matched with the timestamp of the audio / video and the signing date in the image through a time and space alignment algorithm, ensuring the consistency of the event timeline.

[0099] The dynamic quality evaluation and adaptive optimization module further optimizes the fusion results through a dynamic quality evaluation and feedback mechanism. First, the completeness of the knowledge graph and the reliability of the evidence chain are quantitatively evaluated in real time through pre-set multi-dimensional indicators, then when the multi-modal data fusion quality is detected to be low, the entities or evidence involved in the low-quality fusion are highlighted in the knowledge graph, then the low-quality subgraph is frozen and a to-be-reviewed task queue is generated, and the original evidence is checked by manual intervention; finally, the credibility of different data sources is adapted, and the modal error rate library is updated after each conflict resolution, continuously calibrating the weight coefficients, thereby improving the robustness of the method provided by the present application.

[0100] The structured output module displays case entities, legal logical relationships and multi-modal evidence chains in a visual form, supporting judges to quickly locate key information. The cross-agency collaborative data sharing interface is encapsulated in an API data interface or a data interface based on a federated learning framework, allowing cross-agency sharing of multi-modal fusion data without leaking original data, breaking down information silos. The dynamic desensitization report automatically processes desensitized information according to user demands, such as a grassroots court submitting a case summary to a higher court through a report generated by the system while protecting the privacy of the parties.

[0101] Although specific embodiments of the present application are disclosed for the purpose of illustrating the present application, the purpose being to facilitate understanding of the present application and to enable its implementation, it will be understood by those skilled in the art that various substitutions, changes and modifications are possible without departing from the spirit and scope of the present application and the appended claims. Therefore, the present application should not be limited to the disclosed content of the best mode, and the scope of the present application claimed is defined by the scope of the claims.

Claims

1. A multimodal data fusion method for judicial scenarios, comprising the following steps: 1) Collect multimodal data from judicial application scenarios and convert it into data in a unified format; 2) Map the data in the unified format obtained in step 1) to a unified semantic space to achieve cross-modal alignment; then associate the aligned multimodal data to obtain the text description and corresponding image evidence of the same case as the multimodal features of the corresponding case; 3) Based on the multimodal features of each aligned case, a knowledge graph of judicial application scenarios is constructed; Then, based on the knowledge graph, multimodal data fusion is driven and a multimodal evidence chain for each entity is constructed; 4) Quantify the completeness of the knowledge graph and the reliability of the multimodal evidence chain according to preset indicators, mark the abnormal nodes in the knowledge graph according to the quantification results and adjust them accordingly; 5) Generate a structured knowledge graph based on the adjusted knowledge graph and create a dynamic anonymization report; the nodes in the structured knowledge graph contain legal entities and their attributes, the edges in the structured knowledge graph represent the relationships between legal entities, and each relationship is bound to a multimodal evidence chain.

2. The method according to claim 1, characterized in that, In step 2), the alignment effect is optimized through a contrastive learning strategy, the method of which is as follows: 21) Automatically associate text descriptions with corresponding image evidence in the same case to obtain text-image pairs for that case; 22) Process the legal text paragraphs in the text-image pair using a pre-trained Legal-BERT model, and output a D-dimensional text embedding vector E. t ; 23) Use the ViT model to extract the global features E of the image evidence in the text-image pair. i The image is then embedded and mapped to a vector space of dimension D, resulting in a D-dimensional image embedding vector E. i ; 24) Based on the text embedding vector E corresponding to the text-image pair t Image embedding vector E i The cosine similarity S(E) of the text-image pair is calculated. t E i If S(E) t E i )≥R th If the text-image pair is found to be associated, it is determined to be an associated text-image pair; otherwise, it is a non-associated text-image pair. th The preset threshold; 25) Based on the calculated loss value The Legal-BERT model is optimized to make the similarity of associated text-image pairs higher than that of unassociated text-image pairs; where σ is the temperature coefficient, N is the set of text-image pairs, and S(E t′ E i′ Let E be the text embedding vector corresponding to the non-associated text-image pair (t′, i′). t′ Image embedding vector E i′ The calculated cosine similarity.

3. The method according to claim 1, characterized in that, The method for knowledge graph-driven multimodal data fusion and constructing a multimodal evidence chain for each entity is as follows: 31) Extract legal entities from the multimodal features of each case, and attach a multimodal source tag to each entity; 32) Utilize graph convolutional network algorithm models to identify potential relationships between entities, and combine them with predefined relationship types in the legal knowledge base to generate initial relationships between entities; 33) Perform cross-modal verification on each of the initial relationships. If the verification passes, calculate the evidence chain confidence of the initial relationship to obtain a set of legal relationships. Each verified initial relationship in the set of legal relationships is accompanied by an associated multimodal evidence index, and high-confidence initial relationships are automatically selected and injected into the knowledge graph based on the confidence threshold. For initial relationships with evidence chain confidence lower than the confidence threshold or with conflicts, manual review is performed before injection into the knowledge graph. 34) Construct a multimodal evidence chain for each entity based on the knowledge graph obtained in step 33).

4. The method according to claim 3, characterized in that, The method for calculating the confidence level of the evidence chain for this initial relationship is as follows: First, calculate the timestamp t in the court hearing video. v With the time description in the document paragraph t t The error between them is used to determine the multimodal evidence for each initial relationship; then through Calculate the initial relation r i Evidence Chain Confidence i ); where S(r) i ,·) represents the initial relation r i Similarity score with multimodal evidence, where θ is the similarity threshold. For the j-th text data in multimodal evidence, For the k-th image data in multimodal evidence, Let m be the number of the m-th audio signal in the multimodal evidence.

5. The method according to claim 4, characterized in that, The timestamp t in the court hearing video v With the time description in the document paragraph t t Calculate the error Δ(t) v ,t t If Δ(t) v ,t t If ) < ∈, where ∈ is a predefined threshold, then the court hearing video and the document paragraph are determined to be evidence of the same event; calculate the similarity score between the relationship and the evidence S(r,e)=α·S text-image (r,e)+β·S text-av (r,e)+γ·S time (r,e)-σ·ConflictPenalty(r,e), if S(r,e) is greater than the similarity threshold θ, then the entity behavior in text segment r is dynamically associated with the signature region in image evidence and the key segments in audio / video e; where S text-image For text-image semantic cosine similarity, S text-av (r,e) represents the consistency of textual audio and video content, S time (r,e) represents the quantization error between the time of the text description and the audio / video timestamps, ConflictPenalty(r,e) represents the conflict penalty between different modal evidence, and α, β, γ, σ are weighting coefficients that satisfy α+β+γ=0.

9.

6. The method according to claim 5, characterized in that, τ is the attenuation coefficient, t text,r For text detection timestamps, t av,e For audio and video timestamps.

7. The method according to claim 1, 2, or 3, characterized in that, The multimodal data includes structured data, semi-structured data, and unstructured data. First, the multimodal data undergoes preprocessing. For structured data, regular expressions and a rule engine are used to standardize fields, and missing fields are filled in based on a legal knowledge base. For unstructured text data, a segmented semantic parsing model is used to divide the text data into paragraphs, and each paragraph is labeled with a semantic tag. For semi-structured image, audio, and video data, a multi-scale convolutional network is used to extract features from the image data, and speech recognition technology is used to convert the audio data into timestamped text. Then, the preprocessed data is mapped to a unified semantic space to achieve cross-modal alignment, resulting in data in a unified format.

8. A multimodal data fusion system for judicial scenarios, characterized in that, It includes a multimodal judicial data preprocessing and feature representation module, a judicial knowledge graph-driven multimodal data fusion module, a dynamic quality assessment and adaptive optimization module, and a structured output module; The multimodal judicial data preprocessing and feature characterization module is used to collect multimodal data in judicial application scenarios and convert it into data in a unified format. The judicial knowledge graph-driven multimodal data fusion module is used to map the obtained unified format data to a unified semantic space to achieve cross-modal alignment; then, the aligned multimodal data are associated to obtain the text description and corresponding image evidence of the same case as the multimodal features of the corresponding case. Then, based on the multimodal features of each aligned case, a knowledge graph of judicial application scenarios is constructed. Then, based on the knowledge graph, multimodal data fusion is driven and a multimodal evidence chain for each entity is constructed; The dynamic quality assessment and adaptive optimization module quantifies the completeness of the knowledge graph and the reliability of the multimodal evidence chain based on preset indicators, and marks and adjusts abnormal nodes in the knowledge graph based on the quantification results. The structured output module is used to generate a structured knowledge graph based on the adjusted knowledge graph and create a dynamic desensitization report; the nodes in the structured knowledge graph contain legal entities and their attributes, the edges in the structured knowledge graph represent the relationships between legal entities, and each relationship is bound to a multimodal evidence chain.

9. A server, characterized in that, It includes a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the method of any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images