Similar case recommendation method and system

Abstract

The present invention provides a method and system for recommending similar cases. The method obtains multimodal data of various historical cases in procuratorial work; preprocesses and extracts features of the multimodal data using an LMM model and an LLM model to obtain an element set of each historical case; constructs a time series hypergraph of each historical case; converts the time series hypergraph into a comprehensive embedding vector, and constructs a database of historical cases and corresponding comprehensive embedding vectors; obtains a query case and the corresponding comprehensive embedding vector, calculates the similarity between the embedding vector of the query case and the comprehensive embedding vectors of each historical case in the database, and pushes the historical case with the highest similarity score to the user, thereby achieving more accurate, more reliable, and more explanatory similar case recommendations and effectively assisting users in their case handling work.

Description

Technical Field

[0001] The present invention belongs to the technical field of similar case recommendation, and in particular relates to a similar case recommendation method and system. Background Art

[0002] Procuratorial work involves the review, judgment, and handling of numerous complex cases. During case handling, referencing and drawing lessons from previous similar cases ("similar cases") is crucial for standardizing legal application standards, ensuring judicial fairness, and improving case handling efficiency. However, existing similar case recommendation technology has significant limitations when applied to the complex and ever-changing procuratorial work scenarios.

[0003] Currently, mainstream technologies for similar case recommendation include: 1. Text similarity-based techniques, which are currently the most widely used. These methods first perform text preprocessing (such as word segmentation and stop word removal) on legal documents (such as indictments, verdicts, review reports, and interrogation transcripts), and then use different text representation models to extract features. 2. Techniques based on simple graph structures attempt to model the structured information in a case. These methods typically use identified entities (such as parties, lawyers, judges, locations, and specific items) as nodes in the graph, and simple relationships between entities (such as co-occurrence in the same case, the same litigation phase, or direct interaction) as edges. For example, if defendants A and B jointly participated in a crime, an edge is established between A and B. Once the graph is constructed, graph matching algorithms (such as subgraph isomorphism) or graph embedding techniques (such as DeepWalk, Node2Vec, and GCN - Graph Convolutional Network) are used to learn node or graph representations, and then similarity is calculated. Third, traditional machine learning-based techniques rely on domain experts or technicians to manually design and extract a series of variables (features) that describe case characteristics. These features may include: case type, charges involved, number of parties involved, amount of money involved, key details (such as whether the case surrendered voluntarily or was a repeat offender), and evidence type statistics. These features are then used to train classification models (such as SVM, Random Forest, LogisticRegression) or ranking models (such as RankSVM, LambdaMART) to predict the similarity between two cases or calculate similarity scores.

[0004] The defects of the above-mentioned technologies are that they are unable to effectively integrate and utilize various forms of data and evidence such as text, images, audio and video that are commonly found in cases, resulting in incomplete case information representation; it is difficult to effectively model and analyze high-order relationships involving multiple entities in the case (such as joint crimes, multi-party contract disputes, and complex chains of evidence); they ignore the dynamic process of the evolution of case elements, events, and evidence presentation over time, and are unable to analyze the similarity of temporal patterns; they mainly rely on surface text information or predefined simple structures, making it difficult to deeply explore the inherent legal logic, factual structure, and deep semantic similarities of the case. Summary of the Invention

[0005] Based on this, an embodiment of the present invention provides a similar case recommendation method and system, which aims to solve the problem in the existing technology that the accuracy, comprehensiveness and intelligence of similar case recommendations are limited, and it is difficult to meet the urgent needs of users for high-quality similar case references in complex case handling.

[0006] A first aspect of an embodiment of the present invention provides a similar case recommendation method, which is applied to a prosecutorial industry scenario. The method includes:

[0007] Acquire multimodal data of each historical case in procuratorial work, wherein the multimodal data includes at least text data, image data, audio data, video data, and structured metadata;

[0008] LMM and LLM models are used to preprocess and extract features from multimodal data to obtain an element set for each historical case. The element set includes at least an entity list, an event list, time information, relationship information, and feature embeddings of each multimodal data.

[0009] Constructing a temporal hypergraph for each historical case, wherein target elements in the element set are mapped to nodes of the temporal hypergraph, hyperedges between the nodes are defined, and time attributes between the hyperedges are determined;

[0010] Converting the temporal hypergraph into a comprehensive embedding vector and constructing a database of historical cases and corresponding comprehensive embedding vectors;

[0011] Obtain the query case and the corresponding comprehensive embedding vector, calculate the similarity between the embedding vector of the query case and the comprehensive embedding vector of each historical case in the database, and push the historical case with the highest similarity score to the user.

[0012] Furthermore, in the step of using the LMM model and the LLM model to preprocess and extract features from the multimodal data to obtain the element set of each historical case, the text data is subjected to entity recognition, relationship extraction, event extraction, semantic role labeling, time information normalization, text summary generation, text embedding, and instruction-driven information extraction;

[0013] Performing target detection and recognition, image description generation, key information extraction, visual feature embedding, and cross-modal alignment on the image data;

[0014] Performing automatic speech recognition, speaker recognition, sentiment analysis, keyword extraction, and audio feature embedding on the audio data;

[0015] Performing spatiotemporal behavior recognition, target tracking, scene understanding, video description, and video feature embedding on the video data;

[0016] performing encoding processing on the structured metadata;

[0017] The result of entity recognition constitutes the entity list, the result of relationship extraction obtains the relationship information, the result of event extraction constitutes the event list, and the result of time information normalization constitutes the time information.

[0018] Furthermore, the hyperedge is used to represent a high-order relationship between nodes, and a hyperedge can connect two or more nodes.

[0019] Furthermore, the step of converting the temporal hypergraph into a comprehensive embedding vector includes:

[0020] Extracting global structural features of the temporal hypergraph and calculating the Laplacian matrix;

[0021] Performing eigendecomposition on the Laplace matrix and determining a preset number of eigenvectors to constitute a spectral embedding of a node;

[0022] Using the node attributes of the temporal hypergraph and / or the spectral embedding as input to a hypergraph neural network model, and obtaining a final node embedding after propagation through the hypergraph neural network model;

[0023] The nodes are embedded and aggregated by a graph readout function to obtain an embedding vector of the entire temporal hypergraph;

[0024] The temporal hypergraph is regarded as a series of snapshots at discrete time points, an embedding vector is calculated for each snapshot, and then a sequence model is used to learn the representation of the entire sequence to obtain the temporal embedding;

[0025] The spectrum embedding, the node embedding, and the time series embedding are fused to obtain the comprehensive embedding vector.

[0026] Furthermore, in the step of fusing the spectral embedding, the node embedding, and the temporal embedding to obtain the comprehensive embedding vector, one of splicing, weighted averaging, or attention mechanism is used for fusion.

[0027] Furthermore, the steps of obtaining the query case and the corresponding comprehensive embedding vector and calculating the similarity between the embedding vector of the query case and the comprehensive embedding vectors of each historical case in the database include:

[0028] Calculate the cosine similarity between the comprehensive embedding vector of the query case and the comprehensive embedding vectors of each historical case in the database;

[0029] Based on the temporal hypergraph of the query case and the temporal hypergraphs of each historical case in the database, the structural similarity of the two temporal hypergraphs at the structural level is calculated;

[0030] Based on the time series hypergraph of the query case and the time series hypergraph of each historical case in the database, the normalized distance between the time points of similar events is calculated to obtain the time pattern similarity;

[0031] The cosine similarity, the structural similarity and the temporal pattern similarity are weightedly fused to obtain a final similarity.

[0032] Furthermore, the operation of the core convolutional layer in the hypergraph neural network model is expressed as:

[0033] ;

[0034] in, is the feature representation matrix of all nodes in the lth layer, d l is the feature dimension, is the weight matrix of the lth layer, is a nonlinear activation function, D v is the node degree diagonal matrix, D e is the hyperedge degree / weight diagonal matrix, and B is the incidence matrix.

[0035] A second aspect of an embodiment of the present invention provides a similar case recommendation system for implementing the similar case recommendation method described in the first aspect, the system comprising:

[0036] An acquisition module is used to acquire multimodal data of each historical case in the prosecution business, wherein the multimodal data includes at least text data, image data, audio data, video data and structured metadata;

[0037] A data processing module is used to preprocess and extract features from multimodal data using LMM and LLM models to obtain an element set for each historical case. The element set includes at least an entity list, an event list, time information, relationship information, and feature embeddings of each multimodal data.

[0038] A construction module, configured to construct a temporal hypergraph of each historical case, wherein target elements in the element set are mapped to nodes of the temporal hypergraph, hyperedges between the nodes are defined, and time attributes between the hyperedges are determined;

[0039] a conversion module, configured to convert the temporal hypergraph into a comprehensive embedding vector and construct a database of historical cases and corresponding comprehensive embedding vectors;

[0040] The calculation module is used to obtain the query case and the corresponding comprehensive embedding vector, calculate the similarity between the embedding vector of the query case and the comprehensive embedding vector of each historical case in the database, and push the historical case with the highest similarity score to the user.

[0041] A third aspect of an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the similar case recommendation method provided in the first aspect.

[0042] A fourth aspect of an embodiment of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein when the processor executes the program, the similar case recommendation method provided in the first aspect is implemented.

[0043] A similar case recommendation method and system provided in an embodiment of the present invention obtains multimodal data of various historical cases in procuratorial business; uses LMM model and LLM model to preprocess and extract features of the multimodal data to obtain an element set of each historical case; constructs a time series hypergraph of each historical case; converts the time series hypergraph into a comprehensive embedding vector, and constructs a database of historical cases and corresponding comprehensive embedding vectors; obtains query cases and corresponding comprehensive embedding vectors, calculates the similarity between the embedding vector of the query case and the comprehensive embedding vectors of each historical case in the database, and pushes the historical case with the highest similarity score to the user, thereby achieving more accurate, reliable and explanatory similar case recommendations, and effectively assisting users in their case handling work. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A flowchart of a similar case recommendation method according to the first embodiment of the present invention;

[0045] Figure 2 This is a structural block diagram of a similar case recommendation system provided in Example 2 of the present invention;

[0046] Figure 3 This is a structural block diagram of an electronic device provided in Example 3 of the present invention. DETAILED DESCRIPTION

[0047] To facilitate understanding of the present invention, the present invention will be described more fully below with reference to the accompanying drawings. The drawings illustrate several embodiments of the present invention. However, the present invention may be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and comprehensive understanding of the present invention.

[0048] It should be noted that when an element is referred to as being "fixed to" another element, it may be directly on the other element or there may be an intermediate element. When an element is referred to as being "connected to" another element, it may be directly connected to the other element or there may be an intermediate element. The terms "vertical," "horizontal," "left," "right," and similar expressions used herein are for illustrative purposes only.

[0049] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one skilled in the art to which this invention pertains. The terms used in this specification of the present invention are for the purpose of describing specific embodiments only and are not intended to limit the present invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0050] Example 1

[0051] According to an embodiment of the present invention, an embodiment of a similar case recommendation method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0052] In this embodiment 1, a similar case recommendation method is provided, which can be used in electronic devices, such as computers. Figure 1 , Figure 1 The flowchart of the implementation of a similar case recommendation method provided in the first embodiment of the present invention is shown, which specifically includes steps S01 to S05.

[0053] Step S01, obtaining multimodal data of each historical case in the prosecution business, wherein the multimodal data includes at least text data, image data, audio data, video data and structured metadata.

[0054] Specifically, we first access or collect multimodal data of various historical cases in the prosecution business, among which text data include case summaries, prosecution opinions, review reports, interrogation / questioning records, witness testimony, judicial appraisal opinions (text part), legal documents (judgments, rulings), etc.; image data include on-site inspection photos, physical evidence photos, injury photos, surveillance screenshots, electronic data screenshots, scanned paper documents, charts in appraisal reports, etc.; audio data include interrogation / questioning recordings, reporting phone recordings, secretly recorded evidence, etc.; video data includes interrogation / questioning videos, surveillance videos, dashcam videos, law enforcement recorder videos, electronic data demonstration videos, etc.; structured metadata includes case number, case type, involved crimes, time of crime, location of crime, basic information of the parties (name, ID number, role), case handling stage, case handling personnel, legal provisions citations, etc.

[0055] In step S02, the LMM model and the LLM model are used to preprocess and extract features from the multimodal data to obtain an element set of each historical case. The element set includes at least an entity list, an event list, time information, relationship information, and feature embedding of each multimodal data.

[0056] Specifically, the LMM model is a large multimodal model, including LLaVA, GPT-4V, domain-specific models, etc., and the LLM model is a large language model, including GPT-4, Claude, ERNIE, and fine-tuned legal LLMs.

[0057] It should be noted that the text is processed through the LMM model and the LLM model, wherein the text data is subjected to entity recognition, relationship extraction, event extraction, semantic role labeling, time information normalization, text summary generation, text embedding, and instruction-driven information extraction. It can be understood that entity recognition refers to the identification and classification of key entities in the text, such as: names of people (suspects, victims, witnesses), organizations (companies, units), locations (scenes of crimes, hiding places), time (specific date and time, relative time), objects (tools used in committing crimes, stolen goods), legal concepts (charges, elements of crimes), behaviors (theft, injury, bribery), names of evidence, etc.; relationship extraction refers to the identification of relationships between entities, such as employment relationships, kinship relationships, joint crime relationships, causal relationships, subordinate relationships, etc.; event extraction refers to the identification of key events and their elements described in the text. (time, place, participants, behavior), for example, identifying the event "On the evening of May 10, 2023, Zhang San stole Li Si's electric car on XX Road"; semantic role labeling refers to analyzing sentence structure and marking the semantic roles played by words in the event (agent, object, time, place, method, etc.); time information normalization refers to converting relative time and fuzzy time in the text into a standard time format; text summary generation refers to generating a concise summary of the core content of the case; text embedding refers to using the LLM model to encode text fragments (sentences, paragraphs, documents) into high-dimensional semantic vectors x i In addition, complex prompts can be designed to guide the LLM / LMM to accurately extract specific information, for example: "Please analyze the following interrogation transcript, extract the defendant's confession regarding the motive for committing the crime, and evaluate its credibility."

[0058] The image is processed through the LMM model, wherein the image data is subjected to target detection and recognition, image description generation, key information extraction, visual feature embedding, and cross-modal alignment. It can be understood that target detection and recognition refers to the detection of objects (such as vehicles, weapons, faces), text (such as license plates, file contents), and scenes (such as streets, indoors) in the image; image description generation refers to the generation of natural language text describing the image content; key information extraction refers to the extraction of text information from evidence photos and the identification of spatial layout from scene maps; visual feature embedding refers to the use of LMM models or specialized visual models (such as ResNet, ViT) to extract the deep visual feature vector g of the image. i ; Cross-modal alignment refers to the LMM model's ability to understand the image content and associate it with the text description, for example, identifying the "red car" in the photo and matching it with the "red car driven by the suspect" mentioned in the transcript.

[0059] The audio is processed through the ASR + NLP / LMM model, wherein the audio data is subjected to automatic speech recognition, speaker recognition, sentiment analysis, keyword extraction, and audio feature embedding. It can be understood that automatic speech recognition refers to transcribing speech data such as interrogation recordings into text; speaker recognition refers to distinguishing different speakers in the recording; sentiment analysis refers to analyzing the speaker's emotional state (which may serve as auxiliary evidence); keyword extraction refers to extracting key information from the transcribed text or directly from the audio signal; audio feature embedding refers to extracting the acoustic feature vector of the audio signal. .

[0060] The video is processed through the Video Understanding Models / LMM model, wherein the video data is subjected to spatiotemporal behavior recognition, target tracking, scene understanding, video description, and video feature embedding. It can be understood that spatiotemporal behavior recognition refers to identifying actions and events in the video (such as fighting, running a red light, and trading); target tracking refers to tracking the movement trajectory of specific objects (people, cars) in the video; scene understanding refers to analyzing the scene and environment in which the video occurs; video description refers to generating a summary or description of the video content; video feature embedding refers to extracting the spatiotemporal feature vector v of the video. i .

[0061] Encode the structured metadata, for example, by using a One-Hot encoding or embedding layer to convert it into a feature vector s i .

[0062] The result of entity recognition constitutes the entity list, the result of relationship extraction obtains the relationship information, the result of event extraction constitutes the event list, and the result of time information normalization constitutes the time information.

[0063] It's important to note that a key advantage of LMMs is their ability to process and understand mixed-modal input. For example, you can provide an LMM with a text description and a photo of an incident and ask, "Does the photo confirm the on-site conditions described in the text?" The LMM can perform preliminary cross-modal verification and reasoning, and the results can serve as input or attributes for subsequent hypergraph construction.

[0064] After this step, each case d i The original multimodal data is converted into a set of structured elements containing rich semantic information, including: a list of recognized entities , event list , time information , relationship information , and the feature embedding x of each modal data i , gi , a i , v i , s i These elements will serve as the basis for building the temporal hypergraph.

[0065] Step S03: construct a temporal hypergraph of each historical case, wherein the target elements in the element set are mapped to nodes of the temporal hypergraph, hyperedges between the nodes are defined, and time attributes between the hyperedges are determined.

[0066] It should be noted that the types of nodes can be very diverse. For example, entity nodes: people (suspects, victims, witnesses, defense attorneys, prosecutors), organizations, locations, objects (evidence, stolen goods), vehicles, etc.; event nodes: represent specific events that occurred in the case (such as theft, injury, transaction events, reporting events, arrest events); concept nodes: legal concepts (such as specific crimes, self-defense, and surrender), behavior patterns (such as premeditation, contingency), and states (such as drunkenness and injury); evidence nodes: represent specific evidence items (such as interrogation records A, surveillance videos B, and appraisal reports C); time nodes: represent important time points or time periods. Each node can carry rich attribute information, for example, node ID, node type; text description of the node (such as entity name, event summary); associated timestamps or time intervals; feature embeddings extracted from LMM / LLM models (such as text embeddings x i , visual embedding g i etc., if the node directly corresponds to a certain modal data); metadata of the node (such as the role of the character, the value of the item, etc.).

[0067] Specifically, super edge Used to represent high-order relationships between nodes, a hyperedge can connect two or more nodes. Hyperedges are defined based on different dimensions and semantic associations: Event Participation Hyperedge: Connects all nodes that participate in the same event. For example, for the event "Zhang San and Li Si jointly rob Wang Wu", a hyperedge can be constructed:

[0068] ; Semantic Role Hyperedge: Connects nodes that play the same or related semantic roles in an event or relationship. For example, connects all "co-perpetrator" nodes; Evidence Relation Hyperedge: Connects an evidence node with the fact node it proves or the entity node involved. For example,

[0069] Behavior / State Hyperedge: connects related nodes (subject, time, location, degree, etc.) that describe the same specific behavior (such as drinking, speeding) or state (such as injury, drunkenness). For example, Temporal Relation Hyperedge: connects event nodes that have a clear temporal order or causal relationship. For example, ; Multimodal Correlation Hyperedge: connects information nodes or feature nodes of different modalities that describe or verify the same fact or event. For example, This hyperedge explicitly links evidence from different sources to the same core fact. A Legal Logic Hyperedge connects the nodes that satisfy the elements of a crime. For example, for the crime of theft, a hyperedge might connect the nodes {suspect, covert theft, property, large / huge amount, intentional state of mind}.

[0070] Each hyperedge can also carry attributes, for example, including hyperedge ID, hyperedge type (event, evidence, time series, etc.); hyperedge weight (indicating the strength or confidence of the relationship); time attribute: This is the key to the temporal hypergraph. Associate one or more timestamps t or time intervals for each hyperedge. , indicating the time point or time period when the relationship occurs or is valid.

[0071] For each case d i , according to the above definition, integrate all extracted nodes , defined hyperedge , and their temporal properties , construct a temporal hypergraph representation of the case Among them, the time series hypergraph can be represented by a variety of mathematical forms, the most commonly used is the incidence matrix of the ith case (Incidence Matrix) Matrix elements If the node v belongs to the hyperedge e, otherwise it is 0 (can be expanded to a weighted form). Time information It can be stored as a property of a node / hyperedge, or by constructing a series of time snapshot hypergraphs (GraphSnapshots) Another representation is the adjacency tensor, which can better capture high-order interactions but has higher computational complexity.

[0072] Step S04: convert the temporal hypergraph into a comprehensive embedding vector, and construct a database of historical cases and corresponding comprehensive embedding vectors.

[0073] In the embodiment of the present invention, the spectral theory of hypergraph is used to extract the global structural features of the temporal hypergraph and calculate the Laplacian matrix, wherein the normalized Laplacian matrix is ​​defined as:

[0074] ;

[0075] Where I is the identity matrix, D v is the node degree diagonal matrix, D e is the hyperedge degree / weight diagonal matrix, L i is the normalized Laplace matrix of the i-th case;

[0076] The Laplace matrix is ​​subjected to eigendecomposition, and a preset number of eigenvectors are determined to form the spectral embedding of the node. It can be understood that the Laplace matrix Perform eigenvalue decomposition to obtain the eigenvalue λ and the corresponding eigenvector u. The eigenvalues ​​and eigenvectors (especially the eigenvectors corresponding to smaller eigenvalues) contain the global structural information of the hypergraph (such as connectivity and clustering structure). In addition, the first k eigenvectors (or those based on the eigenvalue size) can be selected to form the spectral embedding of the node, or the eigenvalue distribution can be used as the spectral feature of the entire hypergraph.

[0077] The node attributes (text embedding, visual feature reference, etc.) of the temporal hypergraph and / or the spectral embedding are used as the input of the hypergraph neural network model, i.e., as the initial node features of the HGNN. In addition, hyperedge attributes (such as type and weight) can also be integrated into the message passing process. After propagation through the hypergraph neural network model, the final node embedding is obtained. The hypergraph neural network (HGNN) model is used to learn the deep representation of nodes and the entire hypergraph. HGNN aggregates neighborhood information (including high-order neighbors within the same hyperedge) by propagating information (message passing) on ​​the hypergraph structure. Various HGNN architectures can be selected, such as HGNN, HyperGCN, and UniGNN. Taking HGNN as an example, its core convolutional layer operation can be expressed as: ;

[0078] in, is the feature representation matrix of all nodes in the lth layer, d l is the feature dimension, is the weight matrix of the lth layer, σ is the nonlinear activation function, D v is the node degree diagonal matrix, D e is the hyperedge degree / weight diagonal matrix, and B is the incidence matrix;

[0079] The node embeddings are aggregated through graph readout functions (such as summation, averaging, and max pooling) to obtain the embedding vector of the entire temporal hypergraph;

[0080] Viewing a temporal hypergraph as a series of snapshots at discrete time points , calculate an embedding vector for each snapshot , then use a sequence model, such as a recurrent neural network (RNN, LSTM, GRU) or Transformer, to learn the representation of the entire sequence and obtain a temporal embedding;

[0081] The spectral embedding, the node embedding, and the temporal embedding are fused to obtain the comprehensive embedding vector, wherein the fusion is performed by using one of splicing, weighted averaging, or attention mechanism. The specific splicing operation is expressed as:

[0082] ;

[0083] in, is the spectral embedding or spectral feature, is the embedding vector of the entire temporal hypergraph, For time series embedding, it should be noted that if the dimensions and sources are different, they need to be aligned or used selectively. is the final graph embedding, and both temporal and spectral information have been considered, then explicit fusion may not be required, usually is the main embedding.

[0084] The weighted average operation is expressed as:

[0085] ;

[0086] in, 、 as well as is the corresponding weight, which can be preset or obtained through learning.

[0087] In the attention mechanism operation, the attention network is used to dynamically learn the importance of different embedding components and perform weighted fusion.

[0088] Step S05: Obtain the query case and the corresponding comprehensive embedding vector, calculate the similarity between the embedding vector of the query case and the comprehensive embedding vectors of each historical case in the database, and push the historical case with the highest similarity score to the user.

[0089] Specifically, the cosine similarity between the comprehensive embedding vector of the query case and the comprehensive embedding vector of each historical case in the database is calculated. The cosine similarity calculation formula is:

[0090] ;

[0091] z q is the comprehensive embedding vector of the query case, z c is the comprehensive embedding vector of each historical case in the database. The cosine similarity ranges from [-1, 1]. The closer the value is to 1, the more similar it is. It should be noted that the Euclidean distance between the comprehensive embedding vector of the query case and the comprehensive embedding vector of each historical case in the database can also be calculated, and then the Euclidean distance can be converted into similarity. The Euclidean distance calculation formula is:

[0092] ;

[0093] The smaller the Euclidean distance, the more similar it is. The formula for converting Euclidean distance to similarity is:

[0094] ;

[0095] or

[0096] ;

[0097] is a hyperparameter;

[0098] Based on the temporal hypergraph of the query case and the temporal hypergraphs of each historical case in the database, the structural similarity of the two temporal hypergraphs at the structural level is calculated. Specifically, a matching algorithm based on hyperedge type, node type, and time attribute can be designed. For example, the overlap degree of hyperedges of the same type and close in time in the two hypergraphs is calculated;

[0099] Based on the time series hypergraph of the query case and the time series hypergraph of each historical case in the database, the normalized distance between the time points of similar events is calculated to obtain the temporal pattern similarity. Specifically, the time series of key events (such as using Dynamic Time Warping (DTW)), case duration, and relative order patterns of events are compared. For example, the normalized distance between the time points of similar events in two cases is calculated, which is expressed as:

[0100] ;

[0101] To query the time point when similar events occurred in the case, The time point when similar events occurred in historical cases;

[0102] The cosine similarity, the structural similarity and the temporal pattern similarity are weightedly fused to obtain the final similarity. The calculation formula for weighted fusion is:

[0103] ;

[0104] in, Temporal Hypergraph and The structural similarity of is the corresponding weight, satisfying , can be set empirically (e.g., mainly relying on embedding similarity, set higher), or learned through machine learning methods (such as training a small model on labeled case pairs) to optimize recommendation results.

[0105] Furthermore, according to the calculated final similarity score qc , for all historical cases d in the candidate case library c Relative to query case d q Sort the cases in descending order and select the K cases with the highest scores as the recommendation results. The value of K can be set by the user or adjusted adaptively by the system based on the situation.

[0106] In summary, the similar case recommendation method in the above-mentioned embodiment of the present invention obtains multimodal data of various historical cases in the prosecution business; adopts LMM model and LLM model to preprocess and extract features of the multimodal data to obtain an element set of each historical case; constructs a time series hypergraph of each historical case; converts the time series hypergraph into a comprehensive embedding vector, and constructs a database of historical cases and corresponding comprehensive embedding vectors; obtains the query case and the corresponding comprehensive embedding vector, calculates the similarity between the embedding vector of the query case and the comprehensive embedding vector of each historical case in the database, and pushes the historical case with the highest similarity score to the user, thereby achieving more accurate, more reliable and more explanatory similar case recommendations, and effectively assisting users in handling cases.

[0107] Example 2

[0108] See also Figure 2 , Figure 2 This is a structural block diagram of a similar case recommendation system provided in Example 2 of the present invention. The similar case recommendation system 200 is used to implement the above-mentioned embodiments and preferred implementation methods, and the details that have been explained will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and conceivable.

[0109] Specifically, the similar case recommendation system 200 includes: an acquisition module 21, a data processing module 22, a construction module 23, a conversion module 24, and a calculation module 25, wherein:

[0110] An acquisition module 21 is used to acquire multimodal data of each historical case in the prosecution business, wherein the multimodal data includes at least text data, image data, audio data, video data and structured metadata;

[0111] A data processing module 22 is configured to preprocess and extract features from multimodal data using an LMM model and an LLM model to obtain an element set for each historical case, wherein the element set includes at least an entity list, an event list, time information, relationship information, and feature embedding of each multimodal data. The text data is subjected to entity recognition, relationship extraction, event extraction, semantic role labeling, time information normalization, text summary generation, text embedding, and instruction-driven information extraction.

[0112] Performing target detection and recognition, image description generation, key information extraction, visual feature embedding, and cross-modal alignment on the image data;

[0113] Performing automatic speech recognition, speaker recognition, sentiment analysis, keyword extraction, and audio feature embedding on the audio data;

[0114] Performing spatiotemporal behavior recognition, target tracking, scene understanding, video description, and video feature embedding on the video data;

[0115] performing encoding processing on the structured metadata;

[0116] The result of entity recognition constitutes the entity list, the result of relationship extraction obtains the relationship information, the result of event extraction constitutes the event list, and the result of time information normalization constitutes the time information;

[0117] A construction module 23 is configured to construct a temporal hypergraph for each historical case, wherein target elements in the element set are mapped to nodes of the temporal hypergraph, hyperedges between the nodes are defined, and time attributes between the hyperedges are determined. The hyperedges are used to represent high-order relationships between nodes, and a hyperedge can connect two or more nodes.

[0118] a conversion module 24 for converting the temporal hypergraph into a comprehensive embedding vector and constructing a database of historical cases and corresponding comprehensive embedding vectors;

[0119] The calculation module 25 is used to obtain the query case and the corresponding comprehensive embedding vector, calculate the similarity between the embedding vector of the query case and the comprehensive embedding vector of each historical case in the database, and push the historical case with the highest similarity score to the user.

[0120] Furthermore, in some other embodiments of the present invention, the conversion module 24 includes:

[0121] An extraction unit, configured to extract global structural features of the temporal hypergraph and calculate a Laplace matrix;

[0122] an eigendecomposition unit, configured to perform eigendecomposition on the Laplace matrix and determine a preset number of eigenvectors constituting a spectral embedding of a node;

[0123] A propagation unit is configured to use the node attributes of the temporal hypergraph and / or the spectral embedding as input to a hypergraph neural network model, and obtain a final node embedding after propagation through the hypergraph neural network model. The operation of the core convolutional layer in the hypergraph neural network model is expressed as follows:

[0124] ;

[0125] in, is the feature representation matrix of all nodes in the lth layer, d l is the feature dimension, is the weight matrix of the lth layer, σ is the nonlinear activation function, D v is the node degree diagonal matrix, D e is the hyperedge degree / weight diagonal matrix, and B is the incidence matrix;

[0126] an aggregation unit, configured to aggregate the node embeddings through a graph readout function to obtain an embedding vector of the entire temporal hypergraph;

[0127] The first computing unit is used to treat the temporal hypergraph as a series of snapshots at discrete time points, calculate an embedding vector for each snapshot, and then use the sequence model to learn the representation of the entire sequence to obtain the temporal embedding;

[0128] The first fusion unit is used to fuse the spectrum embedding, the node embedding and the time series embedding to obtain the comprehensive embedding vector, wherein the fusion is performed by using one of splicing, weighted averaging or attention mechanism.

[0129] Furthermore, in some other embodiments of the present invention, the calculation module 25 includes:

[0130] A second calculation unit is used to calculate the cosine similarity between the comprehensive embedding vector of the query case and the comprehensive embedding vectors of each historical case in the database;

[0131] A third calculation unit is used to calculate the structural similarity of the two time series hypergraphs at the structural level based on the time series hypergraph of the query case and the time series hypergraphs of each historical case in the database;

[0132] A fourth calculation unit is used to calculate the normalized distance between the time points of occurrence of similar events based on the time series hypergraph of the query case and the time series hypergraphs of each historical case in the database to obtain the time pattern similarity;

[0133] The second fusion unit is configured to perform weighted fusion on the cosine similarity, the structural similarity and the temporal pattern similarity to obtain a final similarity.

[0134] Example 3

[0135] Another aspect of the present invention provides an electronic device, see Figure 3 , shown is an electronic device in embodiment 3 of the present invention, including a memory 20, a processor 10, and a computer program 30 stored in the memory and executable on the processor. When the processor 10 executes the computer program 30, the similar case recommendation method described above is implemented.

[0136] In some embodiments, the processor 10 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip, used to run program codes or process data stored in the memory 20, such as executing access restriction programs.

[0137] The memory 20 includes at least one type of readable storage medium, including flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 20 may be an internal storage unit of the electronic device, such as the hard disk of the electronic device. In other embodiments, the memory 20 may also be an external storage device of the electronic device, such as a plug-in hard disk equipped on the electronic device, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. Furthermore, the memory 20 may include both an internal storage unit of the electronic device and an external storage device. The memory 20 can be used not only to store application software and various types of data of the electronic device, but also to temporarily store data that has been output or is about to be output.

[0138] It should be pointed out that Figure 3 The structure shown does not constitute a limitation to the electronic device. In other embodiments, the electronic device may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0139] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which implements the similar case recommendation method as described above when the program is executed by a processor.

[0140] Those skilled in the art will appreciate that the logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device), or in conjunction with such instruction execution system, apparatus, or device. For purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by an instruction execution system, apparatus, or device, or in conjunction with such instruction execution system, apparatus, or device.

[0141] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting, or processing it in another suitable manner as necessary, and then storing it in a computer memory.

[0142] It should be understood that various components of the present invention may be implemented using hardware, software, firmware, or a combination thereof. In the aforementioned embodiments, multiple steps or methods may be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one or a combination of the following technologies known in the art may be used: a discrete logic circuit having logic gate circuits for implementing logic functions on data signals, an application-specific integrated circuit having suitable combinational logic gate circuits, a programmable gate array (PGA), a field-programmable gate array (FPGA), etc.

[0143] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0144] The above embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art would be able to make numerous modifications and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A similar case recommendation method, characterized in that: Applied to the procuratorial industry scenario, the method includes: Acquire multimodal data of each historical case in procuratorial work, wherein the multimodal data includes at least text data, image data, audio data, video data, and structured metadata; LMM and LLM models are used to preprocess and extract features from multimodal data to obtain an element set for each historical case. The element set includes at least an entity list, an event list, time information, relationship information, and feature embeddings of each multimodal data. Constructing a temporal hypergraph for each historical case, wherein target elements in the element set are mapped to nodes of the temporal hypergraph, hyperedges between the nodes are defined, and time attributes between the hyperedges are determined; Converting the temporal hypergraph into a comprehensive embedding vector and constructing a database of historical cases and corresponding comprehensive embedding vectors; Obtain the query case and the corresponding comprehensive embedding vector, calculate the similarity between the embedding vector of the query case and the comprehensive embedding vector of each historical case in the database, and push the historical case with the highest similarity score to the user.

2. The similar case recommendation method according to claim 1, characterized in that: In the step of using the LMM model and the LLM model to preprocess and extract features from the multimodal data to obtain the element set of each historical case, the text data is subjected to entity recognition, relationship extraction, event extraction, semantic role labeling, time information normalization, text summary generation, text embedding, and instruction-driven information extraction; Performing target detection and recognition, image description generation, key information extraction, visual feature embedding, and cross-modal alignment on the image data; Performing automatic speech recognition, speaker recognition, sentiment analysis, keyword extraction, and audio feature embedding on the audio data; Performing spatiotemporal behavior recognition, target tracking, scene understanding, video description, and video feature embedding on the video data; performing encoding processing on the structured metadata; The result of entity recognition constitutes the entity list, the result of relationship extraction obtains the relationship information, the result of event extraction constitutes the event list, and the result of time information normalization constitutes the time information.

3. The similar case recommendation method according to claim 2, characterized in that: The hyperedge is used to represent a high-order relationship between nodes, and a hyperedge can connect two or more nodes.

4. The similar case recommendation method according to claim 3, characterized in that: The step of converting the temporal hypergraph into a comprehensive embedding vector comprises: Extracting global structural features of the temporal hypergraph and calculating the Laplacian matrix; Performing eigendecomposition on the Laplace matrix and determining a preset number of eigenvectors to constitute a spectral embedding of a node; Using the node attributes of the temporal hypergraph and / or the spectral embedding as input to a hypergraph neural network model, and obtaining a final node embedding after propagation through the hypergraph neural network model; The nodes are embedded and aggregated by a graph readout function to obtain an embedding vector of the entire temporal hypergraph; The temporal hypergraph is regarded as a series of snapshots at discrete time points, an embedding vector is calculated for each snapshot, and then a sequence model is used to learn the representation of the entire sequence to obtain the temporal embedding; The spectrum embedding, the node embedding, and the time series embedding are fused to obtain the comprehensive embedding vector.

5. The similar case recommendation method according to claim 4, characterized in that: In the step of fusing the spectral embedding, the node embedding, and the temporal embedding to obtain the comprehensive embedding vector, one of splicing, weighted averaging, or attention mechanism is used for fusion.

6. The similar case recommendation method according to claim 5, characterized in that: The steps of obtaining the query case and the corresponding comprehensive embedding vector and calculating the similarity between the embedding vector of the query case and the comprehensive embedding vectors of each historical case in the database include: Calculate the cosine similarity between the comprehensive embedding vector of the query case and the comprehensive embedding vectors of each historical case in the database; Based on the temporal hypergraph of the query case and the temporal hypergraphs of each historical case in the database, the structural similarity of the two temporal hypergraphs at the structural level is calculated; Based on the time series hypergraph of the query case and the time series hypergraph of each historical case in the database, the normalized distance between the time points of similar events is calculated to obtain the time pattern similarity; The cosine similarity, the structural similarity and the temporal pattern similarity are weightedly fused to obtain a final similarity.

7. The similar case recommendation method according to claim 4, characterized in that: The operation of the core convolutional layer in the hypergraph neural network model is expressed as: ； in, is the feature representation matrix of all nodes in the lth layer, d l is the feature dimension, is the weight matrix of the lth layer, σ is the nonlinear activation function, D v is the node degree diagonal matrix, D e is the hyperedge degree / weight diagonal matrix, and B is the incidence matrix.

8. A similar case recommendation system, characterized in that: For implementing the similar case recommendation method according to any one of claims 1 to 7, the system comprises: An acquisition module is used to acquire multimodal data of each historical case in the prosecution business, wherein the multimodal data includes at least text data, image data, audio data, video data and structured metadata; A data processing module is used to preprocess and extract features from multimodal data using LMM and LLM models to obtain an element set for each historical case. The element set includes at least an entity list, an event list, time information, relationship information, and feature embeddings of each multimodal data. A construction module, configured to construct a temporal hypergraph of each historical case, wherein target elements in the element set are mapped to nodes of the temporal hypergraph, hyperedges between the nodes are defined, and time attributes between the hyperedges are determined; a conversion module, configured to convert the temporal hypergraph into a comprehensive embedding vector and construct a database of historical cases and corresponding comprehensive embedding vectors; The calculation module is used to obtain the query case and the corresponding comprehensive embedding vector, calculate the similarity between the embedding vector of the query case and the comprehensive embedding vector of each historical case in the database, and push the historical case with the highest similarity score to the user.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the similar case recommendation method according to any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and running on the processor, wherein when the processor executes the program, the method for recommending similar cases according to any one of claims 1 to 7 is implemented.

Images