Multi-modal retrieval augmentation generation system for mass law enforcement audio-video data

Abstract

The application relates to the technical field of multi-modal information processing, and particularly discloses a multi-modal retrieval enhancement generation system for massive law enforcement audio and video data, which comprises a multi-modal index module based on voiceprint assistance, a multi-modal fusion encoder, a knowledge graph database, a multi-modal retrieval and generation module, and a knowledge graph database module.The multi-modal index module is used for generating multi-modal representation according to a recorder audio and video; the multi-modal fusion encoder is used for processing the multi-modal representation and constructing the knowledge graph database; the multi-modal retrieval and generation module is used for receiving a user query, extracting multi-modal keywords, querying the knowledge graph database, obtaining a fusion retrieval set, generating a related event comprehensive analysis report by using a VLM and a LLM; and the knowledge graph database module is used for storing and managing the knowledge graph database.The application realizes the structured index, semantic association, cross-scene speaker association and efficient content generation of law enforcement audio and video content by using a voiceprint-assisted knowledge graph construction and a knowledge-driven retrieval mechanism, and finally outputs a related event comprehensive analysis report.

Description

Technical Field

[0001] This invention relates to the field of multimodal information processing technology, and in particular to a multimodal retrieval and enhancement generation system for massive amounts of law enforcement audio and video data. Background Technology

[0002] With the rapid growth of audio and video data in law enforcement, processing and understanding massive amounts of such data has become a major challenge. This data is characterized by its long duration, diverse sources, and complex content. Traditional archiving methods store audio and video data as isolated fragments, leading to the loss of contextual information and making it difficult to capture semantic relationships or speaker identities across scenarios. Existing Retrieval Augmentation (RAG) technologies primarily target text data and lack the ability to comprehensively process multimodal information in law enforcement audio and video.

[0003] Speech recognition (ASR), speaker separation, and voiceprint analysis technologies have provided new tools for audio and video analysis, but these technologies are often used independently and lack systematic integration, making it difficult to meet the needs of law enforcement scenarios for cross-scenario identity association and multimodal semantic modeling.

[0004] The core challenges facing law enforcement audio and video data processing are as follows:

[0005] 1. Multimodal complexity: The data includes video footage, voice dialogue, ambient sound, and voiceprint features, which need to be comprehensively analyzed to extract key information, such as event description and the identity of the parties involved.

[0006] 2. Cross-scenario semantic association: Different law enforcement scenarios (such as street patrols and interrogation rooms) may involve the same people or events, so semantic and identity associations need to be established.

[0007] 3. Real-time performance and privacy protection: Law enforcement data processing requires the rapid retrieval of relevant fragments while protecting sensitive information, such as personal identity.

[0008] 4. Large-scale data management: The data volume is huge, requiring efficient indexing and retrieval mechanisms to support real-time queries. Summary of the Invention

[0009] This invention aims to address the limitations of traditional methods in multimodal information processing, cross-scene association, and semantic consistency. To this end, this invention provides a multimodal retrieval enhancement and generation system for massive amounts of law enforcement audio and video data. It integrates speech recognition, voiceprint analysis, knowledge graph databases, and multimodal retrieval technologies. Through voiceprint-assisted knowledge graph construction and a knowledge-driven retrieval mechanism, it achieves structured indexing, semantic association, cross-scene speaker association, and efficient content generation of law enforcement audio and video content, ultimately outputting a comprehensive analysis report of related events. This invention is applicable to scenarios such as law enforcement record analysis, surveillance video retrieval, and law enforcement event tracking, and provides comprehensive event insights and archiving suggestions by generating a comprehensive analysis report of related events.

[0010] This invention provides a multimodal retrieval and enhancement generation system for massive amounts of law enforcement audio and video data. The technical solution adopted is as follows: it includes a voiceprint-assisted multimodal indexing module, a multimodal retrieval and generation module, and a knowledge graph database module.

[0011] The voiceprint-assisted multimodal indexing module is used to generate subtitles, transcriptions, and voiceprint vectors based on the audio and video of the recorder, merge the subtitles, transcriptions, and voiceprint vectors to form a multimodal representation, and use a multimodal fusion encoder to process the multimodal representation to construct a knowledge graph database.

[0012] The multimodal retrieval and generation module is used to receive user queries, extract multimodal keywords, query the knowledge graph database based on the multimodal keywords to obtain text retrieval sets, visual retrieval sets, and voiceprint retrieval sets, and integrate them to form a fusion retrieval set; based on the fusion retrieval set, it uses VLM and LLM to generate a comprehensive analysis report of related events.

[0013] The knowledge graph database module is used to store and manage the knowledge graph database.

[0014] Furthermore, the process of forming multimodal representations is as follows:

[0015] The audio and video recordings from the recorder are divided into multiple segments;

[0016] Visual frames are sampled from each segment, input into a visual language model, and subtitles are generated.

[0017] Transcriptions for each segment are generated using automatic speech recognition;

[0018] Speaker segmentation is performed on each segment to generate a voiceprint vector;

[0019] The subtitle, transcription, and voiceprint vectors are merged to form a multimodal representation.

[0020] Furthermore, the process of sampling visual frames is as follows: calculate the average grayscale difference between the current frame and the previous sampled frame as the rate of change. If the rate of change exceeds the rate of change threshold and the time interval exceeds the minimum interval, then the current frame is used as the sampling frame. The first frame is always sampled.

[0021] Furthermore, the nodes in a knowledge graph database include speaker nodes, speech segment nodes, event nodes, and object nodes;

[0022] Edges include identity-related edges, voice generation edges, behavior-related edges, and event-related edges.

[0023] Furthermore, the processing procedure of the multimodal fusion encoder is as follows:

[0024] Subtitles and transcriptions are processed through embedding layers to obtain subtitle embeddings and audio transcription embeddings, respectively. Then, the first layer output is obtained through a cross-attention mechanism.

[0025] The first-layer output and the voiceprint vector are integrated using a causal multi-head self-attention mechanism to obtain the identity information, resulting in the second-layer output.

[0026] The second-layer output and the semantic relationship embedding of the knowledge graph are passed through a graph attention network to obtain the updated node embedding and attention coefficient matrix.

[0027] Furthermore, the multimodal keywords include text keywords, visual descriptions, and potential voiceprint identifiers;

[0028] Based on text keywords, corresponding fragments are extracted from the knowledge graph database to form a text retrieval set;

[0029] Visual descriptions are converted into visual embedding vectors, and corresponding fragments are extracted from a knowledge graph database to generate a visual retrieval set.

[0030] Based on potential voiceprint identifiers, the voiceprint vectors of relevant speaker nodes are extracted from the knowledge graph database, and all related segments of the voiceprint vectors are extracted to form a voiceprint retrieval set.

[0031] Furthermore, the text retrieval set, visual retrieval set, and voiceprint retrieval set are integrated, and a weighted fusion mechanism is used to remove redundant fragments to form a fused retrieval set.

[0032] Furthermore, the comprehensive analysis report of related events includes at least one of the following analytical dimensions: event timeline, entity correlation, event summary, key findings, and archiving recommendations.

[0033] Furthermore, the event timeline is based on the timeline sorting of edges in the knowledge graph database, which sorts cross-scene events or segments in chronological order to form an event timeline;

[0034] Entity association is determined by starting from the matching nodes of the fused retrieval set, traversing the edges of the knowledge graph database, extracting relevant subgraphs, and displaying the associations between entities;

[0035] Event summaries are generated by using LLM to summarize search results and subgraphs, producing a concise description of the event's cause, development process, and outcome.

[0036] The key discovery is the detection of anomalies or patterns in events, achieved based on the results of subgraph extraction, text blocks, and embedding similarity comparison;

[0037] The archiving recommendations are based on the strength of the event association, suggesting whether to merge videos or events into a single archive.

[0038] Furthermore, it also includes a task management module, which is used to receive and initialize law enforcement audio and video processing tasks and manage task status.

[0039] The above-described one or more technical solutions in the embodiments of the present invention have at least one of the following technical effects:

[0040] 1. Voiceprint-Assisted Knowledge Graph Database: This invention integrates visual, audio, and voiceprint information from audio and video into a knowledge graph database using a visual language model, speech recognition, voiceprint analysis, and a multimodal fusion encoder (MEnc), preserving cross-scene semantic and identity associations. The invention designs a novel multimodal fusion encoder (MEnc) that employs a hierarchical attention mechanism to fuse subtitle embedding, audio transcription embedding, and voiceprint vectors layer by layer. Specifically, the first-layer attention module focuses on visual-audio alignment, the second layer integrates voiceprint identity information, and the third layer injects semantic relationship embeddings from the knowledge graph, thereby achieving more accurate cross-modal interaction representation. This structure significantly improves the recall and accuracy of multimodal retrieval, reducing parameter redundancy and improving generalization ability compared to traditional concatenation fusion methods.

[0041] 2. Knowledge-driven multimodal retrieval and generation: By leveraging a knowledge graph database and combining text semantic matching, visual content retrieval, and voiceprint association, relevant audio and video clips are accurately extracted and a comprehensive response is generated, forming a comprehensive analysis report of related events.

[0042] 3. Cross-scene speaker association: Utilizing voiceprint analysis and knowledge graph databases, automatically identify and associate the same speakers in different law enforcement scenarios to construct a semantic network across videos.

[0043] 4. Efficient task processing workflow: Through a pipeline architecture built on task initialization, AI model processing, and knowledge graph database, the system ensures efficiency, robustness, and privacy protection, and outputs structured analysis reports.

[0044] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0046] Figure 1 This is a structural block diagram provided by the present invention.

[0047] Figure 2 This is a flowchart of the knowledge graph database construction process provided by the present invention.

[0048] Figure 3 This is a schematic diagram of the structure of the multimodal fusion encoder provided by the present invention.

[0049] Figure 4 This is a flowchart of the multimodal retrieval and generation process provided by the present invention. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention. The following embodiments are used to illustrate this invention but should not be used to limit the scope of this invention.

[0051] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0052] The following is combined Figures 1 to 4The present invention will be further described in detail below, providing a multimodal retrieval and enhancement generation system for massive amounts of law enforcement audio and video data:

[0053] In this embodiment, as Figure 1 As shown, a multimodal retrieval and enhancement generation system for massive law enforcement audio and video data is provided, including: a voiceprint-assisted multimodal indexing module, a task management module, a multimodal retrieval and generation module, and a knowledge graph database module.

[0054] The voiceprint-assisted multimodal indexing module is used to generate subtitles, transcriptions, and voiceprint vectors based on the audio and video of the recorder, merge the subtitles, transcriptions, and voiceprint vectors to form a multimodal representation, and use a multimodal fusion encoder to process the multimodal representation to construct a knowledge graph database.

[0055] The multimodal retrieval and generation module is used to receive user queries, extract multimodal keywords, query the knowledge graph database based on the multimodal keywords to obtain text retrieval sets, visual retrieval sets, and voiceprint retrieval sets, and integrate them to form a fusion retrieval set; based on the fusion retrieval set, it uses VLM and LLM to generate a comprehensive analysis report of related events.

[0056] The knowledge graph database module is used to store and manage the knowledge graph database;

[0057] The task management module is used to receive and initialize law enforcement audio and video processing tasks and manage task status.

[0058] The specific working process of this system is as follows:

[0059] Step S1: The task management module receives the law enforcement audio and video processing task, which includes audio and video metadata (video_meta) and the processing directory (handle_dir). The task status is set to "Processing" via the task status manager, with an initial progress of 0%. Then, the audio and video file paths are verified. Upon successful verification, an encrypted temporary directory (audio_asr_temp) is created to store intermediate files, ensuring the security of sensitive data.

[0060] Step S2: The multimodal indexing module with voiceprint assistance reads the audio and video metadata, and then performs the following operations, such as... Figure 2 As shown:

[0061] Audio and video splitting: This process splits the input audio and video data from the recorder. Segmented into multiple short segments Each segment contains visual frames and audio content. This represents the j-th segment. This represents the total number of segments.

[0062] Visual-text semantic extraction: Visual frames are sampled from each segment. This embodiment employs an intelligent frame sampling algorithm based on inter-frame change rate and time interval. Specifically, it includes: calculating the average grayscale difference between the current frame and the previous sampled frame as the change rate; if the change rate exceeds a change rate threshold and the time interval exceeds the minimum interval, the current frame is used as the sampled frame, with the first frame always being sampled. Then, face detection and pose determination are performed on the sampled frames. A pose score is calculated for each detected face, forming a list of each person's faces. Only the photo with the highest pose score is stored to reduce storage space. This photo can be inferred using a sentiment classification model to form a sentiment label. In this embodiment, the change rate threshold is 0.2, and the minimum interval is 20 frames.

[0063] All sampled frames are input into the Visual Language Model (VLM) to generate natural language captions for the video, describing the dynamics, behaviors, and key objects in the law enforcement scene, such as "Law enforcement officers question the parties involved on the street, holding a notebook."

[0064] Assuming that intelligent sampling methods are used to obtain... Sampling ,in, This represents the k-th sampled frame. The subtitle for the j-th segment is obtained by inputting the visual language model. , , Represents a visual language model.

[0065] Audio-text and voiceprint semantic extraction: Using automatic speech recognition (ASR) to generate time-aligned transcripts for each segment, capturing dialogue or ambient sounds. ,in, This represents the transcription of the j-th segment. This represents an automatic speech recognition model.

[0066] Speaker segmentation is performed for each segment, generating speaker identifiers and voiceprint vectors. Speakers are grouped, and high-quality segments with a duration of ≥1.5 seconds are filtered out. Aggregated voiceprints are calculated, and a robust averaging strategy is used to remove 10% of extreme vectors to obtain the final voiceprint vector.

[0067] Unified multimodal representation: Merging subtitle, transcription, and voiceprint vectors to form a unified multimodal representation. ,in, For the audio and video of the recorder A multimodal representation, For the first Subtitles for a segment, For the first Transcription of a segment, For the first The voiceprint vector of each segment.

[0068] Multimodal fusion encoder (MEnc) is used to process multimodal representations and construct a knowledge graph database. , .

[0069] The representation nodes include speakers, speech segments, events, and objects, and store metadata for multimodal representations and recorder audio and video. They are generated by a visual language model, an automatic speech recognition model, and a multimodal fusion encoder.

[0070] Edges represent semantic relationships between nodes, including behavioral relationships (such as "inquire"), identity associations (such as "IS_SAME_AS"), event associations (such as "cause"), and speech generation (such as "HAS_SPEECH"), which are generated by LLM (Large Language Model) relation extraction and voiceprint matching.

[0071] like Figure 3 As shown, this embodiment uses a multimodal fusion encoder to generate embeddings, capturing interactive features of visual, audio, text, and voiceprint. MEnc employs a three-layer attention mechanism, fusing caption, transcription, voiceprint, and knowledge graph relationship features to generate unified entity embeddings, supporting subsequent node attribute definitions. This design reduces parameter redundancy and ensures efficient cross-modal interaction.

[0072] Subtitles and transcription are processed separately through an embedding layer to obtain subtitle embeddings. and audio transcription embedding ,in, , , This indicates that it is done through an embedding layer.

[0073] The first layer of the multimodal fusion encoder achieves subtitle embedding through a cross-attention mechanism. With audio transcription embedding Alignment, reducing modal bias, and calculating attention weights. ,in, For visual queries, , To query the weights, For audio keys, , For key weights, For audio values, , The weights are set to values, and the first layer output is... The first layer of the multimodal fusion encoder consists of stacked layer normalization, cross-attention layers, and additional layers.

[0074] The second layer of the multimodal fusion encoder will input the... With voiceprint vector Fusion, by injecting voiceprints (speaker identity features) into the pre-alignment result ( This achieves content and identity association; it uses a causal multi-head self-attention mechanism to integrate identity information, with the second layer output being... The second layer of the multimodal fusion encoder consists of stacked causal multi-head self-attention layers, an addition layer, and layer normalization.

[0075] The third layer of the multimodal fusion encoder combines semantic relationships from the knowledge graph to optimize the relevance of entity embeddings; the input is... semantic relationship embedding of knowledge graph .in, As the initial node embedding of the third layer, it provides the basis for multimodal features for the Graph Attention Network (GAT); It is a vector representation of relation edges in a knowledge graph, encoding semantic information of semantic relations and related metadata, such as confidence and timestamps. The node embeddings are updated by propagating relationships through a Graph Attention Network (GAT), using the following formula: ,in, The updated embedding for node i, Let be the set of neighboring nodes of node i. Let be the attention coefficient between node i and node j. The weight matrix is ​​a linear transformation matrix. The original embedding of neighbor node j. Guided Graph Attention Network (GAT) computation To propagate semantic relationships between nodes; at the same time, enhance It captures semantic interactions across modalities and scenarios, supporting multimodal retrieval and the generation of comprehensive analysis reports on related events. The third layer outputs updated node embeddings and attention coefficient matrices. The second layer of the multimodal fusion encoder consists of stacked feedforward layers and graph attention layers.

[0076] The updated node embedding integrates semantic information from its neighbors, and is no longer an isolated single-node feature, but rather a fusion of features from all its neighboring nodes. The updated node embedding maintains the same dimensionality as the original input node embedding and includes multimodal information such as captions, transcription, and voiceprints, avoiding the one-sidedness of single-modal features. It also achieves differentiated fusion with attention weights, allowing neighbors with high semantic relevance to contribute more through attention coefficients.

[0077] The attention coefficient matrix is ​​a matrix composed of the attention coefficients of all node pairs. Its purpose is to supplement the explanation of the importance weight of neighboring nodes to the target node. Its dimension matches the number of nodes and reflects the semantic association strength.

[0078] A knowledge graph database is constructed using the updated node embeddings and attention coefficient matrix.

[0079] In this embodiment, the nodes of the knowledge graph database include speaker nodes, speech segment nodes, event nodes, and object nodes.

[0080] The edges between nodes include identity association edges, speech generation edges, behavior association edges, and event association edges, as detailed below:

[0081] Identity-related edges: connect speaker nodes, relying on voiceprint vector matching + subtitle / transcribed text clues.

[0082] Speech generation edge: connects speaker nodes and speech segment nodes, relying on voiceprint vector matching + caption / transcription time alignment.

[0083] Behavior-related edges: connect speaker nodes to speaker nodes and speaker nodes to event nodes, relying on text behavior descriptions of subtitles / transcriptions + voiceprint vector identity verification.

[0084] Event-related edges: connect event nodes to event nodes, audio segment nodes to event nodes, and object nodes to event nodes, relying on textual causal clues from subtitles / transcriptions + timestamp alignment.

[0085] This construction phase serves as the foundation for subsequent retrieval and generation, processing the raw recorder audio and video into a structured knowledge graph database. Nodes contain multimodal features, and relational edges record semantic associations. This offline or semi-real-time process directly supports query-driven extraction in the retrieval phase, ensuring knowledge-driven processing. The constructed multimodal representation and layer-by-layer fusion of MEnc provide an embedding foundation for subsequent text, visual, and voiceprint matching. Encrypted temporary directories and robust averaging strategies extend to threshold adjustment and index acceleration in retrieval, achieving a shared optimization of privacy and efficiency.

[0086] Audio and video metadata typically includes multiple recorder audio and video clips. Each recorder audio and video clip forms a part of a knowledge graph database, and multiple parts are connected by edges to form a complete knowledge graph database. Nodes in the knowledge graph database are identified by their corresponding recorder audio and video clips through their metadata, and each node is associated with a segment of a recorder audio and video clip.

[0087] The knowledge graph database constructed by the voiceprint-assisted multimodal indexing module is stored in the knowledge graph database module.

[0088] Step S3: The multimodal retrieval and generation module queries the knowledge graph database in the knowledge graph database module based on the received user query, and generates a comprehensive analysis report of related events based on the query results. For example... Figure 4 As shown, the specific process is as follows:

[0089] Query preprocessing: Receive user queries and extract multimodal keywords from the user queries, including text keywords (corresponding to...). Visual description (corresponding) ) and potential voiceprint identifiers (corresponding to This step is directly related to the multimodal representation in the construction phase, preparing embedding vectors for subsequent matching, similar to MEnc pairs. , , Layer-by-layer attention fusion.

[0090] To improve semantic matching accuracy, LLM can be used to reconstruct user queries, converting them into declarative sentences, such as converting "the dialogue between parties in a street inquiry" into "retrieving the dialogue content of parties in a street inquiry scenario", and then extracting multimodal keywords from the reconstructed query.

[0091] If no potential voiceprint identifier is found, the potential voiceprint identifier is inferred from the user query.

[0092] The knowledge graph database is queried using multimodal keywords to obtain text retrieval sets, visual retrieval sets, and voiceprint retrieval sets. The specific process is as follows:

[0093] Text semantic matching: Based on text keywords, the embedding similarity between the keywords and entity descriptions in the knowledge graph database is calculated. This embodiment uses cosine similarity or BERT-based semantic matching, selecting relevant text blocks with similarity higher than a threshold (e.g., 0.8). Corresponding audio and video segments are extracted from the selected text blocks to form a text retrieval set. If the text block contains cross-scene references, then... Expand the search scope. Similarity calculation is equivalent to... Semantic matching of text portions in the model; LLM processing during the build phase. generate The entity description provides the semantic basis for this matching. Without time-aligned transcription, Unable to capture the dialogue context.

[0094] Text blocks refer to structured text fragments extracted from law enforcement audio and video data, stored in a knowledge graph database, and used to describe the semantic content of audio and video segments. They mainly originate from subtitles and transcriptions and are associated with nodes and edges in the knowledge graph database. A text block represents a set of text fragments that semantically match a user's query during the retrieval phase. Cross-scene referencing refers to the semantic information contained in a text block being associated with the content of other scenes or video files through edges in the knowledge graph. In other words, cross-scene referencing is triggered when the semantic content of a text block (such as dialogue, events, or speaker identity) is associated with nodes or edges in different scenes (e.g., different times, locations, or video files).

[0095] Visual content retrieval: Converting visual descriptions into visual embedding vectors using MEnc. , ,in, For visual description, a hierarchical indexing method is used: first, coarse-grained matching of scene type, that is, comparing the visual embedding vector with the subtitle embedding, using the first layer of MEnc for embedding comparison; then fine-grained matching of details, comparing with the visual embedding of the sampled frame corresponding to the subtitle embedding (e.g., cosine similarity threshold of 0.7). , For visual embedding; if the similarity exceeds 0.7, the corresponding fragments are extracted from the knowledge graph database to generate a visual retrieval set. To improve retrieval speed and accuracy, this embodiment uses the photo with the highest pose score to calculate its visual embedding, and then compares it with the visual embedding vector.

[0096] Voiceprint-assisted retrieval: Voiceprint vectors of relevant speaker nodes are extracted from the knowledge graph database based on potential voiceprint identifiers. This embodiment uses cosine similarity for query comparison, with a threshold set to 0.75. If the similarity exceeds the threshold, all associated segments of the voiceprint vector are extracted to form a voiceprint retrieval set. During the search, matching speaker nodes in other audio and video files are retrieved by expanding the search through edges.

[0097] Search result fusion: The text search set, visual search set, and voiceprint search set are integrated, and a weighted fusion mechanism is used to remove redundant fragments to form a fused search set. The weights are based on modal relevance. In this embodiment, the text weight is 0.4, the visual weight is 0.3, and the voiceprint weight is 0.3; these can be adjusted later based on actual usage (accuracy of search results). The essence of constructing the fused search set is the construction phase. Subset filtering and fusion rely on the node embedding update of the third-layer GAT of MEnc. Without multimodal representation during the construction phase, fusion cannot achieve cross-modal interaction.

[0098] Redundant fragments refer to multiple retrieval sets (text, visual, voiceprint) in the multimodal retrieval stage. , , The search results may contain repetitive or highly similar audio and video clips. These clips may overlap in semantic content, timestamps, scenes, or speaker identities, leading to redundant search results. For example: Contains fragments (Dialogue: "Please show your identification"). Contains the same segment (scene "questioning on the street at night"). Containing the same segment (based on the individual's voiceprint). If these segments point to the same... If the timestamp and video ID are the same, then it is considered redundant.

[0099] A fragment deduplication method based on embedding similarity: Each fragment in the retrieval set is associated with multimodal embeddings, including: caption embedding, audio transcription embedding, and voiceprint embedding (voiceprint vector). The caption embedding, audio transcription embedding, and voiceprint embedding are initially weighted and then input into MEnc. The third layer of MEnc outputs the updated embedding of the node corresponding to each fragment. Cosine similarity is used to calculate the similarity of the updated embeddings. If the similarity exceeds a threshold, such as 0.9, the fragment is considered redundant, and only one is retained. Generally, the fragment with the highest confidence is retained. For example: Contains fragments (Dialogue: "Please show your identification") The two segments contain the same fragment (scene "nighttime street questioning"). Calculate and compare the updated embeddings of the two segments. If the similarity is greater than 0.9, it is confirmed that they point to the same fragment, and one is retained.

[0100] The same similarity threshold is used for both the construction and retrieval of the knowledge graph database. The threshold is adjusted during the retrieval stage based on the accuracy of the retrieval results, and the threshold during the construction stage is updated synchronously.

[0101] Based on the fused retrieval set, a comprehensive analysis report of related events is generated using VLM and LLM.

[0102] Extract key information from the fused search set, including dialogue content (from... Extracted from), scene description (from) Extract), speaker identity (from Extracting (from) and sentiment analysis (optional, using an additional sentiment classification model). Integrating multimodal information using VLM and LLM to generate a comprehensive response: ,in, As an extended form of knowledge graph database, For retrieval functions, For user queries, This is a large language model. The specific process is as follows: LLM first constructs a prompt template, injects the search results (fusion search set), then generates a natural language response, and obtains a comprehensive analysis report of related events.

[0103] The report should include at least one of the following analytical dimensions: event chronology, entity relationships, event summary, key findings, and archiving recommendations. Specifically:

[0104] (1) The event timeline is based on the edge timeline sorting of the knowledge graph database. Events or segments across scenes are sorted in chronological order to form an event timeline, and transitions are marked, such as from street patrol to interrogation room.

[0105] Data sources: Metadata (such as timestamps, video metadata) of edges (e.g., “cause”, “IS_SAME_AS”, “participating events”) and nodes from a knowledge graph database; each audio / video segment is associated with a timestamp (obtained from video metadata or time-aligned transcription).

[0106] The sorting process involves using a graph database (Neo4j) to query and traverse the knowledge graph database. Starting from the matching speaker node or event node, related fragments are collected along the edges, such as connecting the same party across videos through the "IS_SAME_AS" edge.

[0107] Timeline algorithms (such as timestamp-based topological sorting) are applied to the collected clips or event nodes to generate ordered sequences. For example, clips within the same video are first sorted, and then cross-video events are connected through logical edges. LLM injects prompt templates based on the sorting results to generate a natural language timeline.

[0108] Cross-scene processing: Extend to other videos via the "IS_SAME_AS" edge (e.g., from the timestamp T1 of the street patrol to the timestamp T2 of the interrogation room), and annotate transitions (e.g., "time interval approximately 20 minutes"). Example: From Extract the "interrogation" event (T1: street patrol) and the "questioning" event (T2: interrogation room), and generate "First law enforcement action: [time 1], subsequent: [time 2]" by sorting by timestamp.

[0109] Efficiency optimization: Graph database indexes accelerate traversal, with an average response time of less than 1 second.

[0110] (2) Entity association is based on the extraction of subgraphs from the knowledge graph database. Starting from the matching nodes of the fusion retrieval set, the edges of the knowledge graph database are traversed to extract relevant subgraphs and display the associations between entities (such as speakers, events, and objects), such as links between people, locations, and events.

[0111] Data source: Nodes and edges in a knowledge graph database.

[0112] Subgraph extraction and processing: Using graph database queries, starting from matching nodes (such as speaker nodes) in the fused retrieval set, edges are traversed to extract relevant subgraphs. For example, cross-scene related entities are extracted from the "Party Member" node along the "IS_SAME_AS" edge. Specifically, Depth-First Search (DFS) or Breadth-First Search (BFS) is used to traverse the neighboring nodes of the node, limiting the depth (e.g., 3 levels) to avoid excessively large subgraphs. This utilizes GAT's... Filter out low-relevance entities. LLM processes subgraph data to generate natural language descriptions, such as "person association: the same person and multiple law enforcement officers; location association: from street A to square B".

[0113] Cross-scene processing: Connect entities in different videos through the "IS_SAME_AS" edge (e.g., "the parties involved in the street - IS_SAME_AS -> the parties involved in the square"), and extract subgraphs to highlight continuity.

[0114] Example: Extract a subgraph from the knowledge graph database, including the "Party Affiliation" node, the "Law Enforcement Officer" node, the "IS_SAME_AS" edge, and the "Encirclement" edge, to generate "Entity Association: The same party (confirmed by voiceprint matching) and multiple law enforcement officers; Event Association: Follow-up verification caused by failure to cooperate with the inquiry".

[0115] Efficiency optimization: Indexing accelerates subgraph extraction, and privacy is protected (e.g., encrypted voiceprint-related edges).

[0116] (3) Event summary is a concise description of the cause, development process and handling result of an event, which is summarized by using LLM to summarize the search results and subgraphs.

[0117] Data source: Key information from the fusion retrieval set and results extracted from subgraphs.

[0118] Summary of the process: LLM first constructs a prompt template (such as "Generate an event summary based on the following event chain and text block: cause, development, and result"), injects the retrieval results (such as text blocks, subgraph nodes, and edges), and uses LLM to generate a natural language summary.

[0119] Cross-scene processing: Summary integration of cross-video events, such as descriptions of the street and interrogation room connected by the "IS_SAME_AS" edge.

[0120] Example: LLM input subgraph ("interrogation -> inquiry") and text block ( =“Relative A is out of contact”, generating “Cause: Dispute caused by relative A being out of contact; Development process: Persuasion and mediation by law enforcement officers; Resolution: Resolved through legal channels”.

[0121] Efficiency optimization: LLM hints for template pre-optimization, reducing computational iterations.

[0122] (4) Key findings are to detect anomalies or patterns in events, such as contradictions (inconsistencies in dialogue) and recurrences (such as the same party being involved in the case multiple times).

[0123] Data sources: results of subgraph extraction, text block and embedding similarity comparison.

[0124] Anomaly detection process: 1. Inconsistency detection: Compare the semantic consistency between text blocks, using LLM or embedding similarity to identify inconsistencies, such as street transcription of "denying violation" versus interrogation room "admitting violation". 2. Pattern recognition: Through GAT... Scan subgraphs to detect recurring patterns, such as multiple "IS_SAME_AS" edges indicating repeated involvement by the same party. 3. LLM summary anomalies: Inject prompt templates (such as "Detect contradictions and recurring patterns in the following subgraphs") and generate key finding descriptions.

[0125] Cross-scene processing: Detect cross-video anomalies using "IS_SAME_AS", such as contradictions in the behavior of the same party.

[0126] Example: Detect repeated edges in the subgraph for "Party member - violation -> traffic rules" (using "IS_SAME_AS") and generate "Key finding: The party member has multiple similar violation records (cross-video correlation finding), and it is recommended to strengthen monitoring".

[0127] Efficiency optimization: Automatic filtering of thresholds (e.g., similarity <0.7 indicates contradiction), and GAT-accelerated scanning mode.

[0128] (5) The archiving recommendation is to recommend merging archives based on the correlation strength. Based on the correlation strength of events, it is recommended whether to merge videos or events into a single archive and mark high-risk entities.

[0129] Data source: Subgraph extraction results and GAT attention coefficients.

[0130] Association Strength Calculation: Association strength is quantified using an attention coefficient, such as a high attention coefficient for the edge "IS_SAME_AS" indicating a strong association. If the attention coefficient is greater than a threshold (e.g., 0.7), merging is recommended (e.g., merging street and interrogation room videos into the same event). LLM Suggestion Generation: A prompt template (e.g., "Based on association strength, recommend archiving methods") is injected, and natural language suggestions are output.

[0131] Cross-scene processing: Calculate the cross-video intensity using the "IS_SAME_AS" edge. If the intensity is high, it is recommended to "merge them into a single case file".

[0132] Example: The attention coefficient of "street party - IS_SAME_AS -> square party" in the sub-image is 0.8, generating "Archiving suggestion: merge the street and square videos into a single case file and mark high-risk individuals".

[0133] Efficiency optimization: threshold automation, privacy protection (such as encrypting high-risk entities).

[0134] This embodiment links retrieval generation with the construction of a voiceprint-assisted knowledge graph database: the retrieval generation stage directly relies on the output of the construction stage, using the knowledge graph database as the core data source to perform multi-channel retrieval of user queries, and then fusing the results to generate a response. This query triggering process and the seamless connection between the construction stage and the retrieval stage support an end-to-end pipeline: the output of the construction stage is directly fed into the retrieval function; cross-scene voiceprint matching in the construction stage enhances the expansion of the retrieval (e.g., the voiceprint retrieval set covers multiple videos), which is reflected in the report as key findings and archiving recommendations; MEnc's multi-layer attention reduces parameter redundancy, improves generalization, and achieves a recall rate greater than 95%, which is reflected in the construction of symbols (e.g., The query result for "law enforcement officers questioning people on the street at night" can be directly used for processing and supports automatic archiving.

[0135] This embodiment achieves cross-scene semantic and speaker association:

[0136] Speech Segment Node Management: Create a speech segment node for each speech segment, storing text transcription, timestamp, sentiment tag (inferred using a sentiment analysis model, such as "angry" or "calm"), and voiceprint vector. Establish a "generative" relationship edge (HAS_SPEECH) from the speaker node to the speech segment node and inject metadata such as scene type ("street patrol").

[0137] Cross-scene voiceprint matching: Iterates through speaker nodes in different law enforcement scenarios (based on video metadata classification) and calculates the cosine similarity between aggregated voiceprints. If the similarity exceeds a threshold, a "similar" relationship edge (IS_SAME_AS) is created to automatically associate the same speaker. Simultaneously, node attributes are updated to record the association confidence. To handle noise, a robust matching strategy is used: the threshold is adjusted based on context (such as location and temporal proximity); if the similarity is between 0.7 and 0.75, a manual verification prompt is triggered.

[0138] Semantic association extension: through the semantic relation edges of the knowledge graph This involves linking events and entities across different scenarios. For example, linking the same person's "questioning" behavior in a street patrol video with their "confession" behavior in an interrogation room video. A cross-scenario subgraph is constructed: starting from matching speaker nodes, connected entities and relationships are traversed to form an event chain. This is then integrated into a comprehensive analysis report of related events. For example, cross-scenario transitions are marked in the event timeline, cross-video links are listed in the entity correlation section, and recurring patterns are highlighted in key findings, such as "the same person involved in multiple cases."

[0139] Privacy and efficiency optimization: Voiceprint data is encrypted during the association process and decrypted only during authorized queries; graph database indexes are used to accelerate cross-scenario queries, with an average response time of less than 1 second.

[0140] Step S4: The task management module updates the task status to "completed", records the progress as 100% and the result statistics, including the generated comprehensive analysis report of related events.

[0141] If an error occurs, the task status is set to "failed", the error message is logged, and sensitive data is cleaned up.

[0142] The implementation process of this system is illustrated below through two specific examples.

[0143] Example 1: Analysis of law enforcement records

[0144] Input: A set of body camera videos, totaling 50 hours in length, including street patrol and interrogation scenes.

[0145] Processing flow:

[0146] Task initialization: Receive video metadata and create an encrypted temporary directory.

[0147] Knowledge graph database construction: The audio and video are segmented into multiple segments. Visual frames are sampled using an intelligent sampling algorithm based on inter-frame change rate and time interval. Then, facial pose is scored to form a list of each person's face. Only the photo with the highest pose score is stored. Subtitles, audio transcription, and voiceprint vectors are generated to construct a knowledge graph database (entities: law enforcement officers, parties involved; relationship: questioning).

[0148] Voiceprint analysis and association: Identify 8 speakers, generate aggregated voiceprint vectors, create speaker and speech segment nodes, and associate the same person in street patrols and interrogations through voiceprint similarity.

[0149] Query processing: When a user queries "the dialogue between the parties involved in a street interrogation", the system retrieves relevant fragments from the knowledge graph database, generates a comprehensive response, and outputs a comprehensive analysis report of related events.

[0150] Example of a comprehensive analysis report on related events:

[0151] Event sequence: First enforcement action: [Time 1], law enforcement officers handle a family dispute; Subsequent enforcement action: [Time 2], handling internal family conflicts, with an interval of approximately 20 minutes.

[0152] Entity relevance: Personnel relevance: involving the same family member, such as "relative A" and the person who reported the incident; Location relevance: [Location A]; Event relevance: continuous stages of internal family conflict.

[0153] Event Summary: Cause: Dispute arising from the disappearance of relative A; Development Process: Persuasion and mediation by law enforcement officers; Outcome: Resolved through legal channels without forced intervention.

[0154] Key findings: The incidents involved a series of conflicts within the same family, and the person who reported the incident was a key instigator.

[0155] Archiving suggestion: Merge archives into the same event to provide complete background.

[0156] Results: The system achieved high accuracy on 500 queries, significantly outperforming the traditional RAG method.

[0157] Example 2: Surveillance video retrieval

[0158] Input: A set of surveillance videos, totaling 30 hours in length, including scenes of city streets and public places, such as nighttime street patrols and public square incident responses.

[0159] Processing flow:

[0160] Task initialization: Receive video metadata, including timestamps and location information, and create an encrypted temporary directory to store intermediate audio files and voiceprint data.

[0161] Knowledge graph database construction: Video is segmented, and visual frames are sampled using an intelligent sampling algorithm based on inter-frame change rate and time interval (e.g., sampling flashing light frames from a nighttime street clip). Facial pose scores are then determined, creating a list of each person's face. Only the photo with the highest pose score is stored (e.g., selecting a clear, frontal photo of the person in question). Subtitles (e.g., "Law enforcement vehicles flashing lights on a nighttime street, law enforcement officers surrounding the person in question"), audio transcription (capturing dialogue such as "Please show your ID"), and voiceprint vectors are generated to construct a knowledge graph database (entities: person in question, law enforcement officers, vehicle; relationships: surrounding, questioning).

[0162] Voiceprint analysis and association: Identify 6 speakers (including repeating individuals), generate aggregated voiceprint vectors, and create speaker and speech segment nodes. Use cosine similarity (threshold 0.75) to associate the voiceprints of individuals in urban street videos with the same person in public square videos, achieving cross-scene identity linking (e.g., associating "uncooperative inquiries" on the street with "follow-up verification" in the square).

[0163] Query Processing: When a user queries "behavior and dialogue of individuals on a street at night," a multimodal retrieval is performed using a knowledge graph database: text matching extracts dialogue transcripts, visual retrieval matches flashing light scenes, and voiceprint retrieval associates the identities of the individuals involved. A comprehensive response is generated, including links to key segments, and a comprehensive analysis report of the related events is output.

[0164] Example of a comprehensive analysis report on related events:

[0165] Event sequence: Initial monitoring: [Time 1], the person involved was discovered during a nighttime street patrol; Subsequent response: [Time 2], follow-up verification action was carried out in the public square, with an interval of approximately 15 minutes.

[0166] Entity correlation: Personnel correlation: The same party (confirmed by voiceprint matching) and multiple law enforcement officers; Location correlation: Transfer from [street A] to [square B]; Event correlation: Failure to cooperate with the inquiry, resulting in subsequent follow-up verification.

[0167] Event Summary: Cause: The person involved violated traffic rules and was stopped for questioning; Development: The person involved fled, and law enforcement officers tracked them down to the square; Outcome: The person involved cooperated with the verification and record.

[0168] Key finding: The individual had multiple similar violation records (found through cross-video correlation), and it is recommended to strengthen monitoring.

[0169] Archiving recommendation: Combine street and square videos into a single case file and label high-risk individuals.

[0170] Results: The system accurately retrieves key fragments and generates comprehensive responses and reports that include dialogue, scene descriptions, and speaker identities. It achieves a recall rate of 95% in 300 queries, which is better than the baseline method.

[0171] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-modal retrieval augmentation generation system for mass law enforcement audio-video data, characterized in that, The method comprises the following steps: Based on the multi-modal index module of voiceprint assistance, multi-modal retrieval and generation module and knowledge graph database module, The multi-modal index module based on voiceprint assistance is used to generate subtitles, transcripts and voiceprint vectors from the audio and video of the recorder, merge the subtitles, transcripts and voiceprint vectors to form a multi-modal representation, use a multi-modal fusion encoder to process the multi-modal representation, and construct a knowledge graph database; The multi-modal retrieval and generation module is used to receive a user query, extract multi-modal keywords, query the knowledge graph database according to the multi-modal keywords, obtain a text retrieval set, a visual retrieval set and a voiceprint retrieval set, and integrate them to form a fusion retrieval set; According to the fusion retrieval set, a VLM and a LLM are used to generate a comprehensive analysis report of related events; The multi-modal keywords include a potential voiceprint identifier; the voiceprint vector of a related speaker node is extracted from the knowledge graph database according to the potential voiceprint identifier, all related segments of the voiceprint vector are extracted, and a voiceprint retrieval set is formed; The knowledge graph database module is used to store and manage the knowledge graph database; The nodes of the knowledge graph database include speaker nodes, voice segment nodes, event nodes and object nodes; The edges include identity association edges, voice generation edges, behavior association edges and event association edges; The voice generation edge connects the speaker node and the voice segment node, and depends on voiceprint vector matching + subtitle / transcript time alignment.

2. A multi-modal search and retrieval augmentation generation system for mass law enforcement audio-visual data as claimed in claim 1 wherein, The process of forming a multi-modal representation is as follows: The recorder audio and video is divided into multiple segments; Visual frames are sampled from each segment, input into a visual language model, and subtitles are generated; Automatic speech recognition is used to generate the transcript of each segment; Speaker separation is performed on each segment to generate a voiceprint vector; The subtitles, transcripts and voiceprint vectors are merged to form a multi-modal representation.

3. A multi-modal search and retrieval augmentation generation system for mass law enforcement audio-visual data as claimed in claim 2 wherein, The process of sampling visual frames is as follows: the average gray difference between the current frame and the last sampled frame is calculated as the change rate, if the change rate exceeds the change rate threshold and the time interval exceeds the minimum interval, the current frame is taken as the sampling frame, and the first frame is always sampled.

4. A multi-modal search and retrieval augmentation generation system for mass law enforcement audio-visual data as claimed in claim 1 wherein, The processing process of the multi-modal fusion encoder is as follows: The subtitles and transcripts are respectively input into embedding layers to obtain subtitle embeddings and audio transcript embeddings, then the first layer output is obtained through a cross-attention mechanism; The first layer output and the voiceprint vector are integrated with identity information using a causal multi-head self-attention mechanism to obtain a second layer output; The second layer output and the semantic relationship embedding of the knowledge graph are input into a graph attention network to obtain updated node embeddings and an attention coefficient matrix.

5. A multi-modal search and retrieval augmentation generation system for mass law enforcement audio-visual data as claimed in claim 1 wherein, The multi-modal keywords include text keywords and visual descriptions; The corresponding segments are extracted from the knowledge graph database according to the text keywords to form a text retrieval set; The visual description is converted into a visual embedding vector, and the corresponding segments are extracted from the knowledge graph database to generate a visual retrieval set.

6. A multi-modal search and retrieval augmentation generation system for mass law enforcement audio-visual data as claimed in claim 1 or 5 wherein, The text retrieval set, the visual retrieval set and the voiceprint retrieval set are integrated, redundant segments are removed using a weighted fusion mechanism, and a fusion retrieval set is formed.

7. A multi-modal search and retrieval augmentation generation system for mass law enforcement audio-visual data as defined in claim 1, wherein, The comprehensive analysis report of related events contains at least one of the following analysis dimensions: event timing, entity association, event summary, key findings and archiving suggestions.

8. A multi-modal search enhancement generation system for mass law enforcement audiovisual data as defined in claim 7, wherein, Event timeline is a timeline ordering based on the edges of the knowledge graph database, which orders the cross-scene events or fragments in chronological order to form an event timeline; Entity relevance is to extract relevant subgraphs by traversing the edges of the knowledge graph database from the matching nodes of the fusion retrieval set, and to show the association between entities; Event summary is to use LLM to summarize the retrieval results and subgraphs to generate a concise description of the cause, development process and disposal result of the event; Key findings are to detect anomalies or patterns in events based on the results of subgraph extraction, text blocks and embedded similarity comparisons; Archiving recommendations are based on the strength of event association to recommend whether to merge videos or events into a single archive.

9. A multi-modal search and retrieval augmentation generation system for mass law enforcement audio-visual data as defined in claim 1, wherein, It also includes a task management module for receiving and initializing law enforcement audio and video processing tasks and managing task status.

Images