Method and system for intelligently generating alarm processing process report based on law enforcement recorder video

Abstract

The invention relates to the technical field of artificial intelligence, and particularly discloses an alarm processing process report intelligent generation method and system based on a law enforcement recorder video, and the method comprises the steps: obtaining law enforcement recorder video data and a query text; separating the video data of the law enforcement recorder to obtain video stream data and audio stream data; identifying the audio stream data to obtain an audio text; performing semantic analysis on the video stream data and the audio text to obtain multi-modal sentiment analysis semantic features; obtaining visual semantic features corresponding to the segments according to the video stream data and the query text; and according to the visual semantic features and the multi-modal sentiment analysis semantic features corresponding to the segments, determining the case type, calling a corresponding alarm processing process report template, and generating an alarm processing process report. According to the method, the audio and video data in the video of the law enforcement recorder can be quickly analyzed, key information is extracted, and an accurate, detailed and timeliness alarm processing process report is generated.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method and system for intelligently generating police response process reports based on law enforcement recorder video. Background Technology

[0002] Body-on-camera (BOC) devices are widely used to record audio and video data during law enforcement processes. They not only standardize law enforcement behavior and improve transparency, but also provide strong evidentiary support for cases. However, existing BOC video data often fails to be fully analyzed and utilized, leaving a large amount of audio and video data idle and unable to be effectively transformed into powerful support for improving law enforcement efficiency and quality. Traditional manual review and processing methods are time-consuming, labor-intensive, and easily influenced by subjective factors, failing to effectively improve the speed and quality of case processing.

[0003] With the development of artificial intelligence technology, especially the advancements in speech recognition and large language modeling, it has become possible to convert law enforcement recorder videos into analyzable text and generate related reports. However, existing technologies still have the following shortcomings: (1) Speech recognition technology is not very accurate and has difficulty handling complex background noise and multiple speech sources.

[0004] (2) Existing semantic analysis methods are difficult to accurately extract key information and cannot quickly locate key events in the law enforcement process.

[0005] (3) The technology for automatically generating reports has not yet fully addressed the need for personalization, and the report content often lacks flexibility.

[0006] Therefore, how to efficiently and accurately extract key information from law enforcement recorder videos and generate valuable police response reports has become a pressing technical challenge. Summary of the Invention

[0007] The present invention aims to solve the aforementioned technical problems. To this end, the present invention provides an intelligent generation method and system for police response reports based on law enforcement recorder video. Through speech recognition, large language models, and semantic retrieval technologies, it can quickly analyze audio and video data from law enforcement recorder videos, extract key information, and generate accurate, detailed, and timely police response reports. This improves the efficiency and utilization value of law enforcement recorder video data analysis, reduces manual intervention, and enhances law enforcement efficiency.

[0008] This invention provides a method for intelligently generating police response reports based on law enforcement recorder video. The technical solution adopted is as follows: (Including...) Obtain video data and query text from law enforcement recorders; Based on the video data from the law enforcement recorder, video stream data and audio stream data are separated. The audio stream data is identified to obtain audio text; Semantic analysis is performed on the video stream data and audio text to obtain multimodal sentiment analysis semantic features; Based on video stream data and query text, visual feature extraction, text feature extraction, feature fusion and regression prediction are used to retrieve segments related to the query text from the video stream data and obtain the visual semantic features corresponding to the segments. Based on the visual semantic features and multimodal sentiment analysis semantic features corresponding to the fragments, the case type is determined, the corresponding police response process report template is called, and a police response process report is generated.

[0009] Furthermore, based on the video data from the law enforcement recorder, video stream data and audio stream data are separated. The specific process is as follows: The video data from the law enforcement recorder is split into video data and audio data, and the video data is used as video stream data; The video data is used to extract features through a facial feature extraction network and a motion feature extraction network to obtain facial features and motion features respectively; The audio data is processed by an audio feature extraction network to extract audio features. The facial features, motion features, and audio features are fused across modally using a cross-modal fusion module to obtain audiovisual features. The audio stream data is obtained by processing the audiovisual features and audio features through a speech separation network.

[0010] Furthermore, the audio stream data is identified to obtain audio text, the specific process of which is as follows: The audio stream data is transformed using a short-time Fourier transform to generate a two-dimensional spectrogram. The two-dimensional spectrogram generates a spectrogram of a clear speech signal through an hourglass-shaped speech recognition model; The spectrogram of the clear speech signal is decoded to generate audio text.

[0011] Furthermore, the hourglass-shaped speech recognition model includes five simple recursive units connected in sequence. The first and fifth simple recursive units are skipped, as are the second and fourth simple recursive units. Attention gates are added to the skipped connections.

[0012] Furthermore, semantic analysis is performed on the video stream data and audio text to obtain multimodal sentiment analysis semantic features. The specific process is as follows: Audio text is encoded using a text encoder to generate text feature representations; video stream data is encoded using an image encoder to generate image feature representations. Text feature representation and image feature representation are generated through a unimodal variational attention layer to generate text latent representation and image latent representation, respectively; Text latent representation and image latent representation are fused across a cross-modal attention layer to generate a multimodal representation; The multimodal representation is sequentially passed through a multilayer perceptron and a conditional random field module to obtain the semantic features of multimodal sentiment analysis.

[0013] Furthermore, the specific process for obtaining the visual semantic features corresponding to the fragment is as follows: The video stream data is processed by a visual feature extraction module to extract global motion features. The query text is processed by the text feature extraction module to obtain a text feature vector; Global action features and text feature vectors are fused through the feature fusion module using an action-language attention mechanism to obtain visual semantic features; Visual semantic features are used by a regression prediction module to obtain fragments related to the query text; Based on the fragments related to the query text, obtain the visual semantic features corresponding to the fragments.

[0014] Furthermore, the specific process for generating an emergency response report is as follows: Based on the visual semantic features and multimodal sentiment analysis semantic features corresponding to the fragments, structured features are obtained by fusing them together; The case type is determined based on the structured characteristics, and the corresponding police response process report template is invoked. Generate an emergency response report based on structured features and an emergency response process report template.

[0015] Furthermore, by using the visual semantic features corresponding to the fragments, the most relevant historical police dispatch records are retrieved from the historical police dispatch record data. The police dispatch process report features of the most relevant historical police dispatch records are fused with the structured features, and the case type is determined based on the fused features and structured features.

[0016] Furthermore, the types of cases include civil disputes, criminal cases, and traffic accidents.

[0017] This invention also provides an intelligent report generation system for police response processes based on law enforcement recorder video. The technical solution adopted is as follows: it includes a data acquisition module, a video-audio separation module, a speech recognition module, a semantic analysis module, a video semantic retrieval module, and a report generation module. The data acquisition module is used to acquire video data from the law enforcement recorder and query text; The video and audio separation module is used to separate video stream data and audio stream data based on the video data from the law enforcement recorder. The speech recognition module is used to recognize the audio stream data to obtain audio text; The semantic analysis module is used to perform semantic analysis on the video stream data and audio text to obtain multimodal sentiment analysis semantic features; The video semantic retrieval module is used to retrieve segments related to the query text from the video stream data based on the video stream data and the query text through visual feature extraction, text feature extraction, feature fusion and regression prediction, and obtain the visual semantic features corresponding to the segments. The report generation module is used to determine the case type based on the visual semantic features and multimodal sentiment analysis semantic features corresponding to the fragment, call the corresponding police response process report template, and generate a police response process report.

[0018] The above-described one or more technical solutions in the embodiments of the present invention have at least one of the following technical effects: 1. This invention utilizes advanced speech recognition, semantic analysis, and large language model technologies to significantly improve the analysis efficiency and utilization value of law enforcement recorder data, providing support for more efficient and accurate law enforcement, and has broad application prospects and social value.

[0019] 2. When extracting audio stream data, this invention introduces an attention mechanism in the cross-modal fusion module, which can make full use of the correlation information between the visual modality and the audio modality, and improve the accuracy and robustness of the fusion features.

[0020] 3. The report generation module of this invention adopts a novel event-driven alert and dispatch report generation framework, which ensures the logical consistency, completeness, and accuracy of the generated reports. Furthermore, to further improve the completeness and flexibility of report generation, cross-modal feature enhancement technology is used in the report generation module. By retrieving historical dispatch records of similar cases from the database, the accuracy and quality of the current case handling process report are enhanced.

[0021] 4. This invention also has the following features: Improved work efficiency: Automated audio transcription and report generation greatly reduce the workload of law enforcement officers in the case handling process, saving a significant amount of time; Reduced human intervention: Through automated technology for analysis and report generation, subjective biases that may occur in manual analysis are reduced, ensuring the accuracy of reports; Enhanced law enforcement transparency: The audio and video data from law enforcement recorders can be fully analyzed, ensuring that every law enforcement action is transparently recorded and reasonably evaluated; Personalized report generation: Based on the different characteristics and requirements of cases, personalized police response process reports are generated to meet different law enforcement needs.

[0022] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0024] Figure 1 This is a flowchart of the method provided by the present invention.

[0025] Figure 2 This is a flowchart of the video and audio separation module provided by the present invention.

[0026] Figure 3 This is a flowchart of the speech recognition module provided by the present invention.

[0027] Figure 4 This is a flowchart of the semantic analysis module provided by the present invention.

[0028] Figure 5 This is a flowchart of the video semantic retrieval module provided by the present invention.

[0029] Figure 6 This is a flowchart of the report generation module provided by the present invention.

[0030] Figure 7 This is a structural block diagram of the system provided by the present invention.

[0031] Figure label: 1. Data acquisition module; 2. Video and audio separation module; 3. Speech recognition module; 4. Semantic analysis module; 5. Video semantic retrieval module; 6. Report generation module. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention. The following embodiments are used to illustrate this invention but should not be used to limit the scope of this invention.

[0033] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0034] The following is combined Figures 1 to 7 The present invention will be further described in detail below, which describes a method and system for intelligently generating police response reports based on law enforcement recorder video: In this embodiment, as Figure 1 As shown, a method for intelligently generating police response reports based on law enforcement recorder video is provided, including the following steps: Step 1: Obtain video data and query text from the law enforcement recorder.

[0035] Body camera video data refers to the audio and video data recorded by body cameras. Query text is text provided by the user.

[0036] Step 2: Using the video-audio separation module, separate the video stream data and audio stream data from the law enforcement recorder video data.

[0037] Law enforcement recorder video data can be split into two parts: video data and audio data. The video data requires no further processing and can be directly used as the video stream. The video and audio data are processed by the facial feature extraction network, audio feature extraction network, motion feature extraction network, cross-modal fusion module, and speech separation network of the video-audio separation module to obtain the audio stream data.

[0038] like Figure 2 As shown, the specific process is as follows: The video data from the law enforcement recorder is split into video data and audio data, and the video data is used as video stream data; The video data is used to extract features through a facial feature extraction network and a motion feature extraction network to obtain facial features and motion features respectively; The audio data is processed by an audio feature extraction network to extract audio features. The facial features, motion features, and audio features are fused across modally using a cross-modal fusion module to obtain audiovisual features. The audio stream data is obtained by processing the audiovisual features and audio features through a speech separation network.

[0039] In this embodiment, the facial feature extraction network consists of a 3D convolutional layer and a lightweight ShuffleNetv2 network. By inputting consecutively stacked facial image frames, the facial feature extraction network generates facial feature vectors, which characterize the spatial and temporal dynamics of the facial region within the video frame, providing rich visual feature support for subsequent feature fusion.

[0040] To capture motion features in videos, this embodiment uses a pre-trained I3D model as the motion feature extraction network. The I3D model supplements the missing temporal dimension by extending a two-dimensional convolutional network to three dimensions, while simultaneously estimating optical flow information using grayscale images. The motion feature vector output by the motion feature extraction network is adjusted to the same dimension as facial features by multiplying the temporal dimension by the channel dimension. Motion features and facial features are physically and semantically complementary, and are ultimately input together into the cross-modal fusion module.

[0041] In the cross-modal fusion module, a multi-head attention mechanism is first used to perform affine transformation on facial and motion features to generate visual feature vectors. Subsequently, the visual feature vectors are fused with audio features across modalities to generate audiovisual features. Through the attention mechanism, the cross-modal fusion module can fully utilize the correlation information between visual modalities (facial and motion) and audio modalities, improving the accuracy and robustness of the fused features.

[0042] The speech separation network, based on U-Net, consists of an encoder and a decoder. The encoder receives audio features as input and compresses and reduces the dimensionality of the complex spectrogram through multiple convolutional and pooling layers. The decoder takes audiovisual features as input and generates a predicted composite mask. The composite mask is multiplied by the spectrogram and then subjected to an inverse short-time Fourier transform (ISTFT) to generate the separated speech signal, which serves as the audio stream data. This speech separation network can accurately and efficiently separate target speech, improving the clarity and integrity of the speech signal, thus supporting subsequent analysis and applications.

[0043] Step 3: The audio stream data is recognized by the speech recognition module to obtain the audio text.

[0044] like Figure 3 As shown, the specific process is as follows: Step 3.1: The audio stream data is transformed by short-time Fourier transform to generate a two-dimensional spectrogram.

[0045] Step 3.2: The two-dimensional spectrogram is used to generate a spectrogram of a clear speech signal through an hourglass-shaped speech recognition model.

[0046] The hourglass-shaped speech recognition model consists of five simple recursive units (SRUs) connected in sequence. The first and fifth simple recursive units are skipped, as are the second and fourth simple recursive units. Attention gates are added to the skipped connections.

[0047] The hourglass-shaped speech recognition model employs a unique pyramidal architecture. The upper pyramid has an increasing number of neurons and a decreasing time step to capture short-term dependencies, while the lower pyramid has a decreasing number of neurons and an increasing time step to capture long-term correlations in the speech signal. This design enables efficient processing and representation of speech spectral features while reducing computational complexity. In this embodiment, cross-layer skip connections and attention gating mechanisms are added to the traditional hourglass-shaped speech recognition model, further enhancing the extraction capability of key spectral regions, significantly improving the stability of gradient flow, and avoiding the gradient vanishing problem in deep networks. The hourglass-shaped speech recognition model utilizes attention weights to highlight key regions in the spectrum, such as dominant formants in the low-frequency band and sparse distributions in the high-frequency band, thereby enhancing the accuracy of signal processing. Each SRU can efficiently preserve the temporal dependency characteristics of the speech signal while optimizing computational complexity.

[0048] In this embodiment, the number of neurons and time steps of the five SRUs are as follows: the first layer SRU has 512 neurons and 512 time steps, the second layer SRU has 1024 neurons and 256 time steps, the third layer SRU has 2048 neurons and 128 time steps, the fourth layer SRU has 1024 neurons and 256 time steps, and the fifth layer SRU has 512 neurons and 512 time steps.

[0049] Step 3.3: The spectrogram of the clear speech signal is decoded to generate audio text.

[0050] In the decoder, the spectrogram of the clear speech signal is first restored to the clear speech signal through the inverse short-time Fourier transform (ISTFT), and then the audio text is generated.

[0051] The training process for the speech recognition module is as follows: (1) Input signal The input to the speech recognition module is a noisy speech signal. It is a clear signal. and noise signals The mixture is represented as: Noisy speech signals are the target that the speech recognition module needs to process.

[0052] (2) Time-frequency conversion First, the frequency domain representation is obtained by converting the frequency domain representation using a short-time Fourier transform (STFT). STFT decomposes a time-domain signal into a two-dimensional representation of time and frequency, facilitating subsequent spectral analysis and processing. (Spectrum of a noisy speech signal) It can be written as: in, It is the spectrum of a clear signal. It is the spectrum of the noise signal.

[0053] (3) Robust feature extraction from The combined feature set is extracted, which includes: Mel frequency cepstral coefficients (MFCC), relative spectrum transform sensing linear prediction coefficients (RASTA-PLP), Gammatone filter bank energy (GFE), and amplitude modulation spectrogram (AMS). The combination of these features provides rich acoustic information, which helps the model better learn the relationship between noise and clear signals.

[0054] (4) Hourglass-shaped speech recognition model processing The extracted features are input into an hourglass-shaped speech recognition model for processing.

[0055] The hourglass-shaped speech recognition model can effectively capture the long-term time dependence in speech signals.

[0056] The core structure of the hourglass-shaped speech recognition model includes: Hourglass architecture: The number of neurons and time step of the SRU vary between different layers, forming an hourglass shape.

[0057] Skip connections: Introducing skip connections between non-adjacent layers can alleviate the gradient vanishing problem.

[0058] Attention gating: Incorporating attention gating mechanisms into skip connections to highlight important spectral regions and features.

[0059] The output of the hourglass-shaped speech recognition model is an estimated time-frequency mask (TF Mask), which includes an ideal ratio mask (IRM) and an ideal binary mask (IBM).

[0060] (5) Time-frequency mask Estimated time-frequency mask and Element-wise multiplication yields the enhanced speech spectrum. The purpose of this step is to suppress noise components and preserve or enhance speech components.

[0061] (6) Sound wave reconstruction Enhanced speech spectrum The enhanced speech signal is obtained by converting it back to the time domain using the inverse short-time Fourier transform (ISTFT). ISTFT resynthesizes the frequency domain signal into a time domain signal, completing the final output of speech enhancement.

[0062] (7) Output signal The module outputs an enhanced speech signal. It is better than the input noisy speech signal. It offers greater clarity and intelligibility. This signal can be directly used for subsequent semantic analysis.

[0063] The design of the speech recognition module significantly improves its accuracy and efficiency while also meeting the requirements of lightweight design and low latency. This makes it ideal for deployment in practical law enforcement recorder systems, providing strong technical support for voice data processing during intelligent law enforcement. The speech recognition module demonstrated excellent performance in experiments, especially in real-time processing environments, effectively balancing computational complexity and performance requirements. It not only significantly improved the intelligibility and perceptual quality of the speech signal but also provided high-quality foundational data for subsequent semantic analysis and the generation of police response reports.

[0064] Step 4: Perform semantic analysis on the video stream data and audio text using the semantic analysis module to obtain multimodal sentiment analysis semantic features.

[0065] The semantic analysis module combines text and video features to extract key semantic information, reduce multi-level noise and multi-granularity semantic gaps, and achieve Joint Multimodal Aspect-Sentiment Analysis (JMASA). By combining multimodal sentiment analysis and constrained optimization strategies, the semantic analysis module aims to address noise issues and multi-granularity semantic gaps in multimodal data, thereby improving the accuracy and robustness of sentiment analysis.

[0066] like Figure 4 As shown, the specific process is as follows: Audio text is encoded using a text encoder to generate text feature representations; video stream data is encoded using an image encoder to generate image feature representations. In this embodiment, the text encoder uses the RoBERTa model, and the image encoder uses the ViT model.

[0067] Text feature representation and image feature representation are generated through a unimodal variational attention layer to produce text latent representation and image latent representation, respectively.

[0068] Text latent representation and image latent representation are fused across a cross-modal attention layer to generate a multimodal representation; The multimodal representation is sequentially passed through a multilayer perceptron (MLP) and a conditional random field (CRF) module to obtain the semantic features of multimodal sentiment analysis.

[0069] During the training of the semantic analysis module, three key constraints—Global Relevance Constraint (GR-Con), Information Bottleneck Constraint (IB-Con), and Semantic Consistency Constraint (SC-Con)—are used to optimize the emotion recognition process for audio and video modalities. During training, GR-Con is explicitly applied, while IB-Con and SC-Con are implicitly applied. Figure 4 Only GR-Con is highlighted in the text. GR-Con is explicitly applied in the cross-modal fusion attention layer by calculating a global relevance score between text and image. To regulate multimodal fusion, IB-Con is primarily applied implicitly in the unimodal variational attention layer, generating optimal latent representations through a variational attention mechanism. SC-Con captures semantic consistency at multiple granularities through a contrastive learning strategy, implicitly reflected in the overall framework design and data flow.

[0070] GR-Con aims to reduce instance-level noise between modalities by measuring the global similarity between audio and video. In multimodal sentiment analysis, audio and video often differ in timeline and emotional expression, leading to mismatches. GR-Con adjusts multimodal fusion by calculating the global similarity between audio features (such as pitch and speech rate) and video features (such as facial expressions and body language), ensuring consistency in emotional expression between audio and video modalities. This constraint helps the model reduce the impact of noise overall and improve the accuracy of sentiment analysis.

[0071] IB-Con optimizes feature-level noise reduction and effectively controls information flow using the Information Bottleneck (IB) theory. In multimodal sentiment analysis, audio and video provide different information, so it's necessary to avoid the propagation of redundant information during fusion. The IB-Con constraint compresses features from audio and video modalities by minimizing the KL divergence, ensuring that each modality only transmits the most critical information and reduces interference from irrelevant information. This constraint optimizes the latent representation of features, allowing the model to focus more on key information relevant to sentiment analysis and improving the accuracy of sentiment recognition.

[0072] SC-Con enhances semantic consistency between modalities by maximizing mutual information (MI) between audio and video. Audio and video modalities differ in their sentiment expression, thus SC-Con is needed to ensure consistency in sentiment analysis. SC-Con utilizes a contrastive learning mechanism to maximize semantic alignment between audio and video modalities by minimizing the InfoNCE loss, thereby enhancing the accuracy of sentiment recognition during multimodal fusion. This constraint ensures coordination in sentiment expression between audio and video modalities, reducing semantic bias.

[0073] Through the synergistic effect of these three constraints, the semantic analysis module effectively addresses noise and semantic gaps in multimodal data, optimizing the accuracy of sentiment analysis. The global relevance constraint reduces instance-level noise between modalities, the information bottleneck constraint reduces redundant information, and the semantic consistency constraint strengthens semantic alignment between modalities, making the sentiment analysis model more accurate. Ultimately, these constraint mechanisms enable the semantic analysis module to perform efficient sentiment analysis and contextual reasoning on law enforcement recorder video and audio data, providing robust data support for subsequent intelligent report generation. This module demonstrates higher robustness and accuracy when handling sentiment recognition tasks in complex scenarios, providing technical assurance for the intelligent and automated enforcement process.

[0074] Step 5: Using the video semantic retrieval module, based on the video stream data and query text, through visual feature extraction, text feature extraction, feature fusion and regression prediction, retrieve the segments related to the query text from the video stream data and obtain the visual semantic features corresponding to the segments.

[0075] The video semantic retrieval module aims to accurately retrieve relevant segments from a large number of video candidate segments in video stream data based on a given query text. For example... Figure 5 As shown, the video semantic retrieval module, by introducing an action-language attention mechanism, achieves more refined cross-modal interaction based on the traditional CTRL framework, significantly improving retrieval accuracy. The video semantic retrieval module comprises four main parts: a visual feature extraction module, a text feature extraction module, a feature fusion module, and a regression prediction module.

[0076] The specific working process of the video semantic retrieval module is as follows: Step 5.1: The video stream data is processed by the visual feature extraction module to extract global motion features.

[0077] In this embodiment, the visual feature extraction module uses a pre-trained C3D network. The C3D network can effectively capture temporal information between video frames through 3D convolution, extracting action features containing both temporal and spatial information. The output of the seventh fully connected layer of the C3D network serves as the global action feature, further enhancing the feature's expressive power. To improve the discriminative power of the features, this embodiment adds a fully connected layer after the C3D network. This results in denser extracted features with stronger discriminative power. The final generated global action features are then used as the visual features of video candidate segments and input into subsequent modules.

[0078] Step 5.2: The text feature vector is obtained by extracting the text feature through the text feature extraction module.

[0079] In this embodiment, the text feature extraction module employs GloVe (Global Vectors for Word Representation) and Bi-GRU (Bidirectional Gated Recurrent Unit). GloVe is a word embedding model used to map words to real-valued vector representations and captures semantic relationships between words through global word frequency statistics. In video semantic retrieval, GloVe is used to convert words in the query text into vector representations for subsequent text feature extraction and semantic matching. Bi-GRU is a bidirectional gated recurrent unit network used to process sequential data, capable of simultaneously considering both positive and negative contextual information to capture contextual dependencies in the text. In video semantic retrieval, Bi-GRU is used to extract features from the query text and generate text feature vectors that incorporate contextual semantic information.

[0080] The text feature extraction module can also employ the BERT (Bidirectional Encoder Representations from Transformers) model to extract features from the query text. The BERT model uses a bidirectional encoding mechanism to capture deep contextual information from the text, transforming it into high-quality text feature vectors. These vectors contain semantic information from the query text, providing accurate semantic representations for cross-modal fusion of video and text. Through text feature extraction, the system can understand the user's query needs, providing strong semantic support for subsequent retrieval tasks.

[0081] Step 5.3: Global action features and text feature vectors are fused through the feature fusion module using an action-language attention mechanism to obtain visual semantic features.

[0082] The feature fusion module establishes a close connection between visual and textual features, enhancing the interaction between video and text by introducing an action-language attention mechanism. During the fusion process, the attention mechanism dynamically adjusts the weights of specific action features in the video based on the similarity between the video content and the query text, thereby more accurately matching the query text. For example, when the query text mentions "police chasing a suspect," the module automatically increases the weight of visual features related to the chasing action, optimizing the semantic matching between the video clip and the text. Ultimately, through the feature fusion module, global action features and textual feature vectors are fused into visual semantic features, providing a more accurate semantic representation for subsequent retrieval.

[0083] Step 5.4: Visual semantic features are used by the regression prediction module to obtain fragments related to the query text.

[0084] The regression prediction module processes the fused visual semantic features to predict the relevance between video clips and query text. Employing a Multilayer Perceptron (MLP) network, the module performs a series of convolution and pooling operations to calculate a similarity score between the video clip and the query text. This prediction score allows the system to rank video clips and recommend the most relevant clips to the user, thus achieving accurate video semantic retrieval. The regression prediction module calculates the localization score and localization bias of the most relevant clips. The localization score represents the semantic matching degree between the candidate video clip and the query text. It reflects the similarity between the video clip and the text description; a higher score indicates a higher match. In video semantic retrieval, the localization score measures the semantic relevance of each candidate video clip to the query text, helping the model select the most matching video clip. The localization bias refers to the deviation between the start and end times of the candidate video clip and the actual labeled start and end times. It is used to adjust the boundaries of the video clip to more closely approximate the actual time range. Through the localization bias, the model can fine-tune the start and end times of the video clips, thereby improving retrieval accuracy.

[0085] Step 5.5: Based on the fragments related to the query text, filter the visual semantic features to obtain the visual semantic features corresponding to the fragments.

[0086] To optimize the performance and improve the retrieval efficiency of the video semantic retrieval module, this embodiment employs an end-to-end training method combined with multimodal loss function optimization. During training, this embodiment enhances the model's robustness through data augmentation techniques and continuously optimizes network parameters using the backpropagation algorithm, ensuring the model provides fast and accurate retrieval results when processing real queries. The video semantic retrieval module significantly improves retrieval efficiency through effective cross-modal fusion and retrieval strategies, and provides strong support for the intelligent analysis of complex video data.

[0087] Step 6: Using the report generation module, determine the case type based on the visual semantic features and multimodal sentiment analysis semantic features corresponding to the fragment, call the corresponding police response process report template, and generate a police response process report.

[0088] To improve the accuracy of case analysis and event reconstruction capabilities in the reports, the report generation module employs a novel event-driven Prompt Response Process Report Generation (PromptERG) framework. This framework relies on an improved encoder-decoder architecture and incorporates cross-modal feature enhancement techniques to ensure the logical coherence, completeness, and accuracy of the generated reports.

[0089] like Figure 6 As shown, the specific process by which the report generation module generates an emergency response report is as follows: Step 6.1: Using the encoder, structured features are obtained by fusing the visual semantic features and multimodal sentiment analysis semantic features corresponding to the segments.

[0090] The visual semantic features corresponding to the video segment and the multimodal sentiment analysis semantic features serve as inputs to the encoder. The visual semantic features include key information about people, actions, and scenes to capture dynamic changes in the video; the multimodal sentiment analysis semantic features include key semantic information extracted from the audio. The encoder fuses the two features to generate structured features. The fusion process is performed using a multimodal fusion network, the Cross-modal Transformer.

[0091] Step 6.2: Determine the case type based on the structured features and call the corresponding police response process report template.

[0092] The structured features are input into the adaptive case type determination module to obtain the classification result, i.e., the case type. Then, based on the case type, the corresponding police response report template is retrieved from the case-driven prompt templates. In this embodiment, the case types include civil disputes, criminal cases, and traffic accidents, and the corresponding police response report templates are also divided into three types.

[0093] To further improve the completeness and flexibility of report generation, this embodiment also includes a cross-modal feature enhancement module. The specific working process of the cross-modal feature enhancement module is as follows: The visual semantic features corresponding to the fragments are input into the cross-modal feature enhancement module. Using these features, historical dispatch records of similar cases are retrieved from the database to improve the accuracy and quality of current case reports. The retrieval process utilizes the pre-trained CLIP model for cross-modal matching, comparing the visual semantic features of the current case fragments with the semantic information of historical reports to identify the most relevant historical cases. The most relevant historical dispatch records are then retrieved from the historical dispatch record data.

[0094] The system fuses the most relevant historical police dispatch report features with structured features. This fusion process utilizes an attention mechanism to ensure that the generated report is based not only on information from the current case but also on key content from historical cases. The fused features are then concatenated with the structured features and input into an adaptive case type determination module. This module determines the case type based on the fused features and the structured features.

[0095] Step 6.3: Using the decoder, generate an alarm response report based on the structured features and the alarm response report template.

[0096] The decoder is responsible for generating the final emergency response report, and the specific process is as follows: Sequence-to-Sequence (Seq2Seq) Model: The decoder adopts the Seq2Seq model structure and generates a report. , Composed of a series of word tokens composition, The length of the report. For the vocabulary list, It includes all possible words and symbols that may appear in the report, such as: common words (e.g., "accident," "suspect," "vehicle," etc.), case-related technical terms (e.g., "civil dispute," "criminal case," "traffic accident," etc.), punctuation marks, and special symbols (e.g., period, comma, [BLA], [POS], etc.). The decoder is based on the structured features of the input and case-driven prompts in the police response report template. Generate tokens for each time step. During each generation step of the decoder, tokens are derived from the reported vocabulary. Select a word or token as the output of the current time step.

[0097] Hypothetical vocabulary Includes the following words: ={"Accident", "Suspect", "Vehicle", "Civil Dispute", "Criminal Case", "Traffic Accident", "Scene", "Description", "Evidence", "Time", "Location", ...} When generating the report, the decoder will... Choose words to build the report. For example: When the case type is entered as "traffic accident", the decoder will generate the following report: "The accident occurred on XX / XX / 20XX at the XX intersection, involving a black sedan and a white SUV." The words "accident," "vehicle," "time," and "location" are all from the vocabulary list. .

[0098] Vocabulary It is a core component of the decoder in the report generation module, defining all possible words that the model can generate. This is achieved through a vocabulary. The model can select appropriate words from predefined vocabulary to generate police response reports that are logical, complete, and accurate.

[0099] Case-driven prompts: Case-driven prompts combine case type (such as civil disputes, criminal cases, traffic accidents, etc.) and semantic clues to provide specific guidance for report generation. For example, for traffic accident cases, prompts may include key information such as "accident location," "accident time," and "vehicles involved." These prompts guide the decoder to generate targeted report content through predefined templates or dynamically generated prompt words.

[0100] The report generation module employs language modeling loss as the primary loss function during training to maximize the similarity between the generated report and the real report. During training, the language modeling loss is based on the vocabulary. The calculation is as follows. The model needs to predict the correct words at each time step and compare them with the true words in the vocabulary. The specific steps are as follows: Language modeling loss: The language modeling loss is used to optimize the lexical-level accuracy of the generated reports. This loss function optimizes the model parameters by maximizing the conditional probability of the generated reports.

[0101] Contrastive Loss: To optimize the consistency of multimodal features, a contrastive loss is introduced. This loss function reduces semantic bias in the report generation process by comparing the feature representations of positive samples (real reports) and negative samples (generated reports).

[0102] Adaptive Case-Balanced Learning (ACBL): To address the imbalance between different case types, Adaptive Case-Balanced Learning (ACBL) is introduced. This algorithm adaptively adjusts its optimization objective based on the learning status of each case type, ensuring that even rare case types are adequately learned. Specifically, ACBL balances the learning of different case types by adjusting the class weights in the loss function.

[0103] The report generation module, by combining an encoder-decoder architecture, case-driven prompts, and cross-modal feature enhancement technology, provides an efficient and accurate automated police dispatch report generation system. This approach not only improves the quality of generated police dispatch reports but also optimizes the analysis efficiency of body camera data, providing reliable support for the recording of law enforcement processes and subsequent case management.

[0104] This embodiment also provides an intelligent report generation system for police response processes based on law enforcement recorder video, such as... Figure 7 As shown, the technical solution adopted is as follows: including: data acquisition module 1, video and audio separation module 2, speech recognition module 3, semantic analysis module 4, video semantic retrieval module 5, and report generation module 6.

[0105] The data acquisition module 1 is used to acquire video data from the law enforcement recorder and query text; The video and audio separation module 2 is used to separate video stream data and audio stream data based on the video data from the law enforcement recorder. The speech recognition module 3 is used to recognize the audio stream data to obtain audio text; The semantic analysis module 4 is used to perform semantic analysis on the video stream data and audio text to obtain multimodal sentiment analysis semantic features; The video semantic retrieval module 5 is used to retrieve segments related to the query text from the video stream data based on the video stream data and the query text through visual feature extraction, text feature extraction, feature fusion and regression prediction, and obtain the visual semantic features corresponding to the segments. The report generation module 6 is used to determine the case type based on the visual semantic features and multimodal sentiment analysis semantic features corresponding to the fragment, call the corresponding police response process report template, and generate a police response process report.

[0106] This system can be integrated and run via a cloud platform or local server, and can seamlessly interface with law enforcement recorder devices. During implementation, after law enforcement officers wear the recorder and start recording, the system automatically separates the recorded audio and video data into video and audio streams, which are then transmitted separately to their respective processing modules, ultimately generating a police response report. This report can be viewed via smartphone, computer, or dedicated terminal.

[0107] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for intelligently generating police handling process reports based on law enforcement recorder videos, characterized in that: include: Obtain law enforcement camera video data and query text; Separating the video data from the law enforcement recorder to obtain video stream data and audio stream data; Identify the audio stream data to obtain an audio text; Performing semantic analysis on the video stream data and the audio text to obtain multimodal sentiment analysis semantic features; According to the video stream data and the query text, through visual feature extraction, text feature extraction, feature fusion and regression prediction, the fragments related to the query text are retrieved from the video stream data to obtain the visual semantic features corresponding to the fragments; According to the visual semantic features and multimodal sentiment analysis semantic features corresponding to the fragment, the case type is determined, the corresponding police handling process report template is called, and the police handling process report is generated.

2. The method for intelligently generating police handling process reports based on law enforcement recorder videos as described in claim 1, characterized in that: According to the law enforcement recorder video data, video stream data and audio stream data are separated, and the specific process is as follows: Splitting the law enforcement recorder video data into video data and audio data, and using the video data as video stream data; Through a facial feature extraction network and a motion feature extraction network, feature extraction is performed on the video data to obtain facial features and motion features; Extracting features from the audio data using an audio feature extraction network to obtain audio features; The facial features, motion features and audio features are cross-modally fused through a cross-modal fusion module to obtain audio-visual features; The audio-visual features and audio features are processed through the speech separation network to obtain audio stream data.

3. The method for intelligently generating police handling process reports based on law enforcement recorder videos as claimed in claim 1, characterized in that: The audio stream data is identified to obtain the audio text, and the specific process is as follows: The audio stream data is transformed into a two-dimensional spectrum diagram by short-time Fourier transform; The two-dimensional spectrum graph generates a spectrogram of a clear speech signal through an hourglass-shaped speech recognition model; The spectrogram of the clear speech signal is passed through a decoder to generate an audio text.

4. The method for intelligently generating police handling process reports based on law enforcement recorder videos as described in claim 3, characterized in that: The hourglass-shaped speech recognition model includes five simple recurrent units connected in sequence. The first-layer simple recurrent units and the fifth-layer simple recurrent units are jump-connected, and the second-layer simple recurrent units and the fourth-layer simple recurrent units are jump-connected. An attention gate is added to the jump connection.

5. The method for intelligently generating police handling process reports based on law enforcement recorder videos as claimed in claim 1, characterized in that: The video stream data and audio text are semantically analyzed to obtain multimodal sentiment analysis semantic features. The specific process is as follows: The audio text is passed through a text encoder to generate text feature representation; the video stream data is passed through an image encoder to generate image feature representation; The text feature representation and image feature representation are respectively generated through the unimodal variational attention layer to generate the text potential representation and image potential representation; The text latent representation and image latent representation are fused through a cross-modal attention layer to generate a multimodal representation; The multimodal representation passes through the multi-layer perceptron and conditional random field modules in sequence to obtain the multimodal sentiment analysis semantic features.

6. The method for intelligently generating police handling process reports based on law enforcement recorder videos as claimed in claim 1, characterized in that: The specific process of obtaining the visual semantic features corresponding to the fragment is: The video stream data is passed through the visual feature extraction module to extract the global action features; The query text passes through the text feature extraction module to obtain a text feature vector; The global action feature and text feature vector are fused through the feature fusion module using the action-language attention mechanism to obtain visual semantic features; The visual semantic features are used through a regression prediction module to obtain fragments related to the query text; According to the fragments related to the query text, the visual semantic features corresponding to the fragments are obtained.

7. The method for intelligently generating police handling process reports based on law enforcement recorder videos as claimed in claim 1, characterized in that: The specific process of generating a police handling process report is as follows: According to the visual semantic features corresponding to the clips and the semantic features of multimodal sentiment analysis, the structural features are fused; Determine the case type based on structural features and call the corresponding police handling process report template; Generate a police handling process report based on structured features and police handling process report template.

8. The method for intelligently generating police handling process reports based on law enforcement recorder videos as claimed in claim 7, characterized in that: Through the visual semantic features corresponding to the fragments, the most relevant historical police records are retrieved from the historical police record data, the police handling process report features of the most relevant historical police records are fused with the structured features, and the case type is determined based on the fused features and structured features.

9. The method for intelligently generating police handling process reports based on law enforcement recorder videos as described in claim 1, 7 or 8, characterized in that: The types of cases described include civil disputes, criminal cases and traffic accidents.

10. An intelligent generation system for police handling process reports based on law enforcement recorder videos, characterized in that: The method for intelligently generating police handling process reports based on law enforcement recorder videos according to any one of claims 1 to 9 comprises: a data acquisition module, a video and audio separation module, a speech recognition module, a semantic analysis module, a video semantic retrieval module and a report generation module. The data acquisition module is used to acquire the law enforcement recorder video data and query text; The video and audio separation module is used to separate the video data of the law enforcement recorder into video stream data and audio stream data; The speech recognition module is used to recognize the audio stream data to obtain an audio text; The semantic analysis module is used to perform semantic analysis on the video stream data and audio text to obtain multimodal sentiment analysis semantic features; The video semantic retrieval module is used to retrieve the segments related to the query text from the video stream data through visual feature extraction, text feature extraction, feature fusion and regression prediction based on the video stream data and the query text, and obtain the visual semantic features corresponding to the segments; The report generation module is used to determine the case type according to the visual semantic features and multimodal sentiment analysis semantic features corresponding to the fragment, call the corresponding police handling process report template, and generate a police handling process report.

Images