Method and system for extracting AI key information of on-duty record video

Abstract

The invention discloses an AI key information extraction method and system for an on-duty record video, and relates to the technical field of intelligent information analysis, and the method comprises the steps: obtaining the on-duty record video, extracting video key frames, carrying out the normalization, and carrying out the noise reduction of an audio signal, time windows are divided to extract multi-modal features, and cross-modal attention is adopted to generate fusion features; the method comprises the following steps: identifying entities for video key frames by using a multi-target tracking algorithm, constructing initial entity features for each entity, constructing a space edge and a time sequence edge by using the entities as nodes, calculating weights to construct an adaptive time sequence diagram, and performing space convolution and time sequence convolution by using a diagram convolution network to update the entity features; and mapping the entity features into key scores by using an MLP, extracting a key node set to form event candidates, and scoring and classifying the event candidates to output key event information. According to the method, the key event identification accuracy and availability are improved, and the duty record backtracking efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent information analysis, and in particular to an AI key information extraction method and system for on-duty recording video. BACKGROUND

[0002] With the wide deployment of front-end sensing devices such as on-duty recorders and vehicle-mounted forensics terminals, the scale of multi-source video data generated by on-duty activities is growing exponentially. In the field of intelligent processing of on-duty recording video, the industry has gradually developed from traditional file-level storage and manual retrieval to automated analysis based on computer vision and speech recognition. For example, target detection and tracking models are used to identify personnel and vehicles in the picture, speech recognition technology is used to transcribe audio into text, and keyword retrieval is used to achieve coarse-grained case backtracking. Some systems also attempt to introduce deep learning algorithms such as behavior recognition and scene classification to automatically alert to scenes such as violent conflicts, pursuits, and gatherings. At the same time, new technologies such as multi-modal learning and graph neural networks are being applied to the field of security video analysis to fuse video, audio, and text information and model the interaction between multiple subjects in the video, providing a technical foundation for improving case retrieval efficiency and automatic forensics capabilities. However, existing on-duty recording video analysis technologies generally have shortcomings. In the time dimension, most methods still remain at the level of independent discrimination of frames or segments, lacking unified modeling of the continuous state evolution of the same entity on the time axis and the interaction between entities, making it difficult to timely depict the gradual risk escalation process in typical on-duty scenarios such as "approach-dispute-physical conflict". SUMMARY

[0003] In view of the above-mentioned existing problems, the present application is proposed.

[0004] Therefore, the present application provides an AI key information extraction method and system for on-duty recording video, solving the problem that existing technologies remain at the level of independent discrimination of frames or segments, lacking unified modeling of the continuous state evolution of the same entity on the time axis and the interaction between entities.

[0005] To solve the above technical problems, the present application provides the following technical solutions: In a first aspect, the present application provides an AI key information extraction method for on-duty recording video, which comprises, extracting video key frames from the on-duty recording video, normalizing the video key frames, and performing noise reduction processing on the audio signal, dividing the time window to extract multi-modal features and generating fusion features using cross-modal attention; using a multi-target tracking algorithm to identify entities from the video key frames, constructing initial entity features for each entity, constructing spatial edges and time sequence edges using the entities as nodes and calculating weights to construct an adaptive time sequence graph, and updating the entity features using a graph convolution network for spatial convolution and time sequence convolution; The entity features are mapped into a key score using an MLP, and a key node set is extracted to form an event candidate, the event candidate is scored and classified to output key event information, and the key event information is arranged and displayed.

[0006] As a preferred scheme of the AI key information extraction method for the on-duty record video, wherein: the on-duty record video is obtained, key frames of the video are extracted and normalized, and the audio signal is denoised, the on-duty record video and the audio are obtained from the on-duty recorder, the video frame is converted into a gray image , the average absolute difference between adjacent video frames is calculated ; The key frame threshold is updated using a sliding window ; The video frame with an average absolute difference greater than the key frame threshold is marked as a key frame and retained, and the key frame is pixel normalized The audio signal is denoised in time domain using a lightweight Wave-U-Net The denoised audio is framed, each frame has a length of L sampling points, and a frame shift of S sampling points, and each frame of audio signal is multiplied by a Hamming window to calculate a short-time Fourier transform The short-time Fourier transform result is projected through a Mel filter bank to obtain the fth Mel channel and take the logarithm to obtain a Mel spectrogram.

[0007] As a preferred scheme of the AI key information extraction method for the on-duty record video, wherein: the time window is divided to extract multi-modal features and generate fusion features using cross-modal attention, the video and audio are divided into continuous windows with a length of , key frames in each continuous window are extracted, and visual features of the key frames are extracted using an EfficientNet-Lite0 model The visual features of all key frames in the continuous window are time-pooled to obtain window visual features ; The audio features of the corresponding Mel spectrogram of each continuous window are extracted The audio in the continuous window is input into an ASR model to obtain a text transcription sequence ; The text transcription sequence is input into a DistilBERT to obtain a text feature vector ; The window visual feature, audio feature and text feature vector of each continuous window are formed into a multi-modal feature triple output, and the three features in the feature triple are mapped to the same dimension Attention scores are calculated for each modality; The attention scores are converted into attention weights of the modalities using softmax; The features of the three modalities are weighted and summed based on the attention weights and linear transformation to obtain the fused features .

[0008] As a preferred scheme of the AI key information extraction method of the on-duty record video, wherein: the multi-target tracking algorithm is used to identify entities for the video key frame, the initial entity feature is constructed for each entity, the spatial edge and the time sequence edge are constructed by taking the entity as the node, and the adaptive time sequence graph is constructed by calculating the weight; in each continuous window, the lightweight target detector (such as MobileNet-SSD) is used for entity detection to obtain the boundary box of the entity; The multi-target tracking algorithm is used to assign a continuous ID to each entity and record the centroid position of the entity ; The initial entity feature is constructed for each entity i in the continuous window k ; The spatial distance of the entities i and j in the time window k is calculated , and the relative speed is defined; The edge feature vector is constructed , the edge score is calculated using a layer of perception machine, and the edge weight is obtained by using sigmoid for score mapping ; If the edge weight between the entities i and j is greater than a set threshold, a spatial edge is added between the entities i and j, and a time sequence edge is established between the adjacent continuous windows k and k+1 for the same entity i The adaptive time sequence graph is constructed by taking the entity as the node .

[0009] As a preferred scheme of the AI key information extraction method of the on-duty record video, wherein: the graph convolution network is used for spatial convolution and time sequence convolution to update the entity feature, which means that the node feature of each time sequence graph is updated using the attention graph convolution; After O-layer aggregation, the spatial aggregation feature is obtained ; The time sequence of the entity i is counted , and the one-dimensional convolution kernel with a convolution kernel size of R is used for time sequence convolution; The time sequence convolution output is taken as the node feature with space-time context semantics Output.

[0010] As a preferred embodiment of the AI-based key information extraction method for duty recording videos described in this invention, the step of using an MLP to map entity features to key scores and extracting a set of key nodes to form event candidates, and then scoring and classifying the event candidates to output key event information, refers to using an MLP network to map node features to key scores. ; A criticality threshold is set, and nodes with criticality scores higher than the threshold are marked as high-scoring nodes. Spatially adjacent and temporally consecutive high-scoring nodes are aggregated into an event candidate set. The set of nodes for the m-th event candidate is . The event level is scored; Aggregate the node features of all nodes in the event candidates to form the event aggregate feature. Based on the event aggregation features, a classifier is used to classify event types.

[0011] As a preferred embodiment of the AI ​​key information extraction method for duty recording videos described in this invention, the step of arranging and displaying key event information refers to sorting events according to event level scores, and adding event type to each event to form a key information list for output and display.

[0012] Secondly, this invention provides an AI-powered key information extraction system for duty recording videos, comprising: The feature extraction module is used to extract key frames from duty recording videos, normalize them, and perform noise reduction on audio signals. It also divides time windows to extract multimodal features and uses cross-modal attention to generate fused features. The graph update module is used to identify entities using a multi-object tracking algorithm on video keyframes. It constructs initial entity features for each entity, uses entities as nodes to construct spatial and temporal edges and calculates weights to construct an adaptive temporal graph, and uses a graph convolutional network to perform spatial and temporal convolution to update entity features. The event recognition module uses MLP to map entity features to key scores and extract key node sets to form event candidates. It scores and classifies the event candidates, outputs key event information, and arranges and displays the key event information.

[0013] Thirdly, the present invention provides a computer device including a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program is executed by the processor, it implements any step of the AI ​​key information extraction method for duty recording video as described in the first aspect of the present invention.

[0014] In a fourth aspect, the present application provides a computer readable storage medium having stored thereon a computer program, wherein the computer program, when executed by a processor, implements any step of the AI key information extraction method for on-duty record video according to the first aspect of the present application.

[0015] The present application has the beneficial effects that: the present application realizes automatic extraction and arrangement display of key event information in on-duty record video by key frame normalization and audio noise reduction, multi-modal feature cross-modal attention fusion, adaptive time sequence graph construction and graph convolution time sequence update, combined with MLP key score and event candidate screening, improves key event recognition accuracy and usability, and improves on-duty record backtracking efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0017] Figure 1 The flowchart of the AI key information extraction method for on-duty record video in embodiment 1.

[0018] Figure 2 The structural diagram of the AI key information extraction system for on-duty record video in embodiment 1. DETAILED DESCRIPTION

[0019] In order to make the above-mentioned purposes, features and advantages of the present application more apparent and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the drawings of the specification.

[0020] In the following description, many specific details are set forth in order to provide a thorough understanding of the present application, but the present application can also be implemented in other ways different from those described herein, and those skilled in the art can make similar generalizations without departing from the connotation of the present application, therefore the present application is not limited to the specific embodiments disclosed below.

[0021] Secondly, the "one embodiment" or "embodiment" referred to herein means that the specific features, structures or characteristics can be included in at least one implementation of the present application. "In one embodiment" appearing in different places in the specification does not mean the same embodiment, nor is it an independent or alternative embodiment that excludes other embodiments.

[0022] Embodiment 1, refer to Figure 1 and Figure 2For the first embodiment of the application, the embodiment provides an AI key information extraction method for on-duty record video, including the following steps: S1, obtain the on-duty record video, extract the video key frame for normalization, and perform noise reduction processing on the audio signal, divide the time window, extract the multi-modal feature, and generate the fusion feature by using the cross-modal attention; Specifically, the on-duty record video is obtained, the video key frame is extracted for normalization, and the audio signal is subjected to noise reduction processing, that is, the on-duty record video and the audio are obtained from the on-duty recorder, the video frame is converted into a gray scale image , the average absolute difference between adjacent video frames is calculated :

[0023] wherein H and W are the height and width of the gray scale image respectively, the gray scale of the pixel point ; The key frame threshold is updated by using a sliding window :

[0024]

[0025] wherein is the sliding window mean value, is the standard deviation, is the sliding window length, is the time, is the sensitivity coefficient; The video frame with the average absolute difference greater than the key frame threshold is marked as a key frame and is retained, and the key frame is subjected to pixel normalization; The audio signal is subjected to time domain noise reduction by using a lightweight Wave-U-Net:

[0026] wherein is the trained lightweight Wave-U-Net model, is the model parameter, is the noise-reduced audio; The noise-reduced audio is subjected to frame division, each frame has a length of L sampling points, and the frame shift is S sampling points, and each frame of audio signal is multiplied by a Hamming window to calculate the short-time Fourier transform:

[0027] wherein is the short-time Fourier transform result, is the frame index, is the sample point index, is the frequency index, j is the imaginary unit; The short-time Fourier transform result is projected through a mel filter bank to obtain the fth mel channel and logarithm to obtain a mel spectrum diagram:

[0028]

[0029] wherein is the fth mel channel energy, is the frequency response of the fth mel filter, is a small constant to prevent zero values of the logarithm, is the mel spectrum of the fth mel channel.

[0030] By converting the on-duty record video into a gray-scale diagram frame by frame and calculating the average gray-scale difference of adjacent frames, while constructing an adaptive threshold based on sliding window statistics, dynamic perception of the degree of picture change is achieved. Compared with the traditional fixed threshold or simple frame extraction strategy, this step does not mechanically select frames at equal intervals of time, but automatically adjusts "what kind of change is significant" according to the average level and fluctuation amplitude of the picture motion amount in a recent period of time. In a large number of smooth scenes such as long-time patrol and road duty, this mechanism can effectively filter out a large number of redundant frames with very small picture differences, significantly reducing the computational burden of subsequent feature extraction and reasoning; while in the sudden appearance of dramatic scene changes such as pursuit, conflict, and vehicle sudden stop, it can quickly raise the key frame selection ratio to ensure that the pictures that are crucial to case classification and responsibility determination are not missed, truly achieving a dynamic balance between "saving computing power" and "retaining key segments". This idea of adaptively selecting key frames according to the statistical characteristics of the scene solves the contradictory problem in existing on-duty video processing, which is either too dense frame extraction leading to large computational overhead or too sparse frame extraction leading to loss of details.

[0031] By performing pixel normalization on the selected key frames, the standardization of image brightness and contrast under different devices, different lighting, and different exposure settings is achieved, so that the subsequent deep network sees more stable distributed inputs. The on-duty recorder often works in indoor and outdoor, multiple time periods, sunny and rainy, and complex backlight conditions. If uniform intensity adjustment is not done, the model may mistake device differences, lighting changes for scene differences and produce false judgments. Normalization processing makes the model pay more attention to structural information such as human posture, object contour, and relative position, and weakens non-essential imaging condition differences, thereby improving the generalization ability under cross-device and cross-scene conditions.

[0032] By employing a lightweight neural network to denoise the original audio signal in the time domain, followed by frame segmentation, windowing, and short-time Fourier transform of the denoised signal, and constructing a Mel spectrogram based on auditory perception characteristics, the goal of extracting stable acoustic and speech features from noisy duty environments is achieved. Compared to relying solely on traditional frequency domain noise reduction or spectrogram representation without auditory scale transformation, this combined approach preserves key signals related to the case, such as speech content, burst noise, and screams, while significantly suppressing background interference such as wind noise, vehicle noise, and distant chatter. This results in cleaner inputs for subsequent speech recognition and acoustic event detection tasks, more closely resembling human auditory perception. Especially in scenarios with unstable video quality, such as nighttime operations, rapid movement, and outdoor intersections, clear Mel spectrogram features provide relatively reliable acoustic support for multimodal fusion. This allows the system to still identify threatening speech, heated arguments, and other implicit risks based on auditory cues even when the image quality degrades, significantly improving the robustness and practical value of the entire duty video key information extraction method in complex environments.

[0033] Furthermore, dividing the time window to extract multimodal features and using cross-modal attention to generate fusion features refers to dividing the video and audio into segments of length [missing information]. continuous windows Keyframes are extracted from each consecutive window, and the EfficientNet-Lite0 model is used to extract visual features from the keyframes.

[0034] in For the EfficientNet-Lite0 model, For normalized keyframes, These are the parameters of the EfficientNet-Lite0 model. Visual features; The visual features of the window are obtained by temporal pooling of the visual features of all keyframes in a continuous window. :

[0035] in For continuous windows Number of keyframes; For each consecutive window Corresponding Mel spectrum Extracting audio features :

[0036] in To make the Wav2Vec 2.0 model lightweight, For lightweight Wav2Vec 2.0 model parameters The continuous window The audio input ASR model (automatic speech recognition model) gets a text transcription sequence :

[0037] Wherein is the ASR model, is the ASR model parameter, is the i-th word of the k-th window, is the number of words; The text transcription sequence is input into DistilBERT to get a text feature vector :

[0038] Wherein is the DistilBERT model (language representation model), is the DistilBERT model parameter; The window visual feature, audio feature and text feature vector of each continuous window are formed into a multi-modal feature triple, and the three features in the feature triple are mapped to the same dimension , the attention score is calculated for each modality;

[0039] Wherein , and are the window visual feature, audio feature and text feature vector mapped to the same dimension, is a trainable vector, , and are the attention scores of the window visual feature, audio feature and text feature vector; The attention score is converted into the attention weight of the modality using softmax:

[0040] Wherein is the attention score, including , and , is the attention weight, including , and ; Based on the attention weight, the features of the three modalities are weighted and summed and linearly transformed to get a fused feature :

[0041]

[0042] wherein is a weighted sum feature, and are linear transformation parameters, is an activation function.

[0043] By uniformly dividing the video frames and audio signals into time windows of one second in length and simultaneously extracting visual, audio and text features in each window, the "fragmentation, multi-modal alignment representation" of the on-duty record scene is realized. This concept is not simply a time slice, but explicitly takes the "second-level event fragment" as the analysis unit, and constrains the picture change, environmental sound and language content within the same time range, so that the subsequent model can use picture posture, tone change and word meaning as clues to determine whether there is a dispute, warning, interrogation and other behaviors at a certain time. Compared with the common practice in the prior art of "first rolling the video as a whole, then separately identifying the audio, and finally splicing the results", the time window division directly solves the problem of multi-modal signal asynchronization and difficulty in alignment, significantly reducing the misjudgment probability caused by time misalignment, such as the case where the picture has already pushed and pulled the body while the voice is still in the previous speaking session. By respectively using convolutional neural networks, audio pre-training models and language representation models to extract visual features, audio features and text features in each time window, and performing time pooling on the multi-frame visual features in the same window to form a single window visual representation, the unified abstraction of "fine-grained frame information" and "coarse-grained semantic fragment" is realized. In the on-duty scene, the same second may contain multiple frames, some of which are severely affected by shaking and occlusion, and some of which are relatively clear. By pooling the multi-frame information into a stable representation, the interference of individual abnormal frames can be effectively eliminated, so that the model focuses on the overall spatial layout and action trend in that time period, rather than on the accidental noise of a single frame. This design has obvious advantages in solving the problem of unstable single-frame features caused by frequent shaking and rapid movement of the on-duty recorder, thereby providing more stable visual input for subsequent cross-modal fusion. By constructing a multi-modal feature triple of "visual features, audio features and text features", and introducing a cross-modal attention mechanism in a unified dimensional space, the dynamic adjustment of the contribution of different modalities is realized, rather than fixed weighting or simple concatenation. Under this mechanism, in each time window, the system automatically calculates the attention scores of the three modalities according to the current feature content, and normalizes them into weights to weight the fusion result. The corresponding technical problem of this concept is that existing on-duty video analysis often defaults to visual information being the most important, while ignoring typical cases such as "fuzzy picture but highly indicative sound" and "clear voice but chaotic background picture". By introducing cross-modal attention, when the picture is clear and the human posture features are obvious, the model will naturally increase the weight of the visual modality; when there is insufficient light at night or there is serious occlusion, the weights of the audio and text modalities will automatically increase, so that the system can still determine whether there is a high-risk event through clues such as loud argument and command password.This adaptive weight assignment produces unexpected results in the context of the present application: in complex noise environments, extreme changes in lighting, and conditions of diverse devices, the robustness of multi-modal fusion is significantly better than that of single modal or fixed weighting models, and the recall rate and accuracy of key events can be quantitatively improved.

[0044] By applying linear transformation and nonlinear activation to the weighted multi-modal features, they are compressed into unified fusion features, which not only provide formal unified and fixed dimension input for subsequent graph-based time series modeling, but also form a clear hierarchy of "from raw perception to high-level semantic representation" in network structure. For subsequent steps such as entity graph construction and event candidate generation, having such a fusion feature that has been time-aligned, multi-modal aligned, and importance reweighted can significantly reduce modeling complexity and avoid repeating the problem of modal imbalance and noise suppression in the downstream network. From the point of view of invention concept, this is equivalent to establishing an "intelligent information aggregation layer" at the front end of the entire method, which solves the problem of multi-modal distribution inconsistency and reliability imbalance, a long-standing problem in the field of on-duty video intelligent analysis, and provides a high-quality feature basis for subsequent key event extraction.

[0045] S2, a multi-target tracking algorithm is used to identify entities for video key frames, initial entity features are constructed for each entity, spatial edges and time edges are constructed using entities as nodes, and adaptive time graphs are constructed by calculating weights, and a graph convolution network is used for spatial convolution and time convolution to update entity features; Specifically, a multi-target tracking algorithm is used to identify entities for video key frames, initial entity features are constructed for each entity, spatial edges and time edges are constructed using entities as nodes, and adaptive time graphs are constructed by calculating weights, which means that in each continuous window, a lightweight target detector (such as MobileNet-SSD) is used for entity detection on key frames to obtain the bounding box of the entity:

[0046] wherein is the bounding box of the i-th entity in the k-th continuous window, and is the upper left corner coordinate, and is the width and height, is the entity category (person, vehicle, etc.); A multi-target tracking algorithm is used to assign a continuous ID to each entity and record the centroid position of the entity :

[0047] An initial entity feature is constructed for each entity i in the continuous window k :

[0048] where is the position information concatenation vector, and are parameters; the spatial distance between entities i and j at time window k is calculated as and the relative velocity is defined as:

[0049] where is the position difference of entity i at adjacent time windows, is the relative velocity between entities i and j; the edge feature vector is constructed as and the edge score is calculated using a one-layer perceptron with sigmoid score mapping to obtain the edge weight :

[0050]

[0051]

[0052] where is the edge score between entities i and j, , and are learnable parameters, is the activation function; if the edge weight between entities i and j is greater than a set threshold, a spatial edge is added between entities i and j, and a temporal edge is established between the same entity i at adjacent consecutive windows k and k+1 to construct an adaptive temporal graph with entities as nodes :

[0053] where is the node set, is the spatial edge set, is the temporal edge set.

[0054] By using a lightweight object detector to perform entity detection on key frames and output bounding boxes within each continuous time window, and then using a multi-object tracking algorithm to assign a persistent ID to each entity and record the centroid position, the conversion from "frame-level objects" to "temporal entities" is achieved. In traditional on-duty video analysis, the common practice is to detect each frame independently, resulting in the same person being treated as multiple unrelated objects in different frames, and unable to form a continuous behavior trajectory. This step introduces entity ID and centroid trajectory, enabling the system to continuously track the same person or the same car on the time axis, thereby identifying dynamic behavior patterns such as "gradual approach", "long tail", and "short time fast approach and then leave" in subsequent analysis. Taking street conflict as an example, without entity tracking, single-frame detection alone cannot distinguish between "accidental passing" and "multiple approaches leading to pushing"; however, with persistent ID and centroid trajectory, it can be found that the two people approach each other multiple times and stay for a long time, which in essence improves the system's sensitivity to potential conflict behavior, solving the problem of long-time behavior correlation that traditional frame-independent detection cannot capture. By introducing scene fusion features into entity features, each entity node is actually embedded in the global context of the current time window; and by explicitly preserving the geometric information of the bounding box, the "who approaches whom" and "who blocks in the middle of the road" can be more accurately determined when constructing relationships between entities. This is more than the existing approach of constructing entity nodes based solely on appearance features, adding a layer of "semantic + geometric" joint coding, which can more accurately distinguish between ordinary pedestrians, core conflict participants, and on-duty personnel in complex on-duty scenes, significantly improving the discrimination basis for subsequent event recognition. By introducing edge features that consider both spatial distance and relative speed, and using a learnable perception mechanism to build edge scoring and map to edge weights, adaptive spatial edge filtering and weight distribution are achieved. This comprehensive edge construction method of "distance + relative speed + node features" effectively overcomes the problem of insufficient ability to describe complex crowd interaction scenes with fixed threshold methods, making the graph structure more consistent with the "true interaction relationship" in on-duty scenes, thereby providing high-quality topology for subsequent graph-based spatio-temporal modeling. By setting edge weight thresholds to dynamically add or delete spatial edges, and fixedly establishing temporal edges between the same entity across windows, an adaptive temporal graph is constructed that contains spatial relationships and behavior continuity, allowing the entire on-duty recorded video to be organized as "entity nodes + adaptive relationships + time links" in the graph structure layer. Spatial edges represent immediate interactions between different subjects, while temporal edges depict the behavior trajectory of a single subject, both of which together form a complete spatio-temporal relationship network.

[0055] Further, using a graph convolution network for spatial convolution and temporal convolution to update entity features The node uses attention graph convolution to update the node feature:

[0056] where is the neighbor set of node i, is the normalized attention coefficient, which is normalized by LeakyReLU after the calculation of node i and node j, is the weight matrix of the l-th layer, is the node feature of the l-th layer, which is initially , is the updated node feature; After performing O-layer aggregation, the spatial aggregation feature is obtained; The time series of entity i is counted and a one-dimensional convolution kernel with a convolution kernel size of R is used for time series convolution:

[0057]

[0058] where is the spatial aggregation feature of the continuous window k, is the convolution weight matrix of the r-th time step, is the convolution output feature; The time series convolution output is taken as the node feature with spatio-temporal context semantics.

[0059] By performing the above steps on each time series graph The attention graph convolution is performed on the nodes, and the node features are aggregated in multiple levels in the spatial dimension. In the conventional on-duty video analysis, if only the fixed weight graph convolution or the simple average neighbor feature is used, in the scene with dense crowd and complex entity relationship, the problem that the important neighbor and the irrelevant neighbor are treated equally may occur. For example, there are both the conflict opposite party and the irrelevant passer-by near the on-duty personnel. If the aggregation is not distinguished, the strong action of the conflict opposite party and the slight movement of the irrelevant passer-by are mixed together, which weakens the key information. The present application introduces the normalized attention coefficient, so that each node assigns different weights to different neighbors according to the matching degree of the own feature and the neighbor feature, so that the neighbor nodes that really participate in the high-risk behaviors such as confrontation, pursuit and pulling are dominant in the aggregation process, and the influence of the distant passer-by and the background vehicle is automatically reduced. The node features after the O-layer spatial aggregation are sequentially arranged in time sequence and a one-dimensional time sequence convolution is applied thereto, which is equivalent to constructing a “behavior pattern filter” for each entity in the time axis. The key event in the on-duty scene is usually not a certain isolated moment, but a combination of a series of continuous actions, such as “approaching-confrontation-pushing”, “stopping-parking-getting off-inspecting”, “stopping-turning back to observe-escaping” and the like. If only the graph structure of a single time window is observed, even if the spatial convolution is very good, only the posture or distance condition at a moment can be recognized, and it is difficult to grasp the behavior evolution as a whole. The time sequence convolution output is output as the node feature with the spatio-temporal context semantic, which provides a high-level representation with “local interaction structure” and “historical behavior trajectory” for the subsequent key score and event candidate construction.

[0060] S3, mapping the entity features into key scores using an MLP and extracting a key node set to form an event candidate, scoring and classifying the event candidate to output key event information, and arranging and displaying the key event information; Specifically, mapping the entity features into key scores using an MLP and extracting a key node set to form an event candidate, scoring and classifying the event candidate to output key event information means mapping the node features into key scores using an MLP network

[0061] wherein is the MLP network parameter; A key threshold is set, and the nodes with the key scores higher than the key threshold are marked as high-score nodes, and the high-score nodes that are adjacent in space and continuous in time are aggregated into an event candidate set wherein the node set of the mth event candidate is The event level is scored:

[0062] wherein​ event level score; aggregate node features of all nodes in the event candidate (e.g., average pooling of node features) to form event aggregate features classify the event type based on the event aggregate features using a classifier (e.g., SVM classifier).

[0063] The automatic learning mapping from "spatio-temporal structured features" to "quantifiable importance" is achieved by using MLP to map node features to criticality scores. Unlike many existing methods that directly make binary classification at frame level or graph level, this step measures importance at a finer "entity-time" granularity, with the network learning "which behaviors, locations, and relationship patterns are more worthy of attention" under sample-driven. By scoring each node individually through MLP, it can be observed that the scores of relevant personnel nodes and the nodes of nearby officers immediately adjacent to them are significantly higher than those of distant passerby nodes, thus numerically embodying the distinction between "core" and "background". By setting a criticality threshold and marking nodes with scores higher than the threshold as high-score nodes, and then aggregating high-score nodes into event candidate sets according to the rule of "spatial adjacency and temporal continuity", the structured organization from "discrete high-score points" to "complete event segments" is achieved. By summarizing the criticality scores of nodes within each event candidate set to obtain event level scores and aggregating node features into event features, quantitative evaluation and representation at the event level are achieved, thus forming a consistent scoring and representation system at both "node" and "event" levels. Based on event aggregate features, a general classifier (e.g., SVM) is introduced to classify event types, enabling "event recognition" to expand from simple score ranking to "semantic category determination". By key node screening and event candidate construction, the problem is reduced to "local high-risk segments", and then fine classification is performed in this subspace. This two-level structure is more robust in the highly redundant and noisy environment of actual duty videos.

[0064] Further, the key event information is arranged and displayed. The events are sorted according to the event level scores, and the event types are attached to each event to form a key information list for output and display.

[0065] The embodiment also provides an AI key information extraction system for duty record videos, comprising: The feature extraction module is configured to obtain the duty record video, extract video key frames, perform normalization, and perform noise reduction processing on audio signals, divide time windows, extract multi-modal features, and generate fusion features using cross-modal attention. The graph updating module is configured to identify entities using a multi-target tracking algorithm on the video key frames, construct initial entity features for each entity, construct spatial edges and temporal edges using the entities as nodes, calculate weights, construct an adaptive temporal graph, and update entity features using a graph convolution network for spatial convolution and temporal convolution. An event identification module is configured to map the entity features into a key score using the MLP, extract a key node set to form an event candidate, score and classify the event candidate, output key event information, and arrange and display the key event information.

[0066] The embodiment also provides a computer device suitable for the AI key information extraction method for on-duty record video, which comprises a memory and a processor.

[0067] The computer device can be a terminal, which comprises a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is configured to perform wired or wireless communication with external terminals. The wireless communication can be achieved through WIFI, a carrier network, NFC (Near Field Communication) or other technologies. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the computer device, or an external keyboard, touchpad or mouse, etc.

[0068] The embodiment also provides a storage medium having a computer program stored thereon. The program is executed by a processor to implement the AI key information extraction method for on-duty record video. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic memory, a flash memory, a magnetic disk or an optical disk.

[0069] To sum up, by key frame normalization and audio noise reduction, multi-modal feature cross-modal attention fusion, adaptive time sequence graph construction and graph convolution time sequence update, and in combination with MLP key score and event candidate screening, automatic extraction and arrangement display of key event information in the on-duty record video are realized, the key event recognition accuracy and usability are improved, and the on-duty record backtracking efficiency is improved.

[0070] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present application but not limit the present application. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced equivalently without departing from the spirit and scope of the technical solutions of the present application, and they should be covered in the scope of the claims of the present application.

Claims

1. An AI key information extraction method for on-duty record video, characterized in that: comprising, The video key frame is identified by using a multi-target tracking algorithm, an initial entity feature is constructed for each entity, the entity is taken as a node to construct a spatial edge and a time sequence edge, and a weight is calculated to construct an adaptive time sequence graph, and a graph convolution network is used for spatial convolution and time sequence convolution to update the entity feature; The entity feature is mapped to a key score by using an MLP, a key node set is extracted to form an event candidate, the event candidate is scored and classified to output key event information, and the key event information is arranged and displayed. The noise-reduced audio is framed, each frame has a length of L sampling points, and a frame shift is S sampling points, and each frame of audio signal is multiplied by a Hamming window to calculate a short-time Fourier transform; 2.The AI key information extraction method of on-duty recording video according to claim 1, wherein: The obtained on-duty record video extracts video key frames for normalization and performs noise reduction processing on an audio signal , calculates the average absolute difference between adjacent video frames ; Updating key frame threshold with sliding window ; average absolute difference video frames greater than a keyframe threshold are labeled as keyframes retain, pixel normalize keyframes; On audio signals Time-domain noise reduction using lightweight Wave-U-Net; The short-time Fourier transform result is projected through a mel filter bank to obtain an fth mel channel and take a logarithm to obtain a mel spectrum graph. The attention score is converted into a modality attention weight by using a softmax; 3.The AI key information extraction method of on-duty recording video according to claim 2, wherein: The dividing time window extracts multi-modal features and generates fusion features by using cross-modal attention. The video and audio are divided into continuous windows with a length of The key frames in each continuous window are extracted, and the EfficientNet-Lite0 model is used for visual feature extraction of the key frames.​ pooling the visual features of all key frames in the continuous window to obtain window visual features ; for each successive window corresponding mel-spectrogram extracting audio features ; obtaining a text transcription sequence from an ASR model for audio input in a continuous window ;​ inputting the text transcription sequence into DistilBERT to obtain a text feature vector ; The window visual feature, the audio feature and the text feature vector of each continuous window are formed into a multi-modal feature triple output, and the three features in the feature triple are mapped to the same dimension Attention scores are calculated for each modality; The key frame is detected by using a lightweight target detector (such as MobileNet-SSD) in each continuous window to obtain a boundary box of the entity. The features of the three modalities are weighted and summed and linearly transformed based on attention weights to obtain fused features . 4.The AI key information extraction method of on-duty recording video according to claim 3, characterized in that: The key event information is arranged and displayed, that is, the events are sorted according to the event level score, and an event type is added to each event to form a key information list for output and display. assigning a persistent ID to each entity using a multi-object tracking algorithm and recording the centroid position of the entity ; Construct initial entity features for each entity i in a consecutive window k ; Spatial distance of the computing entities i and j at the time window k and defining a relative velocity; Constructing edge feature vectors , using a layer of perceptron to calculate edge score and using sigmoid for score mapping to obtain edge weight ; If the edge weight between entities i and j If the value exceeds a set threshold, a spatial edge is added between entities i and j, and a temporal edge is established between adjacent consecutive windows k and k+1 for the same entity i. Construct an adaptive sequence graph by using entities as nodes. . 5.The AI key information extraction method of on-duty recording video according to claim 4, characterized in that: The updating entity features using spatial convolution and temporal convolution of the graph convolution network refers to updating node features of each time series graph using attention graph convolution ; performing the O-layer aggregation to obtain a spatial aggregation feature ; Time series of statistical entities i and using a one-dimensional convolution kernel with a kernel size of R for a temporal convolution; Temporal convolutional output as node features with spatio-temporal context semantics Output. 6.The AI key information extraction method of on-duty recording video according to claim 5, wherein: The using MLP to map entity features into key scores and extract a key node set to form an event candidate, scoring and classifying the event candidate to output key event information refers to using an MLP network to map node features into key scores ; Setting a criticality threshold, marking nodes with criticality scores higher than the criticality threshold as high-score nodes, and aggregating high-score nodes that are spatially adjacent and temporally continuous into an event candidate set wherein the node set of the mth event candidate is Scoring the event level; aggregating the node features of all nodes in the event candidate to form event aggregated features classifying the event type using a classifier based on the event aggregated features. 7.The AI key information extraction method of on-duty recording video according to claim 6, wherein: comprising, 8. An AI key information extraction system for on-duty recording video, based on the AI key information extraction method for on-duty recording video according to any one of claims 1 to 7, characterized in that: The feature extraction module is configured to acquire a duty record video, extract video key frames, perform normalization, and perform noise reduction processing on an audio signal, divide a time window, extract multi-modal features, and generate fusion features by using cross-modal attention. The graph updating module is configured to identify entities by using a multi-target tracking algorithm on the video key frames, construct an initial entity feature for each entity, construct a spatial edge and a time sequence edge by taking the entity as a node, calculate a weight to construct an adaptive time sequence graph, and update the entity feature by using a graph convolution network for spatial convolution and time sequence convolution. The event identification module is configured to map the entity feature to a key score by using an MLP, extract a key node set to form an event candidate, score and classify the event candidate to output key event information, and arrange and display the key event information. The processor executes the computer program to implement the steps of the AI key information extraction method of the duty record video according to any one of claims 1-7. 9.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is characterized in that: The computer program is executed by the processor to implement the steps of the AI key information extraction method of the duty record video according to any one of claims 1-7.

10. A computer readable storage medium having stored thereon a computer program, characterized in that: ​

Images