Coarse segmentation method for police sample videos and intelligent slicing method for body camera videos

Abstract

This invention discloses a coarse segmentation method for police sample videos and an intelligent slicing method for law enforcement recorder videos, belonging to the field of video analysis technology. This invention prioritizes dividing police sample videos into several confidence intervals that are relatively independent in semantic and physical context based on the semantic coherence of the text. When semantic information is missing or unreliable, it seamlessly switches to secondary segmentation based on visual features and metadata. Furthermore, it optimizes the multimodal segmentation results based on semantic dominance and interval merging rules to obtain more accurate coarse segmentation results, significantly reducing the workload of manual annotation. This invention designs an intelligent slicing network that integrates multimodal and gating decision-making, utilizing multimodal data for complementarity and introducing a gating network to achieve dynamic adaptive multimodal fusion, effectively suppressing noise interference, thereby significantly improving the accuracy and reliability of the model's boundary detection.

Description

Technical Field

[0001] This invention relates to the field of video analysis technology, and more specifically, to: 1. a coarse segmentation method for police sample videos based on multimodal analysis; 2. an intelligent slicing method for law enforcement recorder videos that integrates multimodal analysis and gating decision-making. Background Technology

[0002] Body cameras are evidence-gathering devices worn by police officers during official duties, and the police videos they record are crucial pieces of evidence. Currently, police video analysis heavily relies on manual review, resulting in low processing efficiency. This makes it difficult to handle the large-scale processing demands of massive amounts of video data and fails to meet the urgent requirements of increasingly sophisticated smart policing and digital governance scenarios for automated video content analysis and structuring.

[0003] While automated video analysis methods based on deep learning have emerged—using trained neural networks to process videos and segment them—they suffer from fundamental limitations. First, neural networks rely on labeled sample data, and the segmentation of police sample videos depends on professional personnel. Police sample videos are characterized by long time spans and significant changes in shooting angles, making event boundary labeling extremely time-consuming and labor-intensive, severely restricting the application of video analysis in the police field. Second, most existing automated video analysis methods are single-modal, relying solely on visual or audio cues. Their reliability is poor in complex law enforcement scenarios such as poor lighting, violent shaking, and noisy environments, making it difficult to produce stable structured data and thus affecting the video analysis results. Third, although some automated video analysis methods use multimodal approaches, their multimodal fusion strategies are rigid—using fixed-weight feature splicing or simple addition, failing to dynamically evaluate the confidence level of each modality based on the actual scenario. When the signal quality of a particular modality is poor, noise severely interferes with judgment, affecting the accuracy of structured information and thus impacting the video analysis results. Summary of the Invention

[0004] Based on this, it is necessary to address the issues that existing police sample video event boundary marking relies on manual methods and that existing police video analysis methods based on single or multimodal modes are ineffective. Therefore, a coarse segmentation method and an intelligent slicing method for police sample videos are proposed.

[0005] This invention is achieved using the following technical solution: In a first aspect, the present invention provides a coarse segmentation method for police sample videos based on multimodality, comprising: Obtain police sample videos and corresponding device sample metadata; Speech recognition is performed on police sample videos to obtain a continuous text stream, and the continuous text stream is processed based on the principle of semantic coherence to obtain several ordered confidence intervals; each confidence interval is marked as high semantic or low semantic, and two adjacent confidence intervals are separated by one segmentation time point. Visual content analysis was performed on police sample videos to obtain several content change points, and motion state analysis was performed on device sample metadata to obtain several motion state transition points. Only content change points and motion state transition points that are marked as low semantic confidence intervals were retained. If a certain confidence interval contains a sudden change in screen content or / and a transition point in motion state, then this point is used as the segmentation time point to further divide the confidence interval into several sub-intervals; otherwise, the confidence interval is treated as a single sub-interval. The process involves sequentially traversing all sub-intervals for selective merging to obtain the final coarse segmentation result. If two sub-intervals with different labels are adjacent and the sub-interval labeled with higher semantics comes first, then the two sub-intervals are merged into a new sub-interval labeled with higher semantics. Otherwise, the two sub-intervals remain unchanged.

[0006] This multimodal-based coarse segmentation method for police sample videos implements the method or process according to embodiments of this disclosure.

[0007] Secondly, this invention discloses a method for intelligent slicing of law enforcement recorder videos that integrates multimodal and gating decision-making, including: Acquire police video streams and device metadata streams recorded by law enforcement recorders within a preset time period; Visual feature sequence f was extracted from police video streams. v and text feature sequence f t Extracting motion feature sequences f from device metadata stream m And form a three-modal input sequence F=[f v , f t , f m ]; F is input into a trained intelligent slicing network for processing to obtain the boundary probability sequence P; Based on P, event boundary points are determined to complete intelligent slicing of police video streams; The intelligent slicing network is trained based on police sample videos; some or all segmentation time points of the police sample videos are obtained using the multimodal coarse segmentation method for police sample videos disclosed in the first aspect.

[0008] This method for intelligent slicing of law enforcement recorder videos, which integrates multimodal and gating decision-making, implements the method or process according to embodiments of this disclosure.

[0009] Thirdly, the present invention discloses a computer program product comprising a computer program. When executed by a processor, the computer program implements the steps of the coarse segmentation method for police sample videos based on multimodality as disclosed in the first aspect, or the steps of the law enforcement recorder video slicing method integrating multimodality and gating decision as disclosed in the second aspect.

[0010] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention provides a method for boundary labeling of police sample videos. It prioritizes dividing the police sample videos into several confidence intervals that are relatively independent in semantic and physical context based on the semantic coherence of the text. When semantic information is missing or unreliable, it seamlessly switches to secondary segmentation based on visual features and metadata. Furthermore, it optimizes the multimodal segmentation results based on semantic dominance and interval merging rules to obtain more accurate coarse segmentation results, which significantly reduces the workload of manual labeling.

[0011] 2. This invention designs an intelligent slicing network that integrates multimodal and gated decision-making. It utilizes multimodal data to achieve complementarity and introduces a gated network to achieve dynamic adaptive multimodal fusion, effectively suppressing noise interference and thus significantly improving the accuracy and reliability of the model's boundary detection. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 The flowchart shows the coarse segmentation method for police sample videos based on multimodality provided in Embodiment 1 of the present invention. Figure 2 This is a flowchart of the intelligent video slicing method for law enforcement recorders that integrates multimodal and gating decision-making, provided in Embodiment 2 of the present invention. Detailed Implementation

[0014] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0015] It should be noted that when a component is said to be "installed on" another component, it can be directly on the other component or it may be in a component that is centered on it. When a component is said to be "set on" another component, it can be directly set on the other component or it may also be in a component that is centered on it. When a component is said to be "fixed to" another component, it can be directly fixed to the other component or it may also be in a component that is centered on it.

[0016] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the specification of this invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "or / and" as used herein includes any and all combinations of one or more of the associated listed items.

[0017] Example 1 Referring to the background art, current methods for marking event boundaries in police sample videos rely on manual intervention. See also... Figure 1 This embodiment 1 provides a coarse segmentation method for police sample videos based on multimodality, which realizes automated segmentation of police sample videos to obtain coarse segmentation results, thus effectively reducing the amount of manual work.

[0018] like Figure 1 As shown, the coarse segmentation method for police sample videos based on multimodal approaches includes: S1, acquire police sample videos and corresponding device sample metadata.

[0019] It should be noted that, as mentioned in the background section, the police sample video is captured by a law enforcement recorder and has a timestamp. Therefore, it also synchronously stores device sample metadata that represents relevant motion information—the specific data types of which include, but are not limited to, GPS location, speed, acceleration, and attitude.

[0020] Of course, police sample videos may involve personal privacy information and should only be used with the consent of the individuals involved.

[0021] S2, performs preliminary segmentation from the perspective of text modality.

[0022] S2 mainly determines whether the semantics are coherent based on semantic similarity, thereby dividing the intervals. It includes: performing speech recognition on police sample videos to obtain a continuous text stream, and processing the continuous text stream based on the principle of semantic coherence to obtain several ordered confidence intervals.

[0023] Each confidence interval is labeled as high semantic or low semantic, and two adjacent confidence intervals are separated by a single splitting time point.

[0024] Specifically, processing continuous text streams based on the principle of semantic coherence to obtain several ordered confidence intervals includes: S201 divides a continuous text stream into an ordered sequence of text units U using sentence boundary detection methods (commonly known as pySBD); where U includes n text units U1~U2. n ; n > 1.

[0025] Each text unit represents a complete semantic expression unit, which usually corresponds to a complete sentence or a semantically complete phrase.

[0026] S202, each text unit is semantically encoded using a pre-trained speech model (such as BERT or its variants), resulting in n semantic vectors V1~V2. n .

[0027] Among them, V i U was captured i The deep semantic information; i∈[1,n].

[0028] S203, taking two adjacent semantic vectors as one semantic slice to obtain an ordered sequence of semantic slices W; where W includes: n-1 semantic slices (V1, V2) ~ (V n-1 V n ).

[0029] It should be noted that there is some overlap between two adjacent semantic vectors.

[0030] S204, the cosine similarity function is used to calculate the similarity between two adjacent semantic vectors in each semantic slice to obtain an ordered similarity sequence Sim; where Sim includes: n-1 similarity values ​​sim(V1, V2) ~ sim(V n-1 V n ).

[0031] sim(V1, V2) represents the degree of similarity between U1 and U2. Other similarity values ​​are similar and will not be explained again.

[0032] In fact, connecting Sims sequentially can construct a semantic coherence curve—a curve that intuitively reflects the changing trend of semantic coherence in a continuous text stream. The i-th data point on this curve is sim(V) i V i+1 ), also means U i U i+1 Semantic transition coherence.

[0033] S205, by judging the similarity of each semantic slice and the similarity threshold T_conf (usually an empirical value), W is divided into intervals to obtain several ordered confidence intervals.

[0034] Specifically, if a confidence interval is marked as high semantic, then the semantic segments within it are continuous and their similarity is not less than T_conf. In other words, a confidence interval marked as high semantic contains semantically clear and coherent segments.

[0035] If a confidence interval is marked as low semantics, then the semantic segments within it are continuous and all have a similarity less than T_conf. In other words, confidence intervals marked as low semantics contain segments with ambiguous semantics or where speech recognition fails.

[0036] S205 actually divides the intervals from the perspective of semantic coherence by traversing in sequence, specifically including: Use (V1, V2) as the starting point of the first confidence interval; If sim(V1, V2)≥T_conf, then continue traversing backwards until a semantic slice with a similarity less than T_conf is found. This semantic slice is used as the starting point of the second confidence interval, the semantic slice preceding this semantic slice is used as the ending point of the first confidence interval, and the first confidence interval is marked as high semantic. If sim(V1, V2) < T_conf, then continue traversing backwards until a semantic slice with a similarity not less than T_conf is found. This semantic slice is used as the starting point of the second confidence interval, the semantic slice preceding this semantic slice is used as the ending point of the first confidence interval, and the first confidence interval is marked as low semantic. This process continues until W is traversed to obtain a number of ordered confidence intervals.

[0037] For ease of understanding, assume W: (V1, V2), (V2, V3), (V3, V4), (V4, V5), (V5, V6), and satisfy: sim(V1, V2)≥T_conf, sim(V2, V3)≥T_conf, sim(V3, V4)≥T_conf; sim(V4, V5)<T_conf; sim(V5, V6)≥T_conf. Then, according to the above division, it is divided into 3 confidence intervals and 4 splitting time points—specifically including: The confidence interval is [V1, V4], which includes: (V1, V2), (V2, V3), and (V3, V4). The second confidence interval is [V4, V5], which includes (V4, V5). Confidence interval three is [V4, V5], which includes (V5, V6); The first split point is the time corresponding to V1; the second split point is the split point between confidence interval 1 and confidence interval 2, which is the time corresponding to V4; the third split point is the split point between confidence interval 2 and confidence interval 3, which is the time corresponding to V5; and the fourth split point is the time corresponding to V6.

[0038] S3 optimizes and corrects the initial segmentation results based on visual and metadata modalities.

[0039] S3 primarily supplements semantic information through visual and metadata modalities when semantic information is missing or unreliable. Specifically, it includes: S301, Visual content analysis is performed on police sample videos to obtain several points of abrupt change in the content of the images.

[0040] Abrupt changes in visual content can be one or more of the following: potential scene transition points, points of dramatic change in motion intensity, and points of abrupt change in human posture. Therefore, methods used for visual content analysis include, but are not limited to: histogram difference method, scene detection algorithms, dense optical flow algorithms, and posture estimation algorithms.

[0041] Specifically, histogram difference methods or scene detection algorithms (such as PySceneDetect) can be used to detect scene content—if the difference between adjacent frames exceeds the scene threshold θ_scene, it is marked as a potential scene transition point. Dense optical flow algorithms (such as Farneback optical flow) can also be used to calculate inter-frame motion vectors—if the average optical flow amplitude exceeds the optical flow threshold θ_flow, it is determined as a point of abrupt change in motion intensity. Pose estimation algorithms (such as the OpenPose model) can also be used to detect specific behavioral patterns of people—if a sudden change in posture occurs (such as raising a hand, running, etc.), it is identified as a point of sudden change in human posture.

[0042] S302, perform motion state analysis on the device sample element information to obtain several motion state transition points.

[0043] Specifically, S302 includes: S3021, perform filtering and noise reduction processing on the device sample metadata (e.g., using Kalman filtering or moving average filtering).

[0044] S3022, based on the motion state threshold, state judgment is performed to determine several motion state transition points.

[0045] For example, first set a speed threshold V_thresh and a duration threshold T_stop. Then, if the speed changes from below V_thresh to above V_thresh, it is marked as the start point of motion; when the speed changes from above V_thresh to below V_thresh and the duration exceeds T_stop, it is marked as the start point of stillness. These points are all motion state transition points.

[0046] S303 selectively retains abrupt changes in image content and transition points in motion state based on semantic-dominated rules.

[0047] Specifically, if a sudden change in screen content or a transition in motion state falls within a confidence interval marked as low semantic meaning, that point is deleted. In other words, only sudden changes in screen content and transitions in motion state that fall within a confidence interval marked as low semantic meaning are retained.

[0048] This rule ensures that, within a high semantic confidence interval, segmentation decisions are primarily based on semantic information.

[0049] S304 performs secondary segmentation based on retained abrupt changes in image content and motion state transition points.

[0050] If a certain confidence interval contains a sudden change in image content or / and a transition point in motion state, then this point is used as the segmentation time point to further divide the confidence interval into several sub-intervals; otherwise, the confidence interval is treated as a single sub-interval.

[0051] The subintervals still have labels and will retain the labels of their respective confidence intervals.

[0052] It's important to note that only confidence intervals marked with low semantic meaning will exhibit secondary segmentation. For confidence intervals marked with high semantic meaning, they are directly treated as a sub-interval.

[0053] S305, selectively merges sub-intervals based on the semantic continuation principle.

[0054] In other words, all sub-intervals are traversed sequentially for selective merging to obtain the final coarse segmentation result. Specifically, if two sub-intervals with different labels are adjacent and the sub-interval with the higher semantic label comes first, then the two sub-intervals are merged into a new sub-interval with the higher semantic label; otherwise, the two sub-intervals are left unchanged.

[0055] It should be emphasized that if a low semantic confidence interval is divided into two or more sub-intervals, only the first low semantic sub-interval immediately following the high semantic sub-interval will be merged.

[0056] This selective merging is based on the understanding that brief semantic vacuums (such as ambient noise or pauses) are usually a continuation of previous high semantic events, so subsequent low semantic sub-regions can be merged into preceding high semantic sub-regions.

[0057] Therefore, the coarse segmentation result obtained so far includes: several sub-intervals, and two adjacent sub-intervals are separated by a segmentation time point - these segmentation time points are the boundary markers finally obtained by this method.

[0058] It should be noted that the coarse segmentation results cannot guarantee 100% accuracy, so manual verification of the coarse segmentation results can be performed when necessary. Compared to annotation, the workload of verification is relatively small.

[0059] Example 2 Referring to the background art, existing police video analysis methods based on single-modal or multi-modal approaches are ineffective. See also... Figure 2 This embodiment 2 provides a method for intelligent slicing of law enforcement recorder videos that integrates multimodal and gating decision-making. It utilizes multimodal data to achieve complementarity and introduces a gating network to achieve dynamic adaptive multimodal fusion, thereby improving the accuracy and reliability of the model's boundary detection.

[0060] like Figure 2 As shown, the intelligent video slicing method for law enforcement recorders that integrates multimodal and gating decision-making includes: Step 1: Obtain the police video stream and device metadata data stream recorded by the law enforcement recorder within a preset time period (denoted as ΔT).

[0061] The police video stream and equipment metadata data stream are analogous to the police sample video and equipment sample metadata in Example 1, and will not be described again here.

[0062] Step 2: Extract the visual feature sequence f from the police video stream. v and text feature sequence f t Extracting motion feature sequences f from device metadata stream m And form a three-modal input sequence F=[f v , f t , f m ].

[0063] In this embodiment 1, it is recommended to use a pre-trained I3D network to extract f. v —Its dimension is 1024; it is recommended to use a pre-trained BERT model to extract f. t —Its dimensions are 768; it is recommended to standardize the device metadata flow to obtain f. m —Its dimensions are 11 (including 2-dimensional latitude and longitude, 6-dimensional velocity and acceleration, and 3-dimensional attitude).

[0064] Of course, f v , f t , f m All of these are temporal features, each containing a series of visual features, text features, and motion features corresponding to a specific moment.

[0065] In other words, f v =[f v _1, f v _2,…, f v _ΔT];f t =[f t _1, f t _2,…, f t _ΔT];f m =[f m _1,f m _2,…, f m [_ΔT]; where j∈[1, ΔT]; f v _j represents the visual feature corresponding to the j-th time point within ΔT; f t _j represents the text feature corresponding to the j-th time point within ΔT; f m _j represents the motion characteristic corresponding to the j-th time within ΔT.

[0066] Step 3: Input F into the trained intelligent slicing network for processing to obtain the boundary probability sequence P; determine the event boundary points based on P to complete the intelligent slicing of the police video stream.

[0067] In this embodiment 2, an intelligent slicing network is constructed, which adopts multimodal input and introduces a gating network to achieve adaptive weight allocation.

[0068] See Figure 2 The intelligent slicing network consists of: 3 gating networks, 1 Softmax function layer, 1 weighted fusion layer, 1 time encoder, and 1 predictor.

[0069] The first gated network is used for f-based v Generate visual scalar weights s v ; The second gated network is used for f-based... t Generate text scalar weights s t ; The third gated network is used for f-based... m Generate motion scalar weights s m ; The Softmax function layer is used to activate s through the Softmax activation function. v s t s mNormalized into gated visual weights g v Gated text weight g t Gated motion weights g m .

[0070] Among them, g v g t g m This can form a gating weight vector g, which reflects the system's confidence in each mode under the current input, and satisfies g v +g t +g m =1.

[0071] The above process can also be expressed by a formula: s v =GRU(f v ); s t =GRU(f t ); s m =GRU(f m ); g=[g v , g t , g m ] = Softmax(s v , s t , s m ); In the formula, GRU(.) represents the gating network processing procedure; Softmax(.) represents the Softmax function layer processing procedure.

[0072] The weighted fusion layer is used to combine g v with f v Multiply, and g t with f t Multiply, and g m with f m The three products are multiplied and then summed to obtain the fused feature representation h. This weighted fusion allows the network to suppress the contribution of noisy modes according to the weights, while retaining the original information channels of each mode in the final feature for further processing by subsequent network layers.

[0073] The above process can also be expressed by a formula: h=g v ·f v ⊕g t ·f t ⊕g m ·f m .

[0074] h is actually a temporal feature, which contains the fused features corresponding to a series of time points. That is, h = [h1, h2, ..., h...]. ΔT ]; where h j This represents the fusion feature corresponding to the j-th time within ΔT.

[0075] The timing encoder is a Transformer encoder, which is used to encode and model h to obtain the timing encoded sequence H.

[0076] The above process can also be expressed by a formula: H = TransformerEncoder(h); In the formula, TransformerEncoder(.) represents the timing encoder processing procedure.

[0077] H is also a temporal feature, which contains the encoded features corresponding to a series of time points. That is, H=[H1, H2,…,H ΔT ]; where H j This represents the encoded feature corresponding to the j-th time point within ΔT.

[0078] The predictor consists of one linear layer and one sigmoid function layer. The linear layer is used to perform a linear transformation on H. The sigmoid function layer is used to process the output of the linear layer through the sigmoid activation function to obtain P.

[0079] The above process can also be expressed by a formula: P = σ(linear(H)); In the formula, linear(.) represents the linear layer processing procedure; σ(.) represents the Sigmoid function layer processing procedure.

[0080] P is also a time-series feature, which contains a series of probability values ​​corresponding to different times. That is, P=[P1, P2,…,P…]. ΔT ]; where P j This represents the probability that the event boundary is at time j within ΔT.

[0081] So, P j Compare with the probability threshold θ_p, if P j If the value is greater than θ_p, it means that the j-th moment within ΔT is an event boundary; otherwise, it means that the j-th moment within ΔT is not an event boundary. This completes the intelligent slicing of police videos.

[0082] Of course, step three uses a trained intelligent slice network—which is trained on police sample videos using a conventional end-to-end training method to ensure optimal network performance.

[0083] For police sample videos, they contain accurate labels (i.e., correct segmentation time points). These accurate labels can be manually annotated or obtained through coarse segmentation using the multimodal police sample video coarse segmentation method provided in Example 1 (with manual verification if necessary). In other words, some or all segmentation time points of the police sample video are obtained using the multimodal police sample video coarse segmentation method provided in Example 1.

[0084] It is important to note that the loss function L used during training of the intelligent slicing network is... total for: L total =L boundary +λ×L gate ; In the formula, L boundary L represents the boundary detection loss; λ represents the weighting coefficient; L gate This represents the gated regularization loss.

[0085] ① For L boundary In this regard, it uses weighted binary cross-entropy loss to solve the problem of sparse boundary points.

[0086] L boundary The expression is: ; In the formula, y j This represents the true boundary label at the j-th time point within ΔT—1 if it is a segmentation time point, 0 otherwise; W p W represents the weight of positive samples; n W represents the weight of negative samples; p +W n =1.

[0087] ② For L gate In other words, it aims to encourage the network to reduce the corresponding weights when a mode fails.

[0088] L gate The expression is: ; In the formula, X represents the set of three modalities (including: text modality, visual modality, and metadata modality); x represents any modal data in X; (x,j) represents x at the j-th time within ΔT; quality(x,j) represents the modal quality corresponding to (x,j); θ represents the modal quality threshold; I{quality(x,j),θ} is the modal quality indicator function—if quality(x,j) < θ, then I{quality(x,j),θ} is 1, otherwise I{quality(x,j),θ} is 0; g x(j) represents the true gating weight label corresponding to x at time j within ΔT.

[0089] It should be noted that, since the coarse segmentation method for police sample videos based on multimodality provided in Example 1 involves multimodality, the actual gating weight labels are set as follows: If (x,j) corresponds to a split point determined by a high semantic confidence interval, then g x (j) The true gating weight label in semantics is 1, and the true gating weight labels in visual and metadata are both 0; If (x,j) corresponds to a screen content transition point, then g x (j) The true gating weight label in visual sense is 1, and the true gating weight labels in semantic sense and metadata are both 0; If (x,j) corresponds to a motion state transition point, then g x (j) The true gating weight label in metadata is 1, and the true gating weight labels in semantics and vision are both 0.

[0090] Of course, the sample police videos can also be augmented with data augmentation strategies to increase their diversity and optimize the training dataset. These data augmentation strategies include, but are not limited to: random insertion mechanisms, adversarial replacement mechanisms, modal noise injection strategies, and sample distribution optimization mechanisms.

[0091] Specifically: ① The random insertion mechanism uses cross-domain fragment insertion to enhance the model's generalization ability.

[0092] For example, clips related to police scenes can be extracted from copyrighted films and television works (such as action movies, dramas, documentaries, etc.) and randomly inserted into sample police videos; the start and end times of each inserted clip can be accurately recorded, and boundary labels can be created at the start and end positions of the inserted clips.

[0093] ② The adversarial replacement mechanism increases the difficulty of the data by replacing only a single modality of data, so that the model can learn the features of each modality.

[0094] For example, one could retain the audio and metadata of the original segments in a sample police video and replace only the visual content; or retain the video and metadata of the original segments in a sample police video and replace only the audio content. Regardless of the replacement method used, it is essential to ensure that the replaced boundary labels remain consistent with the original semantics.

[0095] ③ The modal noise injection strategy involves adding noise data to some or all modalities while keeping the interval unchanged, thus enhancing the model's anti-interference ability.

[0096] For example, noise can be added to the audiovisual aspects of sample police videos. Specifically, visually, rectangular occlusion areas can be randomly added for 2-5 seconds to simulate equipment occlusion; directional blur effects can be added based on optical flow estimation to simulate motion blur; and screen brightness and contrast can be adjusted to simulate low-light scenes. Auditorily, environmental noises such as sirens, crowd noise, and wind and rain can be injected, or intermittent recordings and sudden silence segments can be generated, superimposed with background dialogue and radio interference to simulate equipment malfunctions.

[0097] ④ The sample distribution optimization mechanism adjusts the event distribution of sample police videos based on the actual occurrence of police incidents.

[0098] For example, by analyzing the distribution patterns of various events in historical law enforcement data, rare but important event types (such as firearms use and high-risk arrests) can be added to the sample police videos to ensure that the training samples are both comprehensive and focused.

[0099] Example 3 This embodiment 3 discloses a computer device, including a memory and a processor. The memory stores a computer program. When the processor executes the computer program, it implements the steps of the coarse segmentation method for police sample videos based on multimodality disclosed in embodiment 1, or the steps of the law enforcement recorder video slicing method that integrates multimodality and gating decision disclosed in embodiment 2.

[0100] The computer equipment can be either a mobile terminal or a fixed terminal. Examples of the former include mobile phones, laptops, digital radio receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), and in-vehicle terminals (such as in-vehicle navigation terminals); examples of the latter include digital TVs and desktop computers.

[0101] This embodiment 3 also discloses a readable storage medium storing computer program instructions. When the computer program instructions are read and executed by a processor, they perform the steps of the coarse segmentation method for police sample videos based on multimodality disclosed in embodiment 1, or the steps of the law enforcement recorder video slicing method that integrates multimodality and gating decision disclosed in embodiment 2.

[0102] The readable storage medium may include, but is not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above.

[0103] This embodiment 3 also discloses a computer program product, including a computer program. When executed by a processor, the computer program implements the steps of the coarse segmentation method for police sample videos based on multimodality disclosed in embodiment 1, or the steps of the law enforcement recorder video slicing method integrating multimodality and gating decision disclosed in embodiment 2.

[0104] It should be noted that the computer program used to execute the above can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, C++, and Python—as well as conventional procedural programming languages—such as C or similar languages. The computer program can be executed entirely on the user's computer, partially on the user's computer, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer through any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN).

[0105] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A coarse segmentation method for police sample videos based on multimodal data, characterized in that, It includes: Obtain police sample videos and corresponding device sample metadata; Speech recognition is performed on police sample videos to obtain a continuous text stream, and the continuous text stream is processed based on the principle of semantic coherence to obtain several ordered confidence intervals; each confidence interval is marked as high semantic or low semantic, and two adjacent confidence intervals are separated by one segmentation time point. Visual content analysis was performed on police sample videos to obtain several content change points, and motion state analysis was performed on device sample metadata to obtain several motion state transition points. Only content change points and motion state transition points that are marked as low semantic confidence intervals were retained. If a certain confidence interval contains a sudden change in screen content or / and a transition point in motion state, then this point is used as the segmentation time point to further divide the confidence interval into several sub-intervals; otherwise, the confidence interval is treated as a single sub-interval. The process involves sequentially traversing all sub-intervals for selective merging to obtain the final coarse segmentation result. If two sub-intervals with different labels are adjacent and the sub-interval labeled with higher semantics comes first, then the two sub-intervals are merged into a new sub-interval labeled with higher semantics. Otherwise, the two sub-intervals remain unchanged.

2. The coarse segmentation method for police sample videos based on multimodal processing according to claim 1, characterized in that, The data types of device sample metadata include: GPS location, speed, acceleration, and attitude.

3. The coarse segmentation method for police sample videos based on multimodal processing according to claim 1, characterized in that, Based on the principle of semantic coherence, continuous text streams are processed to obtain several ordered confidence intervals, including: S201, using sentence boundary detection, divides a continuous text stream into an ordered sequence of text units U; where U includes n text units U1~U2. n n > 1; S202, each text unit is semantically encoded using a pre-trained speech model, resulting in n semantic vectors V1~V2. n ; S203, taking two adjacent semantic vectors as one semantic slice to obtain an ordered sequence of semantic slices W; where W includes: n-1 semantic slices (V1, V2) ~ (V n-1 V n ); S204, the cosine similarity function is used to calculate the similarity between two adjacent semantic vectors in each semantic slice to obtain an ordered similarity sequence Sim; where Sim includes: n-1 similarity values ​​sim(V1, V2) ~ sim(V n-1 V n ); S205, By judging the similarity of each semantic slice and the size of the similarity threshold T_conf, W is divided into intervals to obtain several ordered confidence intervals; If a certain confidence interval is marked as high semantic, then the semantic slices within it are continuous and the similarity is not less than T_conf. If a certain confidence interval is marked as low semantics, then the semantic slices within it are continuous and all have a similarity of less than T_conf.

4. The coarse segmentation method for police sample videos based on multimodal processing according to claim 3, characterized in that, S205 includes: Use (V1, V2) as the starting point of the first confidence interval; If sim(V1, V2)≥T_conf, then continue traversing backwards until a semantic slice with a similarity less than T_conf is found. This semantic slice is used as the starting point of the second confidence interval, the semantic slice preceding this semantic slice is used as the ending point of the first confidence interval, and the first confidence interval is marked as high semantic. If sim(V1, V2) < T_conf, then continue traversing backwards until a semantic slice with a similarity not less than T_conf is found. This semantic slice is used as the starting point of the second confidence interval, the semantic slice preceding this semantic slice is used as the ending point of the first confidence interval, and the first confidence interval is marked as low semantic. This process continues until W is traversed to obtain a number of ordered confidence intervals.

5. The coarse segmentation method for police sample videos based on multimodal processing according to claim 1, characterized in that, The methods used for visual content analysis are: histogram difference method, scene detection algorithm, dense optical flow algorithm, pose estimation algorithm, or any one or more of these.

6. The coarse segmentation method for police sample videos based on multimodal processing according to claim 1, characterized in that, Motion state analysis is performed on the device sample metadata to obtain several motion state transition points, including: The device sample metadata is filtered and denoised, and several motion state transition points are determined based on motion state thresholds.

7. A method for intelligent slicing of law enforcement recorder videos integrating multimodal and gating decision-making, characterized in that, It includes: Acquire police video streams and device metadata streams recorded by law enforcement recorders within a preset time period; Visual feature sequence f was extracted from police video streams. v and text feature sequence f t Extract motion feature sequence f from device metadata stream m And form a three-modal input sequence F=[f v , f t , f m ]; F is input into a trained intelligent slicing network for processing to obtain the boundary probability sequence P; Based on P, event boundary points are determined to complete intelligent slicing of police video streams; Among them, the intelligent slicing network is trained based on police sample videos; The partial or complete segmentation time points of the police sample video are obtained using the coarse segmentation method for police sample videos based on multimodality as described in any one of claims 1-6.

8. The intelligent video slicing method for law enforcement recorders integrating multimodal and gating decision-making as described in claim 7, characterized in that, The intelligent slicing network consists of: 3 gated networks, 1 Softmax function layer, 1 weighted fusion layer, 1 temporal encoder, and 1 predictor; The first gated network is used for f-based v Generate visual scalar weights s v ; The second gated network is used for f-based... t Generate text scalar weights s t ; The third gated network is used for f-based... m Generate motion scalar weights s m ; The Softmax function layer is used to activate s through the Softmax activation function. v s t s m Normalized into gated visual weights g v Gated text weight g t Gated motion weights g m ; The weighted fusion layer is used to combine g v with f v Multiply, and g t with f t Multiply, and g m with f m Multiply the products and sum the three sets of products to obtain the fused feature representation h; The timing encoder is a Transformer encoder, which is used to encode and model h to obtain the timing encoded sequence H; The predictor consists of one linear layer and one sigmoid function layer. The linear layer is used to perform a linear transformation on H. The sigmoid function layer is used to process the output of the linear layer through the sigmoid activation function to obtain P.

9. The intelligent video slicing method for law enforcement recorders integrating multimodal and gating decision-making as described in claim 7, characterized in that, The loss function L used during the training of the intelligent slice network total for: L total =L boundary +λ×L gate ; In the formula, L boundary L represents the boundary detection loss; λ represents the weighting coefficient; L gate Indicates the gate regularization loss; Or / and, sampled police videos also employ data augmentation strategies to increase their diversity.

10. A computer program product comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the coarse segmentation method for police sample videos based on multimodality as described in any one of claims 1-6, or the steps of the intelligent slicing method for law enforcement recorder videos that integrates multimodality and gating decision-making as described in any one of claims 7-9.

Images