Video abstract frame extraction system and method for law enforcement and case handling abnormal behavior analysis

Abstract

The invention relates to a video abstract frame extraction system and method for law enforcement and case handling abnormal behavior analysis. The video abstract frame extraction system comprises a video access module, a video decoding fragment segmentation module, an abstract frame extraction module and a video frame output module. According to the invention, accurate frame extraction sampling can be carried out on the motion video, the problem that key motion information is lost due to inaccurate extracted video frames in the prior art is solved, and abnormal behavior analysis in the field of law enforcement and case handling can be more accurately facilitated.

Description

Technical Field

[0001] The invention belongs to the technical field of video surveillance, and relates to a video summary frame extraction system and method, in particular to a video summary frame extraction system and method for abnormal behavior analysis in law enforcement and case handling. Background Art

[0002] With the widespread application of video surveillance, video surveillance data is growing at a very fast rate. The traditional method of directly storing, transmitting and browsing raw videos is very inefficient.

[0003] In order to make surveillance videos have better applications in the field of abnormal behavior analysis in law enforcement, a video summary frame extraction method suitable for surveillance videos is urgently needed to reduce the amount of data. This video frame extraction method requires high accuracy and real-time performance.

[0004] The video summarization frame extraction method mainly processes and extracts each frame in the video, obtains key frames and corresponding feature information, and provides basic data for subsequent video analysis and processing.

[0005] However, existing video summarization frame extraction methods have many defects. For example, the sampling method based on time points ignores the motion properties of people and objects in the video, so the generated summary frames are of low accuracy and cannot well support the abnormal behavior analysis function of law enforcement.

[0006] Therefore, in order to solve the above technical problems, the present invention proposes a video summarization frame extraction system and method with high accuracy for abnormal behavior analysis in law enforcement and case handling.

[0007] After searching, no public documents of the prior art identical or similar to the present invention were found. Summary of the invention

[0008] The purpose of the present invention is to overcome the shortcomings of the prior art and propose a real-time video summarization frame extraction system and method for abnormal behavior analysis in law enforcement and case handling, which can extract frames according to the motion in the video stream and improve the efficiency and accuracy of abnormal behavior analysis services.

[0009] The present invention solves the practical problem by adopting the following technical solutions:

[0010] A video summary frame extraction system for abnormal behavior analysis in law enforcement and case handling, comprising: a video access module, a video decoding segment segmentation module, a summary frame extraction module and a video frame output module;

[0011] The video access module is used to access the monitoring video signal of the video monitoring device. The output end of the video access module is connected to the video decoding segment segmentation module. The video decoding segment segmentation module is used to parse the video stream and split the video stream into motion segments and static segments.

[0012] The output end of the video decoding segment segmentation module is connected to a summary frame extraction module, and the summary frame extraction module performs summary frame extraction on the dynamic segment;

[0013] The output end of the summary frame extraction module is connected to the video frame output module, and the video frame output module is used to output the extracted key frames to the law enforcement and case handling abnormal behavior analysis module.

[0014] A video summarization frame extraction method for analyzing abnormal behavior in law enforcement and case handling comprises the following steps:

[0015] Step 1: The video access module receives the monitoring video code stream data output by the video monitoring device;

[0016] Step 2: The video decoding segment segmentation module decodes and splits the monitoring video code stream data, and divides the video code stream into motion segments and static segments;

[0017] Step 3, retaining the video frames of the motion clips in the data real-time cache;

[0018] Step 4: a frame abstraction module abstracts the data in the real-time buffer of the motion frame by using a time sampling method, and outputs the extracted video key frame data;

[0019] Step 5: The video frame output module transmits the extracted video key frame data to the law enforcement and case handling abnormal behavior analysis module, thereby completing a real-time video summary frame extraction method for law enforcement and case handling abnormal behavior analysis.

[0020] Moreover, the specific method of step 2 is:

[0021] Step 2.1: The video decoding segment segmentation module decodes and splits the monitoring video code stream data. When it is determined that the input video frame type is an I frame, the first frame is retained in the motion segment. When it is not the first frame, it is determined whether it is a motion segment based on the frame activity. If it is a motion segment, it is retained in the motion segment.

[0022] Step 2.2: When the input video frame type is determined to be a P frame, determine whether the first P frame is a motion frame. If it is a motion frame, it is considered that the motion segment begins, and the subsequent P frames are still considered to be motion segments until the motion segment ends; otherwise, it is considered to be a static segment.

[0023] Step 2.3: When it is considered to be a motion segment, start distinguishing P frames into still frames and motion frames. When it is a still frame, start counting n=n+1; n represents the number of consecutive still frames, starting from 0. When n is greater than a fixed threshold, the motion segment is considered to be over, and this video is retained in the motion segment. The next P frame is re-judged to be a motion frame. If it is a motion frame, the motion segment starts again, the count n is stopped and reset to zero, and this video is retained in the motion segment.

[0024] When it is considered to be a still segment, it is also necessary to distinguish P frames as still frames and motion frames. When the frame type is judged to be a motion frame, it means that a new motion segment has started, and the video is retained to the motion segment. When the frame type is judged to be a still frame, the segment is abandoned.

[0025] Moreover, the specific method of dividing the P frame into a still frame and a motion frame in step 2.3 is:

[0026] (1) Use a block matching algorithm to perform motion analysis on the decoded video to obtain a two-dimensional motion vector field;

[0027] (2) Based on the obtained two-dimensional motion vector field of the current frame, an intensity label map STI is formed, and the intensity label map STI is scanned with a spatial template to obtain a local activity histogram to describe the spatial distribution of the motion vector of the current frame;

[0028] (3) Based on the spatial distribution of the motion vector of the current frame described by the obtained local activity histogram, the intensity information of the motion vector of the current frame and LAH are combined to form the video motion activity description value of the current frame, and the motion frame and the still frame are distinguished according to the activity description value of each frame.

[0029] Moreover, the specific method of step 2.3 (1) is:

[0030] The first is image segmentation, which divides each frame of the decoded video sequence into many non-overlapping blocks;

[0031] Then, the matching blocks are searched and the most similar matching block is found for each block of the current frame in the search area of ​​the previous frame.

[0032] Finally, the motion vector is calculated. The motion displacement is calculated by the relative position of the matching block and the current block. The obtained motion displacement is the motion vector of the current block. The motion vector of each block is integrated to obtain the two-dimensional motion vector field of the current frame.

[0033] Moreover, the specific method of step 2.3 (2) is:

[0034] First, based on the obtained two-dimensional motion vector field of the current frame, according to the intensity of the motion vector, it is quantized and assigned an intensity level mark R(K), the number of intensity marks corresponds to the number of quantization levels of the motion vector, and each quantization level corresponds to an intensity mark; an intensity mark map is obtained from the two-dimensional motion vector field of the current frame;

[0035] Then, a 4-neighborhood spatial template structure is determined, and the template is used to scan the intensity mark map STI of the current frame, and the number of template types with different mark distributions obtained at each scanning position is counted to obtain the local activity histogram of the current frame to describe the spatial distribution of the motion vector of the current frame;

[0036] The local activity statistics histogram of the current frame is defined as follows:

[0037] LAH=(h1,h2,…,h n )

[0038]

[0039] Among them, h i represents the value of the i-th level of the local activity template histogram, Ω i refers to the i-th local activity template defined under the spatial template P, num(.) is the counting function, Ω i The formal definition of the binary power set is as follows:

[0040]

[0041] Here R(i) is the motion vector strength level mark, Represents the number of marks with activity level marked as R(i) in the spatial template P; the num(.) counting function determines the template level based on the number of types of intensity marks appearing in the spatial template and the number of vectors corresponding to such marks.

[0042] When using n-level neighborhood space template and r-level intensity label, N of LAN is determined by the following formula:

[0043]

[0044] Moreover, the specific method of step 2.3 (3) is:

[0045] The concept of frame motion is introduced and represented by three parameters: the sum of block motion vector moduli, the mean of block motion vector moduli and the maximum block motion vector moduli;

[0046] The activity of the current frame is comprehensively measured by these three parameters, and the video frames are divided into motion frames and still frames;

[0047] The formula for the modulus sum of block motion vectors is:

[0048] The block motion vector modulus mean is:

[0049] The maximum block motion vector modulus is:

[0050] m and n are the number of blocks in the horizontal and vertical directions in the current frame, 0≤i≤m, 0≤j≤n, v ij (x) is the horizontal value of the motion vector of the current block, v ij (y) is the value of the motion vector of the current block in the vertical direction, and N is the number of blocks whose motion vector modulus is not 0;

[0051] When the moving object occupies a large area in the image, the value of the sum of the block motion vector moduli will be particularly large, and the frame is considered to have a large motion activity;

[0052] When the moving object is small, the value of the sum of the block motion vector moduli will not be particularly large, but the mean value of the block motion vector moduli will still be relatively large. In this case, the motion activity of the frame is also considered to be large.

[0053] When the values ​​of both are not large, but the maximum block motion vector modulus is large, it can still be determined that the motion activity of the frame is large.

[0054] By measuring whether the motion activity of a frame is large or not by these three situations, all video frames can be divided into motion frames, i.e. frames with large motion activity, and still frames, i.e. frames with small motion activity;

[0055] Through the above method, the video code stream is divided into motion segments and static segments.

[0056] Advantages and beneficial effects of the present invention:

[0057] The present invention proposes a video summarization frame extraction system and method for abnormal behavior analysis in law enforcement and case handling. By adding an analysis algorithm based on video activity to split video motion segments on the basis of the existing fixed-time sampling method, accurate frame sampling of motion videos is achieved, which solves the problem of inaccurate loss of key motion information in the extracted video frames in the prior art, and can be more accurately beneficial to abnormal behavior analysis in the field of law enforcement and case handling. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 is a process flow chart of the present invention;

[0059] Figure 2 A processing flow chart of video segmentation of the present invention;

[0060] Description of reference numerals:

[0061] Video surveillance equipment 1, video access module 2, video decoding segmentation module 3, summary frame extraction module 4, video frame output module 5, law enforcement and case handling abnormal behavior analysis module 6. DETAILED DESCRIPTION

[0062] The embodiments of the present invention are further described in detail below with reference to the accompanying drawings:

[0063] A video summary frame extraction system for abnormal behavior analysis in law enforcement and case handling, comprising: a video access module, a video decoding segment segmentation module, a summary frame extraction module and a video frame output module;

[0064] The video access module is used to access the monitoring video signal of the video monitoring device. The output end of the video access module is connected to the video decoding segment segmentation module. The video decoding segment segmentation module is used to parse the video stream and split the video stream into motion segments and static segments.

[0065] The output end of the video decoding segment segmentation module is connected to a summary frame extraction module, and the summary frame extraction module performs summary frame extraction on the dynamic segment;

[0066] The output end of the summary frame extraction module is connected to the video frame output module, and the video frame output module is used to output the extracted key frames to the law enforcement and case handling abnormal behavior analysis module.

[0067] A video summarization frame extraction method for abnormal behavior analysis in law enforcement, such as Figure 1 As shown, the following steps are included:

[0068] Step 1: The video access module receives the monitoring video code stream data output by the video monitoring device;

[0069] Step 2: The video decoding segment segmentation module decodes and splits the monitoring video code stream data, and divides the video code stream into motion segments and static segments;

[0070] like Figure 2 As shown, the specific method of step 2 is:

[0071] Step 2.1: The video decoding segment segmentation module decodes and splits the monitoring video code stream data. When it is determined that the input video frame type is an I frame, the first frame is retained in the motion segment. When it is not the first frame, it is determined whether it is a motion segment based on the frame activity. If it is a motion segment, it is retained in the motion segment.

[0072] Step 2.2: When the input video frame type is determined to be a P frame, determine whether the first P frame is a motion frame. If it is a motion frame, it is considered that the motion segment begins, and the subsequent P frames are still considered to be motion segments until the motion segment ends; otherwise, it is considered to be a static segment.

[0073] Step 2.3: When it is considered to be a motion segment, start distinguishing P frames into still frames and motion frames. When it is a still frame, start counting n=n+1 (n represents the number of consecutive still frames, starting from 0). When n is greater than a fixed threshold, it is considered that the motion segment is over, and this video is retained in the motion segment. The next P frame is re-determined whether it is a motion frame. If it is a motion frame, the motion segment starts again, the count n is stopped and reset to zero, and this video is retained in the motion segment.

[0074] When it is considered to be a still segment, it is also necessary to distinguish P frames as still frames and motion frames. When the frame type is judged to be a motion frame, it means that a new motion segment has started, and the video is retained to the motion segment. When the frame type is judged to be a still frame, the segment is abandoned.

[0075] In this embodiment, I frame, which means a key frame, is an independent frame in video encoding and can be independently decoded to generate a complete image without referring to other frames.

[0076] In this embodiment, the P frame is the difference between the current frame and a previous key frame or P frame, that is, a difference frame. The P frame does not have complete picture data, but only data of the picture difference with the previous frame.

[0077] The specific method of distinguishing the p-frames as still frames and motion frames in step 2.3 is as follows:

[0078] In this embodiment, the decoded video code stream is split into motion frames and still frames using a local activity histogram method; the specific steps include:

[0079] (1) Use a block matching algorithm to perform motion analysis on the decoded video to obtain a two-dimensional motion vector field;

[0080] The specific steps include:

[0081] The first is image segmentation, which divides each frame of the decoded video sequence into many non-overlapping blocks;

[0082] Then, the matching blocks are searched and the most similar matching block is found for each block of the current frame in the search area of ​​the previous frame.

[0083] Finally, the motion vector is calculated. The motion displacement is calculated by the relative position of the matching block and the current block. The obtained motion displacement is the motion vector of the current block. The motion vector of each block is integrated to obtain the two-dimensional motion vector field of the current frame.

[0084] (2) Based on the obtained two-dimensional motion vector field of the current frame, an intensity label map STI is formed, and the intensity label map STI is scanned with a spatial template to obtain a local activity histogram to describe the spatial distribution of the motion vector of the current frame;

[0085] The specific steps include:

[0086] First, based on the obtained two-dimensional motion vector field of the current frame, according to the strength of the motion vector, it is quantized and assigned an intensity level mark R(K). The number of intensity marks corresponds to the number of quantization levels of the motion vector, and each quantization level corresponds to an intensity mark. In this way, we can obtain a strength tag image (STI) from the two-dimensional motion vector field of the current frame.

[0087] Then, a 4-neighborhood spatial template structure is determined, and the template is used to scan the intensity mark map STI of the current frame. The number of template types with different mark distributions obtained at each scanning position is counted to obtain the local activity histogram (LAH) of the current frame to describe the spatial distribution of the motion vector of the current frame.

[0088] In this embodiment, the local activity statistical histogram of the current frame is defined as follows:

[0089] LAH=(h1,h2,…,h n )

[0090]

[0091] Among them, h i represents the value of the i-th level of the local activity template histogram, Ω i refers to the i-th local activity template defined under the spatial template P, num(.) is the counting function, Ω i The formal definition of the binary power set is as follows:

[0092]

[0093] Here R(i) is the motion vector strength level mark, represents the number of markers with activity level R(i) in the spatial template P. Thus, the num(.) counting function determines the template level according to the number of types of intensity markers appearing in the spatial template and the number of vectors corresponding to such markers.

[0094] When using n-level neighborhood space template and r-level intensity label, N of LAN is determined by the following formula:

[0095]

[0096] (3) Based on the spatial distribution of the motion vector of the current frame described by the obtained local activity histogram, the intensity information of the motion vector of the current frame and LAH are combined to form the video motion activity description value of the current frame, and the motion frame and the still frame are distinguished according to the activity description value of each frame.

[0097] The specific method is:

[0098] The concept of frame motion is introduced and represented by three parameters: the sum of block motion vector moduli, the mean of block motion vector moduli and the maximum block motion vector modulus.

[0099] By comprehensively measuring the activity of the current frame through these three parameters, the video frames can be divided into motion frames (frames with greater motion activity) and still frames (frames with less motion activity).

[0100] The formula for the modulus sum of block motion vectors is:

[0101] The block motion vector modulus mean is:

[0102] The maximum block motion vector modulus is:

[0103] m and n are the number of blocks in the horizontal and vertical directions in the current frame, 0≤i≤m, 0≤j≤n, v ij (x) is the horizontal value of the motion vector of the current block, v ij (y) is the value of the motion vector of the current block in the vertical direction, and N is the number of blocks whose motion vector modulus is not 0.

[0104] When the area occupied by the moving object in the image is large, the value of the sum of the block motion vector moduli will be particularly large, and the frame is considered to have a large motion activity. When the moving object is small, the value of the sum of the block motion vector moduli will not be particularly large, but the mean block motion vector modulus will still be relatively large, and the frame is also considered to have a large motion activity. When both values ​​are not large, but the maximum block motion vector modulus is large, it can still be determined that the frame has a large motion activity. By measuring whether a frame has a large motion activity using these three situations, all video frames can be divided into motion frames (frames with large motion activity) and static frames (frames with small motion activity). 。

[0105] Through the above method, the video code stream is divided into motion segments and static segments.

[0106] Step 3, retaining the video frames of the motion clips in the data real-time cache;

[0107] Step 4: The abstract frame extraction module abstracts the data in the real-time buffer of the motion frame by using the time sampling method, and outputs the extracted video key frame data.

[0108] Step 5: The video frame output module transmits the extracted video key frame data to the law enforcement and case handling abnormal behavior analysis module, thereby completing a real-time video summary frame extraction method for law enforcement and case handling abnormal behavior analysis.

[0109] It should be emphasized that the embodiments of the present invention are illustrative rather than restrictive. Therefore, the present invention includes but is not limited to the embodiments described in the specific implementation modes. Any other implementation modes derived by those skilled in the art based on the technical solutions of the present invention also fall within the scope of protection of the present invention.

Claims

1. A video summarization frame extraction system for abnormal behavior analysis in law enforcement, characterized by: include: Video access module, video decoding segmentation module, summary frame extraction module and video frame output module; The video access module is used to access the monitoring video signal of the video monitoring device. The output end of the video access module is connected to the video decoding segment segmentation module. The video decoding segment segmentation module is used to parse the video stream and split the video stream into motion segments and static segments. The output end of the video decoding segment segmentation module is connected to a summary frame extraction module, and the summary frame extraction module performs summary frame extraction on the dynamic segment; The output end of the summary frame extraction module is connected to the video frame output module, and the video frame output module is used to output the extracted key frames to the law enforcement and case handling abnormal behavior analysis module.

2. A video summarization frame extraction method for abnormal behavior analysis in law enforcement, characterized by: The following steps are involved: Step 1: The video access module receives the monitoring video code stream data output by the video monitoring device; Step 2: The video decoding segment segmentation module decodes and splits the monitoring video code stream data, and divides the video code stream into motion segments and static segments; Step 3, retaining the video frames of the motion clips in the data real-time cache; Step 4: a frame abstraction module abstracts the data in the real-time buffer of the motion frame by using a time sampling method, and outputs the extracted video key frame data; Step 5: The video frame output module transmits the extracted video key frame data to the law enforcement and case handling abnormal behavior analysis module, thereby completing a real-time video summary frame extraction method for law enforcement and case handling abnormal behavior analysis.

3. The video summarization frame extraction method for abnormal behavior analysis in law enforcement and case handling according to claim 2 is characterized by: The specific method of step 2 is: Step 2.1: The video decoding segment segmentation module decodes and splits the monitoring video code stream data. When it is determined that the input video frame type is an I frame, the first frame is retained in the motion segment. When it is not the first frame, it is determined whether it is a motion segment based on the frame activity. If it is a motion segment, it is retained in the motion segment. Step 2.2: When the input video frame type is determined to be a P frame, determine whether the first P frame is a motion frame. If it is a motion frame, it is considered that the motion segment begins, and the subsequent P frames are still considered to be motion segments until the motion segment ends; otherwise, it is considered to be a static segment. Step 2.3: When it is considered to be a motion segment, start distinguishing the P frame into a still frame and a motion frame. When it is a still frame, start counting n=n+1; n represents the number of consecutive still frames, starting from 0. When n is greater than a fixed threshold, the motion segment is considered to be over, and this video is retained in the motion segment. The next P frame is re-judged to be a motion frame. If it is a motion frame, the motion segment starts again, the count n is stopped and reset to zero, and this video is retained in the motion segment. When it is considered to be a still segment, it is also necessary to distinguish P frames as still frames and motion frames. When the frame type is judged to be a motion frame, it means that a new motion segment has started, and the video is retained to the motion segment. When the frame type is judged to be a still frame, the segment is abandoned.

4. The video summarization frame extraction method for abnormal behavior analysis in law enforcement and case handling according to claim 3 is characterized by: The specific method of dividing the P frames into still frames and motion frames in step 2.3 is as follows: (1) Use a block matching algorithm to perform motion analysis on the decoded video to obtain a two-dimensional motion vector field; (2) Based on the obtained two-dimensional motion vector field of the current frame, an intensity label map STI is formed, and the intensity label map STI is scanned with a spatial template to obtain a local activity histogram to describe the spatial distribution of the motion vector of the current frame; (3) Based on the spatial distribution of the motion vector of the current frame described by the obtained local activity histogram, the intensity information of the motion vector of the current frame and LAH are combined to form the video motion activity description value of the current frame, and the motion frame and the still frame are distinguished according to the activity description value of each frame.

5. The video summarization frame extraction method for abnormal behavior analysis in law enforcement and case handling according to claim 4 is characterized by: The specific method of step 2.3 (1) is: The first is image segmentation, which divides each frame of the decoded video sequence into many non-overlapping blocks; Then search for matching blocks and find the most similar matching block for each block of the current frame in the search area of ​​the previous frame; Finally, the motion vector is calculated, and the motion displacement is calculated by the relative position of the matching block and the current block. The obtained motion displacement is the motion vector of the current block. The motion vector of each block is integrated to obtain the two-dimensional motion vector field of the current frame.

6. The video summarization frame extraction method for abnormal behavior analysis in law enforcement and case handling according to claim 4 is characterized by: The specific method of step 2.3 (2) is: First, based on the obtained two-dimensional motion vector field of the current frame, according to the intensity of the motion vector, it is quantized and assigned an intensity level mark R(K), the number of intensity marks corresponds to the number of quantization levels of the motion vector, and each quantization level corresponds to an intensity mark; an intensity mark map is obtained from the two-dimensional motion vector field of the current frame; Then, a 4-neighborhood spatial template structure is determined, and the template is used to scan the intensity mark map STI of the current frame, and the number of template types with different mark distributions obtained at each scanning position is counted to obtain the local activity histogram of the current frame to describe the spatial distribution of the motion vector of the current frame; The local activity statistics histogram of the current frame is defined as follows: LAH=(h1,h2,…,h n ) Among them, h i represents the value of the i-th level of the local activity template histogram, Ω i refers to the i-th local activity template defined under the spatial template P, num(.) is the counting function, Ω i The formal definition of the binary power set is as follows: Here R(i) is the motion vector strength level mark, represents the number of markers with activity level R(i) in the spatial template P; the num(.) counting function determines the template level according to the number of types of intensity markers appearing in the spatial template and the number of vectors corresponding to such markers; When using n-level neighborhood space template and r-level intensity label, N of LAN is determined by the following formula:

7. The video summarization frame extraction method for abnormal behavior analysis in law enforcement and case handling according to claim 4 is characterized by: The specific method of step 2.3 (3) is: The concept of frame motion is introduced and represented by three parameters: the sum of block motion vector moduli, the mean of block motion vector moduli and the maximum block motion vector moduli; The activity of the current frame is comprehensively measured by these three parameters, and the video frames are divided into motion frames and still frames; The formula for the sum of block motion vector modulo is: The block motion vector modulus mean is: The maximum block motion vector modulus is: m and n are the number of blocks in the horizontal and vertical directions in the current frame, 0≤i≤m, 0≤j≤n, v ij (x) is the horizontal value of the motion vector of the current block, v ij (y) is the value of the motion vector of the current block in the vertical direction, and N is the number of blocks whose motion vector modulus is not 0; When the moving object occupies a large area in the image, the value of the sum of the block motion vector moduli will be particularly large, and the frame is considered to have a large motion activity; When the moving object is small, the value of the sum of the block motion vector moduli will not be particularly large, but the mean value of the block motion vector moduli will still be relatively large. In this case, the motion activity of the frame is also considered to be large. When the values ​​of both are not large, but the maximum block motion vector modulus is large, it can still be determined that the motion activity of the frame is large; By measuring whether the motion activity of a frame is large or not by these three situations, all video frames can be divided into motion frames, i.e. frames with large motion activity, and still frames, i.e. frames with small motion activity; Through the above method, the video code stream is divided into motion segments and static segments.

Images