Sparse and dense centripetal key frame sampling method for video action recognition

Abstract

The invention belongs to the technical field of video key frame sampling for video understanding and action recognition, and particularly relates to a sparse and dense centripetal key frame sampling method for video action recognition, and the method comprises the following specific steps: S1, firstly, determining the total frame number of a target video and the number of key frames which need to be collected and preset by a user; the stage is the basis of the whole sampling method, and necessary data support can be provided for subsequent key frame selection by mastering the basic information of the video; then, whether the number of key frames needing to be collected exceeds the actual total frame number of the video or not is checked, so that the feasibility and rationality of subsequent processing are ensured; according to the method, model construction is carried out on video key frame sampling again, key frame sampling of a target video is carried out in a mode of regularly constructing sparse, dense and relatively loose key frame sampling, and effective samples are provided for later tasks.

Description

Technical Field

[0001] The present invention relates to the technical field of video key frame sampling for video understanding and action recognition, and in particular to a sparse dense centripetal key frame sampling method for video action recognition. Background Art

[0002] The key problem of video-based action recognition is to efficiently and compactly represent the most informative spatiotemporal features in the video. An efficient action recognition model should be used to process the key information in the video sequence, focus on the important areas in the key frames, that is, the central area of ​​the video sequence, and extract high-dimensional fine-grained features in the key frames that are useful for model recognition in continuous time frames.

[0003] Since there is a lot of redundant information in video actions, this information not only includes the start and end stages of the action in the video, but also contains adjacent frames with high similarity. These redundant frames make it difficult for the network model to converge, and the network model's ability to extract video information degrades. Therefore, we propose a sparse dense concentric key frame sampling method for video action recognition to solve the above problem. Summary of the invention

[0004] 1. Technical issues to be resolved

[0005] In view of the deficiencies of the prior art, the present invention provides a sparse dense centripetal key frame sampling method for video action recognition, which solves the problems raised in the above background technology.

[0006] (II) Technical solution

[0007] In order to achieve the above-mentioned purpose, the present invention specifically adopts the following technical solutions:

[0008] A sparse dense centripetal key frame sampling method for video action recognition. The specific process of the method is as follows:

[0009] S1. First, it is necessary to determine the total number of frames of the target video and the number of key frames that the user presets to be collected; this stage is the basis of the entire sampling method. By mastering the basic information of the video, it can provide necessary data support for the subsequent key frame selection; then, check whether the number of key frames to be collected exceeds the actual total number of frames of the video to ensure the feasibility and rationality of subsequent processing;

[0010] S2. Based on the information obtained in S1, i.e., the relationship between the number of key frames to be collected and the total number of frames of the target video, a branch is selected to perform the key frame extraction work; there are two different paths, each of which is optimized for a specific situation, aiming to effectively extract the most representative key frames regardless of whether the number of video frames is small or large;

[0011] S3. When the number of key frames to be collected is less than or equal to the total number of video frames, the system will adopt a denser sampling strategy to ensure that important details in the video can be captured while maintaining a high temporal resolution, which is crucial for fine-grained action recognition;

[0012] S4. If the number of key frames required to be collected exceeds the total number of video frames, this is usually an abnormal situation. The algorithm will also have a corresponding processing mechanism to ensure that each video frame is regarded as a key frame, or reselect the key frame according to the actual situation to adapt to this special scenario.

[0013] Further, in said S2,

[0014] When the number of samples is less than the total number of frames, when entering the first branch, the core task of this stage is to calculate the first batch of sampling ratios and perform key frame sampling based on this ratio. The specific steps are as follows:

[0015] S2.1. Determine the first batch sampling ratio α

[0016] Input information: the number of key frames to be collected and the total number of frames of the video;

[0017] Determine the first batch sampling ratio: First, determine the first batch sampling ratio α; this first batch sampling ratio determines the frequency of selecting each frame from the original video as a key frame. The larger the first batch sampling ratio value, the more evenly the first batch key frame positions can be distributed throughout the entire video key frame, which is also the key to determining the first batch key frame positions;

[0018] S2.11. Get the first batch of sampling intervals

[0019] Determine the first batch sampling interval: Based on the determined first batch sampling ratio α, the first batch sampling interval epart = frames / (*α*sam) can be determined; after calculation, if epart is an integer, epart is directly used as the first batch sampling interval; if epart is not an integer, the final first batch sampling interval can be determined by rounding up; this is done to ensure the uniformity of sampling, while also taking into account the feasibility in actual operation;

[0020] S2.12 Sampling the first batch of key frames

[0021] Equal sampling: Use the determined epart representation to divide the total number of frames of the target video into epart parts. Starting from the first frame of the video, one frame is selected as a key frame every epart frame until the number of key frames collected is reached; this ensures that the key frames are evenly distributed on the video timeline, which helps capture important moments of action changes;

[0022] Sampling principle: The principle followed in the sampling process is to ensure that the key frames can cover the time range of the entire video as evenly as possible, avoiding deviations in action recognition due to excessive sampling concentration on a certain part of the video; in addition, the characteristics of the video content, such as the complexity and change rate of the action, should also be considered, and the sampling strategy should be appropriately adjusted to adapt to different types of videos; Specific work:

[0023] S2.13, redistribute the order of the first batch of key frames, and perform centripetal sorting according to the sequence numbers of the first batch of key frames, that is, reorder the sequence number values ​​in the new sequence from the center to the left and right sides, and reorder according to the rule of sorting method abs(i-centor_pos), where abs represents the absolute value, i is the sequence number position of the i-th key frame in the target video, and centor_pos is the sequence number of the key frame in the target video whose sequence number is located in the center area;

[0024] S2.14, for the sequence numbers of the first batch of key frames reordered, a midpoint operation is performed between two adjacent sequence numbers to obtain a new sequence number, and the new key frame sequence numbers obtained by midpoint sampling, the second batch of key frame sequence numbers represent the second batch of key frame sequences obtained by sampling, and the second batch of key frame sequences are reordered by the sorting method in S2.13;

[0025] S2.15 The second batch of key frames obtained in S2.14 are formed into a new sequence, and the new sequence is subjected to proximity sampling from the center to the left and right sides, that is, for the sequence number i in the sequence, {i-1, i+1} proximity sampling is performed, and the third batch of key frames are obtained through proximity sampling;

[0026] S2.16, according to the key frame acquisition method of S2.14 and S2.15, alternately extract key frames. During the extraction process, it is necessary to determine whether the number of key frames required to be collected in S1 is met. If the number is sufficient, exit the key frame sampling process and output the key frames; if the number of collected key frames is insufficient, continue to execute the key frame sampling process; finally merge the first batch, the second batch, ..., the nth batch of key frames to obtain the final collected key frames;

[0027] S2.17. The above S2.11-S2.16 are the key frame extraction steps required when the number of key frames to be collected is less than the total number of video frames; the process can be expressed as:

[0028]

[0029] Where keyframes is the key frame obtained by sampling, i is the key frame number that has been collected, M is the total number of frames of the target video, and f nare the key frames that have been collected and their labels, epart is the first sampling interval, and c is the key frame number in the center area of ​​the target video.

[0030] Further, in said S2,

[0031] S2.2 When the number of key frames required to be collected is greater than or equal to the total number of video frames, that is, when the total number of frames is insufficient or just meets the requirement, the second rule key frame sampling method is required.

[0032] S2.21. First, obtain the number of key frames of the target video and the total number of frames of the video; count all frames in the target video to form a sequence;

[0033] S2.22 For all frames in the target video obtained in S2.21, which are the first batch of key frames to be processed, these key frames are firstly re-ordered in a centripetal order of sequence numbers to obtain the first batch of key frames;

[0034] S2.23 copies the first batch of key frames to the left and right sides starting from the video center sequence number, and repeats this process until the number of captured frames is sufficient. The process can be expressed as:

[0035]

[0036] Where keyframes is the key frame obtained by sampling, pretotal is the sequence of all frames of the target video, i is the number of the key frame that has been collected, M is the total number of frames of the target video, epart is the first batch of sampling intervals, and c is the key frame number in the center area of ​​the target video.

[0037] (III) Beneficial effects

[0038] Compared with the prior art, the present invention provides a sparse dense centripetal key frame sampling method for video action recognition, which has the following beneficial effects:

[0039] The present invention constructs a model by re-sampling video key frames, and samples the key frames of the target video by regularly constructing sparse and dense and relatively loose key frame sampling methods, thereby providing effective samples for later tasks.

[0040] The sampling process does not involve image processing, so this key frame sampling method for auxiliary tasks has almost no impact on the real-time performance of downstream tasks. It can be effectively used for videos of any length and has good generalization performance.

[0041] The key frames extracted in the whole process are not affected by the total number of video frames, and focus on the central area of ​​the video sequence, that is, sparsely and densely distributed and relatively loosely distributed over the entire video sequence, so that the sampled key frames are sufficient to represent the key information of the entire video sequence. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is the principle flow chart of the present invention;

[0043] Figure 2 The present invention is a schematic diagram of the sampling process of a video sequence example with a total number of video frames of 55 and 245;

[0044] Figure 3 Schematic diagram of the density distribution of the sampling key frames in the video sequence of the present invention, wherein a is the density distribution diagram of the key frames when 16 frames are sampled when the total number of frames is 55, b is the density distribution diagram when 100 frames are sampled when the total number of frames is 55, and c is the density distribution diagram when 16 frames are sampled when the total number of frames is 245;

[0045] Figure 4 This is a schematic diagram of a sequence density curve when 16 frames are sampled when the total number of video frames is 55 or 245. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] Example

[0048] like Figure 1-4 As shown, a sparse dense centripetal key frame sampling method for video action recognition is proposed in one embodiment of the present invention.

[0049] 1. Take the sparse dense centripetal key frame sampling method as an example by inputting the example video "HeadMassage". The example is a video sequence with a total number of frames of 245 and 55, and 16 frames are sampled.

[0050] S1. Get the total number of frames of the target video (denoted as N total The formula uses frames to represent) and the number of key frames required (denoted as N req The formula uses sam to represent), first analyze N total =55, N req =16 embodiment.

[0051] S2, such as Figure 1 The figure shows the flow chart of the principle of sparse dense centripetal key frame sampling method. total and N reqConfirm the sampling method, here N total >N req , and the process of the "yes" branch is sampled. The detailed process is introduced below.

[0052] S2.1. First, the sampling rate of the target video is calculated by first calculating the commonly used coefficient α, where α=0.3.

[0053] S2.2 Calculate the sampling rate by dividing the sampling rate into equal parts: epart = frames / (α*sam) (here epart = 5), where epart is the first sampling interval, that is, all frames are divided into five parts, each containing 11 frames. The first sampling of equal points is performed by dividing the sampling rate into equal parts, that is, Figure 2 The key frames numbered 1, 2, 3, 4, 5, and 6 in frames:55 complete the first batch of key point sampling.

[0054] S2.2, sort the first batch of key frames (labeled [1,2,3,4,5,6]) in a centripetal manner, and re-sort them according to the rule of sorting method abs(i-centor_pos), where i is the label of the sampled key frame obtained in the first batch of sampling, centor_pos is the label of the key frame at the center position in the sampled key frame label, and the key frame label sequence is [3,4,2,5,1,6] (here "label" is Figure 2 The number in the lower right corner does not represent the sequence number of the key frame in the entire video. [1,2,3,4,5,6] corresponds to the sequence number [1,11,22,33,44,55] in the real video).

[0055] S2.3, obtain the second batch of key frames. The second batch of key frames are sampled by sorting the first batch of key frame numbers, and the second sampling is performed by taking the midpoints of the key frames corresponding to the two adjacent numbers. The rule is [1, 11, 22, 33, 44, 55], and x = (i + (i + 1)) / 2, and the midpoint is determined by rounding up to get Figure 2 The key frames are numbered [7, 8, 9, 10, 11], and the corresponding key frame numbers in the target video are [6, 17, 28, 39, 50]. The second batch of key frames is obtained by obtaining the midpoints of adjacent related key frames.

[0056] S2.4, obtain the third batch of key frames. The obtained second batch of key frame labels are centripetally sorted to obtain the sorted label sequence [9, 10, 8, 11, 7], and the corresponding key frame numbers are [28, 17, 39, 6, 50]. The sorted new sequence is sampled from the center to the left and right sides, that is, for sequence number i, the adjacent sampling of {i-1, i+1} is performed, and the third batch of key frames is obtained through adjacent sampling, as shown below. Figure 2 The key frames numbered [12, 13, 14, 15, 16] correspond to the key frames numbered [27, 29, 16, 18, 38] in the target video. The third batch of key frames is obtained.

[0057] After the entire S2, all 16 key frames are obtained, labeled Figure 2 The label numbers 1-16 correspond to the sequence numbers in the target video [1, 11, 22, 33, 44, 55, 28, 39, 17, 50, 6, 27, 29, 16, 18, 38], and after sorting, they are [1, 6, 11, 16, 17, 18, 22, 27, 28, 29, 33, 38, 39, 44, 50, 55].

[0058] Through the above method, we can continue to derive when N total =55, N req =16, the sampling key frame numbers are [1, 25, 49, 73, 74, 75, 98, 122, 123, 124, 147, 171, 172, 169, 221, 245].

[0059] The above steps can be expressed by the following formula:

[0060]

[0061] 2. Taking the sparse dense centripetal key frame sampling method as an example by inputting the instance video "HeadMassage", when the total number of frames in the instance is 55, 100 frames are sampled.

[0062] S1. Get the total number of frames of the target video (denoted as N total The formula uses frames to represent) and the number of key frames required (denoted as N req The formula uses sam to represent), first analyze N total =55, N req =100 embodiment.

[0063] S2, such as Figure 1 The figure shows the flow chart of the principle of sparse dense centripetal key frame sampling method. total and N req Confirm the sampling method, here N total <N req , and the process of the "no" branch is sampled. The detailed process is introduced below.

[0064] S2.1. First, obtain the number of key frames of the target video (denoted as N req ) and the total number of frames of the video (denoted as N total ). And count the sequence numbers of all video frames.

[0065] S2.2. All the video key frames obtained in S2.1 are the first batch of key frames to be processed. First, these key frames are re-sorted in a centripetal order of sequence numbers to obtain the first batch of key frames.

[0066] S3, copy the first batch of key frames to the left and right sides starting from the video center sequence number, and repeat this process until the number of captured frames is sufficient. The process can be expressed as:

[0067]

[0068] like Figure 3 , 4 As shown, in some embodiments, a key frame density distribution diagram of 16-frame sampling when the total number of frames is 55 and 245 is shown, which can reflect the key frame sampling characteristics of the sparse dense centripetal key frame sampling method, and better focus on the important area, that is, the central area;

[0069] The two different branch sampling methods can comprehensively analyze and process videos with different frame numbers, and perform sparse, dense and centripetal key frame sampling. The obtained key frames can be effectively concentrated in the central area, or key frame sampling can be performed in the target video in a "cluster" manner to obtain evenly distributed dense "clusters", thereby efficiently realizing key frame sampling of target videos with different frame numbers.

[0070] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A sparse dense centripetal keyframe sampling method for video action recognition, characterized by: The specific process of this method is as follows: S1. First, it is necessary to determine the total number of frames of the target video and the number of key frames that the user presets to be collected; this stage is the basis of the entire sampling method. By mastering the basic information of the video, it can provide necessary data support for the subsequent key frame selection; then, check whether the number of key frames to be collected exceeds the actual total number of frames of the video to ensure the feasibility and rationality of subsequent processing; S2. Based on the information obtained in S1, i.e., the relationship between the number of key frames to be collected and the total number of frames of the target video, a branch is selected to perform the key frame extraction work; there are two different paths, each of which is optimized for a specific situation, aiming to effectively extract the most representative key frames regardless of whether the number of video frames is small or large; S3. When the number of key frames to be collected is less than or equal to the total number of video frames, the system will adopt a denser sampling strategy to ensure that important details in the video can be captured while maintaining a high temporal resolution, which is crucial for fine-grained action recognition; S4. If the number of key frames required to be collected exceeds the total number of video frames, this is usually an abnormal situation. The algorithm will also have a corresponding processing mechanism to ensure that each video frame is regarded as a key frame, or reselect the key frame according to the actual situation to adapt to this special scenario.

2. The sparse dense centripetal key frame sampling method for video action recognition according to claim 1, characterized in that: In S2, When the number of samples is less than the total number of frames, when entering the first branch, the core task of this stage is to calculate the first batch of sampling ratios and perform key frame sampling based on this ratio. The specific steps are as follows: S2.

1. Determine the first batch sampling ratio α Input information: the number of key frames to be collected and the total number of frames of the video; Determine the first batch sampling ratio: First, determine the first batch sampling ratio α; this first batch sampling ratio determines the frequency of selecting each frame from the original video as a key frame. The larger the first batch sampling ratio value, the more evenly the first batch key frame positions can be distributed throughout the entire video key frame, which is also the key to determining the first batch key frame positions; S2.

11. Get the first batch of sampling intervals Determine the first batch sampling interval: Based on the determined first batch sampling ratio α, the first batch sampling interval epart = frames / (*α*sam) can be determined; after calculation, if epart is an integer, epart is directly used as the first batch sampling interval; if epart is not an integer, the final first batch sampling interval can be determined by rounding up; this is done to ensure the uniformity of sampling, while also taking into account the feasibility in actual operation; S2.12 Sampling the first batch of key frames Equal sampling: Use the determined epart representation to divide the total number of frames of the target video into epart parts. Starting from the first frame of the video, one frame is selected as a key frame every epart frame until the number of key frames collected is reached; this ensures that the key frames are evenly distributed on the video timeline, which helps capture important moments of action changes; Sampling principle: The principle followed in the sampling process is to ensure that the key frames can cover the time range of the entire video as evenly as possible, avoiding deviations in action recognition due to excessive sampling concentration on a certain part of the video; in addition, the characteristics of the video content, such as the complexity and change rate of the action, should also be considered, and the sampling strategy should be appropriately adjusted to adapt to different types of videos; Specific work: S2.13, redistribute the order of the first batch of key frames, and perform centripetal sorting according to the sequence numbers of the first batch of key frames, that is, reorder the sequence number values ​​in the new sequence from the center to the left and right sides, and reorder according to the rule of sorting method abs(i-centor_pos), where abs represents the absolute value, i is the sequence number position of the i-th key frame in the target video, and centor_pos is the sequence number of the key frame in the target video whose sequence number is located in the center area; S2.14, for the sequence numbers of the first batch of key frames reordered, a midpoint operation is performed between two adjacent sequence numbers to obtain a new sequence number, and the new key frame sequence numbers obtained by midpoint sampling, the second batch of key frame sequence numbers represent the second batch of key frame sequences obtained by sampling, and the second batch of key frame sequences are reordered by the sorting method in S2.13; S2.15 The second batch of key frames obtained in S2.14 are formed into a new sequence, and the new sequence is subjected to proximity sampling from the center to the left and right sides, that is, for the sequence number i in the sequence, {i-1, i+1} proximity sampling is performed, and the third batch of key frames are obtained through proximity sampling; S2.16, according to the key frame acquisition method of S2.14 and S2.15, alternately extract key frames. During the extraction process, it is necessary to determine whether the number of key frames required to be collected in S1 is met. If the number is sufficient, exit the key frame sampling process and output the key frames; If the number of collected key frames is insufficient, the key frame sampling process needs to be continued; Finally, the first batch, the second batch, ..., and the nth batch of key frames are merged to obtain the final collected key frames; S2.

17. The above S2.11-S2.16 are the key frame extraction steps required when the number of key frames to be collected is less than the total number of video frames; the process can be expressed as: Where keyframes is the key frame obtained by sampling, i is the key frame number that has been collected, M is the total number of frames of the target video, and f n are the key frames that have been collected and their labels, epart is the first sampling interval, and c is the key frame number in the center area of ​​the target video.

3. The sparse dense centripetal key frame sampling method for video action recognition according to claim 1, characterized in that: In S2, S2.2 When the number of key frames required to be collected is greater than or equal to the total number of video frames, that is, when the total number of frames is insufficient or just meets the requirement, the second rule key frame sampling method is required. S2.

21. First, obtain the number of key frames of the target video and the total number of frames of the video; count all frames in the target video to form a sequence; S2.22 For all frames in the target video obtained in S2.21, which are the first batch of key frames to be processed, these key frames are firstly re-ordered in a centripetal order of sequence numbers to obtain the first batch of key frames; S2.23 copies the first batch of key frames to the left and right sides starting from the video center sequence number, and repeats this process until the number of captured frames is sufficient. The process can be expressed as: Where keyframes is the key frame obtained by sampling, pretotal is the sequence of all frames of the target video, i is the number of the key frame that has been collected, M is the total number of frames of the target video, epart is the first batch of sampling intervals, and c is the key frame number in the center area of ​​the target video.

Images