A sparse-dense centripetal keyframe sampling method for video action recognition

Abstract

The application belongs to the technical field of video key frame sampling of video understanding and action recognition, and particularly relates to a sparse dense centripetal key frame sampling method for video action recognition, the specific process of which is as follows, S1, first, the total frame number of a target video and the number of key frames required to be collected by a user are determined; this stage is the basis of the whole sampling method, and through mastering the basic information of the video, necessary data support can be provided for subsequent key frame selection; then, whether the number of key frames required to be collected exceeds the actual total frame number of the video is checked to ensure the feasibility and rationality of subsequent processing. The application re-builds a model for video key frame sampling, regularly builds a sparse and dense and relatively loose key frame sampling mode, and performs key frame sampling on the target video, thereby providing effective samples for a later task.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of video key frame sampling for video understanding and action recognition, and particularly relates to a sparse and dense centripetal key frame sampling method for video action recognition. BACKGROUND

[0002] A key problem of video-based action recognition is to efficiently and compactly represent the most information-rich spatiotemporal features in a video. An efficient action recognition model should be used to process the key information in a video sequence, focus attention on the important regions in the key frames, that is, the central regions of the video sequence, and extract high-dimensional fine-grained features in the key frames that are useful for model recognition in continuous time frames.

[0003] Since there is a large amount of redundant information in video actions, the information not only includes the starting and ending stages of the action in the video, but also includes adjacent frames with high similarity. The redundant frames cause the network model to be difficult to converge, and the network model's ability to extract video information is degraded. Therefore, the application provides a sparse and dense centripetal key frame sampling method for video action recognition to solve the above problems. SUMMARY

[0004] (I) Technical problems solved

[0005] In view of the deficiencies in the prior art, the application provides a sparse and dense centripetal key frame sampling method for video action recognition, which solves the problems proposed in the background.

[0006] (II) Technical solutions

[0007] In order to achieve the above purpose, the application specifically adopts the following technical solutions:

[0008] A sparse and dense centripetal key frame sampling method for video action recognition, the specific process of the method is as follows,

[0009] S1, first, the total number of target video frames and the number of key frames required to be collected by the user need to be determined; this stage is the basis of the entire sampling method, and through the mastery of the basic information of the video, necessary data support can be provided for the subsequent key frame selection; then, it is checked whether the number of key frames required to be collected exceeds the actual total number of frames of the video, so as to ensure the feasibility and rationality of subsequent processing;

[0010] S2, according to the information obtained in S1, that is, the relationship between the number of key frames required to be collected and the total number of target video frames, a branch is selected to perform the extraction work of the key frames; here, there are two different paths, each path is optimized for a specific situation, and aims to effectively extract the most representative key frames in the case of fewer or more video frames;

[0011] S3、When the number of key frames required to be collected is less than or equal to the total number of frames of the video, the system will adopt a more intensive sampling strategy to ensure that important details in the video can be captured while maintaining a high temporal resolution, which is crucial for fine motion recognition;

[0012] S4、If the number of key frames required to be collected exceeds the total number of frames of the video, which is usually an exceptional case, the algorithm will also have corresponding processing mechanism to ensure that each frame of the video is considered as a key frame, or to reselect key frames according to actual situation to adapt to this special scene.

[0013] Further, in the S2,

[0014] When the number of samples is less than the total number of frames, enter the first branch, the core task of this stage is to calculate the first batch sampling ratio and sample key frames based on this ratio, the specific steps are as follows:

[0015] S2.1、Determine the first batch sampling ratio a

[0016] Input information: the number of key frames required to be collected and the total number of frames of the video;

[0017] Determine the first batch sampling ratio: first, determine the first batch sampling ratio a; this first batch sampling ratio determines the frequency of selecting each frame as a key frame from the original video, the larger the first batch sampling ratio value, the more evenly distributed the first batch key frame positions in the entire video, which is also the key to determining the first batch key frame positions;

[0018] S2.11、Get the first batch sampling interval

[0019] Determine the first batch sampling interval: based on the determined first batch sampling ratio a, the first batch sampling interval epart = frames / (*a*sam) can be determined; after calculation, if epart is an integer, epart is directly used as the first batch sampling interval; if epart is not an integer, the final first batch sampling interval can be determined by rounding up; this is done to ensure the uniformity of sampling, while also considering the feasibility in actual operation;

[0020] S2.12 Perform first batch key frame sampling

[0021] Equal interval sampling: use the determined epart to divide the total number of frames of the target video into epart parts, starting from the first frame of the video, select a frame every epart frames as a key frame until the number of collected key frames reaches the required number; this can ensure the uniform distribution of key frames on the video time axis, which helps to capture important moments of motion changes;

[0022] The principle followed in the sampling process is to ensure that the key frames can cover the entire time range of the video as evenly as possible, avoiding the deviation of motion recognition caused by too concentrated sampling in a certain part of the video; in addition, the characteristics of the video content, such as the complexity and change rate of the motion, should be considered to adjust the sampling strategy appropriately to adapt to different types of videos; the specific work is as follows:

[0023] S2.13, re-distribute the arrangement order of the first batch of key frames, and perform centripetal sorting through the sequence number of the first batch of key frames, that is, re-sort the sequence number value in the new sequence from the center to the left and right sides, and re-sort according to the sorting mode abs(i-centor_pos), wherein abs represents taking the absolute value, i is the sequence number position of the i-th key frame in the target video, and centor_pos is the sequence number of the key frame in the center region of the target video;

[0024] S2.14, taking the midpoint between the adjacent two sequence numbers of the first batch of key frames, obtaining a new sequence number, and obtaining a new key frame sequence number by midpoint sampling, the second batch of key frame sequence numbers represent the second batch of key frames sampled, and the second batch of key frame sequence is re-sequenced according to the sorting mode in S2.13;

[0025] S2.15, the second batch of key frames obtained in S2.14 are grouped into a new sequence, and adjacent sampling is performed from the center to the left and right sides of the new sequence, that is, the sequence number i in the sequence is adjacent sampled as {i-1, i+1}, and the third batch of key frames are obtained by adjacent sampling;

[0026] S2.16, according to the key frame acquisition mode of S2.14 and S2.15, the key frames are extracted alternately, and it is necessary to judge whether the number of key frames required to be collected in S1 is met during the extraction process, if the number is sufficient, the key frame sampling process is exited, and the key frames are output; if the number of collected key frames is not enough, the key frame sampling process needs to be continued; finally, the first batch, the second batch,..., and the nth batch of key frames are merged to obtain the final collected key frames;

[0027] S2.17, the above S2.11-S2.16 are the key frame extraction steps required when the number of key frames to be collected is less than the total number of frames of the video; the process can be represented as:

[0028]

[0029] Wherein keyframes is the key frame sampled, i is the key frame label collected, M is the total number of frames of the target video, f nFor the collected key frame and its label, epart is the first batch sampling interval, c is the key frame sequence number in the center area of the target video.

[0030] Further, in S2,

[0031] S2.2 When the required number of collected key frames is greater than or equal to the total number of frames of the video, that is, the total number of frames is not enough or just enough, the second rule key frame sampling method needs to be performed.

[0032] S2.21, first obtain the number of key frames of the target video and the total number of frames of the video; count all frames in the target video to form a sequence;

[0033] S2.22 For all frames in the target video obtained in S2.21, which are the first batch of key frames to be processed, first reorder these key frames in a centripetal ordering manner according to the sequence number to obtain the first batch of key frames;

[0034] S2.23 Copy the adjacent frames to the left and right sides from the center sequence number of the video, and repeat the process until the number of collected frames is sufficient. The process can be represented as:

[0035]

[0036] Where keyframes is the sampled key frame, pretotal is the sequence formed by the total frames of the target video, i is the collected key frame label, M is the total number of frames of the target video, epart is the first batch sampling interval, and c is the key frame sequence number in the center area of the target video.

[0037] (Three) beneficial effects

[0038] Compared with the prior art, the present application provides a sparse dense centripetal key frame sampling method for video action recognition, which has the following beneficial effects:

[0039] The present application renews the model construction of video key frame sampling, and constructs a sparse and dense key frame sampling method with relatively loose distribution, which is used for key frame sampling of the target video, and provides effective samples for the later task.

[0040] The sampling process does not involve image processing, so this key frame sampling method for auxiliary tasks almost does not affect the real-time performance of the downstream task, and can be effectively used for any length of video, and has good generalization.

[0041] The key frames extracted in the whole process can be not affected by the total number of frames of the video, and focus on the center area of the video sequence, that is, the sparse and dense distribution on the whole video sequence, so as to realize that the sampled key frames can represent the key information of the whole video sequence. Attached Figure Description

[0042] Figure 1 This is a flowchart illustrating the principle of the present invention;

[0043] Figure 2 This is a schematic diagram illustrating the sampling process of video sequence instances with a total number of 55 and 245 frames in this invention;

[0044] Figure 3 This is a circular schematic diagram of the density distribution of keyframes sampled in a video sequence according to the present invention. Here, a is the density distribution of keyframes when 16 frames are sampled when the total number of frames is 55, b is the density distribution when 100 frames are sampled when the total number of frames is 55, and c is the density distribution when 16 frames are sampled when the total number of frames is 245.

[0045] Figure 4 This is a schematic diagram of the sequence density curve when sampling keyframes of the present invention for 16 frames when the total number of video frames is 55 and 245. Detailed Implementation

[0046] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0047] Example

[0048] like Figures 1-4 As shown in one embodiment of the present invention, a sparse-dense centripetal keyframe sampling method for video action recognition is proposed.

[0049] 1. Taking the sparse-dense centripetal keyframe sampling method by inputting the example video "HeadMassage" as an example, the examples are video sequences with a total of 245 and 55 frames respectively, and 16 frames are sampled.

[0050] S1. Obtain the total number of frames in the target video (denoted as N). total The formula uses frames (represented by N) and the required number of keyframes (denoted as N). req The formula uses sam to represent N. First, we analyze N. total =55, N req =16 Examples.

[0051] S2, such as Figure 1 The diagram shows the flowchart illustrating the principle of the sparse-dense centripetal keyframe sampling method. By comparing N... total and N reqConfirm the sampling mode is executed, here N total >N req , the process of the "yes" branch is carried out for sampling. The detailed process is introduced below.

[0052] S2.1, first, the target video is divided into sampling rate calculation, by first calculating by the commonly used coefficient α, and α = 0.3.

[0053] S2.2, the calculation of the equal division sampling rate epart = frames / (α*sam) is carried out by determining α (here epart = 5), epart is the first batch of sampling interval, that is, all frames are divided into five parts, each part contains 11 frames. The first equal point sampling is carried out by the equal division sampling rate, that is, as Figure 2 The key frames with labels 1, 2, 3, 4, 5 and 6 in frames:55 are sampled as the first batch of key points.

[0054] S2.2, the obtained first batch of key frames (labels [1, 2, 3, 4, 5, 6]) are sorted in a centripetal manner, and the reordering is carried out according to the rule of abs(i-centor_pos), wherein i is the sampling key frame label obtained in the first batch of sampling process, and centor_pos is the label of the key frame located at the center position in the sampling key frame label. The key frame label sequence obtained is [3, 4, 2, 5, 1, 6] (here "label" is Figure Two The lower right corner label in frames:55 does not represent the sequence number of the key frame in the entire video, and [1, 2, 3, 4, 5, 6] corresponds to the sequence number [1, 11, 22, 33, 44, 55] in the real video).

[0055] S2.3, the second batch of key frames is obtained. The second batch of key frames is sampled by the sorted first batch of key frame labels, and the midpoint of the key frames corresponding to the adjacent two labels is taken to carry out the second sampling, and the rule is x = (i + (i + 1)) / 2 in [1, 11, 22, 33, 44, 55], and the midpoint is determined by taking the upper integer, and the label is obtained Figure 2 [7, 8, 9, 10, 11] in frames:55, which corresponds to the key frame sequence number [6, 17, 28, 39, 50] in the target video. Thus, the second batch of key frames is obtained by obtaining the midpoint of the adjacent related key frames.

[0056] S2.4, the third batch of key frames is obtained. The second batch of key frame labels is sorted in a centripetal manner to obtain the sorted label sequence [9, 10, 8, 11, 7], which corresponds to the key frame sequence number [28, 17, 39, 6, 50]. The new sequence is sorted from the center to the left and right sides, that is, for the sequence number i, the adjacent sampling of {i-1, i+1} is carried out, and the third batch of key frames is obtained by adjacent sampling, asFigure 2 The keyframes numbered [12, 13, 14, 15, 16] correspond to keyframes [27, 29, 16, 18, 38] in the target video. This yields the third batch of keyframes.

[0057] After completing S2, all 16 keyframes are obtained, labeled as follows: Figure 2 The numbers 1-16 correspond to the sequence numbers in the target video as [1, 11, 22, 33, 44, 55, 28, 39, 17, 50, 6, 27, 29, 16, 18, 38], which are sorted as [1, 6, 11, 16, 17, 18, 22, 27, 28, 29, 33, 38, 39, 44, 50, 55].

[0058] The above method can be used to further deduce when N total =55, N req In the embodiment with 16, the sampling keyframe numbers are [1, 25, 49, 73, 74, 75, 98, 122, 123, 124, 147, 171, 172, 169, 221, 245].

[0059] The above steps can be represented by the following formula:

[0060]

[0061] 2. Taking the sparse-dense centripetal keyframe sampling method by inputting the example video "HeadMassage" as an example, when the video sequence with a total of 55 frames in the example is sampled for 100 frames.

[0062] S1. Obtain the total number of frames in the target video (denoted as N). total The formula uses frames (represented by N) and the required number of keyframes (denoted as N). req The formula uses sam to represent N. First, we analyze N. total =55, N req Example of 100.

[0063] S2, such as Figure 1 The diagram shows the flowchart illustrating the principle of the sparse-dense centripetal keyframe sampling method. By comparing N... total and N req Confirm the sampling method to be executed, N here total <N req The sampling process for the "no" branch is performed. The detailed process is described below.

[0064] S2.1 First, obtain the number of keyframes in the target video (denoted as N). req ) and the total number of frames in the video (denoted as N) total And count the sequence number of all video frames.

[0065] S2.2, for all the video key frames obtained in S2.1, which are the first batch of key frames to be processed, first reordering these key frames in the centripetal ordering mode according to the sequence number to obtain the first batch of key frames.

[0066] S3, the first batch of key frames is copied to the left and right sides by the video center sequence number to obtain adjacent frames, and the process is repeated until the number of collected frames is sufficient. The process can be represented as:

[0067]

[0068] As shown in Figure 3 , 4 In some embodiments, the key frame density distribution diagram of 16 frame sampling when the total frame number is 55 and 245 is shown, which can reflect the key frame sampling characteristics of the sparse and dense centripetal key frame sampling method, and better focus attention on the important area, i.e. the center area.

[0069] The two different branch sampling methods can be used to analyze and process different frame number videos, and the sparse and dense centripetal key frame sampling method is used to obtain key frames that can effectively focus on the center area, or in the form of "cluster" in the target video to obtain evenly distributed dense "cluster", and efficiently realize the key frame sampling of different frame number target videos.

[0070] Finally, it should be noted that: the above only describes the preferred embodiments of the present application, and is not used to limit the present application, although the present application has been described in detail with reference to the foregoing embodiments, for those skilled in the art, the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A sparse-dense centripetal keyframe sampling method for video action recognition, characterized in that: The specific process of the method is as follows, S1, first, the total number of target video frames and the number of key frames required by user pre-setting need to be determined; this stage is the basis of the entire sampling method, by mastering the basic information of the video, necessary data support can be provided for the subsequent key frame selection; then, check whether the number of key frames required to be collected exceeds the actual total number of frames of the video, to ensure the feasibility and rationality of subsequent processing; S2, according to the information obtained in S1 stage, that is, the relationship between the number of key frames required to be collected and the total number of target video frames, select branch to execute the extraction work of key frame; here there are two different paths, each path is optimized for a specific situation, aiming to effectively extract the most representative key frame whether the video frame number is less or more; S3, when the number of key frames required to be collected is less than or equal to the total number of video frames, the system will adopt a more intensive sampling strategy to ensure that important details in the video can be captured, while maintaining a high time resolution, which is crucial for fine identification of action; S4, if the number of key frames required to be collected exceeds the total number of video frames, it is an abnormal situation, the algorithm will also have corresponding processing mechanism to ensure that each video frame is regarded as a key frame, or to reselect key frame according to the actual situation to adapt to this special scene; S2.1, determining the first batch sampling ratio a input information: the number of key frames required to be collected and the total number of frames of the video; determining the first batch sampling ratio: first, determine the first batch sampling ratio a; this first batch sampling ratio determines the frequency of selecting each frame as a key frame from the original video, the larger the first batch sampling ratio value, the more evenly distributed the first batch key frame position in the entire video, which is the key to determining the first batch key frame position; S2.11, get the first batch sampling interval determine the first batch sampling interval: based on the determined first batch sampling ratio a, the first batch sampling interval epart = frames / (*a*sam) can be determined; after calculation, if epart is an integer, epart is directly used as the first batch sampling interval; if epart is not an integer, the final first batch sampling interval can be determined by rounding up; this is done to ensure uniformity of sampling, while also considering the feasibility of actual operation; S2.12, first batch key frame sampling equal interval sampling: using the determined epart to divide the total number of frames of the target video into epart parts, starting from the first frame of the video, every epart frame is selected as a key frame until the number of collected key frames reaches the required number; this can ensure the uniform distribution of key frames on the video time axis, which helps to capture important moments of action changes; sampling principle: the principle followed in the sampling process is to ensure that key frames can cover the entire video time range as evenly as possible, avoiding the deviation of action recognition caused by too concentrated sampling in a certain part of the video; in addition, the characteristics of the video content, the complexity and rate of change of the action should also be considered, and the sampling strategy should be adjusted appropriately to adapt to different types of videos; specific work: S2.13, rearrange the order of the first batch of key frames, rearrange the order of the first batch of key frames by the sequence number of the first batch of key frames, that is, reorder the sequence number values in the new sequence from the center to the left and right sides, and reorder according to the sorting method abs(i-centor_pos), wherein abs represents the absolute value, i is the sequence number position of the i-th key frame in the target video, and centor_pos is the sequence number of the key frame in the center region of the target video; S2.14, take the midpoint between the sequence numbers of the first batch of key frames reordered in the sequence number, to get a new sequence number, the new key frame sequence number obtained by the midpoint sampling of these new key frame sequence numbers represents the second batch of key frame sequence, and the second batch of key frame sequence is reordered according to the sorting method in S2.13; S2.15, to the S2.14 in the acquisition of the second batch of key frames, the composition of the new sequence, the new sequence is by the center to the left and right side of the adjacent sampling, namely the sequence number i in the sequence, the {i-1, i+1} adjacent sampling, the third batch of key frames is obtained by adjacent sampling;S2.16, according to the key frame acquisition method of S2.14, S2.15, the key frame is extracted alternately, and it is needed to judge whether the number of key frames required to be collected in S1 is met in the extraction process, if the number is sufficient, the key frame sampling process is exited, and the key frame is output; if the number of key frames collected is not enough, the key frame sampling process needs to be continued; The first batch, the second batch, the n batch of key frames are finally merged to obtain the final collected key frames;S2.17, the above S2.11-S2.16 is the key frame extraction step when the number of key frames required to be collected is less than the total number of video frames; The process can be represented as: wherein keyframes is the key frame obtained by sampling, i is the key frame label collected, M is the total number of target video frames, fn is the key frame and its label collected, epart is the first batch sampling interval, and c is the key frame sequence number in the center region of the target video. In S2, S2.2, when the number of key frames required to be collected is greater than or equal to the total number of video frames, that is, the total number of frames is not enough or just enough, the second rule key frame sampling method needs to be performed, S2.21, first, the number of key frames of the target video and the total number of video frames are obtained; all frames of the target video are counted to form a sequence;S2.22, all frames of the target video obtained in S2.21 are the first batch of key frames to be processed, which are first reordered by the centripetal ordering mode according to the sequence number to obtain the first batch of key frames;S2.23, the first batch of key frames is copied to the left and right sides of the video center sequence number, and the process is repeated until the number of collected frames is sufficient, and the process can be represented as: wherein keyframes is the key frame obtained by sampling, pretotal is the sequence formed by the total frames of the target video, i is the key frame label collected, M is the total number of target video frames, epart is the first batch sampling interval, and c is the key frame sequence number in the center region of the target video.

Images