VIDEO PROCESSING APPARATUS, VIDEO PROCESSING METHOD AND MACHINE-READABLE STOR

Abstract

To provide a video processing apparatus, a video processing method, and a machine-readable storage medium.SOLUTION: A video processing apparatus includes: an acquisition unit for acquiring an initial marker frame on the basis of an action video clip; a setting nit for setting a sliding window on the basis of information on the action video clip; a matching unit which executes matching on similarity between the initial marker frame with each frame in the sliding window of the action video clip; and a selection unit which selects one frame in the sliding window of the action video clip, as a marker frame, on the basis of a result of the similarity matching.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to the technical field of video processing, and in particular to a video processing apparatus, a video processing method and a machine-readable storage medium for action counting. [Background technology]

[0002] Periodic or quasi-periodic actions are widely present in video understanding tasks. In recent years, the demand for detection and statistics of such actions has gradually increased in fields such as video surveillance and virtual reality. In the prior art, deep learning-based methods can relatively accurately position the start and end of different actions in a time series. However, when it is necessary to position the start and end of an action instance at one time (one of multiple times) in a video segment (a video clip) of the same action that overlaps (repeates) multiple times, such an action is called a periodic action, and such a video clip is called a periodic action video clip, the positioning is often more difficult. Therefore, how to accurately capture an action instance at one time in a periodic action video clip has become an important research topic in this field. Summary of the Invention [Problem to be solved by the invention]

[0003] SUMMARY OF THE PRESENTLY PREFERRED EMBODIMENTS It is an object of the present invention to provide a video processing apparatus, a video processing method and a machine-readable storage medium that can accurately capture an action instance at a certain time in a cyclic action video clip. [Means for solving the problem]

[0004] According to one aspect of the present invention there is provided a video processing apparatus comprising: an obtaining unit for obtaining an initial marker frame based on an action video clip; a setting unit for setting a sliding window based on information about the action video clip; a matching unit for performing similarity matching between the initial marker frame and each frame in the sliding window of the action video clip; and A selection unit is included for selecting, based on a result of similarity matching, one frame in the sliding window of the action video clip as a marker frame.

[0005] According to another aspect of the present invention there is provided a video processing method comprising the steps of: Get the initial marker frame based on the action video clip; setting a sliding window based on information about the action video clip; performing similarity matching between the initial marker frame and each frame in the sliding window of the action video clip; and Selecting a frame in the sliding window of the action video clip as a marker frame based on a result of the similarity matching.

[0006] According to another aspect of the present invention, there is provided a machine-readable storage medium carrying a program product including machine-readable instruction codes which, when read and executed by a computer, cause the computer to perform a video processing method according to the present invention.

[0007] By using the video processing device, the video processing method and the machine-readable storage medium according to the present invention, an action instance can be accurately captured at a certain time in an action video clip by searching for a marker frame that satisfies a condition in the action video clip. [Brief description of the drawings]

[0008] [Figure 1] 1 is a block diagram showing a configuration of a video processing device according to an embodiment of the present invention. [Diagram 2] FIG. 13 is a block diagram showing the configuration of a video processing device according to another embodiment of the present invention. [Diagram 3] 2 is a block diagram showing a configuration of an acquisition unit in a video processing device according to an embodiment of the present invention; [Figure 4] FIG. 13 is a block diagram showing a configuration of a video processing device according to another embodiment of the present invention. [Diagram 5] FIG. 11 is a block diagram showing a detailed configuration of a part of a video processing device according to another embodiment of the present invention. [Figure 6] 2 is a flowchart of a video processing method in an embodiment of the present invention. [Figure 7] 4 is a flowchart of a video processing method in another embodiment of the present invention. [Figure 8] FIG. 1 is a block diagram illustrating an exemplary configuration of a general purpose computer on which video processing apparatus and methods according to embodiments of the present invention may be implemented. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. Note that such embodiments are merely examples and do not limit the present invention.

[0010] The present invention provides a technique for counting the number of times of periodic actions based on similarity matching. It is assumed that in an action video clip containing only one known action that is repeated multiple times, there always exists a marker frame, i.e., it appears only once during one execution of the action. Based on this assumption, the problem of counting the number of times of overlapping actions in an action video clip can be transformed into a problem of searching for a marker frame that satisfies a condition. It has been found through observation that when most people observe overlapping actions in a video, they always consciously select a frame that satisfies some characteristics as the start of one action instance, and when this frame appears again, it is the start of the next action instance, and thus counting is performed. Therefore, the above assumption is reasonable. Hereinafter, with reference to FIG. 1, a description will be given of how a video processing device in an embodiment of the present invention accurately captures an action instance in a periodic action video clip.

[0011] 1 is a diagram showing a configuration of a video processing device 100 in an embodiment of the present invention. As shown in FIG. 1, the video processing device 100 in the embodiment of the present invention may include an acquisition unit 110, a setting unit 120, a matching unit 130 and a selection unit 140.

[0012] The acquisition unit 110 can obtain an initial marker frame Mark0 based on an action video clip. In the present invention, an action video clip may include the same kind of action that is repeated multiple times. Usually, the type of the action is known. Also, one action may include multiple frames. Specifically, the acquisition unit 110 may select the first frame in the action video clip as the initial marker frame, or may obtain the initial marker frame based on the action video clip in other ways, which will be described later with reference to FIG. 3.

[0013] The setting unit 120 can also set the sliding window based on information about the action video clip, where the information about the action video clip indicates the information required to set the sliding window.

[0014] In addition, the matching unit 130 can perform similarity matching between the initial marker frame and each frame in the sliding window of the action video clip. The image similarity measure of two frames may be a commonly used measure such as frame difference, correlation coefficient, etc.

[0015] It should be noted that in an action video clip, if the background itself is changing or the background is moving due to the camera position not being fixed, it is necessary to adopt techniques such as ROI detection, image segmentation, etc. to lock the region for calculating the similarity to the vicinity of the moving target, instead of using the entire image. In such a case, the video processing device 100 in the embodiment of the present invention may further include a region of interest processing unit (not shown), which can perform ROI (region of interest) detection or segmentation for the initial marker frame and each frame in the sliding window of the action video clip.

[0016] In addition, the selection unit 140 may select a frame in the sliding window of the action video clip as a marker frame based on the result of similarity matching. For example, the selection unit 140 may search for a frame in the sliding window that is most similar to Mark0.

[0017] The processes performed by the setting unit 120, the matching unit 130 and the selection unit 140 will now be described based on an exemplary embodiment.

[0018] First, to set the sliding window, two parameters need to be determined, namely, the start position of the sliding window and the length of the sliding window. A common way is to set the start position at one frame after Mark0 and make the length as long as possible to ensure that the marker frame close to Mark0 appears in the sliding window. However, such an approach, similar to exhaustive, may lead to excessive time consumption and introduce more noise. To avoid such problems, the setting unit 120 in the embodiment of the present invention can set the start position and length of the sliding window based on the speed of the action contained in the action video clip, thereby making the length of the sliding window as short as possible and ensuring that the marker frame is in the sliding window.

[0019] Specifically, assume that the lower and upper limits of the number of frames required for one execution of an action in the current action video clip are Speed_inf and Speed_sup, respectively. This means that, in theory, the number of frames between two marker frames is equal to or greater than Speed_inf and equal to or less than Speed_sup. In this case, it is obvious that when searching for the next marker frame closest to Mark0, the setting unit 120 may set the search interval, i.e., the sliding window, as an interval from the Speed_inf-th frame to the Speed_sup-th frame behind Mark0. In addition, in order to reduce the influence of errors, a bias term b may be added, for example, to the initialization of Mark0. Finally, the setting unit 120 may set the sliding window as an interval from the Speed_inf-b-th frame to the Speed_sup+b-th frame behind Mark0.

[0020] Then, the matching unit 130 can calculate the similarities between Mark0 and all frames in the sliding window [Speed_inf-b, Speed_sup+b] to obtain one similarity sequence.

[0021] Ideally, only one peak value appears in the similarity sequence, which is the target marker frame to be searched. However, in many cases, once Speed_sup becomes more than a multiple of Speed_inf, multiple peak values may appear in the sequence. To find the most suitable one peak value, the selection unit 140 can find the peak value that best meets the conditions in the similarity sequence according to the following idea, that is, the occurrence position of the similarity peak value is at the positions of {loc_peak1, loc_peak2, …, loc_peakn} behind Mark0, among which, Speed_inf - b < loc_peak1 < … < loc_peakn < Speed_sup + b. Also, when the middle position of the sliding window is denoted as Mid, obviously, Mid = (Speed_inf + Speed_sup) / 2. Assuming that the position where the maximum value of the peak values on the left side of Mid is loc_peaki and the position where the maximum value of the peak values on the right side of Mid is loc_peakj, and loc_peakj - loc_peaki = loc_peaki - 0 + δ, where δ is a relatively small integer, this equation means that loc_peaki and loc_peakj may both be marker frames. Otherwise, it is considered that there is no periodicity within [Speed_inf - b, loc_peakj]. In this case, loc_peakj is the next marker frame behind Mark0. In the former case, since the search target is the marker frame closest to Mark0, the search interval behind loc_peaki is excluded, but there is still periodicity within the interval of [Speed_inf - b, loc_peaki]. At this time, calculate Mid’ = (loc_peaki - (Speed_inf - b)) / 2 until only one peak value remains within the interval of [Speed_inf - b, loc_peaki]. Therefore, by such a peak value selection method similar to iteration, the selection unit 140 can finally determine the position where the marker frame Mark1 behind Mark0 is located.

[0022] It should be noted that in matching and searching for marker frames, the video processing device 100 in the embodiment of the present invention does not train one deep learning classifier to determine whether two frames are marker frames or similar, but performs matching for marker frames by calculating the similarity between each two images. There are two main reasons for this: first, even for the same type of action, different people have different habits, so it is difficult to set up one unified standard to define marker frames, and it goes without saying that it takes a lot of time to collect enough marker frame samples for various types of actions. Second, the appearance difference between marker frames and non-marker frames in the same action segment is usually very subtle, and although the algorithm framework based on convolutional networks is good at solving classification problems at the semantic level, it is not sensitive enough to the positional relationship in the image. Therefore, it is more effective and practical to determine and search for marker frames using a method based on marker frames that do not need to be predefined and pixel-level features.

[0023] Thus, the video processing device 100 according to the embodiment of the present invention can accurately capture an action instance at a certain time in an action video clip by searching for a marker frame that satisfies a condition in the action video clip.

[0024] Preferably, as shown in Fig. 2, the video processing device 200 in another embodiment of the present invention may further include an input video processing unit 250. The acquisition unit 210, the setting unit 220, the matching unit 230 and the selection unit 240 shown in Fig. 2 correspond to the acquisition unit 110, the setting unit 120, the matching unit 130 and the selection unit 140 shown in Fig. 1, respectively.

[0025] The input video processing unit 250 can obtain action video clips and information about the action video clips by performing deep learning processing on the input video using a deep neural network DNN. For example, for any given length of input video, the input video processing unit 250 can first perform positioning and segmentation for different actions to obtain multiple different action video clips, so that each action video clip has the same kind of actions. This positioning and segmentation can usually be realized by some existing deep learning networks in this field. In addition, the input video processing unit 250 can further output other information about the action video clips, for example, a deep learning classification layer can output the confidence that the current frame belongs to a certain kind of action, etc.

[0026] In addition, the input video processing unit 250 can provide the action video clips and various information related to the action video clips to the acquisition unit 210, the setting unit 220 and the matching unit 230.

[0027] The video processing device 200 in the embodiment of the present invention is based on the time series positioning result based on deep learning. By using deep learning technology, the type of action is known, and considering that the same type of action usually has some commonalities, the present invention can utilize more useful prior knowledge by using these commonalities to determine or search for marker frames.

[0028] In order to better understand the technical solution of the present invention, the video processing device according to the present invention will be described in further detail below.

[0029] According to a preferred embodiment of the present invention, Fig. 3 shows a configuration of an acquisition unit in a video processing device in an embodiment of the present invention. The acquisition unit 300 shown in Fig. 3 corresponds to the acquisition unit 110 shown in Fig. 1. As shown in Fig. 3, the acquisition unit 300 includes: an action video clip selection unit 310, a confidence acquisition unit 320, a speed score calculation unit 330 and an initial marker frame determination unit 340.

[0030] The acquisition unit 300 can determine the initial marker frame based on at least one of the marker confidence and the speed score of each frame in the action video clip. Hereinafter, the processes performed by the action video clip selection unit 310, the confidence acquisition unit 320, the speed score calculation unit 330 and the initial marker frame determination unit 240 will be described in detail.

[0031] Through observation, we found that a frame selected as a marker frame usually has at least the following two characteristics: characteristic 1: when in the marker frame state, the rate (speed) of the action is relatively slow; characteristic 2: the marker frame has a more prominent appearance characteristic compared to adjacent frames. These two characteristics can give the strongest response to human vision within a certain period of time, and the number of overlapping actions can be counted every time the marker frame appears.

[0032] Based on these two features, the acquisition unit 300 performs the marker frame initialization method as follows.

[0033] First, the action video clip selection unit 310 may select the previous m frames of the action video clip, i.e., {a0, a1, ..., am} (m is usually not very large to avoid periodicity appearing therein), and then the action video clip selection unit 310 may provide the previous m frames of the selected action video clip to the speed score calculation unit 330 and the confidence acquisition unit 320.

[0034] In addition, the speed score calculation unit 330 can calculate the speed score for the previous m frames of the obtained action video clip. Specifically, the speed score calculation unit 330 can calculate the speed score through motion analysis based on optical flow. The speed score calculation unit 330 uses an optical flow extraction algorithm to calculate the displacement (vx, vy) of each pixel in each frame with respect to the pixels in the previous frame. Among them, vx and vy represent the displacements in the horizontal and vertical directions respectively. For a certain frame ai, the speed score calculation unit 330 calculates the norm of the displacement of each pixel (the norm of the vector), retains only the speeds greater than a predetermined threshold, thereby obtaining a motion block with obvious motion in the image, and by obtaining the average value of the speeds within the motion block, the average speeds {V0, …, Vm} of the respective motion blocks of a0, …, am can be obtained, which is the quantization result of Feature 1.

[0035] Regarding Feature 2, the reliability acquisition unit 320 can obtain the marker reliability of each of the previous m frames of the action video clip. For example, the reliability acquisition unit 320 can obtain from the input video processing unit 250 shown in FIG. 2 the reliability that each of the m frames belongs to a certain type of action. Usually, for an image with prominent appearance features, there is a relatively strong response and the reliability is higher. Therefore, the output result of deep learning can be selected as the quantization of the appearance features, represented as {conf0, …, confm}, where 0 < confi <= 1, indicating the reliability that a certain frame is predicted to belong to the action type to which the current action video clip belongs.

[0036] In addition, the initial marker frame determination unit 340 can determine the most suitable frame as the initial marker frame by designing a certain rule based on these two features. The rule here may be a rule that sets a threshold value or that takes a weighted average after normalization. In addition, in the case of an action with a uniform moving speed (e.g., a uniform circular movement), only the appearance feature needs to be considered.

[0037] Thus, the video processing device in the embodiment of the present invention including the acquisition unit 300 can initialize the marker frame based on the motion analysis and the marker confidence based on the optical flow. The most appropriate initial marker frame can be efficiently obtained by fusing the motion analysis result with the prediction result of deep learning. This can help to more accurately capture an action instance at a certain time in an action video clip, and thus improve the accuracy of counting periodic actions.

[0038] According to a preferred embodiment of the present invention, FIG. 4 shows a configuration of a video processing device 400 in another embodiment of the present invention, and FIG. 5 shows a detailed configuration 500 of a part of the video processing device shown in FIG.

[0039] The video processing device 400 shown in Fig. 4 may further include a determining unit 450 and a calculating unit 460. The obtaining unit 410, the setting unit 420, the matching unit 430 and the selecting unit 440 shown in Fig. 4 correspond to the obtaining unit 110, the setting unit 120, the matching unit 130 and the selecting unit 140 shown in Fig. 1, respectively. In addition, the video processing device 400 may further include an input video processing unit, such as the input video processing unit 250 shown in Fig. 2.

[0040] Hereinafter, the processes performed by the determining unit 450 and the calculating unit 460 will be described in detail with reference to FIG. 4, and the processes performed by the setting unit 420, the matching unit 430 and the selecting unit 440 shown in FIG. 4 will be described in detail with reference to FIG. 5.

[0041] The acquisition unit 410 in FIG. 4 can perform the same processing as the acquisition unit 110 shown in FIG. 1 and the acquisition unit 300 shown in FIG. 3 to obtain the initial marker frame Mark0, but a detailed description thereof will be omitted here.

[0042] As shown in Fig. 5, the video processing device 400 shown in Fig. 4 includes a plurality of setting units 520-1 to 520-n, a plurality of matching units 530-1 to 530-n, and a plurality of selection units 540-1 to 540-n, so as to find all the marker frames included in an action video clip. The plurality of setting units 520-1 to 520-n correspond to the setting unit 420 in Fig. 4, the plurality of matching units 530-1 to 530-n correspond to the matching unit 430 in Fig. 4, and the plurality of selection units 540-1 to 540-n correspond to the selection unit 440 in Fig. 4.

[0043] Specifically, the setting unit 520-1, the matching unit 530-1 and the selection unit 540-1 can find the marker frame Mark1 by performing processing similar to that performed by the setting unit 120, the matching unit 130 and the selection unit 140 in the video processing device 100 in Figure 1, but detailed description thereof will be omitted here.

[0044] Then, the setting unit 520-2 can reset the sliding window based on the selection for the marker frame Mark1. Specifically, the setting unit 520-2 can determine a new sliding window based on Mark0 and Mark1. Similarly, the starting position and length of the sliding window can be determined based on the number of frames required for one execution of the action. However, this sliding window is different from the first sliding window, and the number of frames between Mark0 and Mark1 (denoted as Speed1) is the number of frames required for one execution of the current action in the current action video clip. Similarly, if a bias term b' is added as one tolerance, the sliding window can be easily determined as the section from the Speed1-b'th frame to the Speed1+b'th frame after Mark1.

[0045] Subsequently, similar to matching unit 130, matching unit 530-2 can perform similarity matching between marker frame Mark1 and each frame in the reset sliding window of the action video clip.

[0046] Next, similar to the selection unit 140, the selection unit 540-2 can obtain the next marker frame Mark2 by selecting one frame in the reset sliding window of the action video clip as a new marker frame based on the result of similarity matching.

[0047] For the new marker frames found by the above-mentioned selection units, the setting units 520-3 to 520-n, the matching units 530-3 to 530-n, and the selection units 540-3 to 540-n can perform processing similar to that performed by the setting unit 520-2, the matching unit 530-2, and the selection unit 540-2, respectively.

[0048] In addition, the judgment unit 450 in FIG. 4 can judge whether the sliding window has completely slid through the action video clip (i.e., has slid to the last frame of the action video clip), and if the sliding window has not completely slid through the action video clip (i.e., has not reached the last frame of the action video clip), the video processing device 400 repeats the above-mentioned processes, i.e., sliding window resetting, similarity matching, and marker frame selection, for a new marker frame.

[0049] The calculation unit 460 may also calculate the sum of the number of marker frames and the new marker frames as the number of overlaps of the action in the action video clip.

[0050] In this manner, the setting units 520-1 to 520-n, the matching units 530-1 to 530-n, and the selection units 540-1 to 540-n perform the above-mentioned process until the sliding window slides to the end position of the action video clip (i.e., the action video clip has been completely slid), and the number of marker frames searched at this time is the number of times the action is repeated in the action video clip.

[0051] In this way, the video processing device 400 in the embodiment of the present invention can count the number of periodic actions in the action video clip. In addition, in the overall process of searching for a marker frame, the latest marker frame obtained is always taken as the main basis for searching for the next marker frame. This is because in a real scenario, the subject of the action may shift over time, or the camera may move slightly and become out of focus. Referring to the latest marker frame can help track similar changes, leading to more accurate results.

[0052] Hereinafter, a video processing method according to an embodiment of the present invention will be described with reference to FIGS.

[0053] As shown in Figure 6, the video processing method in the embodiment of the present invention starts in step S110, in which an initial marker frame is obtained based on an action video clip.

[0054] Then, in step S120, a sliding window is set based on information about the action video clip.

[0055] Next, in step S130, similarity matching is performed between the initial marker frame and each frame in the sliding window of the action video clip.

[0056] Then, in step S140, one frame in the sliding window of the action video clip is selected as a marker frame based on the result of the similarity matching, after which the process ends.

[0057] According to an embodiment of the present invention, the method may further include performing a deep learning process on the input video using a deep neural network DNN to obtain action video clips and information related to the action video clips.

[0058] According to an embodiment of the present invention, obtaining an initial marker frame based on an action video clip includes selecting a first frame in the action video clip as the initial marker frame; or determining the initial marker frame based on at least one of a marker confidence and a speed score of each frame in the action video clip.

[0059] According to an embodiment of the present invention, the method calculates the speed score by optical flow based motion analysis.

[0060] According to an embodiment of the present invention, the step of performing similarity matching includes performing similarity matching based on pixels of the initial marker frame and pixels of each frame in the sliding window of the action video clip.

[0061] According to an embodiment of the present invention, prior to the step of performing similarity matching, the method further includes performing region of interest (ROI) detection or segmentation for the initial marker frame and for each frame in the sliding window of the action video clip.

[0062] Therefore, the video processing method according to the embodiment of the present invention can accurately capture an action instance at a certain time in an action video clip by searching for a marker frame that satisfies a condition in the action video clip.

[0063] 7 is a flowchart of a video processing method in another embodiment of the present invention. Steps S210 to S240 shown in FIG 7 are the same as steps S110 to S140 in FIG 6, and detailed description thereof will be omitted here.

[0064] Then, in step S250, the sliding window is again set based on the selection for the marker frame.

[0065] Next, in step S260, similarity matching is performed between the marker frame and each frame in the reset sliding window of the action video clip.

[0066] Then, in step S270, based on the result of the similarity matching, one frame in the reset sliding window of the action video clip is selected as a new marker frame.

[0067] Then, in step S280, it is determined whether the sliding window has completely slid through the action video clip.

[0068] If it is determined in step S280 that the sliding window has not completely slid through the action video clip, then return to step S250. For the new marker frame, repeatedly perform the steps of resetting the sliding window in step S250, the step of similarity matching in step S260, and the step of selecting the marker frame in step S270.

[0069] Also, if it is determined in step S280 that the sliding window has completely slid through the action video clip, proceed to step S290. In step S290, the sum of the number of marker frames and the number of new marker frames is calculated as the number of overlaps of the action in the action video clip. Then, the process ends.

[0070] According to an embodiment of the present invention, the step of performing similarity matching includes performing similarity matching based on pixels of the marker frame and pixels of each frame in the reset sliding window of the action video clip.

[0071] According to an embodiment of the present invention, prior to the step of performing similarity matching, the method further includes performing region of interest (ROI) detection or segmentation for the marker frame and for each frame in the re-established sliding window of the action video clip.

[0072] Thus, the video processing method in the embodiment of the present invention can count the number of periodic actions in an action video clip. In addition, in the overall process of searching for a marker frame, the latest marker frame obtained is always used as the main basis for searching for the next marker frame. This is because in a real scenario, the subject of the action may shift over time, or the camera may move slightly and become out of focus. Referring to the latest marker frame can help track similar changes, leading to more accurate results.

[0073] It should be noted that the various specific implementation manners of the above steps of the video processing method in the embodiment of the present invention have been described in detail above, so that the description will be omitted here.

[0074] Also, as will be apparent, each operation (step) of the video processing method according to the present invention can be realized by a computer-executable program stored in various machine-readable storage media.

[0075] Furthermore, the object of the present invention may be realized in the following manner: a storage medium storing the above-mentioned executable program code is provided directly or indirectly to a system or device, and a computer or central processing unit (CPU) in the system or device reads and executes the above-mentioned program code. In this case, as long as the system or device has a function of executing a program, the embodiment of the present invention is not limited to a program, and the program may be in any form, such as an object-oriented program, a program executed by an interpreter, a script program provided to an operating system, etc.

[0076] These machine-readable storage media may include, but are not limited to, various types of memory devices, storage units, semiconductor devices, magnetic units, such as optical, magnetic and magneto-optical disks, and other devices suitable for storing information.

[0077] In addition, when a computer can connect to a relevant site via the Internet, the present invention can also realize the technical solution of the present invention by having the computer download the program code and execute the program after installing it on the computer.

[0078] FIG. 8 is a diagram showing a hardware configuration (general-purpose computer) 1300 capable of implementing a video processing device and a video processing method according to an embodiment of the present invention.

[0079] 8, a central processing unit (CPU) 1301 can perform various processes based on a program stored in a ROM 1302 or a program loaded from a storage unit 1308 to a RAM 1303. The RAM 1303 can also store data required when the CPU 1301 performs various processes, depending on the needs. The CPU 1301, the ROM 1302, and the RAM 1303 are connected to each other via a bus 1304. An input / output interface 1305 is also connected to the bus 1304.

[0080] Further, the following components are connected to the input / output interface 1305: an input unit 1306 including a keyboard, an output unit 1307 including a display such as a liquid crystal display (LCD) and a speaker, a storage unit 1308 including a hard disk, and a communication unit 1309 including a network interface card such as a LAN card, a modem, etc. The communication unit 1309 performs communication processing via a network such as the Internet or a LAN.

[0081] The drive 1310 may be connected to the input / output interface 1305 as needed. A removable medium 1311, for example, a semiconductor memory, can be set in the drive 1310 as needed to install a computer program read from the removable medium 1311 into the memory 1308.

[0082] The present invention further provides a program product including a machine-readable instruction code, which, when read and executed by a machine, can execute the method in the above-mentioned embodiment of the present invention. Accordingly, various storage media, such as magnetic disks (including floppy disks (registered trademark)), optical disks (including CD-ROMs and DVDs), magneto-optical disks (including MDs (registered trademark)), and semiconductor storage devices, that carry such a program product are also included in the present invention.

[0083] The above-mentioned storage medium may include, for example, a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory device, and the like, but is not limited to these.

[0084] Additionally, each operation (process) in the above-described method may be implemented in the form of a computer-executable program stored in various machine-readable storage media.

[0085] Furthermore, with respect to the above-mentioned embodiments, the following supplementary notes are disclosed.

[0086] (Appendix 1) 1. A video processing device comprising: an acquisition unit for obtaining an initial marker frame based on an action video clip; a setting unit for setting a sliding window based on information about the action video clip; a matching unit for performing similarity matching between the initial marker frame and each frame in the sliding window of the action video clip; and The apparatus includes a selection unit for selecting, based on a result of similarity matching, one frame in the sliding window of the action video clip as a marker frame.

[0087] (Appendix 2) 2. A video processing device according to claim 1, comprising: further comprising an input video processing unit; The input video processing unit performs deep learning processing on the input video using a deep neural network DNN to obtain the action video clip and information about the action video clip.

[0088] (Appendix 3) 2. A video processing device according to claim 1, comprising: The apparatus, wherein the acquisition unit selects a first frame in the action video clip as the initial marker frame, or determines the initial marker frame based on at least one of a marker confidence and a speed score of each frame in the action video clip.

[0089] (Appendix 4) 2. A video processing device according to claim 1, comprising: The setting unit resets the sliding window based on the selection for the marker frame; and the matching unit performs similarity matching between the marker frame and each frame in the reset sliding window of the action video clip; The selection unit selects one frame in the reset sliding window of the action video clip as a new marker frame based on a result of similarity matching.

[0090] (Appendix 5) 5. A video processing device according to claim 4, comprising: Further comprising a judgment unit, The determining unit determines whether the sliding window has completely slid through the action video clip; When the sliding window has not completely slid through the action video clip, for a new marker frame, the video processing device repeats the process of the setting unit resetting the sliding window, the matching unit performing similarity matching, and the selection unit selecting a marker frame.

[0091] (Appendix 6) 6. A video processing device according to claim 5, comprising: further comprising a computing unit; The calculation unit calculates a sum of the number of the marker frames and the number of new marker frames as a number of repetitions of an action in the action video clip.

[0092] (Appendix 7) 4. A video processing device according to claim 3, comprising: The acquisition unit calculates the speed score by optical flow based motion analysis.

[0093] (Appendix 8) 2. A video processing device according to claim 1, comprising: further comprising a region of interest processing unit; The region of interest processing unit performs ROI detection or segmentation on the initial marker frame and each frame in the sliding window of the action video clip.

[0094] (Appendix 9) 1. A video processing method comprising the steps of: Get the initial marker frame based on the action video clip; setting a sliding window based on information about the action video clip; similarity matching the initial marker frame with each frame in the sliding window of the action video clip; and The method includes selecting a frame in the sliding window of the action video clip as a marker frame based on a result of similarity matching.

[0095] (Appendix 10) 10. The method according to claim 9, The method further includes performing a deep learning process on an input video using a deep neural network DNN to obtain the action video clip and information about the action video clip.

[0096] (Appendix 13) 10. The method according to claim 9, Obtaining the initial marker frame based on the action video clip includes: selecting a first frame in the action video clip as the initial marker frame; or The method further includes determining the initial marker frame based on at least one of a marker confidence and a speed score of each frame in the action video clip.

[0097] (Appendix 12) 10. The method according to claim 9, resetting the sliding window based on the selection for the marker frame; Similarity matching the marker frame with each frame in the reconfigured sliding window of the action video clip; and The method further includes selecting a frame in the reset sliding window of the action video clip as a new marker frame based on a result of the similarity matching.

[0098] (Appendix 13) 13. The method of claim 12, The method further comprising repeating the steps of resetting the sliding window, matching similarity and selecting a marker frame for each new marker frame until the sliding window slides completely through the action video clip.

[0099] (Appendix 14) 14. The method according to claim 13, comprising: The method further includes calculating a sum of the number of the marker frames and the number of new marker frames as an action overlap count in the action video clip.

[0100] (Appendix 15) 14. The method according to claim 13, comprising: The method further comprising: calculating the speed score by optical flow based motion analysis.

[0101] (Appendix 16) 10. The method according to claim 9, The step of performing similarity matching includes: The method includes performing similarity matching based on pixels of the initial marker frame and pixels of each frame in the sliding window of the action video clip.

[0102] (Appendix 17) 10. The method according to claim 9, Prior to the step of performing similarity matching, the method further comprises: The method further includes performing region of interest (ROI) detection or segmentation on the initial marker frame and each frame in the sliding window of the action video clip.

[0103] (Appendix 18) 13. The method of claim 12, The step of performing similarity matching includes: The method includes performing similarity matching based on pixels of the marker frame and pixels of each frame in a reset sliding window of the action video clip.

[0104] (Appendix 19) 13. The method of claim 12, Prior to the step of performing similarity matching, the method further comprises: The method further includes performing region of interest (ROI) detection or segmentation for the marker frame and each frame in the reset sliding window of the action video clip.

[0105] (Appendix 20) 1. A machine-readable storage medium, comprising: A program product having machine readable instruction code stored therein, A storage medium, the instruction code being capable of causing the computer to execute the video processing method of any one of claims 9 to 19 when read and executed by a computer.

[0106] Although the preferred embodiment of the present invention has been described above, the present invention is not limited to this embodiment, and any modification to the present invention falls within the technical scope of the present invention as long as it does not depart from the spirit of the present invention.

Claims

1. 1. A video processing device comprising: an obtaining unit for obtaining an initial marker frame based on an action video clip including a periodic action; a setting unit for setting a predetermined section of the action video clip based on information about the action video clip; a matching unit for performing similarity matching between the initial marker frame and each frame in a predetermined section of the action video clip; and a selection unit for selecting one frame in a predetermined section of the action video clip as a marker frame based on a result of the similarity matching, the marker frame being a frame that appears only once during one execution of an action; The setting unit resets a predetermined section of the action video clip based on a selection for the marker frame; the matching unit performs similarity matching between the marker frame and each frame in a predetermined section of the reset action video clip; The selection unit selects a frame in a predetermined section of the reset action video clip as a new marker frame based on a result of the similarity matching; The video processing device includes: The video processing device further comprises a calculation unit for selecting a sum of the number of the marker frames and the new marker frames as a number of repetitions of an action in the action video clip.

2. 2. A video processing device according to claim 1, The video processing device further includes an input video processing unit that performs deep learning processing on an input video using a deep neural network to obtain the action video clip and information related to the action video clip.

3. 2. A video processing device according to claim 1, The video processing device, wherein the acquisition unit selects a first frame in the action video clip as the initial marker frame, or determines the initial marker frame based on at least one of a marker confidence and a speed score of each frame in the action video clip.

4. 2. A video processing device according to claim 1, a determining unit for determining whether the predetermined section of the action video clip is at the end of the action video clip; if the predetermined section of the action video clip is not at the end of the action video clip, The video processor, for the new marker frame, A process in which the setting unit resets a predetermined section of the action video clip; The matching unit performs similarity matching; and The selection unit selects a marker frame. A video processing device that repeats the above steps.

5. 4. A video processing device according to claim 3, A video processing device, wherein the acquisition unit calculates the speed score by optical flow based motion analysis.

6. 2. A video processing device according to claim 1, The video processing apparatus further comprises a region of interest processing unit configured to perform region of interest detection or image segmentation on the initial marker frame and on each frame in a predetermined section of the action video clip.

7. A computer implemented video processing method comprising: Obtaining an initial marker frame based on an action video clip including a periodic action; setting a predetermined section of the action video clip based on information about the action video clip; performing similarity matching between the initial marker frame and each frame in a predetermined section of the action video clip; and selecting a frame in a predetermined section of the action video clip as a marker frame based on a result of the similarity matching, the marker frame being a frame that appears only once during one execution of an action; redefining a predetermined section of the action video clip based on a selection made for the marker frame; performing similarity matching between the marker frame and each frame in a predetermined section of the reset action video clip; Selecting a frame in a predetermined section of the reset action video clip as a new marker frame based on a result of the similarity matching; The video processing method includes: The video processing method further comprising: selecting the sum of the number of the marker frames and the new marker frames as a number of repetitions of an action in the action video clip.

8. A program for causing a computer to execute the video processing method according to claim 7.

Images