Video processing device and program

Abstract

A video processing device according to an aspect of the present invention comprises an acquisition unit, a calculation unit, and a generation unit. The acquisition unit acquires an original video including a repetitive action in which a posture change is repeated. The calculation unit calculates a sampling interval for sampling the original video to a predetermined length. When the length of a second section other than a first section representing the repetitive action is less than the sampling interval, the generation unit generates an extended section longer than the sampling interval by alternately connecting the second section and an inverted section obtained by inverting the second section in a time direction.

Description

Image processing device and program

[0001] The present invention relates to a video processing device and a program.

[0002] For example, a repetitive action counting (RAC) method (see, for example, Non-Patent Document 1) is known that predicts the number of cycles of a repetitive action from a video showing a repetitive action with repeated posture changes, such as a pull-up. In this type of RAC method, the original video to be predicted is generally sampled to have a fixed number of frames (for example, 64 frames), and then the number of cycles is predicted. Furthermore, a machine learning model (hereinafter simply referred to as a model) is used to predict the number of cycles.

[0003] Hu et al. , “TransRAC: Encoding Multi-scale Temporal Correlation with Transformers for Repetitive Action Counting”. In CVPR, 2022.

[0004] In the RAC method described above, the model may be undertrained for sections (interval sections) in the original video where repetitive movements are not occurring. Causes of undertraining include a lack of data for interval sections due to the length of the interval sections in the original video being relatively short compared to the length of the sections where repetitive movements are occurring (repetitive movement sections), and missing data where interval sections shorter than the sampling interval are not reflected in the post-sampling data. A model that is undertrained for interval sections may erroneously recognize interval sections as repetitive movement sections during prediction, resulting in reduced performance in predicting the number of cycles (counting performance).

[0005] Therefore, an object of the present invention is to provide a technique that contributes to improving the performance of a model that predicts the number of cycles of a repetitive operation.

[0006] According to one aspect of the present invention, there is provided a video processing device including an acquisition unit, a calculation unit, and a generation unit. The acquisition unit acquires an original video including a repetitive motion with repeated posture changes. The calculation unit calculates a sampling interval for sampling the original video to a predetermined length. When the length of a second interval other than a first interval representing the repetitive motion is less than the sampling interval, the generation unit generates an extended interval longer than the sampling interval by alternately combining the second interval and an inverted interval obtained by inverting the second interval in the time direction.

[0007] According to the present invention, a technique is provided that contributes to improving the performance of a model that predicts the number of cycles of a repetitive operation.

[0008] FIG. 1 is a block diagram showing an example of the hardware configuration of a video processing device according to an embodiment. FIG. 2 is a block diagram showing an example of the functional configuration of the video processing device of FIG. 1. FIG. 3 is a flowchart showing an example of extended video generation processing in the video processing device according to an embodiment. FIG. 4 is a diagram for explaining a specific example of an original video in the embodiment. FIG. 5 is a diagram for explaining a specific example of an interval section in the embodiment. FIG. 6 is a diagram for explaining a specific example of an inversion section in the embodiment. FIG. 7 is a diagram for explaining a specific example of an extended interval section in the embodiment. FIG. 8 is a diagram for explaining a specific example of an extended video in the embodiment.

[0009] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the following description, the term "video processing device" may be appropriately replaced with any other term such as "image processing device" or "information processing device." Similarly, the term "video" may be appropriately replaced with any other term such as "time-series image" or "moving image." In other words, terms with the same meaning may be appropriately replaced.

[0010] <Embodiment> [Configuration] Fig. 1 is a block diagram showing an example of the hardware configuration of a video processing device according to an embodiment. The video processing device 100 shown in Fig. 1 is, for example, a personal computer. Note that the video processing device 100 may also be configured as a server computer installed on the web or on the cloud.

[0011] The video processing device 100 includes a control unit 110 that uses a hardware processor such as a central processing unit (CPU), and a storage unit having a program storage unit 120 and a data storage unit 130, and an input / output interface (hereinafter, interface will be abbreviated as I / F) unit 140 are connected to the control unit 110 via a bus 150.

[0012] The control unit 110 is a circuit (control circuit) that controls the overall components of the video processing device 100. The control unit 110 includes a CPU, a RAM (Random Access Memory), a ROM (Read Only Memory), and the like. The CPU of the control unit 110 controls the entire video processing device 100 in accordance with a program stored in the ROM of the control unit 110. The RAM of the control unit 110 is used as a working area for the CPU of the control unit 110.

[0013] The input / output I / F unit 140 is connected to the input device 102 and the display device 104 .

[0014] The input device 102 is made up of, for example, a keyboard and a mouse, and is used to input information specifying the original video that the user wishes to acquire, commands for instructing the video processing device 100 to perform various operations, and the like.

[0015] The display device 104 is used to display information indicating various operating states of the video processing device 100 and information necessary for input operations, as well as to appropriately display original video and video based on each frame extracted from the original video.

[0016] The input / output I / F unit 140 may also include a communication interface. By including a communication interface, even if a video storage device that stores original video or an original video transmission device that can transmit original video, such as a camera that captures original video, is located far away, the video data transmitted from the original video transmission device can be acquired via a network.

[0017] The program storage unit 120 is configured, for example, by combining a nonvolatile memory such as a solid-state drive (SSD) that can be written and read at any time as a storage medium with a nonvolatile memory such as a ROM, and stores middleware such as an operating system (OS), as well as application programs necessary for executing various control operations according to the embodiment. Hereinafter, the OS and each application program will be collectively referred to as the "program." The program may be installed on the computer in advance from a network or a non-transitory computer-readable storage medium, or may be pre-recorded on the computer. In either case, the program is executed by the CPU of the control unit 110 to cause the computer to function as the video processing device 100.

[0018] The data storage unit 130 is, for example, a combination of a nonvolatile memory such as an SSD that can be written to and read from at any time and a volatile memory such as RAM as a storage medium. The data storage unit 130 stores data such as original video and video based on each frame extracted from the original video. The original video may be acquired from an external storage device, or may be acquired via a network from another terminal device, a server computer on the web, or a cloud.

[0019] Fig. 2 is a block diagram showing an example of the functional configuration of the video processing device 100 of Fig. 1. The video processing device 100 of Fig. 2 includes an acquisition unit 210, a calculation unit 220, and a generation unit 230.

[0020] To realize the functional configuration of Fig. 2, the CPU of the control unit 110 in Fig. 1 loads a program stored in the ROM of the control unit 110 or the program storage unit 120 into the RAM of the control unit 110. The CPU of the control unit 110 then interprets and executes the program loaded into the RAM of the control unit 110. As a result, the video processing device 100 functions as a computer including an acquisition unit 210, a calculation unit 220, and a generation unit 230. Note that the video processing device 100 is not limited to the processes of the acquisition unit 210, the calculation unit 220, and the generation unit 230, and is capable of executing any process in response to an operation by an operator.

[0021] Note that the acquisition unit 210, the calculation unit 220, and the generation unit 230 may be partly or entirely realized using hardware such as an LSI or an ASIC (Application Specific Integrated Circuit).

[0022] The acquisition unit 210 acquires original video including repetitive movements that repeatedly change postures in response to a user's operation. For example, the acquisition unit 210 may acquire the original video from the data storage unit 130. Alternatively, for example, the acquisition unit 210 may acquire original video transmitted from a video data transmission device (not shown).

[0023] In the following specific examples, the original video is assumed to be composed of a repetitive motion section (which may be referred to as a "first section") representing a repetitive motion and an interval section (which may be referred to as a "second section") during which no repetitive motion is performed. For example, if the original video is a film of a series of squats, each set consisting of multiple squats, the repetitive motion section corresponds to the video of one set of squats being performed, and the interval section corresponds to the video of the rest period between sets. Other examples of interval sections include video of the preparation time before a repetitive motion, video after a repetitive motion, and video unrelated to the repetitive motion. In other words, the interval section is a section other than the repetitive motion section.

[0024] The calculation unit 220 calculates a sampling interval for sampling the original video to a predetermined length. The predetermined length is, for example, a fixed number of frames to be input to a machine learning model. The specific value of the number of frames may be determined arbitrarily depending on the computational resources and computation time. Sampling the video corresponds to frame thinning. Specifically, the calculation unit 220 calculates the sampling interval based on the number of frames of the original video and the predetermined fixed number of frames.

[0025] The generator 230 generates an extended video by extending an interval section in the original video. Specifically, the generator 230 generates an inverted section by inverting the interval section in the time direction. Next, the generator 230 generates an extended interval section (which may also be referred to as an "extended section") by alternately combining the interval section and the inverted section. Then, the generator 230 generates an extended video by replacing the interval section in the original video with the extended interval section.

[0026] The video processing device 100 may determine whether to generate an extended video in the generation unit 230. Specifically, the video processing device 100 compares the length of the interval section with the sampling interval, and determines to generate an extended video if the length of the interval section is less than the sampling interval, and otherwise determines not to generate an extended video.

[0027] The configuration of the video processing device has been described above. Next, the operation of the video processing device will be described.

[0028] [Operation] Fig. 3 is a flowchart showing an example of an extended video generation process in the video processing device according to the embodiment. The extended video generation process is a process for generating an extended video by extending an interval section in an original video based on the original video. The process of the flowchart in Fig. 3 starts, for example, when a user executes the extended video generation process.

[0029] (Step ST110) The acquisition unit 210 acquires an original video including a repetitive motion with repeated posture changes. A specific example of the original video will be described below with reference to FIG.

[0030] FIG. 4 is a diagram illustrating a specific example of an original video in an embodiment. FIG. 4 shows the correspondence between the original video, which indicates continuous data in the time direction, and each section of the original video. Specifically, the original video in FIG. 4 includes two repeating motion sections and an interval section IX sandwiched between these two repeating motion sections. The original video has a length of the number of frames FA, and the interval section IX has a length of the number of frames FI. Note that the video obtained after sampling the original video (sampled video) has a length of the number of frames FS. This number of frames FS is a predetermined fixed number of frames.

[0031] (Step ST120) The calculation unit 220 calculates a sampling interval for sampling the original video to a predetermined length. For example, the calculation unit 220 calculates the sampling interval based on the number of frames in the original video and the number of frames of a predetermined fixed length. In the example of Fig. 4, the calculation unit 220 calculates the sampling interval (FA / FS) based on the number of frames FA in the original video and the number of frames FS in the sampled video.

[0032] (Step ST130) The video processing device 100 determines whether the length of the interval is less than the sampling interval. If it is determined that the length of the interval is less than the sampling interval, the process proceeds to step ST140. If it is determined that the length of the interval is not less than the sampling interval, the process of the flowchart in FIG. 3 ends. In the example of FIG. 4, the video processing device 100 determines whether the number of frames FI in the interval IX is less than the sampling interval (FA / FS). In the following specific examples, it is assumed that the number of frames FI in the interval IX is less than the sampling interval (FA / FS).

[0033] (Step ST140) After determining that the length of the interval is less than the sampling interval, the generation unit 230 generates an inversion interval by inverting the interval in the time direction. Specific examples of the interval and the inversion interval will be described below with reference to FIGS. 5 and 6, respectively.

[0034] 5 is a diagram for explaining a specific example of an interval section in the embodiment. In the interval section IX shown in FIG. 5, a frame x corresponding to the number of frames FI is included. １ , frame x ２ , ..., frame x ＦＩ－１ , and frame x ＦＩ The frame order of the interval IX is frame x. １ , frame x ２ , ..., frame x ＦＩ－１ , and frame x ＦＩ For example, the generating unit 230 generates the inversion section RX by inverting each frame of the interval section IX in the time direction.

[0035] 6 is a diagram for explaining a specific example of an inversion section in the embodiment. The inversion section RX shown in FIG. 6 includes a frame x corresponding to the frame number FI. ＦＩ , frame x ＦＩ－１ , ..., frame x ２ , frame x １ The frame order in the inversion section RX is frame x ＦＩ , frame x ＦＩ－１ , ..., frame x ２ , frame x １ That is, the reverse section RX reverses the frame order of the interval section IX.

[0036] (Step ST150) After generating the inversion intervals, the generation unit 230 generates an extended interval interval by alternately combining interval intervals and inversion intervals. Specifically, the generation unit 230 generates the extended interval interval by alternately combining interval intervals and inversion intervals N times (N is an even number greater than or equal to 2). For example, when N = 2, the generation unit 230 combines the interval interval, inversion interval, and interval interval in that order. Furthermore, when N = 4, the generation unit 230 combines the interval interval, inversion interval, interval interval, inversion interval, and interval interval in that order. By setting N to an even number greater than or equal to 2 in this way, the beginning and end of the extended interval interval can always be interval interval data.

[0037] The number of combinations is determined according to the sampling interval. Specifically, the generation unit 230 determines the number of combinations so that the length of the extended interval is longer than the sampling interval. The upper limit of the number of combinations may be determined arbitrarily according to the computational resources and computation time. A specific example of an extended interval will be described below with reference to FIG. 7.

[0038] 7 is a diagram illustrating a specific example of an extended interval in an embodiment. The extended interval EX shown in Fig. 7 is formed by combining an interval IX and an inversion interval RX N times (N is an even number equal to or greater than 2).

[0039] (Step ST160) After generating the extended interval section, the generation unit 230 generates an extended video in which the interval section in the original video is replaced with the extended interval section. After step ST160, the processing of the flowchart in Fig. 3 ends. A specific example of the extended video will be described below with reference to Fig. 8.

[0040] 8 is a diagram illustrating a specific example of an extended image in an embodiment. FIG. 8 illustrates the correspondence between an extended image showing continuous data in the time direction and each section of the extended image. Specifically, the extended image in FIG. 8 includes two repeated motion sections and an extended interval section EX included in these two repeated motion sections. The extended interval section EX is formed by combining an interval section IX and a reversed section RX twice.

[0041] The flowcharts described in the above embodiments are merely examples. The order of the steps in the flowcharts described in the embodiments may be changed to the extent possible, and other steps may be added.

[0042] To summarize the above, the image processing device 100 acquires an original image including a repetitive movement that repeats posture changes using the acquisition unit 210, calculates a sampling interval (FA / FS) using the calculation unit 220 based on the number of frames FA of the original image and the number of frames FS of the sampled image, and if the length of an interval section IX other than the repetitive movement section that represents the repetitive movement is less than the sampling interval (FA / FS), the generation unit 230 alternately combines the interval section IX and an inversion section RX that is the interval section IX inverted in the time direction, thereby generating an extended interval section EX that is longer than the sampling interval (FA / FS).

[0043] While the description of the extended video generation process in FIG. 3 illustrates a specific example in which the original video includes one interval, the present invention is not limited to this. For example, the original video may include multiple intervals. If the original video includes multiple intervals that are longer than the sampling interval, the generation unit 230 generates an extended interval for at least one of the multiple intervals. Furthermore, the generation unit 230 may adjust the number of intervals for which an extended interval is generated to be any ratio (e.g., 50%) of the number of multiple intervals.

[0044] [Effect] According to the above embodiment, the video processing device acquires an original video including a repetitive movement that involves repeated posture changes, calculates a sampling interval for sampling the original video to a predetermined length, and if the length of a second section other than a first section representing the repetitive movement is less than the sampling interval, generates an extended section longer than the sampling interval by alternately combining the second section and an inverted section in which the second section is inverted in the time direction.

[0045] Therefore, the video processing device can generate an extended section longer than the sampling interval by extending the second section shorter than the sampling interval. Thus, the video processing device can promote training of a model that predicts the number of cycles of a repetitive movement on the second section other than the repetitive movement, thereby contributing to improving the performance of the model.

[0046] The generating unit may also generate the extended section by alternately combining the second section and the inverted section N times (N is an even number equal to or greater than 2).

[0047] This allows the video processing device to ensure that the end of the generated extension section is data of the second section, thereby ensuring continuity between the extension section and the first section that follows it.

[0048] Furthermore, when there are multiple second intervals in the original video that are longer than the sampling interval, the generating unit may generate an extended interval for at least one of the multiple second intervals.

[0049] This allows the video processing device to prevent an extended video from including an excessive number of extended sections.

[0050] <Modifications> The functional configuration of the video processing device 100, its processing procedures and contents, the type and use of the original video, and the like can be modified in various ways without departing from the gist of the present invention.

[0051] For example, although the embodiment uses a change in human posture as an example, the present invention is not limited to this. For example, the embodiment can be similarly implemented to obtain the same effect for a repetitive motion or periodic change in posture (shape change) of an object other than a change in human posture.

[0052] The present invention is not limited to the above-described embodiments, and various modifications can be made in the implementation stage without departing from the spirit of the invention. Furthermore, the embodiments may be implemented in appropriate combinations, in which case the combined effects can be obtained. Furthermore, the above-described embodiments include various inventions, and various inventions can be extracted by combining selected components from the disclosed components. For example, if the problem can be solved and the effects can be obtained even if some components are removed from all the components shown in the embodiments, the configuration from which these components are removed can be extracted as an invention.

[0053] REFERENCE SIGNS LIST 100...Video processing device 102...Input device 104...Display device 110...Control unit 120...Program storage unit 130...Data storage unit 140...Input / output I / F unit 150...Bus 210...Acquisition unit 220...Calculation unit 230...Generation unit FA, FI...Number of frames IX...Interval section RX...Inversion section EX...Extended interval section

Claims

1. A video processing device comprising: an acquisition unit that acquires an original video including a repetitive motion that repeats posture changes; a calculation unit that calculates a sampling interval for sampling the original video to a predetermined length; and a generation unit that, when the length of a second interval other than a first interval representing the repetitive motion is less than the sampling interval, generates an extended interval longer than the sampling interval by alternately combining the second interval and an inverted interval obtained by inverting the second interval in the time direction.

2. The video processing device according to claim 1, wherein the generation unit generates the extended section by alternately combining the second section and the inverted section N times (N is an even number equal to or greater than 2).

3. The video processing device according to claim 1, wherein, when there are a plurality of second sections in the original video that are longer than the sampling interval, the generation unit generates the extended section for at least one of the plurality of second sections.

4. A program for causing a computer to function as each unit of the video processing device according to any one of claims 1 to 3.

Images