Video processing device, method, and program

Abstract

One embodiment of a video processing device according to the present invention comprises an acquisition unit, an extraction unit, a difference calculation unit, an accumulation unit, a time position calculation unit, a function calculation unit, and a prediction unit. The acquisition unit acquires original video representing an iterative operation in which an orientation change is repeated. The extraction unit extracts a plurality of frames from the original video. The difference calculation unit calculates a pixel value for the difference between adjacent frames among the plurality of frames. The accumulation unit accumulates the pixel values for the difference in each cycle of the iterative operation. The time position calculation unit calculates the time position of an orientation change within the cycle on the basis of the result of accumulating the pixel values for the difference. The function position calculation unit calculates a cumulative distribution function for associating each frame in the cycle with the time position of the orientation change represented by each frame. The prediction unit predicts a time position of each frame in the cycle on the basis of the cumulative distribution function.

Description

Video processing device, method and program

[0001] One aspect of the present invention relates to a video processing device, method, and program.

[0002] For example, a repetitive action counting (RAC) method is known that predicts the number of cycles of a repetitive action, such as a pull-up, from a video showing the action with repeated posture changes (see, for example, Non-Patent Document 1). This type of RAC method can predict the number of cycles N to be four based on the start point s and end point e of one cycle from a 100-frame video showing posture changes with frame number t, as shown in FIG. 10 . Similarly, the RAC method can predict a normal distribution representing each cycle using the start point s and end point e from multiple frames of original video vd, as shown in FIG. 11 . In the example of FIG. 11 , prediction results pd are output by predicting each of five cycles from a 64-frame video using a normal distribution with a mean μ = (s + e) / 2 and a variance σ = (s + e) / 6 (where μ = 3σ).

[0003] Such normal distribution prediction is based on the assumption that posture changes occur at a constant pace throughout the entire cycle. Specifically, the assumptions are (i) to (ii).

[0004] (i) During one cycle of movement, postures change continuously, and a group of postures equal to the number of frames is observed. Below, three postures (posture A (starting point s), posture B, and posture C (ending point e)) are extracted from this group of postures and explained. (ii) Postures A, B, and C change in this order at a constant pace over time.

[0005] The example shown in Fig. 12 shows a state in which posture A is assumed in frame 0, posture B in frame 5, and posture C in frame 10. Note that the posture changes successively between frames 0 to 5 and between frames 5 to 10. In this case, the frame numbers of one repetition range are frames 0 to 10. The correspondence between other frame numbers and postures is expressed in a similar manner.

[0006] For example, let's say the repetitive motion is skipping rope. In this case, one skipping rope cycle is defined as the range from the start point (jumping and leaving the ground) to the end point (landing). The repetitive motion is the motion within this range repeated over and over. In this case, posture A is defined as the "posture at the moment when the jump leaves the ground," posture B as the "posture at the moment when the jump switches from ascending to descending," and posture C as the "posture at the moment when the jump lands." In the case of skipping rope, even if the number of frames required for one cycle increases or decreases, the ascending motion from posture A to posture B and the descending motion from posture B to posture C within the cycle change at approximately the same pace.

[0007] Hu et al., “TransRAC: Encoding Multi-scale Temporal Correlation with Transformers for Repetitive Action Counting”. In CVPR, 2022.

[0008] The RAC method described above does not pose any particular problems, but according to the inventor's investigations, for example, in situations such as pull-ups where a person can change the pace at which they change from posture A to posture B and the pace at which they change from posture B to posture C, the pace at which posture changes within a cycle may differ. In this case, the RAC method, which is based on a normal distribution, cannot take into account the variance in the pace of posture changes and cannot predict which posture each frame in a cycle will be in, so there is room for improvement.

[0009] 13 , for example, the pace of posture change is approximately constant for frames 0 to 10 of the first cycle. Meanwhile, for frames 30 to 50 of the second cycle, the pace of posture change is fast for frames 30 to 35 in the first half of the cycle, and slow for frames 35 to 50 in the second half of the cycle. For frames 60 to 80 of the third cycle, the pace of posture change is slow for frames 60 to 75 in the first half of the cycle, and fast for frames 75 to 80 in the second half of the cycle. When the pace of posture change varies within a cycle, as in the second or third cycle, the RAC method, which is based on a normal distribution, cannot predict the time position of each frame in the cycle in terms of posture change.

[0010] The present invention has been made in consideration of the above circumstances, and provides a technology that can predict the time position of a posture change represented by each frame within a cycle, even if the pace of the posture change varies within the cycle.

[0011] In order to solve the above problem, one aspect of a video processing device according to the present invention includes an acquisition unit, an extraction unit, a difference calculation unit, an accumulation unit, a time position calculation unit, a function calculation unit, and a prediction unit. The acquisition unit acquires an original video representing a repetitive motion with repeated posture changes. The extraction unit extracts a plurality of frames from the original video. The difference calculation unit calculates the pixel value of the difference between adjacent frames among the plurality of frames. The accumulation unit accumulates the pixel values ​​of the difference for each cycle of the repetitive motion. The time position calculation unit calculates the time position of the posture change within the cycle based on the accumulation result of the pixel values ​​of the difference. The function position calculation unit calculates a cumulative distribution function that associates each frame within the cycle with the time position of the posture change represented by that frame. The prediction unit predicts the time position of each frame within the cycle based on the cumulative distribution function.

[0012] According to one aspect of the present invention, the difference pixel values ​​between frames are calculated, and the time position of a posture change within a cycle is calculated based on the accumulated result of the difference pixel values, and each frame is associated with the time position represented by the frame. This makes it possible to predict the time position of each frame within a cycle. Therefore, even if the pace of posture change within a cycle varies, it is possible to predict the time position of a posture change represented by each frame within a cycle.

[0013] That is, according to one aspect of the present invention, even if the pace of posture change within a cycle differs, it is possible to predict the time position of the posture change represented by each frame within the cycle.

[0014] FIG. 1 is a block diagram showing an example of the configuration of a video processing device according to a first embodiment of the present invention. FIG. 2 is a block diagram showing an example of the configuration of a processing unit shown in FIG. 1. FIG. 3 is a flowchart for explaining an example of operation in the first embodiment. FIG. 4 is a schematic diagram for explaining an example of operation in the first embodiment. FIG. 5 is a schematic diagram for explaining an example of operation in the first embodiment. FIG. 6 is a schematic diagram for explaining an example of operation in the first embodiment. FIG. 7 is a schematic diagram for explaining an example of operation in the first embodiment. FIG. 8 is a block diagram showing an example of the configuration of a processing unit in a second embodiment. FIG. 9 is a flowchart for explaining an example of operation in the second embodiment. FIG. 10 is a diagram for explaining a general RAC method. FIG. 11 is a diagram for explaining a general RAC method. FIG. 12 is a diagram for explaining a general RAC method. FIG. 13 is a diagram for explaining problems with the general RAC method.

[0015] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the following description, the term "video processing device" may be appropriately replaced with any other name such as an image processing device, an information processing device, or a video prediction device. Similarly, the term "video" may be appropriately replaced with any other name such as a time-series image or a moving image. In other words, terms with the same meaning may be appropriately replaced.

[0016] 1 is a block diagram showing an example of the configuration of a video processing device according to a first embodiment of the present invention. The video processing device 100 is, for example, a personal computer. Note that the video processing device 100 may also be configured as a server computer installed on the web or in the cloud.

[0017] The video processing device 100 includes a control unit 110 that uses a hardware processor such as a central processing unit (CPU), and a storage unit having a program storage unit 120 and a data storage unit 130, and an input / output interface (hereinafter, interface will be abbreviated as I / F) unit 140 are connected to the control unit 110 via a bus 150.

[0018] The input / output I / F unit 140 is connected to the input device 102 and the display device 104 .

[0019] The input device 102 is made up of, for example, a keyboard and a mouse, and is used by the user to input information specifying the original video that he or she wishes to acquire, commands for instructing the video processing device 100 to perform various operations, and the like.

[0020] The display device 104 is used to display information indicating various operating states of the video processing device 100 and information necessary for input operations, as well as to appropriately display original video, video based on each frame extracted from the original video, etc.

[0021] The input / output I / F unit 140 may also include a communication interface. By including a communication interface, even if an original video transmission device capable of transmitting original video, such as a video storage device that stores original video or a camera that captures original video, is located far away, the input / output I / F unit 140 can acquire video data transmitted from the original video transmission device via a network.

[0022] The program storage unit 120 is configured, for example, by combining a nonvolatile memory such as a solid-state drive (SSD) that can be written and read as needed with a nonvolatile memory such as a read-only memory (ROM) as a storage medium. It stores middleware such as an operating system (OS) as well as application programs necessary for executing various control operations according to an embodiment. Hereinafter, the OS and each application program will be collectively referred to as the "program." The program may be installed on the computer in advance from a network or a non-transitory computer-readable storage medium, or may be pre-recorded on the computer. In either case, the program is executed by the processor to cause the computer to function as the video processing device 100.

[0023] The data storage unit 130 is, for example, a combination of a nonvolatile memory such as an SSD that can be written to and read from at any time and a volatile memory such as a RAM (Random Access Memory) as a storage medium. The data storage unit 130 stores data such as original video and video based on each frame extracted from the original video. The original video may be acquired from an external storage device, or may be acquired via a network from another terminal device, a server computer on the web, or a cloud.

[0024] 2, the control unit 110 includes an acquisition unit 111, a difference calculation unit 112, a function calculation unit 113, and a prediction unit 114. The control unit 110 is not limited to the processes of the above units 111 to 114, and can execute any process in response to an operation by an operator.

[0025] Each of the above-mentioned units 111 to 114 is realized by causing a hardware processor of the control unit 110 to execute an application program stored in the program storage unit 120. Note that some or all of the above-mentioned units 111 to 114 may be realized using hardware such as an LSI (Large Scale Integration) or an ASIC (Application Specific Integrated Circuit).

[0026] The acquisition unit 111 acquires original video representing a repetitive motion with repeated posture changes. For example, the acquisition unit 111 may acquire the original video from the data storage unit 130. Alternatively, for example, the acquisition unit 111 may acquire original video transmitted from a video data transmission device (not shown). The acquisition unit 111 is an example of an acquisition unit.

[0027] The difference calculation unit 112 extracts multiple frames from the original video. The difference calculation unit 112 also calculates the pixel values ​​of the differences between adjacent frames among the multiple frames. The difference calculation unit 112 also accumulates the pixel values ​​of the differences for each cycle of the repetitive operation. The difference calculation unit 112 is an example of an extraction unit, a difference calculation unit, and an accumulation unit.

[0028] The function calculation unit 113 calculates the time position of a posture change within a cycle based on the accumulation result of the difference pixel values. The function calculation unit 113 also calculates a cumulative distribution function that associates each frame within the cycle with the time position of the posture change represented by that frame. The function calculation unit 113 is an example of a time position calculation unit and a function calculation unit.

[0029] The prediction unit 114 predicts the time position of each frame within the cycle based on the cumulative distribution function, and stores the prediction results in the data storage unit 130.

[0030] Next, an example of the operation of the video processing device configured as described above will be described using the flowchart in Figure 3 and the schematic diagrams in Figures 4 to 7. The following description will be given using an original video captured of walking on a treadmill as an example of original video representing a repetitive movement with repeated posture changes. This original video represents a repetitive movement with repeated foot posture changes on the treadmill. The following description will be given in order.

[0031] (1) Acquisition of Original Video The acquisition unit 111 acquires original video representing a repetitive motion with repeated posture changes (step ST1). For example, the acquisition unit 111 reads the original video from the data storage unit 130.

[0032] (2) Frame Extraction After step ST1, the difference calculation unit 112 extracts multiple frames from the acquired original video (step ST2). Note that the original video has several tens of frames per second, meaning that adjacent frames are nearly identical, making it difficult to detect inter-frame differences. Therefore, the difference calculation unit 112 extracts frames from the original video in advance. In this example, the difference calculation unit 112 extracts 64 frames from the original video.

[0033] (3) Calculation of Inter-Frame Differences After step ST2, the difference calculation unit 112 calculates the pixel values ​​of the differences between adjacent frames among the extracted 64 frames (step ST3). FIG. 4 is a schematic diagram illustrating the operation of step ST3, showing a portion of each cycle of the extracted multiple frames in the upper row and a portion of the difference image in the lower row. In FIG. 4, the subscript t indicates the frame number within the cycle and takes a value from 1 to T. The final frame number T within the cycle can be calculated using the total frame number of the start point s and the total frame number of the end point e of each cycle as T = e - s + 1. The total frame number is a number that identifies all 64 extracted frames and takes a value from 1 to 64. For example, if the first cycle has total frame numbers 3 to 13, the final frame number T within the cycle can be calculated as T = e - s + 1 = 13 - 3 + 1 = 11, using the total frame number 3 of the start point s and the total frame number 13 of the end point e.

[0034] The difference calculation unit 112 calculates, for example, a plurality of frames (...f ｔ－１ , f ｔ , f ｔ＋１ , f ｔ＋２ , ...), for each cycle, a certain frame (f ｔ－１ ) and the frame (f ｔ－１ ) adjacent frame (f ｔ ) and calculates the difference in pixel values ​​at the same position. The difference calculation unit 112 also calculates a difference image (df ｔ ) and calculate the sum of the differences D ｔ However, when frame number t=1, there is no frame with frame number t-1, so the total difference value D １ = 0. The difference total value may be called by other names such as "pixel total value" or "pixel value total." Similarly, the difference calculation unit 112 calculates the difference total value for a certain frame (f ｔ ) and the frame (f ｔ ) adjacent frame (f ｔ＋１ ) and calculate the difference in pixel values ​​at the same position, and generate a difference image (df ｔ＋！ ) and calculate the sum of the differences D ｔ＋１Similarly, the difference calculation unit 112 calculates the difference between a frame (f ｔ＋１ ) and the frame (f ｔ＋１ ) adjacent frame (f ｔ＋２ ) and calculate the difference in pixel values ​​at the same position, and generate a difference image (df ｔ＋２ ) and calculate the sum of the differences D ｔ＋２ Similarly, the difference calculation unit 112 updates a frame to the next frame, calculates the difference between adjacent frames, and sums up the pixel values ​​of the obtained difference image. In this way, the difference calculation unit 112 calculates the pixel values ​​of the differences between all adjacent frames from the extracted multiple frames. Note that if the pace of posture change is fast, the total difference value D of the inter-frame differences ｔ When the pace of posture change is slow, the total difference value D ｔ is a small value.

[0035] (4) Accumulation of Difference Pixel Values ​​After step ST3, the difference calculation unit 112 accumulates the difference pixel values ​​for each cycle of the repetitive operation (step ST4). For example, the difference calculation unit 112 accumulates a difference total value D ｔ The accumulated result is the total difference value D ｔ When the value is large, it increases by a large value. Also, the cumulative result is the difference total value D ｔ If is small, it increases by a small amount.

[0036] (5) Calculating the Time Position of Posture Change After step ST4, the function calculation unit 113 calculates the time position of the posture change within the cycle based on the accumulation result of the difference pixel values ​​(step ST5). For example, the function calculation unit 113 calculates the time position of the posture change by normalizing the accumulation result as shown in equation (1).

[0037]

[0038] For example, the function calculation unit 113 sets the normalized cumulative result as the time position of the posture change. In this case, the time position of the posture change takes values ​​ranging from an initial value of 0 to a final value of 1 within a cycle. The initial value of 0 is the time position of the posture change at frame number 1 within the cycle. The final value of 1 is the time position of the posture change from frame number 1 to T within the cycle. If the pace of the posture change in the first half of the cycle is faster than that in the second half of the cycle, the median value of 0.5 of the time position of the posture change is obtained as the time position corresponding to a frame number earlier than the median frame number within the cycle. Furthermore, if the pace of the posture change in the first half of the cycle is slower than that in the second half of the cycle, the median value of 0.5 of the time position of the posture change is obtained as the time position corresponding to a frame number later than the median frame number within the cycle. Note that the function calculation unit 113 may also normalize the frame numbers for each cycle. The normalized frame numbers may be expressed by setting the frame number of the starting point s within the cycle to 0 and the frame number T of the end point e within the cycle to 1. Furthermore, the normalized frame number of a point between the start point s and the end point e in a cycle is obtained by subtracting the overall frame number of the start point s from the overall frame number of this point, and dividing the result by the number of frames in the cycle.

[0039] (6) Calculating the Cumulative Distribution Function After step ST5, the function calculation unit 113 calculates a cumulative distribution function that associates each frame in a cycle with the time position of the posture change represented by that frame (step ST6). This step ST6 is executed for each cycle, and a cumulative distribution function for each cycle is calculated. FIGS. 5 and 6 are schematic diagrams for explaining the operation of step ST5. FIG. 5 shows a cumulative distribution function when posture change occurs at an average pace within a cycle, and FIG. 6 shows a cumulative distribution function when posture change occurs at a faster pace in the first half of the cycle than in the second half. In FIGS. 5 and 6, the horizontal axis represents the normalized frame number t, and the vertical axis represents the normalized time position of the posture change corresponding to the normalized frame number t. Here, the time position of the posture change represents the value of the cumulative distribution function F(t) based on the normalized frame number t. Any configuration can be used as the cumulative distribution function F(t), as long as it associates the frame number of each frame with the time position of the posture change represented by that frame. For example, the cumulative distribution function F(t) may be configured using a formula that can calculate the time position based on the frame number, or may be configured using a table that can read out the time position based on the frame number.

[0040] (7) Predicting Temporal Position After step ST6, the prediction unit 114 predicts the temporal position of each frame within the cycle based on the cumulative distribution function (step ST7). The prediction unit 114 also stores the prediction result in the data storage unit 130. Through steps ST1 to ST7, the control unit 110 can obtain a prediction result pd1 from the original video vd, as shown in FIG. 7 . The prediction result pd1 represents the temporal position of posture change predicted from the total frame number within each cycle of five repetitive movements when 64 frames are extracted from the original video vd. The prediction result pd1 reflects the cumulative distribution function F(t) corresponding to the pace of posture change in each cycle, as shown in FIG. 5 or FIG. 6 . In other words, the prediction result pd1 indicates the true value of posture change according to the pace of posture change. Note that the vertical axis of the prediction result pd1 represents the total frame number, and the horizontal axis represents the cumulative distribution function F(t). Furthermore, the prediction result pd1 sets the time position of the posture change to 0 for all frame numbers other than the cycle.

[0041] On the other hand, as shown in Fig. 11, a general method produces a prediction result pd based on a normal distribution, which does not reflect the pace of posture change, and the prediction result pd always matches the median value of the frame number within a cycle with the median time position of the posture change. In other words, the prediction result pd in Fig. 11 will show a false value of the posture change if the posture change is not at the average pace.

[0042] After step ST7 is completed, the video processing device 100 may terminate the processing. That is, the application of the video processing device 100 is not limited to the usual case of setting a label that predicts the time position of a posture change for each extracted frame, but may also include a case where it is simply desired to predict the time position of a posture change for each extracted frame. An example of the latter case may be a case where it is desired to check whether the posture change is average. In this way, depending on the application, the video processing device 100 may not necessarily proceed to label setting, but may terminate the processing by predicting the time position of a posture change. Conversely, depending on the application, the video processing device 100 may set a label for each frame based on the predicted time position.

[0043] As described above, according to the first embodiment, the acquisition unit 111 acquires an original video image representing a repetitive motion involving repeated posture changes. The difference calculation unit 112 extracts multiple frames from the original video image. The difference calculation unit 112 also calculates the pixel values ​​of the differences between adjacent frames among the multiple frames. The difference calculation unit 112 also accumulates the pixel values ​​of the differences for each cycle of the repetitive motion. The function calculation unit 113 calculates the time position of the posture change within the cycle based on the accumulated result of the pixel values ​​of the differences. The function calculation unit 113 also calculates a cumulative distribution function that associates each frame within the cycle with the time position of the posture change represented by that frame. The prediction unit 114 predicts the time position of each frame within the cycle based on the cumulative distribution function. In this way, the acquisition unit 111 calculates the pixel values ​​of the differences between frames, calculates the time position of the posture change within the cycle based on the accumulated result of the pixel values ​​of the differences, and associates each frame with the time position represented by that frame. This makes it possible to predict the time position of each frame within the cycle. Therefore, even if the pace of posture change varies within a cycle, it is possible to predict the time position of the posture change represented by each frame within the cycle.

[0044] In other words, according to the first embodiment, it is possible to reflect the pace of transformation of posture change in repeated video. That is, according to the first embodiment, the total difference value of pixel values ​​between two adjacent frames is expressed as a cumulative distribution normalized over one cycle, and the value of a sampling point (the time position of the posture change represented by the extracted frame) can be predicted.

[0045] 8 is a block diagram showing an example of the configuration of a control unit in a second embodiment, and parts that are substantially the same as those in Fig. 1 and Fig. 2 are given the same reference numerals and detailed explanations thereof are omitted, and only different parts will be described here. Duplicate explanations will also be omitted for each of the following embodiments.

[0046] The control unit 110 includes the above-described units 111 to 114, as well as a setting unit 115. As described above, each of these units 111 to 115 is realized by causing a hardware processor in the control unit 110 to execute an application program stored in the program storage unit 120. Similarly, some or all of the above-described units 111 to 115 may be realized using hardware such as an LSI or an ASIC.

[0047] The setting unit 115 sets a label corresponding to the time position predicted by the prediction unit 114 to each frame in the cycle. Note that this label may be called a "predicted label" or a "true value label" or other names. In addition to a value indicating the time position, the label may use any attribute such as a cycle ID indicating which cycle it is and a value indicating the pace of posture change (fast / average / slow). Furthermore, for attributes common within a cycle, such as the cycle ID or the pace of posture change, labels may be set only to specific frames, such as the start point s or end point e of the cycle. The setting unit 115 is an example of a setting unit.

[0048] Next, an example of the operation of the video processing device configured as above will be described with reference to the flowchart of FIG.

[0049] Now, it is assumed that steps ST1 to ST7 have been executed in the same manner as described above.

[0050] (8) Setting Labels After step ST7, the setting unit 115 sets a label corresponding to the time position predicted in step ST7 to each frame in the cycle (step ST8). As a result, the setting unit 115 stores the video having the frames with the labels set as output video in the data storage unit 130. Then, step ST8 ends.

[0051] As described above, according to the second embodiment, the setting unit 115 sets, to each frame in a cycle, a label corresponding to the time position predicted by the prediction unit 114. Therefore, in addition to the effects described above, even if the pace of posture change within a cycle differs, it is possible to set, to each frame in the cycle, a label corresponding to the time position of the posture change represented by that frame.

[0052] In other words, according to the second embodiment, it is possible to set a predictive label corresponding to the variance of the pace of posture change for each cycle.

[0053] Other Embodiments The functional configuration of the video processing device 100, its processing procedures and contents, the type and use of the original video, and the like can be modified in various ways without departing from the spirit of the present invention.

[0054] For example, in the first embodiment, an original image showing a repetitive movement of repeatedly changing the posture of the feet on a treadmill is used, but this is not limiting. For example, the posture changes shown in the original image are not limited to the parts or contents of a repetitive movement, such as pull-ups, which involve repeatedly changing the posture of the arms.

[0055] Furthermore, for example, while the first embodiment uses original video images showing repetitive movements at approximately the same position on a treadmill, this is not limiting. For example, even if the repetitive movements include movement to different positions, the first embodiment can be implemented in a similar manner to obtain similar effects by capturing original video images in a range that includes positions before and after the movement.

[0056] Furthermore, for example, although the first embodiment uses human posture changes as an example, the present invention is not limited to this. For example, the first embodiment can be similarly implemented to achieve similar effects for periodic posture changes (shape changes) other than human posture changes, where the pace of the changes can vary.

[0057] In the first embodiment, 64 frames are extracted from the original video, but this is not limiting. For example, the number of frames to be extracted may be other than 64. Any extraction method may be used. For example, frames may be extracted from the original video of frame number L at regular intervals.

[0058] Furthermore, for example, the first embodiment uses the term "time position of posture change," but is not limited to this. For example, instead of the term "time position of posture change," other terms such as "degree of posture change" or "progression of posture change" can be used, as long as the terms have the same meaning, and the first embodiment can be implemented in the same way to obtain the same effect. Note that "terms with the same meaning" are, for example, terms that correspond to the value of the cumulative distribution function F(t).

[0059] For example, in the second embodiment, a label corresponding to the time position of a posture change is set, but this is not limiting. That is, in addition to a value indicating a time position, any attribute such as a cycle ID indicating which cycle it is and a value indicating the pace of posture change (fast / average / slow) may be used as the label. In this case, attributes common to the same cycle, such as the cycle ID and the pace of posture change, may be set only to a specific frame, such as the start point s or end point e of the cycle.

[0060] Furthermore, for example, by setting such labels, a dataset may be obtained in which each frame of the output video is used as input data and feature values ​​(time positions of posture changes) corresponding to the labels set for each frame of the output video are used as output data. With this dataset, a method for setting predictive labels corresponding to the variance in the pace of posture changes for each cycle enables learning that takes such variance into account, even when the pace of posture changes varies from cycle to cycle. For example, a TransRAC model for the RAC method may be trained using this dataset. Based on the output video, such a trained model can output feature values ​​(time positions of posture changes) corresponding to each label as prediction results. Using such a trained model, the RAC method may predict the time position of each frame in a cycle in a posture change, even when the pace of posture change varies within a cycle.

[0061] Although the embodiments of the present invention have been described in detail above, the above description is merely an example of the present invention in every respect. It goes without saying that various improvements and modifications can be made without departing from the scope of the present invention. In other words, when implementing the present invention, specific configurations according to the embodiments may be appropriately adopted.

[0062] In short, this invention is not limited to the above-described embodiments, and in the implementation stage, the components can be modified and embodied without departing from the spirit of the invention. Furthermore, various inventions can be formed by appropriately combining multiple components disclosed in the above-described embodiments. For example, some components may be omitted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined.

[0063] DESCRIPTION OF SYMBOLS 100...Video processing device 102...Input device 104...Display device 110...Control unit 111...Acquisition unit 112...Difference calculation unit 113...Function calculation unit 114...Prediction unit 115...Setting unit 120...Program storage unit 130...Data storage unit 140...Input / output I / F unit 150...Bus vd...Original video pd1...Prediction result

Claims

1. A video processing device comprising: an acquisition unit that acquires original video representing a repetitive motion with repeated posture changes; an extraction unit that extracts a plurality of frames from the original video; a difference calculation unit that calculates the pixel value difference between adjacent frames among the plurality of frames; an accumulation unit that accumulates the pixel values ​​of the difference for each cycle of the repetitive motion; a time position calculation unit that calculates the time position of a posture change within a cycle based on the accumulated result of the pixel values ​​of the difference; a function calculation unit that calculates a cumulative distribution function that associates each frame within the cycle with the time position of the posture change represented by that frame; and a prediction unit that predicts the time position of each frame within the cycle based on the cumulative distribution function.

2. The video processing device according to claim 1, further comprising a setting unit that sets a label corresponding to the predicted time position to each frame in the cycle.

3. A method executed by a video processing device, comprising: acquiring an original video representing a repetitive motion with repeated posture changes; extracting a plurality of frames from the original video; calculating differential pixel values ​​between adjacent frames among the plurality of frames; accumulating the differential pixel values ​​for each cycle of the repetitive motion; calculating the time position of a posture change within a cycle based on the accumulated result of the differential pixel values; calculating a cumulative distribution function that associates each frame within the cycle with the time position of a posture change represented by that frame; and predicting the time position of each frame within the cycle based on the cumulative distribution function.

4. A program for causing a computer to function as the video processing device according to claim 1 or 2.

Images