Estimation device, estimation method, and program

Abstract

To improve an accuracy of estimation of an operation pattern.SOLUTION: An estimation apparatus that estimates an operation pattern of an estimation target generates image data of nonlinear motion history image in which each frame of a component video is superimposed with an estimated influence as a weight for each initial influence function, based on data of the component video capturing an operation for which an aimed result is identical to an operation to be estimated and M types of initial influence functions with an elapsed time from starting of the video as an independent variable and an estimated influence, which indicates a strength of an influence on a result of the estimation, as a dependent variable, and a graph has one peak, estimates an operation pattern captured in each of the nonlinear motion history image, acquires a degree of matching each of the estimated operation pattern and the operation pattern captured in the component video, acquires the degree obtained by using the initial influence function in which the elapsed time giving a peak is i, as an estimation influence of the frame at the elapsed time i of the video of the estimation target, and estimates an operation pattern of the estimation target using the obtained estimation influence.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an estimation device, an estimation method, and a program. [Background technology]

[0002] In recent years, technology using machine learning has been developed to estimate patterns of actions indicated by the movements of objects included in a video. In such technology, since the amount of data of the video itself is large, the amount of data required for learning is huge, and the learning process takes a long time. To solve this problem, as in Non-Patent Document 1, learning may be performed using time-series data called Temporal Template, in which predetermined information is added to the frames that make up the video.

[0003] Although the temporal template is time-series data, there is a technology to express similar information in still images. For example, the motion history image (MHI) described in Non-Patent Document 1 is an image that displays the trajectory of a movement as if an afterimage was left, with the closer the time, the brighter the information, and the older the information, the lighter the brightness. In other words, the motion history image is an image that is the result of superimposing each frame of the temporal template with a change in brightness. Therefore, it is expected that the use of the motion history image will enable learning that focuses on motion information with a smaller amount of information than inputting a video. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] A. Bobick, J. Davis, “The recognition of human movement using temporal templates,” IEEE Transactions on Pattern Analysis and Machine Intelligence, VOL.23, NO.3, 2001. Summary of the Invention [Problem to be solved by the invention]

[0005] The MHI in the conventional technology such as Non-Patent Document 1 is a still image in which each frame is superimposed so that the larger the change or the closer to the final frame, the more important the frame is for estimating the motion pattern, which is a pattern of the motion. However, in estimating the motion pattern, the part with a large change is not necessarily important. In addition, in estimating the motion pattern, the frame close to the final frame is not necessarily important. Therefore, when it is necessary to distinguish small differences between similar motion patterns and estimate the motion pattern, such as a straight shot or a cross shot in a tennis stroke, a uniform method of providing information such as Non-Patent Document 1 may not be able to properly estimate the movement of a truly important target. In other words, when a uniform method of providing information such as Non-Patent Document 1 is used, the timing that strongly affects the estimation result may not be used appropriately, and the accuracy of the estimation result may be poor.

[0006] In view of the above circumstances, an object of the present invention is to provide a technique for improving the accuracy of estimating a movement pattern. [Means for solving the problem]

[0007] One aspect of the present invention is an estimation device that estimates a pattern of a movement to be estimated, the estimation device including: data of constituent moving images, which are moving images capturing a movement whose desired result is the same as the movement to be estimated; and M types of initial influence functions (M is an integer of 2 or more) that are predetermined functions having an independent variable of video time, which is the elapsed time from the start of the moving image, and an estimated influence indicating the strength of the influence on the result of the estimation, as a dependent variable, and whose graph has one peak, the initial influence functions being based on the data of the constituent moving images; and an image set generation processing unit that generates image data of a nonlinear motion history image, which is an image obtained by superimposing each frame of the constituent moving images with the estimated influence as a weight, for each initial influence function; and The estimation device includes an estimation model execution unit that executes an estimation model to estimate a movement pattern that will appear in a nonlinear motion history image to be processed for each image data of a history image; a similarity acquisition unit that acquires the degree of similarity between each of M movement patterns estimated by the estimation model execution unit and the movement pattern appearing in the constituent moving images; an estimated influence acquisition unit that acquires the degree obtained using the initial influence function whose peak is the moving image time i (i is a real number) as an estimated influence of a frame at moving image time i of the moving image to be estimated; and a main estimation process execution unit that estimates the movement pattern of the estimation target using the estimated influence acquired by the estimated influence acquisition unit.

[0008] The estimation method estimates a pattern of a movement of an estimation target, comprising: an image set generation process step of generating image data of a nonlinear motion history image, which is an image obtained by superimposing each frame of the constituent videos with the estimated influence as a weight, for each initial influence function based on data of constituent videos, which are videos capturing movements whose desired result is the same as the movement of the estimation target, and M types of initial influence functions (M is an integer of 2 or more) each of which is a predetermined function with an independent variable being video time, which is the elapsed time from the start of the video, and an estimated influence indicating the strength of the influence on the result of the estimation, as a dependent variable, the graph of which has one peak; and the degree of agreement between each of the M movement patterns estimated by the estimation model execution step and the movement patterns depicted in the constituent moving pictures; an estimated influence acquisition step of acquiring the degree obtained using the initial influence function whose peak is the moving picture time i (i is a real number) as an estimated influence of a frame at moving picture time i of the moving picture to be estimated; and a main estimation process execution step of estimating the movement pattern of the estimation target using the estimated influence acquired by the estimated influence acquisition step.

[0009] One aspect of the present invention is a program for causing a computer to function as the above-described estimation device. Effect of the Invention

[0010] According to the present invention, it is possible to improve the accuracy of estimating a motion pattern. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is an explanatory diagram illustrating an overview of an estimation device according to an embodiment. [Diagram 2] FIG. 4 is a diagram showing an example of impact level information according to the embodiment. [Diagram 3]FIG. 2 is a diagram illustrating an example of a hardware configuration of an estimation apparatus according to an embodiment. [Figure 4] FIG. 2 is a diagram illustrating an example of a functional configuration of a control unit included in the estimation device of the embodiment. [Diagram 5] 4 is a flowchart showing an example of a flow of a process executed by the estimation device in the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] (Embodiment) FIG. 1 is an explanatory diagram for explaining an overview of an estimation device 1 according to an embodiment. The estimation device 1 estimates a pattern of a motion to be estimated. The following patterns of motions are called motion patterns. A motion to be estimated by the estimation device 1 is a motion in which a motion at a certain timing among the entire process of the motion has a stronger influence on the estimation result than motions at other timings. A motion in which a motion at a certain timing among the entire process has a stronger influence on the estimation result than motions at other timings is, for example, a stroke motion made by a tennis player.

[0013] Moreover, the movements to be estimated by the estimation device 1 have approximately the same length from the start to the end of the movement. More specifically, the estimation device 1 estimates the movement pattern of the estimation target based on data of a video capturing the movement of the estimation target. Hereinafter, the video capturing the movement of the estimation target is referred to as a target video. Hereinafter, the video capturing the movement, including the target video, is referred to as a movement video.

[0014] The estimation device 1 executes a pre-estimation process. The pre-estimation process is a process for estimating an estimated influence degree of each frame of a target video. The estimated influence degree is the strength of the influence of a movement pattern captured in a movement video on the estimation result. The higher the estimated influence degree of a frame, the more a change in that frame changes the estimation result.

[0015] The degree of influence of each frame of the motion video refers to the strength of the influence that each timing of the motion captured in the motion video has on the estimation result of the motion pattern captured in the video, because a frame is an image that constitutes a motion video and captures a moment of the motion to be estimated.

[0016] The estimation device 1 generates image data of an image in which each frame of a motion video is superimposed using the estimated influence as a weight. Superimposing images using the estimated influence as a weight specifically means superimposing images with a luminance according to the weight of the estimated influence. For example, the luminance according to the weight of the estimated influence is higher as the weight of the estimated influence is heavier. Hereinafter, an image in which each frame of a motion video is superimposed using the estimated influence as a weight is referred to as a nonlinear motion history image. Hereinafter, the process of generating a nonlinear motion history image is referred to as a nonlinear motion history image generation process.

[0017] One of the targets of the nonlinear motion history image generation process is the data of a target video. Hereinafter, the nonlinear motion history image obtained by using the data of the target video as the target of the nonlinear motion history image generation process is referred to as the target image. In the nonlinear motion history image generation process that uses the data of the target video as the processing target, the estimated influence degree estimated in the pre-estimation process is used as the estimated influence degree.

[0018] The nonlinear motion history image generation process is also executed in the pre-processing. The nonlinear motion history image generation process is also executed for a plurality of motion moving images in the pre-processing. In the nonlinear motion history image generation process in the pre-processing, a predetermined estimated influence degree is used as a weight.

[0019] After executing the pre-estimation process, the estimation device 1 executes a main estimation process. The main estimation process includes a process of generating a target image using a result of the pre-estimation process, and a process of estimating a motion pattern of an estimation target based on the target image.

[0020] <Pre-estimation processing> The advance estimation process will be described in more detail. In the advance estimation process, the estimation device 1 acquires M types (M is an integer equal to or greater than 2) of initial influence functions. The initial influence functions are predetermined functions with the elapsed time from the start of a video as an independent variable and the estimated influence as a dependent variable, and are predetermined functions whose graphs have one peak. Since the initial influence functions are predetermined, the estimated influences of the initial influence functions are predetermined values ​​that correspond one-to-one to each independent variable. Hereinafter, the elapsed time from the start of a video is referred to as video time.

[0021] Hereinafter, the video time that gives the peak of the initial value influence function is called the peak time. The difference between the types of initial influence functions is specifically the difference in the peak time. The graph of the initial influence function is, for example, a normal distribution.

[0022] The initial influence function may be input by, for example, a user, or may be stored in advance in, for example, a predetermined storage device. When the user inputs the initial influence function, the estimation device 1 acquires the input initial influence function. When the initial influence function is stored in advance in a predetermined storage device, the estimation device 1 acquires the initial influence function by reading the initial influence function from the storage device. In the following, for ease of explanation, the estimation device 1 will be explained taking as an example a case where the initial influence function is stored in advance in a predetermined storage device.

[0023] In the pre-estimation process, the estimation device 1 acquires a set of data of videos (hereinafter referred to as "constituent videos") that capture actions that have the same desired results as the action to be estimated, and information (hereinafter referred to as "correct answer information") that indicates the action patterns captured in the constituent videos. Like the target video, the constituent videos are also a type of action video.

[0024] A specific example of a motion whose desired result is the same as the motion to be estimated will be described below. For example, if the motion to be estimated is a stroke in tennis, a stroke performed by a person other than the person who performed the motion to be estimated is an example of a motion whose desired result is the same as the motion to be estimated. This is because both are motions that aim to return a ball by a stroke.

[0025] The set of constituent moving image data and correct answer information may be input by, for example, a user, or may be stored in advance in, for example, a predetermined storage device. When the user inputs the set of constituent moving image data and correct answer information, the estimation device 1 acquires the input constituent moving image data and correct answer information. When the constituent moving image data and correct answer information are stored in advance in a predetermined storage device, the estimation device 1 acquires the constituent moving image by reading the constituent moving image data and correct answer information from the storage device. Below, for simplicity of explanation, the estimation device 1 will be explained using an example in which the constituent moving image data and correct answer information are input by the user.

[0026] In the advance estimation process, the estimation device 1 executes a nonlinear motion history image generation process for each of the acquired M types of initial influence functions, with the processing target being image data of the constituent moving images (001). Hereinafter, the nonlinear motion history image generation process for each of the acquired M types of initial influence functions, with the processing target being image data of the constituent moving images, will be referred to as an image set generation process.

[0027] By executing the image set generation process, image data of nonlinear motion history images is generated for each of the M types of initial influence functions. That is, by executing the image set generation process, image data of M nonlinear motion history images is generated.

[0028] In the pre-estimation process, the estimation device 1 executes an estimation model on image data of the generated M non-linear motion history images (002). The estimation model is a mathematical model whose processing target is the image data of the non-linear motion history image, and is a mathematical model that estimates a movement pattern appearing in the non-linear motion history image that is the processing target. By executing the estimation model on the image data of the M non-linear motion history images, a movement pattern estimated by the estimation model is obtained for each non-linear motion history image.

[0029] In the pre-estimation process, the estimation device 1 calculates the degree of agreement (hereinafter referred to as the "estimated agreement degree") between the motion pattern estimated by the estimation model for the non-linear motion history image and the motion pattern indicated by the correct answer information for each of the M non-linear motion history images (003). That is, in the pre-estimation process, the estimation device 1 obtains the M estimated agreement degrees based on the estimation result of the estimation model.

[0030] In the pre-estimation process, the estimation device 1 acquires an estimated matching degree of the nonlinear motion history image obtained by using an initial value influence function whose peak time is time i as an estimated influence degree of a frame at video time i (i is a real number) of the target video (004). Note that the estimated matching degree of image A means the degree of matching between the motion pattern estimated by the estimation model for image A and the motion pattern indicated by the correct answer information.

[0031] Hereinafter, information indicating the estimated degree of match of the nonlinear motion history image obtained using the initial value influence function with peak time i as the estimated influence at video time i of the target video is referred to as influence information.

[0032] In this way, in the advance estimation process, the estimation device 1 acquires influence information based on the result of estimation by the estimation model. Note that the result of estimation by the estimation model is the operation pattern estimated by the estimation model.

[0033] As described above, the difference between the types of initial influence functions is specifically the difference between the peak times. Therefore, obtaining an estimated degree of agreement using a plurality of types of initial influence functions means obtaining a distribution of estimated degrees of agreement by varying the position of the peak of the graph of the initial influence function. In general, the position of the maximum value of the graph indicates the strength of the action on the target of the action, so that the frame at the peak position has a greater influence on the estimation of the movement pattern than other frames even for nonlinear motion history images. Therefore, although the distribution of estimated degrees of agreement indicates the degree of agreement between the movement pattern estimated using the initial influence function with each movement time as the peak position and the movement pattern indicated by the correct answer information, it can also be said to indicate the strength of the influence that the frame at each movement time has on the result of the estimation of the movement pattern. This is easy to understand if, for example, the initial influence function is assumed to be a δ function. Even if the initial influence function is not a δ function, if it is a function with one peak, a phenomenon that is correlated with the phenomenon in the case of a δ function occurs.

[0034] Fig. 2 is a diagram showing an example of influence information in an embodiment. The horizontal axis of Fig. 2 represents video time. Therefore, the horizontal axis of Fig. 2 is also information indicating video frames. The vertical axis of Fig. 2 represents an estimated influence. In the example of Fig. 2, a function w(t) expressing the estimated influence has a peak at video time t'.

[0035] The estimation model that estimates the movement pattern used to calculate the estimated matching degree may be a trained estimation model or a non-trained estimation model. A trained estimation model is an estimation model after the processing content has been updated by learning using image data of multiple non-linear motion history images generated by the image set generation process. A non-trained estimation model is an estimation model that is not updated by learning. In other words, a non-trained estimation model is an estimation model in which the predetermined processing content is not updated.

[0036] When the estimation model is a trained estimation model, in the learning, the estimation model is updated so as to reduce the difference between the motion pattern estimated by the estimation model and the motion pattern indicated by the correct answer information. That is, when the estimation model is a trained estimation model, the loss function is a function indicating the difference between the motion pattern estimated by the estimation model and the motion pattern indicated by the correct answer information.

[0037] <Main estimation process> The main estimation process will be described in more detail. In the main estimation process, the estimation device 1 executes a nonlinear motion history image generation process in which data of a target video is processed (005). In the nonlinear motion history image generation process in which data of a target video is processed, the estimated influence estimated by the pre-estimation process is used, as described above. Specifically, in the nonlinear motion history image generation process in which data of a target video is processed, an image is generated in which each frame of the target video is superimposed using the estimated influence of each frame indicated by the influence information as a weight. The image in which each frame of the target video is superimposed using the estimated influence of each frame indicated by the influence information as a weight is the target image.

[0038] In the main estimation process, the estimation device 1 executes the estimation model on the obtained target image (006). The result of executing the estimation model on the target image is the result of estimating the motion of the estimation target.

[0039] In addition, the estimation model in the main estimation process may be a trained estimation model if the estimation model in the pre-estimation process has been updated by learning, or may be an estimation model in an initial state before the update by learning. However, when a trained estimation model is used, the influence of the pre-estimation process may appear in the estimation result. Therefore, it is preferable to use an estimation model in an initial state before the update by learning. In addition, when the estimation model in the pre-estimation process is a non-trained estimation model, the estimation model in the main estimation process is the estimation model in the pre-estimation process.

[0040] Note that the estimation device 1 does not necessarily need to execute the advance estimation process again when estimating a new target moving image. When estimating a new target moving image, the main estimation process may be executed without executing the advance estimation process.

[0041] As described above, the estimation device 1 processes the input data of the constituent videos as data of the constituent videos, and processes the input data of the target video as data of the target video. Such a technology may be any technology as long as it can process the input data of the videos as data of the constituent videos, and process the data of the target video as data of the target video. For example, the data of each video may include information indicating whether it is data of a constituent video or data of the target video. In such a case, the estimation device 1 performs a pre-estimation process on the data of the constituent videos, and performs a main estimation process on the data of the target video, based on information indicating whether each piece of input data of a video is data of a constituent video or data of the target video.

[0042] For example, information indicating that input of constituent moving image data has been completed may be input to the estimation device 1. In such a case, the estimation device 1 executes a preliminary estimation process until information indicating that input of constituent moving image data has been completed is input, and executes a main estimation process after information indicating that input of constituent moving image data has been completed is input.

[0043] For example, information indicating the start of input of data of the target moving image may be input to the estimation device 1. In such a case, the estimation device 1 executes the prior estimation process until the information indicating the start of input of data of the target moving image is input, and executes the main estimation process after the information indicating the start of input of data of the target moving image is input.

[0044] For example, the estimation device 1 does not execute the main estimation process from when input of constituent moving image data starts until a pre-determined number of moving image data, such as one, is subjected to a pre-determined estimation process, and executes the main estimation process on moving image data input after the pre-determined number of moving image data is subjected to a pre-determined estimation process. Specifically, the estimation device 1 executes the main estimation process on moving image data input after a pre-determined estimation process is performed on data of one moving image, for example.

[0045] 3 is a diagram illustrating an example of a hardware configuration of the estimation device 1 according to an embodiment. The estimation device 1 includes a control unit 11 including a processor 91, such as a CPU (Central Processing Unit), and a memory 92, which are connected via a bus, and executes a program. The estimation device 1 functions as a device including the control unit 11, an input unit 12, a communication unit 13, a storage unit 14, and an output unit 15 by executing the program.

[0046] More specifically, in the estimation device 1, the processor 91 reads a program stored in the storage unit 14 and stores the read program in the memory 92. The processor 91 executes the program stored in the memory 92, whereby the estimation device 1 functions as a device including a control unit 11, an input unit 12, a communication unit 13, a storage unit 14, and an output unit 15.

[0047] The control unit 11 controls the operations of various functional units included in the estimation device 1. The control unit 11 executes, for example, a preliminary estimation process. The control unit 11 executes, for example, a main estimation process.

[0048] The input unit 12 includes input devices such as a mouse, a keyboard, a touch panel, etc. The input unit 12 may include an interface that connects these input devices to the estimation device 1.

[0049] The communication unit 13 includes an interface for connecting the estimation device 1 to an external device. The communication unit 13 communicates with the external device via wired or wireless communication. The external device is, for example, a device that transmits the constituent moving images. The communication unit 13 acquires a pair of data of the constituent moving images and the correct answer information by communicating with a device that transmits the pair of data of the constituent moving images and the correct answer information. The external device is, for example, a device that transmits data of a moving image capturing the motion of the estimation target. The communication unit 13 acquires data of the moving image capturing the motion of the estimation target by communicating with a device that transmits data of the moving image capturing the motion of the estimation target.

[0050] The storage unit 14 is configured using a computer-readable storage medium device such as a magnetic hard disk device or a semiconductor storage device. The storage unit 14 stores various information related to the estimation device 1. The storage unit 14 stores, for example, various information generated as a result of processing executed by the control unit 11. The storage unit 14 stores, for example, M types of initial influence functions in advance.

[0051] The output unit 15 includes a display device such as a CRT (Cathode Ray Tube) display, a liquid crystal display, an organic EL (Electro-Luminescence) display, etc. The output unit 15 may include an interface that connects these display devices to the estimation device 1.

[0052] 4 is a diagram illustrating an example of a functional configuration of the control unit 11 included in the estimation device 1 of the embodiment. The control unit 11 includes a video data acquisition unit 111, an operation mode determination unit 112, an image set generation processing unit 113, an estimation model execution unit 114, a coincidence acquisition unit 115, an estimation influence acquisition unit 116, a main estimation process execution unit 117, a communication control unit 118, a memory control unit 119, and an output control unit 120.

[0053] The moving image data acquisition unit 111 acquires moving image data input to the communication unit 13. The candidates for moving image data acquired by the moving image data acquisition unit 111 are specifically data of constituent moving images and data of a target moving image.

[0054] The operation mode determination unit 112 determines whether to execute a prior estimation process or a main estimation process on the moving image data acquired by the moving image data acquisition unit 111. The operation mode determination unit 112 determines, for example, whether or not a condition for executing the main estimation process (hereinafter referred to as a "main estimation process execution condition") is satisfied.

[0055] When the operation mode determination unit 112 determines that the main estimation process execution condition is satisfied, the estimation device 1 executes the main estimation process. On the other hand, when the operation mode determination unit 112 does not determine that the main estimation process execution condition is satisfied, the estimation device 1 executes the pre-estimation process.

[0056] The main estimation process execution condition may be, for example, a condition that the video data acquired by the video data acquisition unit 111 includes information indicating that the data is data of the target video. The main estimation process execution condition may be, for example, a condition that information indicating that input of data of the constituent video has ended has been input to a predetermined interface such as the input unit 12 or the communication unit 13. The main estimation process execution condition may be, for example, a condition that information indicating that input of data of the target video has started has been input to a predetermined interface such as the input unit 12 or the communication unit 13. The main estimation process execution condition may be, for example, a condition that pre-estimation process has been executed on data of a predetermined number of videos.

[0057] The image set generation processing unit 113 executes an image set generation process on the data of the constituent moving images acquired by the moving image data acquisition unit 111. By executing the image set generation process, image data of M nonlinear motion history images is generated.

[0058] The estimation model execution unit 114 executes the estimation model for each of the image data of the M nonlinear motion history images generated by the image set generation processing unit 113.

[0059] The matching degree acquiring unit 115 acquires an estimated matching degree, which is the degree of matching between each of the M movement patterns estimated by the estimation model executing unit 114 and the movement patterns captured in the constituent moving images.

[0060] The estimated influence acquisition unit 116 acquires the estimated degree of match obtained using an initial influence function having a peak at the animation time i (i is a real number) as the estimated influence of a frame at animation time i of the estimation target animation. That is, the estimated influence acquisition unit 116 records the estimated degree of match obtained using an initial influence function having a peak at the animation time i (i is a real number) in a predetermined storage device such as the memory unit 14 as the estimated influence of a frame at animation time i of the estimation target animation.

[0061] The main estimation process execution unit 117 estimates a pattern of a motion of the estimation target appearing in the target video indicated by the data of the target video acquired by the video data acquisition unit 111, using the estimated influence acquired by the estimated influence acquisition unit 116. The main estimation process execution unit 117 includes, for example, a target image generation unit 171 and a motion pattern estimation unit 172. The target image generation unit 171 generates a target image that is an image in which each frame of the video of the estimation target is superimposed, using the estimated influence acquired by the estimated influence acquisition unit 116 as a weight. The motion pattern estimation unit 172 estimates a motion pattern appearing in the target video, based on the target image generated by the target image generation unit 171. That is, the motion pattern estimation unit 172 estimates a pattern of a motion of the estimation target, based on the target image generated by the target image generation unit 171.

[0062] The communication control unit 118 controls the operation of the communication unit 13. The memory control unit 119 controls the operation of the memory unit 14. The output control unit 120 controls the operation of the output unit 15.

[0063] 5 is a flowchart showing an example of a flow of processing executed by the estimation device 1 in the embodiment. The video data acquisition unit 111 acquires video data (step S101). Next, the operation mode determination unit 112 determines whether or not a main estimation process execution condition is satisfied (step S102).

[0064] If the operation mode determination unit 112 determines that the main estimation process execution condition is not satisfied (step S102: NO), the image set generation processing unit 113 executes an image set generation process on the video data acquired by the video data acquisition unit 111 (step S103). Next, the estimation model execution unit 114 executes an estimation model on each of the image data of the M nonlinear motion history images generated by the image set generation processing unit 113 (step S104).

[0065] Next, the matching degree acquiring unit 115 acquires an estimated matching degree, which is the degree of matching between each of the M movement patterns estimated by the estimation model executing unit 114 and the movement patterns captured in the constituent moving images (step S105).

[0066] Next, the estimated influence acquisition unit 116 acquires the estimated degree of match obtained using an initial influence function whose peak is at video time i (i is a real number) as the estimated influence of the frame at video time i of the video to be estimated (step S106).

[0067] After step S106, the process returns to step S101.

[0068] On the other hand, if the operation mode determination unit 112 determines that the main estimation process execution condition is satisfied (step S102: YES), the main estimation process execution unit 117 estimates a pattern of movements that will appear in the video indicated by the video data acquired by the video data acquisition unit 111 (step S107). More specifically, the main estimation process execution unit 117 uses the estimated influence obtained in step S106 to estimate a pattern of movements that will appear in the video indicated by the video data acquired by the video data acquisition unit 111. The pattern of movements estimated by the main estimation process execution unit 117 is the pattern of movements to be estimated.

[0069] The estimation device 1 configured in this way obtains an estimated matching degree before executing the main estimation process by using constituent videos, which are videos capturing movements whose desired results are the same as the movement of the estimation target, and a plurality of types of initial influence functions. Therefore, the estimation device 1 can estimate a movement pattern by placing emphasis on frames that have a stronger influence on the estimation result of the movement pattern according to the movement of the estimation target. Therefore, the estimation device 1 can improve the accuracy of the estimation of the movement pattern.

[0070] Furthermore, the estimation device 1 configured in this manner acquires the estimated influence of a frame at a video time i of the video to be estimated based on image data of M nonlinear motion history images, not manually. Therefore, the estimation device 1 can reduce the effort required to estimate a motion pattern, compared to a case where the estimated influence is estimated manually.

[0071] (Modification) Note that the estimation model executed by the estimation model executing unit 114 may be, for example, a trained estimation model obtained by learning using the M nonlinear motion history images generated by the image set generation processing unit 113. The learning using the M nonlinear motion history images generated by the image set generation processing unit 113 is executed by, for example, the estimation model executing unit 114.

[0072] The main inference process execution unit 117 may infer a pattern of a motion of an estimation target using the estimation model executed by the estimation model execution unit 114. That is, when the estimation model executed by the estimation model execution unit 114 is a trained estimation model, the main inference process execution unit 117 may infer a pattern of a motion of an estimation target using the trained estimation model. Also, when the estimation model executed by the estimation model execution unit 114 is a non-trained estimation model, the main inference process execution unit 117 may infer a pattern of a motion of an estimation target using the non-trained estimation model.

[0073] When the estimation model executed by the estimation model executing unit 114 is a trained estimation model, the main estimation process executing unit 117 may estimate a pattern of the motion of the estimation target using the estimation model before it is updated by learning.

[0074] The estimation device 1 may be implemented using a plurality of information processing devices communicably connected via a network. For example, the estimation device 1 may be implemented as a device including a pre-estimation processing execution device that executes a pre-estimation processing and a main estimation processing execution unit that executes a main estimation processing.

[0075] All or part of the functions of the estimation device 1 may be realized using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The program may be recorded on a computer-readable recording medium. Examples of computer-readable recording media include portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into computer systems. The program may be transmitted via a telecommunications line.

[0076] Although an embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and designs that do not deviate from the gist of the present invention are also included. [Explanation of symbols]

[0077] 1...estimation device, 11...control unit, 12...input unit, 13...communication unit, 14...storage unit, 15...output unit, 111...video data acquisition unit, 112...operation mode determination unit, 113...image set generation processing unit, 114...estimation model execution unit, 115...match degree acquisition unit, 116...estimated influence degree acquisition unit, 117...main estimation process execution unit, 118...communication control unit, 119...storage control unit, 120...output control unit, 171...target image generation unit, 172...operation pattern estimation unit, 91...processor, 92...memory

Claims

1. An estimation device that estimates a pattern of a motion of an estimation target, an image set generation processing unit that generates image data of a nonlinear motion history image, which is an image in which each frame of the constituent videos is superimposed with the estimated influence as a weight, for each initial influence function, based on data of constituent videos, which are videos capturing movements whose desired result is the same as the movement to be estimated, and M types of initial influence functions (M is an integer of 2 or more) that are predetermined functions whose independent variable is video time, which is the elapsed time from the start of the video, and whose dependent variable is an estimated influence indicating the strength of the influence on the result of the estimation, and whose graph has one peak; an estimation model execution unit that executes an estimation model for estimating a pattern of a movement appearing in a nonlinear motion history image to be processed for each of the image data of the M nonlinear motion history images generated by the image set generation processing unit; a matching degree acquisition unit that acquires a degree of matching between each of the M movement patterns estimated by the estimation model execution unit and a movement pattern captured in the constituent moving image; an estimated influence acquisition unit that acquires the degree obtained by using the initial influence function in which the video time giving the peak is i (i is a real number) as an estimated influence of a frame in the video time i of the video to be estimated; a main estimation process execution unit that estimates a pattern of the motion of the estimation target using the estimated influence degree acquired by the estimated influence degree acquisition unit; An estimation device comprising:

2. The main estimation process execution unit, a target image generating unit that generates a target image in which each frame of the moving image to be estimated is superimposed on the estimated influence acquired by the estimated influence acquiring unit as a weight; A movement pattern estimation unit that estimates a pattern of the movement of the estimation target based on the target image; The estimation device according to claim 1 , comprising:

3. the estimation model executed by the estimation model execution unit is a trained estimation model trained using the M nonlinear motion history images generated by the image set generation processing unit; The estimation device according to claim 1 or 2.

4. the main estimation process execution unit estimates a pattern of the motion of the estimation target using the estimation model executed by the estimation model execution unit; The estimation device according to any one of claims 1 to 3.

5. when the estimation model executed by the estimation model executing unit is a trained estimation model updated by learning using the M nonlinear motion history images generated by the image set generation processing unit, the main estimation process executing unit estimates a pattern of the motion of the estimation target using the estimation model before the update by the learning. The estimation device according to any one of claims 1 to 3.

6. An estimation method for estimating a pattern of a motion of an estimation target, comprising: an image set generation processing step for generating image data of a nonlinear motion history image, which is an image in which each frame of the constituent videos is superimposed with the estimated influence as a weight, for each initial influence function, based on data of constituent videos, which are videos capturing movements whose desired result is the same as the movement to be estimated, and M types of initial influence functions (M is an integer of 2 or more) which are predetermined functions whose independent variable is the video time, which is the elapsed time from the start of the video, and whose dependent variable is an estimated influence indicating the strength of the influence on the result of the estimation, and whose graph has one peak; an estimation model execution step of executing an estimation model for estimating a pattern of a movement appearing in a nonlinear motion history image to be processed for each of the image data of the M nonlinear motion history images generated in the image set generation processing step; a matching degree acquisition step of acquiring a degree of matching between each of the M movement patterns estimated by the estimation model execution step and a movement pattern captured in the constituent moving image; an estimated influence acquisition step of acquiring the degree obtained by using the initial influence function in which the motion time giving the peak is i (i is a real number) as an estimated influence of a frame in the motion time i of the motion image to be estimated; a main estimation process execution step of estimating a pattern of the motion of the estimation target using the estimated influence degree acquired in the estimated influence degree acquisition step; The estimation method has the following structure:

7. A program for causing a computer to function as the estimation device according to any one of claims 1 to 5.

Images