Image evaluation device, method, and program
The image evaluation device addresses the challenge of accurately assessing human motion quality by isolating the person's region in videos and refining model parameters, enabling effective scoring of human actions across different backgrounds and improving feature extraction for human motion.
Patent Information
- Application Number
- PCT/JP2023/020354
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-31
- Publication Date
- 2024-12-05
AI Technical Summary
Conventional image evaluation methods for human motion in videos fail to accurately assess motion quality due to the lack of explicit feature extraction for human motion, as they rely on entire video images and are influenced by background differences, leading to inappropriate evaluation of operation quality.
An image evaluation device and method that inputs moving images of human actions into a model, applies masking to isolate the person's region, and updates parameters to focus on the differences in evaluation scores between individuals, using a combination of original and masked videos to learn and refine the model's feature extraction for human motion.
Enables accurate evaluation of human motion quality by isolating the person's region and refining the model's parameters, allowing for appropriate scoring of human actions in diverse backgrounds and improving generalization to human movements.
Smart Images

Figure JP2023020354_05122024_PF_FP_ABST
Abstract
Description
Image evaluation device, method, and program
[0001] FIELD Embodiments of the present invention relate to an image evaluation device, method, and program.
[0002] Conventionally, there has been a method called Action Quality Assessment (AQA) for automatically evaluating the quality of human actions in video (images), such as in judged competitions or simulated surgeries performed by medical students. In this method, a score indicating the quality of the action is predicted using a model using a deep neural network (DNN) or the like. Furthermore, parameters of the model are trained so that the true score can be predicted based on the difference between the predicted score and the true score.
[0003] Furthermore, as disclosed in Non-Patent Document 1, for example, there is a technology that focuses on subtle differences in quality between similar movements by inputting two movements with different scores, i.e., the same technique, and learning to predict the difference in score.
[0004] In this technology, for example, input data is input into a model to obtain an estimated score, a loss based on this estimated score and the true score, in this case the MSE (Mean Squared Error) between the estimated score and the true score, is calculated, and the model parameters are updated using backpropagation.
[0005] Yu et al., “Group-aware Contrastive Regression for Action Quality Assessment”, In ICCV. 2021.
[0006] However, in the conventional methods described above, features related to human movements, such as joint coordinates, are not explicitly given, and only the entire video image is used as learning data. As a result, features unrelated to human movements, such as differences in the background of a sports venue, are extracted, making it impossible to generalize to human movements.
[0007] For example, when a first original video of a scene including human movement and a second original video of a different scene are input into individual models that share parameter weighting and the difference in scores according to each original video is predicted, if the backgrounds of the first original video and the second original video are different, the difference in the scores may be predicted by focusing on this difference in background, and the quality of human movement cannot be properly evaluated.
[0008] The present invention has been made in light of the above circumstances, and its purpose is to provide an image evaluation device, method, and program that can appropriately evaluate the quality of human movements in moving images.
[0009] An image evaluation device according to one aspect of the present invention includes a first update unit that inputs a video relating to the movements of a first person and a video relating to the movements of a second person into a model that determines features of each video, calculates an estimate of the difference between an evaluation score for the first person's movement in the video relating to the movements of the first person and an evaluation score for the second person's movement in the video relating to the movements of the second person based on the features output from the model, and updates parameters of the model so that the calculated estimate approaches a true value; and a second update unit that inputs into the model each of the video relating to the movements of the first person and the video relating to the movements of the second person when a mask process is applied to a person area for at least one of the video relating to the movements of the first person and the video relating to the movements of the second person, and calculates an estimate of the difference between the evaluation score for the first person's movement in the video relating to the output model based on the features output from the model based on this input, and updates parameters of the model so that the calculated estimate approaches a true value.
[0010] An image evaluation method according to one aspect of the present invention is a method performed by an image evaluation device, comprising: a first update unit of the image evaluation device inputting a video relating to the actions of a first person and a video relating to the actions of a second person into a model that determines features of each video; calculating, based on the features output from the model, an estimate of the difference between the evaluation score of the first person's action in the video relating to the actions of the first person and the evaluation score of the second person's action in the video relating to the actions of the second person; and updating parameters of the model so that this calculated estimate approaches a true value; and a second update unit of the image evaluation device inputting, into the model, each of the video relating to the actions of the first person and the video relating to the actions of the second person, after masking a person-related area for at least one of the video relating to the actions of the first person and the video relating to the actions of the second person; calculating, based on the features output from the model as a result of this input, an estimate of the difference between the evaluation score of the first person's action and the evaluation score of the second person's action in the video relating to the output model; and updating parameters of the model so that this calculated estimate approaches a true value.
[0011] According to the present invention, the quality of human movements in a moving image can be appropriately evaluated.
[0012] FIG. 1 is a diagram showing an application example of an image evaluation device according to an embodiment of the present invention. FIG. 2 is a diagram showing an example of an original image. FIG. 3 is a diagram showing an example of an athlete's mask image. FIG. 4 is a diagram showing an example of a learning process for a first pattern by a learning unit. FIG. 5 is a diagram showing an example of a learning process for a second pattern by a learning unit. FIG. 6 is a diagram showing an example of a learning process for a third pattern by a learning unit. FIG. 7 is a block diagram showing an example of the hardware configuration of an image evaluation device according to an embodiment of the present invention.
[0013] An embodiment of the present invention will be described below with reference to the drawings. FIG. 1 is a diagram illustrating an application example of an image evaluation device according to an embodiment of the present invention. As shown in FIG. 1, an image evaluation device (video evaluation device) 100 according to an embodiment of the present invention includes a learning processing unit 10 and an estimation processing unit 20. The learning processing unit 10 includes an input unit 11, a learning unit (update unit) 12, and a learning model DB (database) 13. The estimation processing unit 20 includes an input unit 21, an estimation unit 22, and a learning model DB 23. Note that the learning model DB 13 of the learning processing unit 10 and the learning model DB 23 of the estimation processing unit 20 store models used to extract features of multiple moving images including human movements. These may also be shared, i.e., a single model DB.
[0014] The input unit 21 of the estimation processing unit 20 inputs a plurality of moving images including human movements. The estimation unit 22 inputs the input moving images into a model stored in a learning model DB 23, and estimates an evaluation score of the human movements in the moving images, which is an output from the model.
[0015] The input unit 21 of the estimation processing unit 20 inputs a plurality of videos including human movements for the purpose of learning the parameters of the model stored in the learning model DB 13, and also inputs the true values of the scores of the human movements in the videos. The learning unit 12 inputs the input videos into the model stored in the learning model DB 13 and estimates the scores of the human movements in the videos, which are output from the model. Then, based on the difference between the estimated score and the true value of the score, the learning unit 12 updates the parameters of the model so that this difference approaches zero or becomes zero, and stores the updated model in the learning model DB 23 of the estimation processing unit 20.
[0016] In this embodiment, the model parameters are learned using two types of video: an original video and an athlete mask video in which the area showing the human movements in the original video, in this case the athlete's movements, is masked.
[0017] 2 is a diagram showing an example of an original video. In this embodiment, original video A and original video B shown in FIG. 2 can be used as the original video for learning the parameters of the model.
[0018] Original video A is an original video showing the movements of high diving athlete A in a swimming competition, i.e., the athlete jumping from a high diving board into the water. Original video B is an original video showing the movements of high diving athlete B, but the athlete and the timing of the competition are different from those in original video A. The movements of the athletes in original video A and original video B are assumed to be movements related to the execution of the same type of high diving technique.
[0019] 3 is a diagram showing an example of an athlete mask image. In this embodiment, athlete mask image A and athlete mask image B shown in FIG. 3 can be used as the athlete mask image for learning the parameters of the model.
[0020] Athlete mask image A is an image generated externally or within the learning processing unit 10 by masking the area of athlete A in the original image A by filling in a rectangular area. Athlete mask image B is an image generated externally or within the learning processing unit 10 by masking the area of athlete B in the original image B.
[0021] The input unit 11 of the learning processing unit 10 inputs original video A, athlete mask video A, original video B, athlete mask video B, the true value of competition score A, and the true value of competition score B, and passes them to the learning unit 12. The true value of competition score A is the true value of the score of evaluation of the athlete's movements in original video A. The true value of competition score B is the true value of the score of evaluation of the athlete's movements in original video B. The number of athletes in original videos A and B may be one or more. For example, the original video may be video showing multiple athletes, such as synchronized swimming.
[0022] The method for acquiring the athlete's area in generating the athlete mask images A and B is not particularly limited. For example, the mask images may be generated automatically using the technology disclosed in "K. He et al. Mask R-CNN. In ICCV, 2017." Alternatively, the mask images may be generated manually by visually checking all frames of the original video. Furthermore, the method for assigning scores for the evaluation is not particularly limited. For example, the scores given by the judges for the movements in the video may be used.
[0023] The following describes the learning process performed by the learning unit 12. In this embodiment, the learning unit 12 learns the model parameters multiple times by performing the following first pattern learning process, second pattern learning process, and third pattern learning process.
[0024] 4 is a diagram showing an example of a first pattern of learning processing by the learning unit. In this first pattern of learning processing, the learning unit 12 inputs original videos A and B, i.e., multiple original videos, into a model, as in conventional methods, and learns model parameters so as to appropriately predict the difference between the competition scores output from the model. This difference is the difference between competition score A, which is a predicted value of the score for evaluating the athlete's movements in original video A, and competition score B, which is a predicted value of the score for evaluating the athlete's movements in original video B.
[0025] 4, in this embodiment, the model stored in the learning model DB 13 includes a first DNN and a second DNN. The first DNN can input a first image, and the second DNN can input a second image. The model parameters include weightings between nodes of the first DNN and weightings between nodes of the second DNN, and these weightings are shared with each other.
[0026] In the example shown in FIG. 4, the first DNN outputs a feature f 1 is output, and the second DNN outputs a feature f 2These features are input to layers in the learning unit 12, and a predicted value of the difference between competition score A and competition score B is output from these layers. Then, in the learning process of the first pattern, the model parameters are learned so that this predicted value approaches or becomes the same as the difference between the true value of competition score A and the true value of competition score B.
[0027] 5 is a diagram showing an example of a second pattern of learning processing by the learning unit 12. In this second pattern of learning processing, the learning unit 12 inputs athlete mask image A and athlete mask image B, i.e., multiple athlete mask images, into the model, and learns the model parameters so that the difference between the competition scores is predicted to be 0 points, assuming that the score is 0 when the athlete is not visible.
[0028] In the example shown in FIG. 5, the first DNN outputs a feature f 1 is output, and the second DNN outputs a feature f 2 These features are input to layers in the learning unit 12, and a predicted value of the difference in the competition score is output from these layers. Then, in the second pattern of learning processing, the model parameters are learned so that this predicted value approaches or becomes 0 points, which is the true value of the competition score.
[0029] 6 is a diagram showing an example of a third pattern of learning processing by the learning unit 12. In this third pattern of learning processing, the learning unit 12 inputs a combination of original video A and athlete mask video B, or a combination of original video B and athlete mask video A, and learns model parameters so that the competition score of the input original video can be appropriately predicted as the difference between the competition scores, assuming that the competition score is 0 if the athlete is not visible.
[0030] In the example shown in FIG. 6, a combination of original video A and athlete mask video B, or a combination of original video B and athlete mask video A, is input, and a feature f 1 is output, and the second DNN outputs a feature f 2These features are input to layers in the learning unit 12, and a predicted value of the difference in the competition score is output from these layers. Then, in the learning process of the third pattern, the model parameters are learned so that this predicted value approaches or becomes the same as the true value of the competition score for the athlete's action in the original video A.
[0031] In any of the first, second, and third patterns of training processes, the structures of the DNN and layers are not limited. Furthermore, the first and second DNNs may be, for example, the I3D (Two-Stream Inflated 3D Convnet) disclosed in "Carreira et al. "Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset.", In CVPR 2017."
[0032] Furthermore, the layers may be multiple fully connected layers, or the model structure disclosed in the above-mentioned non-patent document 1 may be applied.
[0033] In the learning process, a loss function is calculated from the difference between the predicted competition score and the true competition score, and the network model is learned by error backpropagation. The weighting of the parameters of this learned model is stored in the learning model DB 13 and also in the learning model DB 23 of the estimation processing unit 20. The type of loss function is not particularly limited. For example, the L1 distance, the L2 distance, or the sum of these may be used as the loss function.
[0034] The order in which the learning processes using the first, second, and third patterns are executed is not particularly limited. Furthermore, the timing of updating the weighting of parameters in the learning processes using the various patterns is not particularly limited. For example, the combination may be changed depending on the number of learning epochs, i.e., the number of times learning is performed. For example, in the first 50 epochs, learning may be performed using the learning process using the first pattern to stabilize learning, and then in the following 50 epochs, learning may be performed using all of the learning processes using the first, second, and third patterns.
[0035] Alternatively, the type of learning process pattern may be always randomly adopted throughout the epochs. For example, all of the first, second, and third patterns may be learned in the first epoch, or one of the patterns may be used for each epoch, such as using the first pattern in the first epoch and the second pattern in the second epoch.
[0036] Next, the estimation process by the estimation processing unit 20 after the above learning process is performed will be described. In this estimation process, a combination of original video A and original video B is input to the input unit 21, and the estimation unit 22 inputs this input video into a model stored in the learning model DB 23. A feature f, which is a feature of original video A, is obtained from the first DNN. 1 is output, and the second DNN outputs the feature f 2 These features are input to layers in the estimation unit 22, and a predicted value of the difference in the competition score is output from these layers.
[0037] As described above, in the video evaluation device according to this embodiment, the learning process using the athlete mask video promotes the extraction of features in the video related to human movements, thereby achieving the effect of generalizing to human movements, and enabling accurate scoring in sports competitions, for example. Furthermore, in the above embodiment, an example was shown in which model parameters were learned by executing the learning process using all of the first, second, and third patterns. However, this is not limited to this. Learning processes using some of the first, second, and third patterns, which always include the learning process of the first pattern, i.e., executing the learning process of the first pattern and either the learning process of the second or third pattern, can also be used to learn model parameters. The reason for always including the learning process of the first pattern, as described above, is that the estimation process inputs a combination of original videos and outputs a predicted value, preventing a decrease in estimation accuracy due to a discrepancy in input content between the learning process and the estimation process. Furthermore, from the viewpoint of the effects produced as described above, it goes without saying that the accuracy of learning the model parameters when the learning process is performed using all of the first, second and third patterns as described above is higher than the accuracy of learning the model parameters when the learning process is performed using only some of the first, second and third patterns as described above.
[0038] 7 is a block diagram showing an example of the hardware configuration of an image evaluation device according to an embodiment of the present invention. In the example shown in FIG. 7, the image evaluation device 100 according to the embodiment is configured, for example, by a server computer or a personal computer, and has a hardware processor 111A such as a CPU (Central Processing Unit). A program memory 111B, a data memory 112, an input / output interface 113, and a communication interface 114 are connected to this hardware processor 111A via a bus 115.
[0039] The communication interface 114 includes, for example, one or more wireless communication interface units, and enables transmission and reception of information to and from the communication network NW. As the wireless interface, for example, an interface that adopts a low-power wireless data communication standard such as a wireless LAN (Local Area Network) is used.
[0040] An input device 200 and an output device 300, which are attached to the image evaluation apparatus 100 and used by a user or the like, are connected to the input / output interface 113. The input / output interface 113 can take in operation data input by a user or the like through the input device 200, such as a keyboard, a touch panel, a touchpad, or a mouse, and can output output data to an output device 300, which includes a display device using a liquid crystal or an organic electroluminescence (EL) display, for display. The input device 200 and the output device 300 may be devices built into the image evaluation apparatus 100, or may be input devices and output devices of other information terminals that can communicate with the image evaluation apparatus 100 via a network NW.
[0041] The program memory 111B is a non-transitory tangible storage medium that is a combination of a non-volatile memory that can be written to and read from at any time, such as a hard disk drive (HDD) or a solid state drive (SSD), and a non-volatile memory such as a ROM, and can store programs necessary to execute various control processes, etc., according to one embodiment.
[0042] The data memory 112 is a tangible storage medium, for example, a combination of the above-mentioned nonvolatile memory and a volatile memory such as RAM, and can be used to store various data or information acquired and created during various processes.
[0043] The image evaluation device 100 according to one embodiment of the present invention can be configured as a data processing device having the units shown in FIG. 1 as software-based processing function units.
[0044] The information storage unit used as a work memory or the like by each unit of the image evaluation device 100 can be configured by using the data memory 112 shown in Fig. 7. However, these configured storage areas are not essential components within the image evaluation device 100, and may be areas provided in an external storage medium such as a USB (Universal Serial Bus) memory, or in a storage device such as a database server located in the cloud.
[0045] The processing function units in each of the above units can be realized by reading and executing a program stored in the program memory 111B by the hardware processor 111A. Note that some or all of these processing function units may be realized in various other forms, including integrated circuits such as an application specific integrated circuit (ASIC) or a field-programmable gate array (FPGA).
[0046] The methods described in each embodiment may be stored as a program (software means) that can be executed by a computer on a recording medium such as a magnetic disk (e.g., a floppy disk, a hard disk, etc.), an optical disk (e.g., a CD-ROM, a DVD, an MO, etc.), or a semiconductor memory (e.g., a ROM, a RAM, a flash memory, etc.), or may be transmitted and distributed via a communication medium. The program stored on the medium also includes a configuration program that configures the software means (including not only executable programs but also tables and data structures) that the computer executes. The computer that realizes this device reads the program stored on the recording medium and, in some cases, configures the software means using the configuration program, and executes the above-described processing by having the operation controlled by this software means. The term "recording medium" as used herein is not limited to a storage medium for distribution, but also includes a storage medium such as a magnetic disk or semiconductor memory installed inside the computer or in a device connected via a network.
[0047] The present invention is not limited to the above-described embodiments, and various modifications can be made in the implementation stage without departing from the spirit of the invention. Furthermore, the embodiments may be implemented in appropriate combinations, in which case the combined effects can be obtained. Furthermore, the above-described embodiments include various inventions, and various inventions can be extracted by combining selected elements from the disclosed elements. For example, if the problem can be solved and the desired effect can be obtained even if some elements are deleted from all elements shown in the embodiments, the configuration from which these elements are deleted can be extracted as an invention.
[0048] REFERENCE SIGNS LIST 100... image evaluation device 10... learning processing unit 11, 21... input unit 12... learning unit 13, 23... learning model DB (database) 20... estimation processing unit 22... estimation unit
Claims
1. An image evaluation device comprising: a first update unit that inputs a video relating to a first person's movement and a video relating to a second person's movement into a model that determines features of each of the video images, calculates an estimate of the difference between an evaluation score of the first person's movement in the video relating to the first person's movement and an evaluation score of the second person's movement in the video relating to the second person's movement based on the features output from the model, and updates parameters of the model so that the calculated estimate approaches a true value; and a second update unit that inputs into the model each of the video images when a mask process is applied to a person-related area for at least one of the video images relating to the first person's movement and the video images relating to the second person's movement, and calculates an estimate of the difference between the evaluation score of the first person's movement and the evaluation score of the second person's movement in the video relating to the output model based on the features output from the model in response to the input, and updates parameters of the model so that the calculated estimate approaches a true value.
2. The image evaluation device of claim 1, wherein the second update unit performs at least one of a first update process in which the model receives an input of each of the videos relating to the movements of the first person and the videos relating to the movements of the second person, when the masking process has been applied to both of them, calculates an estimate of the difference between the evaluation score of the first person's movement and the evaluation score of the second person's movement in the video relating to the output model based on features output from the model in response to this input, and updates parameters of the model so that this calculated estimate approaches zero, and a second update process in which the model receives an input of each of the videos relating to the movements of the first person and the videos relating to the movements of the second person, when the masking process has been applied to one of them, calculates an estimate of the difference between the evaluation score of the first person's movement and the evaluation score of the second person's movement in the video relating to the output model based on features output from the model in response to this input, and updates parameters of the model so that this calculated estimate approaches a true value of the evaluation score of the person's movement in the video not subjected to the masking process among the respective videos.
3. A method performed by an image evaluation device, comprising: a first update unit of the image evaluation device inputs a video relating to a first person's movement and a video relating to a second person's movement into a model that determines features of each of the video, calculates an estimate of the difference between the evaluation score of the first person's movement in the video relating to the first person's movement and the evaluation score of the second person's movement in the video relating to the second person's movement based on the features output from the model, and updates parameters of the model so that this calculated estimate approaches a true value; and a second update unit of the image evaluation device inputs into the model each of the video relating to the first person's movement and the video relating to the second person's movement when a mask process is applied to a person area for at least one of the video relating to the first person's movement and the video relating to the second person's movement, and calculates an estimate of the difference between the evaluation score of the first person's movement and the evaluation score of the second person's movement in the video relating to the output model based on the features output from the model in response to this input, and updates parameters of the model so that this calculated estimate approaches a true value.
4. The image evaluation method of claim 3, wherein the second updating unit performs at least one of a first updating process in which the video relating to the movement of the first person and the video relating to the movement of the second person are input to the model when the masking process is applied to both of them, calculates an estimate of the difference between the evaluation score of the movement of the first person and the evaluation score of the movement of the second person in the video relating to the output model based on features output from the model in response to this input, and updates parameters of the model so that this calculated estimate approaches zero, and a second updating process in which the video relating to the movement of the first person and the video relating to the movement of the second person are input to the model when the masking process is applied to one of them, calculates an estimate of the difference between the evaluation score of the movement of the first person and the evaluation score of the movement of the second person in the video relating to the output model based on features output from the model in response to this input, and updates parameters of the model so that this calculated estimate approaches a true value of the evaluation score of the person's movement in the video not subjected to the masking process among the respective videos.
5. An image evaluation processing program that causes a processor to function as each unit of the image evaluation device according to claim 1 or 2.
Citation Information
Patent Citations
Learning device, estimation device, learning model data generation method, estimation method, and program
WO2022244135A1
