Machine learning device, skill assessment device, machine learning method, and machine learning program

Abstract

The machine learning device (1) has a data priority selection unit (101) that selects multiple video pairs from a learning video dataset and selects an image frame to be used for determining the superiority or inferiority of a skill level from each of the selected multiple video pairs, a focus area generation unit (103) that generates a focus area to be used for determining the superiority or inferiority of a skill level in an image frame, a superiority or inferiority judgment unit (106) that judges the superiority or inferiority of the skill level in the focus area using the learning model, and a model learning unit (102) that updates the learning model based on the result of the judgment of the superiority or inferiority of the skill level, and the data priority selection unit (101) selects an image frame to be used for determining the superiority or inferiority of a skill level using a selection probability determined based on one or more of user editing in the image frames that constitute each video pair, time-directional features in the multiple image frames that constitute each video pair, and similarity between the image frames that constitute each video pair.

Description

[Technical field]

[0001] The present disclosure relates to a machine learning device, a skill assessment device, a machine learning method, and a machine learning program. [Background technology]

[0002] Pairwise Deep Ranking (PDR) has been proposed. This technology is one of the methods for skill assessment, which calculates a score for the skill level (i.e., proficiency) of a person's behavior and judges the superiority or inferiority of the skill (for example, see Non-Patent Document 1). [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Masayuki Takada and 3 others (Chubu University), "Visual explanation and high accuracy in skill superiority judgment using Attention Pairwise Ranking," 23rd Image Recognition and Understanding Symposium Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the above conventional technology, the only information given is the superiority or inferiority of the skill and the video pair, and the superiority or inferiority of the skill may be judged based on a location other than the location that should be focused on to judge superiority or inferiority (i.e., a biased area of ​​focus in the video pair). For example, in evaluating the skill of drawing a picture, the superiority or inferiority of the skill may be judged based on a video during the period in which the head of the person moving the pen is shown, rather than the video during the period in which the pen is moving. For this reason, the conventional technology has a problem in that the learning behavior may become unstable.

[0005] The present disclosure aims to stabilize learning behavior when training a learning model for inferring the skill level of an actor's behavior in a video. [Means for solving the problem]

[0006] The machine learning device of the present disclosure is a device that learns a learning model for inferring the skill level of the behavior of an actor in a video, and includes a data priority selection unit that selects a plurality of video pairs from a learning video dataset and selects an image frame to be used for determining the superiority or inferiority of the skill level from each video pair that constitutes the selected plurality of video pairs; a focus area generation unit that generates a focus area in the image frame to be used for determining the superiority or inferiority of the skill level; a superiority / inferiority judgment unit that uses the learning model to judge the superiority or inferiority of the skill level in the focus area for each video pair; and a model learning unit that stores the learning model and updates the learning model based on the results of the judgment of the superiority or inferiority of the skill level, and is characterized in that the data priority selection unit selects the image frame to be used for determining the superiority or inferiority of the skill level using a selection probability determined based on one or more of user editing in the image frames that constitute each video pair, time-directional features in the consecutive multiple image frames that constitute each video pair, and similarity between the image frames that constitute each video pair.

[0007] The machine learning method of the present disclosure is a method for learning a learning model for inferring the skill level of the behavior of an actor in a video, and includes the steps of selecting a plurality of video pairs from a learning video dataset, and selecting an image frame to be used for determining the superiority or inferiority of the skill level from each video pair constituting the selected plurality of video pairs, generating a region of interest in the image frames to be used for determining the superiority or inferiority of the skill level, determining the superiority or inferiority of the skill level in the region of interest for each video pair using the learning model, and storing the learning model and updating the learning model based on the result of the judgment of the superiority or inferiority of the skill level.The step of selecting the image frame to be used for determining the superiority or inferiority of the skill level is characterized in that the image frame to be used for determining the superiority or inferiority of the skill level is selected using a selection probability determined based on one or more of user editing in the image frames constituting each video pair, time-directional features in the consecutive multiple image frames constituting each video pair, and similarity between the image frames constituting each video pair. Effect of the Invention

[0008] According to the present disclosure, it is possible to stabilize the learning behavior when learning a learning model for inferring the skill level of the behavior of a subject of an action in a video. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is an explanatory diagram showing a conventional system (Comparative Example 1) for carrying out skill superiority / inferiority judgment. [Diagram 2] FIG. 2 is a functional block diagram illustrating a schematic configuration of a system (Comparative Example 1). [Diagram 3] FIG. 1 is an explanatory diagram showing the operation of a conventional machine learning device (Comparative Example 2) for determining the skill level of the behavior of a person appearing in a video using transfer learning. [Figure 4] FIG. 4 is a functional block diagram illustrating a schematic configuration of the machine learning device (Comparative Example 2) of FIG. 3. [Diagram 5]1 is a functional block diagram illustrating a schematic configuration of a machine learning device according to a first embodiment. [Figure 6] 1 is a diagram illustrating an example of a hardware configuration of a machine learning device (or a skill assessment device) according to a first embodiment. [Figure 7] FIG. 1A is a diagram showing a segment of a moving image selected from a moving image pair by a random data selection unit of comparison example 1, and FIG. 1B is a diagram showing a portion selected from the moving image pair by a data priority selection unit of embodiment 1. [Figure 8] FIG. 2 is an explanatory diagram showing an effect achieved by the machine learning device according to the first embodiment. [Figure 9] FIG. 2 is an explanatory diagram showing the operation of the machine learning device according to the first embodiment. [Figure 10] 4 is a flowchart showing a learning operation by the machine learning device according to the first embodiment. [Figure 11] 4 is a flowchart showing annotation by the machine learning device according to the first embodiment. [Figure 12] FIG. 11 is a functional block diagram illustrating a schematic configuration of a machine learning device according to a second embodiment. [Figure 13] 11 is a flowchart showing a learning operation by the machine learning device according to the second embodiment. [Figure 14] FIG. 11 is a functional block diagram illustrating a schematic configuration of a machine learning device according to a third embodiment. [Figure 15] 13 is a flowchart showing a learning operation by the machine learning device according to the third embodiment. [Figure 16] FIG. 11 is a functional block diagram illustrating a schematic configuration of a machine learning device according to a fourth embodiment. [Figure 17] 13 is a flowchart showing a learning operation by the machine learning device according to the fourth embodiment. [Figure 18] FIG. 13 is a functional block diagram illustrating a schematic configuration of a machine learning device according to a fifth embodiment. [Figure 19] 13 is a flowchart showing a learning operation by the machine learning device according to the fifth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, a machine learning device, a skill assessment device, a machine learning method, and a machine learning program according to embodiments will be described with reference to the drawings. The following embodiments are merely examples, and the embodiments can be appropriately combined and each embodiment can be appropriately modified.

[0011] The machine learning device according to the embodiment is a device that learns a learning model used in an inference device (also called a "skill assessment device") for inferring the skill level (i.e., proficiency) of the behavior of an actor captured in a video. The machine learning device according to the embodiment is, for example, a computer serving as an information processing device. The actor captured in the video is a person performing work (also called a "worker"). In addition, the actor captured in the video may include a mechanism (e.g., a device such as a robot arm or endoscope) that moves in conjunction with the person's actions to perform work.

[0012] A machine learning method according to an embodiment is a method that can be implemented by a machine learning device. The machine learning method according to an embodiment is a method for learning a learning model for inferring a skill level of an action of an actor in a video.

[0013] The machine learning program according to the embodiment is a software program that can be executed by a computer as a machine learning device. The machine learning program according to the embodiment is a program that learns a learning model for inferring the skill level of an action of an actor captured in a video.

[0014] 1. Comparative Example 1 FIG. 1 is an explanatory diagram showing a conventional system (Comparative Example 1) for carrying out skill superiority / inferiority judgment. In FIG. 1, the configuration proposed in the above-mentioned Non-Patent Document 1 is shown as system 210. FIG. 2 is a functional block diagram showing an outline of the configuration of system 210 (Comparative Example 1). The input of the system of Comparative Example 1 is a video pair (i.e., two videos P i ,P j ), and Pi P j This is a video of a better skill. This system is composed of a processing section that divides the video into segments, a feature extractor that extracts the features of the video, a superior network that evaluates superior actions, and an inferior network that evaluates inferior actions. Each of the superior network and inferior network includes an attention branch and a ranking branch. i The output value obtained when the input is the score (p i ). P j The output value obtained when inputting is Score(p j The system 210 learns the relationship between the scores of these two videos. i The better P j It is better than Score(P i ) and Score(P j If the magnitude of the scores is reversed, the difference in scores is given to the loss function for learning. Examples of loss functions that can be used include Marginal Loss, which only learns differences above a certain level, and SoftPlus, which evaluates losses for differences below a certain level as small.

[0015] 2. Comparative Example 2 Fig. 3 is an explanatory diagram showing the operation of a conventional machine learning device (Comparative Example 2) for determining the skill level of the behavior of a person appearing in a video using transfer learning. Fig. 3 shows the configuration of a machine learning device 220 proposed in Non-Patent Document 2. Fig. 4 is a functional block diagram showing the function of the learning model in Fig. 3.

[0016] [Non-Patent Document 2] Masahiro Mitsuhara, 6 others, “Embedding Human Knowledge into Deep Neural Network via Attention Map”, arXiv:1905.03540, May 9, 2019

[0017] In Comparative Example 2, a human being corrects the region of interest generated for a video by an attention mechanism (a network that generates a region of interest) (i.e., human knowledge is embedded in the learning model), and transfer learning is performed in which learning is performed using the corrected region of interest as corrective data. Transfer learning is a Human-in-the-Loop (HITL) type of learning. Transfer learning generates a learning model that, for example, judges the skill level of a person's behavior in a video while interacting with a user.

[0018] For example, in FIG. 3, the data selection unit selects a video pair (i.e., video #1 and video P i And video #2 is video P j ) is selected. j The skill level of the video P i (This is called "P i <P j " In this case, the video P j The score of the skill shown in j ) is a video i The score of the skill shown in i ) should be judged to be greater than

[0019] In FIG. 3, the region of interest generating unit 100 generates a moving image P j The score of the skill shown in att (P j )=0.1 is the video P i The score of the skill shown in att (P i )=0.8 or less (i.e., the score for the region of interest is reversed from the score it should be) and the output from the FC layer, jThe score of the skill shown in rank (P j )=0.3 is the video P i The score of the skill shown in rank (P i )=0.6 (i.e., examples in which the score output from the FC layer is reversed relative to the score it should be).

[0020] When learning to judge the superiority or inferiority of a skill, the machine learning device first i ) one frame of an image is selected from each of the segments S1, S2, and S3 of the moving image P i Score att (P j ) and Score rank (P j ) is calculated. Figure 3 shows the Score att (P i )=0.8, Score rank (P i )=0.6 is shown as an example.

[0021] Next, the machine learning device detects video #2 (i.e., video P j ) one frame of an image is selected from each of the segments S1, S2, and S3 of the video P. i Score att (P j ) and Score rank (P j ) is calculated. Figure 3 shows the Score att (P i )=0.1, Score rank (P i )=0.3 is shown as an example.

[0022] In this example, Score att (P i )=0.8>Score att (P i )=0.1, Score rank (Pi )=0.6>Score rank (P i )=0.3.

[0023] An example of a method for calculating the difference in loss function from the score and using this difference to learn to judge the superiority or inferiority of the skills shown in a video is described in the above-mentioned non-patent document 1 (where score is represented by f).

[0024] 3. First embodiment In the technique of Comparative Example 1 shown in FIGS. 1 and 2, a moving image pair (moving image P i ,P j ), the skill level of the actions of a person shown on the monitor may be judged based on a location other than the location that should be focused on (i.e., a biased area of ​​focus), which may result in low learning stability.

[0025] On the other hand, the region of interest that is manually edited (i.e., human knowledge is embedded) in the transfer learning of Comparative Example 2 shown in Figures 3 and 4 is considered to be an important part in determining the skill level of human behavior in a video pair. For example, when transfer learning is performed in a machine learning device that generates a learning model for evaluating the skill of a drawing action, it is considered that manual editing is rarely performed on a part where a hand moving a pen is not shown. Therefore, the part that is edited in transfer learning can be considered to be an appropriate part in the video pair for determining the skill level of human behavior.

[0026] In embodiment 1, in a machine learning device that generates a learning model for determining the skill level of a task depicted in a video pair, video data from the video pair where editing by transfer learning has been performed (i.e., the part of the video that the user focuses on) is preferentially selected as focus data, and learning is performed based on the selected focus data, thereby improving the accuracy of determining the superiority or inferiority of the skill and further improving the learning stability.

[0027] The data of interest indicates, for example, an image frame edited (e.g., modified, added, deleted, etc.) by a person in transfer learning, image data of a predetermined time range including an image frame edited in transfer learning (i.e., image data from the X1th image frame to the X2th image frame, where X1 and X2 are predetermined positive integers), or data in which the similarity of intermediate features included in the video is equal to or greater than a predetermined value. For example, a data priority selection unit described later evaluates a weight W of a video data portion (image frame) edited by a user, sets a weight 1 for a video data portion (image frame) that has not been edited, and performs a roulette selection to increase the selection probability of the video data portion edited by the user (for example, a process of making the weight W greater than 1).

[0028] FIG. 5 is a functional block diagram illustrating a schematic configuration of a machine learning device 1 according to the first embodiment. The machine learning device 1 is a device that learns a learning model for inferring the skill level of an action of an actor in a video. The machine learning device 1 includes a data priority selection unit 101 that selects a plurality of video pairs (i.e., a plurality of video data) from a video data set for learning stored in a video data set storage unit 110, and selects an image frame used for judging the superiority or inferiority of a skill level from each of the selected video pairs, a region of interest generation unit 103 that generates a region of interest used for judging the superiority or inferiority of a skill level in an image frame, a superiority or inferiority judgment unit 106 that judges the superiority or inferiority of the skill level in the region of interest using the learning model for each video pair, and a model learning unit 102 that stores the learning model and updates the learning model based on the result of the judgment of the superiority or inferiority of the skill level.

[0029] The machine learning device 1 has a region of interest editing unit 105 that edits the video through user operation while viewing a display screen 111, a region of interest memory unit 106 that stores the edited video, and a region of interest generation unit 103 that generates a region of interest.

[0030] The data priority selection unit 101 selects image frames to be used for judging the superiority or inferiority of skill levels, using a selection probability determined based on user editing of image frames constituting each video pair. The data priority selection unit 101 acquires information indicating which image frames in the video have been edited by the user from the region of interest storage unit 104. For example, the data priority selection unit 101 sets the weight of an edited image frame to W (a value greater than 1) and the weight of an unedited image frame to 1, and calculates the selection probability, which is the probability that an image frame will be selected, for example, for each image frame.

[0031] The weight W is an evaluation index that takes into account, for example, the time spent editing by the user, the degree of match between the edited data and the heat map, etc., and for example, the larger the weight W, the more likely it is to be selected.

[0032] The data priority selection unit 101 uses the selection probability to select the image frame of the segment obtained in the comparative example 1. The weight of the data i edited by the user is calculated by multiplying the time it took to edit it by t e and time t e The maximum value of max(t e ), and the region of interest A generated by the region of interest generating unit 103 i and edited focus area E i and the size of the edited region of interest s relative to the image area S i Using the attribute r of the user who edited the region, it can be calculated, for example, as shown in the following formula (1). In this case, the more gaps there are in the edited region of interest and the narrower the region, the higher the probability of selection. W = (t e / max(t e ))+((E i -A i ) / A i )+S / s i +r (1)

[0033] As shown in equation (1), the data priority selection unit 101 can increase the selection probability of an image frame that has user editing among the image frames constituting each moving image pair, compared to the selection probability of the image frame that has no user editing.

[0034] Furthermore, when there is user editing in the image frames constituting each moving image pair, the data priority selection unit 101 can increase the selection probability of the image frame as the time range in which the user editing is performed becomes longer.

[0035] Furthermore, when there is user editing in image frames constituting each moving image pair, the data priority selection unit 101 can increase the selection probability of an image frame as the time required for user editing increases.

[0036] Furthermore, when there is user editing in the image frames constituting each moving image pair, the data priority selection unit 101 can increase the selection probability of the image frame as the difference between before and after editing of the region of interest becomes greater.

[0037] Furthermore, when there is user editing in the image frames constituting each moving image pair, the data priority selection unit 101 can increase the selection probability of an image frame as the area of ​​the region of interest becomes smaller.

[0038] The model learning unit 102 inputs the video data selected by the data priority selection unit 101 into a convolutional neural network (CNN) and extracts features.

[0039] The region of interest generating unit 103 generates a region of interest using an architecture having a Class Activation Mapping (CAM) structure, such as an Attention branch network, as a branch therein, and stores the generated region of interest in the region of interest storage unit 104 .

[0040] The model learning unit 102 extracts features related to the region of interest by masking the features of the CNN with the region of interest generated by the region of interest generation unit 103. In FIG. 9, the data priority selection unit 101 selects a video pair (i.e., video #1, video P i And video #2 is video P j ) is selected. j The skill level of the video P i (This is called "P i <P j "). In this case, the video P j The score of the skill shown in j ) is a video i The score of the skill shown in i ) should be judged to be greater than

[0041] In FIG. 9, the region of interest generating unit 103 generates a moving image P j The score of the skill shown in att (P j )=0.1 is the video P i The score of the skill shown in att (P i )=0.8 or less (i.e., the score for the region of interest is reversed from the score it should be) and the output from the FC layer, j The score of the skill shown in rank (P j )=0.3 is the video P i The score of the skill shown in rank (P i )=0.6 (i.e., examples in which the score output from the FC layer is reversed relative to the score it should be).

[0042] The superiority / inferiority determination unit 106 converts the feature extraction results of the region of interest in a fully connected layer (FC layer) and extracts superiority / inferiority determination information. The superiority / inferiority determination information is an evaluation result indicating which of the movements shown in video #1 and the movements shown in video #2 has a higher skill level. Based on correct answer data relating to the region of interest or superiority / inferiority judgment given in advance by the user, the superiority / inferiority judgment unit 106 calculates the difference between the region of interest generated by the region of interest generation unit 103 or the superiority / inferiority judgment result judged by the superiority / inferiority judgment unit 106. Based on the calculated loss, the superiority / inferiority judgment unit 106 updates the parameters of the CNN of the model learning unit 102 or the CAM of the region of interest generation unit 103, or its own FC layer by backpropagation. The superiority / inferiority judgment unit 106 checks in advance whether the learning convergence condition is satisfied, and if the condition is satisfied, ends the learning, and if not, repeats the learning from multiple data selections.

[0043] The region of interest editing unit 105 acquires information on the region of interest from the region of interest storage unit 104 and visualizes it for the user. The region of interest editing unit 105 accepts input operations from the user and deletes or adds a region of interest. The region of interest editing unit 105 stores the new region of interest edited by the user in the region of interest storage unit 104 as learning data.

[0044] FIG. 6 is a diagram showing an example of a hardware configuration of the machine learning device 1 according to the first embodiment. The machine learning device 1 according to the first embodiment is a device that executes a learning process to generate a learning model by performing machine learning. The machine learning device 1 in FIG. 6 has a function as a skill assessment device that infers a skill level of an action of an actor in an input video by using the learning model. The machine learning device 1 is a device that can implement the machine learning method according to the first embodiment. The machine learning device 1 is, for example, a computer, but may be a computer system configured by cloud computing using a computer network. Note that FIG. 6 shows an example in which the machine learning device that generates a learning model and the skill assessment device that judges the skill level of the action of an actor in a target video are provided in the same computer, but these may be provided in different computers.

[0045] The machine learning device 1 has a processor 3 such as a CPU (Central Processing Unit) and a storage device 2. The storage device 2 is composed of a semiconductor memory such as a RAM (Random Access Memory), a hard disk drive (HDD), a solid state drive (SSD), etc. The machine learning device 1 may have a communication device that communicates with an external device. An input device 4 such as a mouse and a keyboard, and a display device 5 having a display are connected to the machine learning device 1. The machine learning device 1 may also have a communication device that communicates with other devices.

[0046] Each function of the machine learning device 1 is realized by a processing circuit. The processing circuit is, for example, dedicated hardware. The processing circuit may be a processor 3 that executes a program (for example, a machine learning program according to an embodiment) stored in the storage device 2. The processor 3 may be a processing device, an arithmetic unit, a microprocessor, a microcomputer, or a DSP (Digital Signal Processor).

[0047] When the processing circuit is a dedicated hardware, the processing circuit is, for example, an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0048] When the processing circuit is the processor 3, the machine learning method is executed by software, firmware, or a combination of software and firmware. The software and firmware are written as programs and stored in the storage device 2. The processor 3 can implement the machine learning method according to the first embodiment by reading and executing the programs stored in the storage device 2.

[0049] The machine learning device 1 may be partially realized by dedicated hardware and partially realized by software or firmware. In this way, the processing circuit can realize each of the above-mentioned functions by hardware, software, firmware, or any combination of these.

[0050] A user uses a mouse or a keyboard to register a learning dataset (video dataset) via the processor 3. The processor 3 reads out a machine learning program stored in the storage device 2 and performs learning or inference. Based on the learning dataset input by the user, the processor 3 performs machine learning and inference and stores the results in the storage device as learning results.

[0051] The processor 3 extracts multiple data from the video data set. In the processor 3, the data priority selection unit 101 extracts parts to be compared from the multiple data. In the processor 3, the model learning unit 102, the superiority / inferiority determination unit 106, and the region of interest generation unit 103 compare the selected data, evaluate the superiority / inferiority, and learn to obtain the evaluation value previously given in the data set, and register the region of interest generation result and the created model as the learning result. The region of interest generation result is displayed on the display device 5, and the user receives this and annotates the data with the input device 4. The annotation result is registered in the data set by the region of interest editing unit 105 in the processor 3 as information for new machine learning.

[0052] Fig. 7(A) is a diagram showing moving image segments S1, S2, and S3 selected from a moving image pair by the data selection unit in Comparative Example 2 shown in Fig. 3 and Fig. 4. Fig. 7(B) is a diagram showing parts (shaded areas) selected from each segment of a moving image pair by the data priority selection unit 101 in the first embodiment.

[0053] FIG. 8 is an explanatory diagram showing the effect of the machine learning device 1 according to the first embodiment. As shown in FIG. 8, the data priority selection unit 101 can select image frames that are appropriate paired data, thereby achieving consistency in comparison of superiority / inferiority comparison points and comparison of focus areas when dealing with fine movements, such as a video in which skills are displayed. This can be expected to improve the accuracy of skill judgment and obtain appropriate explanations regarding skill levels. In addition, by comparing image frames edited by the user, the skill level between image frames intended by the user can be easily evaluated, and learning becomes stable. In addition, by comparing an image frame edited by the user with an image frame that has not been edited, learning is performed in which the focus area is taken into consideration even for the unedited area, so that a similar focus area is generated for the unedited area even though the user has not edited it.

[0054] FIG. 9 is an explanatory diagram showing the operation of the machine learning device 1 according to the first embodiment. FIG. 10 is a flowchart showing the learning operation by the machine learning device 1 according to the first embodiment. First, the data priority selection unit 101 selects a plurality of video pairs (video #1, P i And video #2 is P j ) (step S101). Next, the data priority selection unit 101 acquires an edited data list indicating which image frames in the moving images constituting the moving image pair have been edited by the user from the region-of-interest storage unit 104 (step S102).

[0055] Next, the data priority selection unit 101 assigns weights of edited image frames as W and unedited image frames as 1 to calculate a selection probability indicating the probability that an image frame will be selected. The weight W may be changed based on an evaluation index that takes into account the editing time by the user or the degree of agreement between the edited data and the heat map. The data priority selection unit 101 selects image frames of segments divided like a conventional Temporal Segment Network, using the calculated selection probability (step S103).

[0056] The model learning unit 102 inputs the data selected by the data priority selection unit to the CNN and extracts image features (step S104).

[0057] The region of interest generating unit 103 generates a region of interest using an architecture having a branched CAM structure therein, and stores the generated region of interest in the region of interest storage unit 104 (step S105).

[0058] The model learning unit 102 extracts features related to the region of interest (step S106).

[0059] The superiority / inferiority determination unit 106 extracts superiority / inferiority determination information (step S107). The superiority / inferiority determination information is an evaluation result indicating which of the movements shown in the video #1 and the movement shown in the video #2 has a higher skill level.

[0060] Based on correct answer data relating to the region of interest or superiority / inferiority judgment given in advance by the user, superiority / inferiority judgment unit 106 calculates the difference between the region of interest generated by region of interest generation unit 103 or the superiority / inferiority judgment result judged by superiority / inferiority judgment unit 106. Based on the calculated loss, superiority / inferiority judgment unit 106 updates the parameters of the CNN of model learning unit 102 or the CAM of region of interest generation unit 103, and its own FC layer by backpropagation (step S108).

[0061] The superiority / inferiority determination unit 106 checks in advance whether a learning convergence condition is satisfied, and if the condition is satisfied, ends the learning, and if not, repeats the learning from a plurality of selected data (step S109).

[0062] 11 is a flowchart showing annotation by the machine learning device 1 according to the first embodiment. The region of interest editing unit 105 acquires information on the region of interest from the region of interest storage unit 104 and visualizes it (presents it on the display screen 111) to the user (step S151). Next, the region of interest editing unit 105 accepts an input operation from the user and edits the region of interest (step S152). The region of interest editing unit 105 stores the new region of interest edited by the user in the region of interest storage unit 104 as learning data (step S153).

[0063] As described above, according to the first embodiment, it is possible to stabilize the learning behavior when learning a learning model for inferring the skill level of the action of an actor in a video.

[0064] 4. Second embodiment Fig. 12 is a functional block diagram illustrating a schematic configuration of a machine learning device 1a according to embodiment 2. In Fig. 12, elements that are the same as or correspond to elements illustrated in Fig. 5 are assigned the same reference numerals as those illustrated in Fig. 5. The machine learning device 1a according to embodiment 2 differs from the machine learning device 1 according to embodiment 1 in that it includes a data priority selection unit (also referred to as a "paired data consecutive selection unit") 101a that selects consecutive image frames, and a time-series feature extraction unit 121 that extracts features of consecutive image frames.

[0065] The data priority selection unit 101a in the machine learning device 1a according to the second embodiment determines the selection probability of an image frame based on the features in the time direction of a plurality of consecutive image frames that make up each moving image pair.

[0066] In the machine learning device 1a according to the second embodiment, a plurality of consecutive image frames are selected in order to increase the probability (hit rate) of selecting an appropriate video pair. In the case of the first comparative example, the video data constituting the video pair is divided into segments and image frames to be compared are randomly selected. However, in the machine learning device 1a according to the second embodiment, the data priority selection unit 101a selects a plurality of consecutive image frames. Furthermore, the machine learning device 1a further includes a time-series feature extraction unit 121 in order to be able to handle the features of consecutive image frames. Since the machine learning device 1a selects a plurality of consecutive image frames, the probability that an image frame edited with a region of interest is included in any of the image frames is higher than in the case of randomly selecting an image frame edited with a region of interest.

[0067] There are the following methods for selecting multiple consecutive image frames. In the first method, a random location is specified and consecutive data for a predetermined number of image frames is selected, which increases the hit rate of the data being edited by the user compared to randomly selecting one frame at a time. An example of a simulation of the hit rate is shown in Table 1 below.

[0068] [Table 1]

[0069] In the machine learning device 1a according to the second embodiment, since there is a high possibility that skills are included in the vicinity of data edited by a user, the selection is made based on a probability distribution centered on the data edited by the user at this time. By preparing and selecting from a plurality of normal distribution variances in advance, the characteristics of a long time series and the characteristics of a short time series are selected.

[0070] 13 is a flowchart showing the learning operation by the machine learning device 1a according to embodiment 2. The processes of steps S201, S203 to S205, and S207 to S209 in FIG. 13 are similar to the processes of steps S101, S104 to S106, and S107 to S109 in FIG.

[0071] In the second embodiment, the data priority selection unit 101b functioning as the paired data consecutive selection unit selects data of a plurality of consecutive image frames (step S202). By selecting data that is consecutive in the time direction in this manner, edited data of the region of interest annotated by the user is more likely to be included in the judgment of superiority or inferiority.

[0072] Moreover, in the second embodiment, the time-series feature extraction unit 121 performs convolution in the time direction on data of a plurality of consecutive image frames to extract features in the time direction from the moving image (step S206).

[0073] As described above, according to the second embodiment, it is possible to capture continuous, detailed movements such as skill videos. At the same time, the hit rate of the target area is higher than when each time is selected randomly. Therefore, the learning behavior when learning the learning model can be stabilized.

[0074] Other than the above, the second embodiment is the same as the first embodiment. Moreover, the data priority selection unit 101b and the time-series feature extraction unit 121 in the second embodiment are also applicable to the first embodiment.

[0075] 5. Third embodiment Fig. 14 is a functional block diagram showing a schematic configuration of a machine learning device 1b according to embodiment 3. In Fig. 14, elements that are the same as or correspond to elements shown in Fig. 5 are given the same reference numerals as those shown in Fig. 5. The machine learning device 1b according to embodiment 3 differs from the machine learning device 1 according to embodiment 1 in that it has a region-of-interest comparison unit 131 that calculates the similarity between image frames edited by a user, and a data priority selection unit 101b selects data based on the similarity.

[0076] In the second embodiment, a means for increasing the hit rate of image frames edited by a user is included, but the editing contents are not taken into consideration, so that video pairs with different contents may be selected. For this reason, the machine learning device 1b according to the third embodiment includes a region of interest comparison unit 131, which compares the editing contents in addition to comparing the regions of interest. The data priority selection unit 101b adjusts the selection probability so as to preferentially select video pairs with similar editing contents. In this way, similar regions of interest can be compared, and therefore the accuracy of skill level evaluation can be expected to improve.

[0077] In Example 1 of the third embodiment, first, the region of interest comparison unit 131 calculates the similarity between image frames edited by the user. The data priority selection unit 101b selects video data by prioritizing (increasing the selection probability) video pairs with higher similarity between image frames edited by the user. However, if data not edited by the user is included in the video pair, a low value (e.g., "0.01") is given as the similarity value to lower the selection probability. The model learning unit 102 learns about the selected video pair, and the region of interest generation unit 103 generates a region of interest. If the selected pair is not edited by the user, the region of interest comparison unit 131 calculates the similarity with other user-edited image frames or data already selected as a pair.

[0078] In Example 2 of the third embodiment, first, the region of interest comparison unit 131 calculates the similarity between image frames edited by the user. Next, the region of interest comparison unit 131 preferentially selects pairs that are dissimilar with a certain probability. For example, the dissimilarity is calculated using the formula "(dissimilarity)=1.0-(similarity)", and the selection probability is determined based on this. This is because the selection of dissimilar video pairs includes video pairs that have not been edited by the user with a certain probability, making it easier to generate regions of interest that the user did not expect, leading to the discovery of new regions of interest. The processing thereafter is the same as that of Example 1 of the third embodiment.

[0079] In Example 3 of the third embodiment, first, since it is difficult to maintain the similarity of all pairs of image frames, the region of interest comparison unit 131 forms clusters based on the similarity and selects data based on the similarity between the clusters. At this time, the region of interest comparison unit 131 sets the image frames edited by the user as a cluster (the number of data is 1) and calculates the similarity between the clusters. Also, all image frames not edited by the user are assumed to belong to the unedited cluster. Next, the region of interest comparison unit 131 selects clusters with higher similarity with higher priority. However, the similarity between the unedited cluster and other clusters is regarded as a predetermined low value. For example, the region of interest comparison unit 131 randomly selects data included in the two selected clusters A and B, respectively, and selects them as pairs. Here, the region of interest comparison unit 131 may select data that is not similar to the representative point or other data in the clusters, rather than randomly. By such a selection method, learning proceeds with the representative point and a point far from the representative as input, and it is expected that a new region of interest will be discovered. Next, the model learning unit 102 learns the selected pair, and the region of interest generating unit 103 generates a region of interest. Next, if the selected pair is not edited by the user, the region of interest comparing unit 131 calculates the similarity with representative data of other clusters, and assigns the data to the cluster with the highest similarity. Next, the region of interest comparing unit 131 updates the representative data as the data with the highest similarity within the cluster.

[0080] 15 is a flowchart showing a learning operation by the machine learning device 1b according to Embodiment 3. The processes of steps S301 and S305 to S310 in FIG.

[0081] The region of interest comparison unit 131 selects data of a plurality of moving image pairs (step S301). The region of interest comparison unit 131 selects a region of interest image (Attention Map) from the data of a plurality of moving image pairs (step S302). The region of interest comparison unit 131 calculates the similarity between the region of interest images (step S303). As a method for calculating the similarity, IoU (Intersection over Union), which calculates the degree of overlap of the region of interest images, may be used. The region of interest comparison unit 131 selects a region of interest image (Attention Map) at a certain time t k From a certain time t k+1 and time s in video #2 l From a certain time s l+1 The target area comparison unit 131 performs this for the entire interval and normalizes the sum so that the sum is 1. The data priority selection unit 101b uses random numbers to select which interval time t k ~t k+1 or s l ~s l+1 The data priority selection unit 101b may determine the section by weighting the similarity so that an image edited by a user is more likely to be selected. The calculation of the similarity is performed for all combinations of video #1 and video #2, but one video may be selected at random and selected based on the similarity with the other video.

[0082] As described above, according to the third embodiment, it is possible to capture continuous, detailed movements such as skill videos. At the same time, the hit rate of the target area is higher than when each time is selected randomly. Therefore, the learning behavior when learning the learning model can be stabilized.

[0083] Except for the above, the third embodiment is the same as the first embodiment. The data priority selection unit 101b in the third embodiment is also applicable to the first or second embodiment.

[0084] 6. Fourth embodiment Fig. 16 is a functional block diagram showing a schematic configuration of a machine learning device 1c according to embodiment 4. In Fig. 16, elements that are the same as or correspond to elements shown in Fig. 5 are given the same reference numerals as those shown in Fig. 5. The machine learning device 1c according to embodiment 4 differs from the machine learning device 1 according to embodiment 1 shown in Fig. 5 in that it includes a motion extraction unit 141 and a motion comparison unit 142.

[0085] In the first to third embodiments, the movement of the subject of the action in the video is not sufficiently considered. In order to learn a learning model for judging a skill level, it is very important to judge whether the movement is the same as an excellent movement. The machine learning device 1c according to the fourth embodiment includes a movement extraction unit 141 and a movement comparison unit 142, and preferentially selects data with similar movements as a video pair.

[0086] According to the machine learning device 1c of embodiment 4, data with similar movements can be compared, so that similar skill levels are more likely to be selected as evaluation targets, which is expected to improve the accuracy of skill level evaluation.

[0087] As a first processing example for determining the similarity of the actions, a process using hand pose tracking can be considered. The first processing example can be executed by the following steps 11 to 15.

[0088] (Process 11) Extract image frames at time t (t=0, Δt, 2Δt, 3Δt, ..., NΔt) from each video, where N is a positive integer. (Process 12) Calculate a motion vector (i.e., Flow) from image frames from time t=mΔt to time t=(m+1)Δt, where m=0, 1, ..., N-1. (Process 13) All frames of video X (N x ) and all frames of video Y (N y For each pair, the motion vector (Δx l ,Δy l ) cosine distance. The cosine distance is calculated as (N x ×Ny ) are calculated. (Process 14) (Process 13) is repeated for all video pairs. (Process 15) A video pair is selected such that the video pair with the higher similarity obtained in (Process 13) is selected with priority.

[0089] As a second processing example for determining the similarity between actions, a process for reducing the amount of calculation using clustering, etc. The second processing example can be executed in the following steps of (Process 21) to (Process 26).

[0090] (Process 21) The same process as (Process 11) in the first processing example is carried out. (Process 22) The same process as (Process 12) in the first processing example is carried out. (Process 23) Calculate only the similarity between adjacent image frames in the video X and perform hierarchical clustering. The number of clusters is determined in advance by the user, for example by defining it. (Process 24) In (Process 23), the similarity between the data contained in the cluster is calculated, and the data that is similar on average to all the data in the cluster is determined as the representative data. (Process 25) Calculate the similarity between the representative data of the cluster group (Cx) of video X and the cluster group (Cy) of video Y generated in (Process 23) and (Process 24). (Process 26) A cluster is selected using the similarity between the clusters calculated in (Process 25), and data within the cluster is randomly selected to obtain pairs. After the clusters are obtained, the same processes as in the third embodiment can be applied.

[0091] In the fourth embodiment, user editing is not used, but it is also possible to weight the motion vectors by user editing, or to select data pairs using an index that combines the similarity of the third embodiment obtained by user editing with the similarity based on the motion vectors.

[0092] Also, the similarity of a specific range of a video may be obtained using a distance index such as Dynamic Time Warping (DTW). A method to average the motion vectors of parts of video X to obtain the overall motion vector for one frame. min Δt to t=m max The interval up to Δt is selected and segmented, the DTW distance between each segment is calculated, and clustering is performed. The selection probability is set so that data belonging to the same cluster is more likely to be selected as a pair. For example, data within clusters are more likely to be selected as a pair. In this case, pairs of data from different clusters are rarely selected.

[0093] 17 is a flowchart showing the learning operation by the machine learning device 1c according to embodiment 4. The processes of steps S401 and S405 to S410 in FIG.

[0094] The motion extraction unit 141 selects data of a plurality of moving image pairs (step S401). The motion comparison unit 142 extracts motion from the data of the plurality of moving image pairs using a method such as Optical Flow (step S402). The motion extraction unit 141 may extract the direction in which an area obtained by dividing an image into blocks moves, and retain the extracted motion as a feature vector. The motion comparison unit 142 calculates the similarity using the cosine distance or the like of the extracted motion feature vectors (step S403). The motion comparison unit 142 calculates the similarity at a certain time t k From a certain time t k+1 Until the time of video #2, l From a certain time s l+1 The motion comparator 142 performs this for the entire interval and normalizes the sum so that the sum is 1. The data priority selector 101c selects which interval time t k ~t k+1 or s l ~s l+1 Then, it is determined whether to select the pair data section (step S404).

[0095] As described above, according to the fourth embodiment, movements are extracted from data of a plurality of video pairs and used to train a learning model, so that the learning behavior can be stabilized.

[0096] Except for the above, the fourth embodiment is the same as the first embodiment. Moreover, the motion extractor 141 and the motion comparator 142 in the fourth embodiment are applicable to any of the first to third embodiments.

[0097] 7. Fifth embodiment Fig. 18 is a functional block diagram showing a schematic configuration of a machine learning device 1d according to embodiment 5. In Fig. 18, elements that are the same as or correspond to elements shown in Fig. 5 are given the same reference numerals as those shown in Fig. 5. The machine learning device 1d according to embodiment 5 differs from the machine learning device 1 according to embodiment 1 shown in Fig. 5 in that the machine learning device 1d according to embodiment 5 includes a foreground extraction unit 151 and a region of interest comparison unit 152.

[0098] In the first to fourth embodiments, an example in which a learning operation is performed on the entire video has been described. However, when learning a learning model for inferring the skill level of a subject's operation, the result of analyzing the background of the video may become noise, which may reduce the accuracy of determining the skill level. For this reason, the machine learning device 1d according to the fifth embodiment includes a foreground extraction unit 151 that extracts a foreground from a video pair selected from the video data set storage unit 110, and a region of interest comparison unit 152 that calculates the similarity of the region of interest by masking the background, which is a region other than the foreground, and setting the foreground as a region of interest.

[0099] In this way, areas unrelated to skill level are masked, so video pairs that are more likely to be directly linked to skill level are selected, which is expected to improve the accuracy of skill level assessment and the interpretability of the assessment.

[0100] 19 is a flowchart showing the learning operation by the machine learning device 1d according to embodiment 5. The processes of steps S501 and S507 to S512 in FIG. 19 are similar to the processes of steps S101 and S104 to S109 in FIG.

[0101] First, the foreground extraction unit 151 selects a plurality of data from the video data set storage unit 110 and extracts the foreground (steps S501 and S502). The foreground can be extracted by, for example, taking an area in which no change occurs between the past image frame and the current image frame as the background.

[0102] Next, the foreground extraction unit 151 acquires the region of interest image from the region of interest storage unit 104 (step S503), and performs mask processing on the foreground and region of interest images (step S504). The region of interest comparison unit 152 calculates the similarity between the masked region of interest images (step S505). The region of interest comparison unit 152 calculates the similarity between the masked region of interest images at a certain time t k From a certain time t k+1 Until the time of video #2, l From a certain time s l+1 The target region comparison unit 152 performs this for the entire section and normalizes the similarity so that the sum becomes 1. The data priority selection unit 101d selects which section time t k ~t k+1 or s l ~ s l+1 Determine whether to select as the interval for paired data.

[0103] As described above, according to the fifth embodiment, movements are extracted from data of a plurality of video pairs and used to train a learning model, so that the learning behavior can be stabilized.

[0104] Except for the above, the fifth embodiment is the same as the first embodiment. Furthermore, the foreground extraction unit 151 and the region of interest comparison unit 152 in the fifth embodiment are also applicable to any of the first to fourth embodiments. [Explanation of symbols]

[0105] 1, 1a to 1d machine learning device, 2 storage device, 3 processor, 4 input device, 5 display device, 101, 101a to 101d data priority selection unit, 102 model learning unit, 103 region of interest generation unit, 104 region of interest storage unit, 105 region of interest editing unit, 106 superiority / inferiority determination unit, 110 video data set storage unit, 111, 112 display example, 121 time series feature extraction unit, 131 region of interest comparison unit, 141 motion extraction unit, 142 motion comparison unit, 151 foreground extraction unit.

Claims

1. A machine learning device that learns a learning model for inferring a skill level of an action of an actor in a video, comprising: A data priority selection unit that selects a plurality of video pairs from a learning video data set, and selects an image frame to be used for judging the superiority or inferiority of the skill level from each of the selected video pairs. a region of interest generating unit that generates a region of interest in the image frame for use in determining the skill level; A superiority / inferiority determination unit that uses the learning model to determine the superiority / inferiority of the skill level in the region of interest for each video pair; a model learning unit that stores the learning model and updates the learning model based on the result of the judgment of the superiority or inferiority of the skill level; having The data priority selection unit selects the image frames to be used in determining the superiority or inferiority of the skill level using a selection probability determined based on one or more of user editing of the image frames constituting each video pair, features in the time direction of the consecutive multiple image frames constituting each video pair, and similarities between the image frames constituting each video pair. A machine learning device characterized by:

2. The data priority selection unit sets the selection probability of the image frame having the user editing in the image frames constituting each moving image pair to be higher than the selection probability of the image frame having no user editing. The machine learning device according to claim 1 .

3. When the image frames constituting each moving image pair are edited by the user, the data priority selection unit increases the selection probability of the image frame as the time range in which the user edits is longer. The machine learning device according to claim 1 .

4. The data priority selection unit, when the image frames constituting each moving image pair have been edited by the user, increases the selection probability of the image frame as the time required for the user editing increases. The machine learning device according to claim 1 .

5. The data priority selection unit increases the selection probability of the image frame when the image frames constituting each moving image pair are edited by the user, as the difference between before and after the editing of the region of interest increases. The machine learning device according to claim 1 .

6. The data priority selection unit increases the selection probability of the image frame when the image frames constituting each moving image pair have been edited by the user, as the area of ​​the region of interest becomes smaller. The machine learning device according to claim 1 .

7. The data priority selection unit determines the selection probability of the image frames based on characteristics in a time direction of the consecutive image frames constituting each moving image pair. The machine learning device according to claim 1 .

8. Further comprising a region of interest comparison unit that calculates a similarity between the regions of interest of each pair of moving images, The data priority selection unit increases the selection probability as the degree of similarity between the regions of interest of the image frames constituting each moving image pair increases. The machine learning device according to claim 1 .

9. A motion extraction unit that extracts motion in the region of interest of each moving image pair; a motion comparison unit that calculates a similarity of the motion in the region of interest of each pair of moving images; and The data priority selection unit increases the selection probability as the similarity of the movements increases. The machine learning device according to claim 1 .

10. a foreground extraction unit for extracting a foreground of each video pair; a region of interest comparison unit that calculates a similarity of the region of interest in the foreground; and The data priority selection unit increases the selection probability as the similarity of the region of interest in the foreground increases. The machine learning device according to claim 1 .

11. The subject of the movement is a person or a mechanism that moves in conjunction with the movement of a part of the person's body. The machine learning device according to any one of claims 1 to 10.

12. A learning model generated by the machine learning device according to any one of claims 1 to 10, The learning model determines the skill level of the subject's behavior in the target video. A skill assessment device characterized by:

13. A machine learning method for learning a learning model for inferring a skill level of an action of an actor in a video, comprising: A step of selecting a plurality of video pairs from a learning video dataset, and selecting an image frame to be used for judging the superiority or inferiority of the skill level from each of the selected video pairs constituting the plurality of video pairs; generating an area of ​​interest in the image frame for use in determining the skill level; A step of determining the superiority or inferiority of the skill level in the region of interest for each video pair using the learning model; storing the learning model and updating the learning model based on the result of the judgment of the superiority or inferiority of the skill level; having In the step of selecting the image frames to be used for judging the superiority or inferiority of the skill level, the image frames to be used for judging the superiority or inferiority of the skill level are selected using a selection probability determined based on one or more of user editing of the image frames constituting each video pair, features in the time direction of the consecutive multiple image frames constituting each video pair, and similarities between the image frames constituting each video pair. A machine learning method comprising:

14. A machine learning program for causing a computer to learn a learning model for inferring a skill level of an action of an actor in a video, comprising: A step of selecting a plurality of video pairs from a learning video dataset, and selecting an image frame to be used for judging the superiority or inferiority of the skill level from each of the selected video pairs constituting the plurality of video pairs; generating an area of ​​interest in the image frame for use in determining the skill level; A step of determining the superiority or inferiority of the skill level in the region of interest for each video pair using the learning model; storing the learning model and updating the learning model based on the result of the judgment of the superiority or inferiority of the skill level; having In the step of selecting the image frames to be used for judging the superiority or inferiority of the skill level, the image frames to be used for judging the superiority or inferiority of the skill level are selected using a selection probability determined based on one or more of user editing of the image frames constituting each video pair, features in the time direction of the consecutive multiple image frames constituting each video pair, and similarities between the image frames constituting each video pair. A machine learning program characterized by

Images