Machine learning device, skill determination device, machine learning method, and machine learning program

Abstract

A machine learning device (1) comprises: a data priority selection unit (101) that selects a plurality of video pairs from a video dataset for training and selects image frames used for determining the superiority or inferiority of skill levels from each video pair constituting the selected plurality of video pairs; a focus area generation unit (103) that generates a focus area used for determining the superiority or inferiority of the skill levels in the image frames; a superiority-inferiority determination unit (106) that determines the superiority or inferiority of the skill levels in the focus area using a training model; and a model learning unit (102) that updates the training model on the basis of the results of the determination of the superiority or inferiority of the skill levels. The data priority selection unit (101) selects the image frames used for determining the superiority or inferiority of the skill levels using a selection probability determined on the basis of one or more of: user editing in the image frames constituting each video pair; time-direction features in the plurality of image frames constituting each video pair; and a similarity between the image frames constituting each video pair.

Description

Machine learning device, skill assessment device, machine learning method, and machine learning program

[0001] The present disclosure relates to a machine learning device, a skill determination device, a machine learning method, and a machine learning program.

[0002] Pairwise Deep Ranking (PDR) has been proposed as a technique for skill assessment, which calculates a score for the skill level (i.e., proficiency) of a person's behavior and determines the superiority or inferiority of the skill (see, for example, Non-Patent Document 1).

[0003] Masayuki Takada and three others (Chubu University), "Visual explanation and high accuracy in skill superiority judgment using Attention Pairwise Ranking," 23rd Image Recognition and Understanding Symposium

[0004] However, in the above-mentioned conventional technology, the only information provided is the skill level and the video pair, and the skill level may be determined based on a location other than the location that should be focused on to determine the skill level (i.e., a biased area of ​​focus in the video pair). For example, in evaluating the skill of drawing a picture, the skill level may be determined based on the video during the period in which the head of the person moving the pen is visible, rather than the video during the period in which the pen is moving. As a result, the conventional technology has the problem of unstable learning behavior.

[0005] The present disclosure aims to stabilize learning behavior when training a learning model for inferring the skill level of an actor's behavior in a video.

[0006] The machine learning device of the present disclosure is a device that learns a learning model for inferring the skill level of the behavior of an actor in a video, and includes: a data priority selection unit that selects multiple video pairs from a video dataset for learning and selects image frames from each video pair that constitutes the selected multiple video pairs to be used for determining the superiority or inferiority of the skill level; a region of interest generation unit that generates a region of interest in the image frames to be used for determining the superiority or inferiority of the skill level; a superiority or inferiority judgment unit that uses the learning model to judge the superiority or inferiority of the skill level in the region of interest for each video pair; and a model learning unit that stores the learning model and updates the learning model based on the results of the judgment of the superiority or inferiority of the skill level, and is characterized in that the data priority selection unit selects the image frames to be used for determining the superiority or inferiority of the skill level using a selection probability determined based on one or more of user editing in the image frames that constitute each video pair, temporal features in the consecutive multiple image frames that constitute each video pair, and similarity between the image frames that constitute each video pair.

[0007] The machine learning method of the present disclosure is a method for learning a learning model for inferring the skill level of the behavior of an actor in a video, and includes the steps of selecting a plurality of video pairs from a video dataset for learning, and selecting image frames from each video pair that constitutes the selected plurality of video pairs to be used for determining the superiority or inferiority of the skill level, generating a region of interest in the image frames to be used for determining the superiority or inferiority of the skill level, determining the superiority or inferiority of the skill level in the region of interest for each video pair using the learning model, and storing the learning model and updating the learning model based on the results of the determination of the superiority or inferiority of the skill level.The step of selecting image frames to be used for determining the superiority or inferiority of the skill level is characterized in that the image frames to be used for determining the superiority or inferiority of the skill level are selected using a selection probability determined based on one or more of user editing in the image frames that constitute each video pair, temporal features in the consecutive plurality of image frames that constitute each video pair, and similarity between the image frames that constitute each video pair.

[0008] According to the present disclosure, it is possible to stabilize the learning behavior when learning a learning model for inferring the skill level of the behavior of a subject in a video.

[0009] 1 is an explanatory diagram showing a conventional system (Comparative Example 1) for performing skill superiority / inferiority determination. It is a functional block diagram schematically showing the configuration of the system (Comparative Example 1). It is an explanatory diagram showing the operation of a conventional machine learning device (Comparative Example 2) for determining the skill level of the behavior of a person appearing in a video using transfer learning. It is a functional block diagram schematically showing the configuration of the machine learning device (Comparative Example 2) of FIG. 3. It is a functional block diagram schematically showing the configuration of a machine learning device according to a first embodiment. It is a diagram showing an example of the hardware configuration of a machine learning device (or a skill determination device) according to the first embodiment. (A) is a diagram showing video segments selected from a video pair by a random data selection unit of Comparative Example 1, and (B) is a diagram showing a portion selected from a video pair by a data priority selection unit of the first embodiment. It is an explanatory diagram showing the effect of the machine learning device according to the first embodiment. It is an explanatory diagram showing the operation of the machine learning device according to the first embodiment. It is a flowchart showing the learning operation by the machine learning device according to the first embodiment. It is a flowchart showing annotation by the machine learning device according to the first embodiment. It is a functional block diagram schematically showing the configuration of a machine learning device according to a second embodiment. It is a flowchart showing the learning operation by the machine learning device according to the second embodiment. It is a functional block diagram schematically showing the configuration of a machine learning device according to a third embodiment. Fig. 1 is a flowchart showing a learning operation by a machine learning device according to embodiment 3. Fig. 2 is a functional block diagram showing an outline of the configuration of a machine learning device according to embodiment 4. Fig. 3 is a flowchart showing a learning operation by a machine learning device according to embodiment 4. Fig. 4 is a functional block diagram showing an outline of the configuration of a machine learning device according to embodiment 5. Fig. 5 is a flowchart showing a learning operation by a machine learning device according to embodiment 5.

[0010] Hereinafter, a machine learning device, a skill determination device, a machine learning method, and a machine learning program according to embodiments will be described with reference to the drawings. The following embodiments are merely examples, and the embodiments can be appropriately combined and modified.

[0011] The machine learning device according to the embodiment is a device that learns a learning model used in an inference device (also referred to as a "skill assessment device") for inferring the skill level (i.e., proficiency) of the behavior of an actor captured in a video. The machine learning device according to the embodiment is, for example, a computer serving as an information processing device. The actor captured in the video is a person performing work (also referred to as a "worker"). The actor captured in the video may also include a mechanism (e.g., a device such as a robot arm or endoscope) that moves in conjunction with the person's actions to perform work.

[0012] A machine learning method according to an embodiment is a method that can be implemented by a machine learning device, and is a method for learning a learning model for inferring the skill level of an action of an actor in a video.

[0013] A machine learning program according to an embodiment is a software program that can be executed by a computer as a machine learning device, and is a program that learns a learning model for inferring the skill level of an action of an actor captured in a video.

[0014] <<1>> Comparative Example 1> Fig. 1 is an explanatory diagram showing a conventional system (Comparative Example 1) for carrying out skill superiority / inferiority judgment. In Fig. 1, the configuration proposed in the above-mentioned Non-Patent Document 1 is shown as system 210. Fig. 2 is a functional block diagram showing the outline of the configuration of system 210 (Comparative Example 1). The input to the system of Comparative Example 1 is a video pair (i.e., two videos P ｉ , P ｊ ) and P ｉ Is P ｊThis is a video of better skills. This system is composed of a processing section that divides the video into segments, a feature extractor that extracts the features of the video, a superior network that evaluates excellent actions, and an inferior network that evaluates inferior actions. Each of the superior network and inferior network includes an attention branch and a ranking branch. ｉ The output value obtained when the above is input is Score (p ｉ ) is. ｊ The output value obtained when the above is input is Score (p ｊ The system 210 learns the relationship between the scores of these two videos. ｉ It is better to ｊ is better than Score (P ｉ ) and Score (P ｊ ) are reversed, the difference in scores is applied to a loss function for learning. Examples of the loss function that can be used include Marginal Loss, which only learns differences above a certain level, and SoftPlus, which evaluates the loss of differences below a certain level as small.

[0015] <<2>> Comparative Example 2> Fig. 3 is an explanatory diagram showing the operation of a conventional machine learning device (Comparative Example 2) for determining the skill level of the behavior of a person appearing in a video using transfer learning. Fig. 3 shows the configuration of a machine learning device 220 proposed in Non-Patent Document 2. Fig. 4 is a functional block diagram showing the functions of the learning model in Fig. 3.

[0016] Masahiro Mitsuhara, 6 others, "Embedding Human Knowledge into Deep Neural Network via Attention Map", arXiv: 1905.03540, May 9, 2019

[0017] In Comparative Example 2, a human corrects the region of interest generated for a video by the attention mechanism (a network that generates a region of interest) (i.e., human knowledge is embedded in the learning model), and transfer learning is performed in which the corrected region of interest is used as correct data for learning. Transfer learning is a Human-in-the-Loop (HITL) type of learning. Transfer learning generates a learning model that, for example, determines the skill level of a person's behavior in a video while interacting with a user.

[0018] For example, in FIG. 3, the data selection unit selects a video pair (i.e., video #1 and video P) from the data set storage unit. ｉ and Video #2, Video P ｊ ) is selected. ｊ The skill level of the video ｉ (This is called "P ｉ <P ｊ In this case, the video P ｊ The skill score displayed on the screen (P ｊ ) is a video ｉ The skill score shown in ｉ ) should be judged to be greater than

[0019] In FIG. 3, the region of interest generating unit generates a moving image P ｊ Skill score shown on the screen ａｔｔ (P ｊ ) = 0.1 is the video P ｉ Skill score shown on the screen ａｔｔ (P ｉ ) = 0.8 (i.e., the score for the region of interest is reversed relative to the score that should be), and the output from the FC layer ｊ Skill score shown on the screen ｒａｎｋ (P ｊ ) = 0.3 is the video P ｉ Skill score shown on the screen ｒａｎｋ (P ｉ) is smaller than 0.6 (i.e., the score output from the FC layer is reversed relative to the score that should have been obtained).

[0020] When learning to judge the superiority or inferiority of proficiency, the machine learning device first ｉ ) and selects one frame of an image from each of the segments S1, S2, and S3 of the moving image P ｉ Score ａｔｔ (P ｊ ) and Score ｒａｎｋ (P ｊ ) is calculated. ａｔｔ (P ｉ ) = 0.8, Score ｒａｎｋ (P ｉ )=0.6 is shown as an example.

[0021] Next, the machine learning device calculates video #2 (i.e., video P ｊ ) and selects one frame of an image from each of the segments S1, S2, and S3 of the moving image P. ｉ Score ａｔｔ (P ｊ ) and Score ｒａｎｋ (P ｊ ) is calculated. ａｔｔ (P ｉ )=0.1, Score ｒａｎｋ (P ｉ )=0.3 is shown as an example.

[0022] In this example, Score ａｔｔ (P ｉ )=0.8>Score ａｔｔ (P ｉ ) = 0.1, Score ｒａｎｋ (P ｉ )=0.6>Score ｒａｎｋ (P ｉ ) = 0.3.

[0023] An example of a method for calculating the difference in loss function from the score and using this difference to learn to judge the superiority or inferiority of the skills shown in the video is described in the above-mentioned Non-Patent Document 1 (where the score is represented by f).

[0024] <3> First Embodiment In the technique of the first comparative example shown in FIGS. 1 and 2, a moving image pair (moving image P ｉ , P ｊ ) to judge the skill level of the actions of a person shown on the screen, the skill level may be judged based on a location other than the location that should be focused on (i.e., a biased area of ​​focus), which may result in low learning stability.

[0025] On the other hand, the region of interest that is manually edited (i.e., human knowledge is embedded) in the transfer learning of Comparative Example 2 shown in Figures 3 and 4 is considered to be an important part in determining the skill level of a person's behavior in a video pair. For example, when performing transfer learning in a machine learning device that generates a learning model for evaluating the skill of drawing a picture, it is considered that manual editing is rarely performed on areas that do not show the hand moving the pen. Therefore, the area that is edited in transfer learning can be considered to be an appropriate part in the video pair for determining the skill level of a person's behavior.

[0026] In embodiment 1, in a machine learning device that generates a learning model for determining the skill level of a task shown in a video pair, video data from the video pair where editing by transfer learning has been performed (i.e., the part of the video that the user is focusing on) is preferentially selected as focus data, and learning is performed based on the selected focus data, thereby improving the accuracy of determining the superiority or inferiority of the skill and further improving the learning stability.

[0027] The data of interest may be, for example, image frames in which a person has edited (e.g., corrected, added, deleted, etc.) video data in transfer learning, image data of a predetermined time range including image frames edited in transfer learning (i.e., image data from image frame X1 to image frame X2, where X1 and X2 are predetermined positive integers), or data in which the similarity of intermediate features included in the video is equal to or greater than a predetermined value. For example, the data priority selection unit (described later) evaluates a weight W of the video data portion (image frame) edited by the user, sets a weight of 1 for the video data portion (image frame) that has not been edited, and performs a roulette selection to increase the probability of selecting the video data portion edited by the user (e.g., a process of making the weight W greater than 1).

[0028] 5 is a functional block diagram illustrating a schematic configuration of a machine learning device 1 according to embodiment 1. The machine learning device 1 is a device that learns a learning model for inferring the skill level of an action of an actor in a video. The machine learning device 1 includes a data priority selection unit 101 that selects multiple video pairs (i.e., multiple video data) from a learning video dataset stored in a video dataset storage unit 110 and selects image frames from each of the selected video pairs to be used for determining the superiority or inferiority of the skill level; a region-of-interest generation unit 103 that generates a region of interest in the image frame to be used for determining the superiority or inferiority of the skill level; a superiority / inferiority determination unit 106 that uses the learning model to determine the superiority or inferiority of the skill level in the region of interest for each video pair; and a model learning unit 102 that stores the learning model and updates the learning model based on the results of the determination of the superiority or inferiority of the skill level.

[0029] The machine learning device 1 has a region of interest editing unit 105 that edits videos through user operations while viewing a display screen 111, a region of interest storage unit 106 that stores the edited videos, and a region of interest generation unit 103 that generates regions of interest.

[0030] The data priority selection unit 101 selects image frames to be used in determining the skill level using a selection probability determined based on user editing of image frames constituting each video pair. The data priority selection unit 101 acquires information indicating which image frames in the video have been edited by the user from the region-of-interest storage unit 104. For example, the data priority selection unit 101 sets a weight W (a value greater than 1) for edited image frames and a weight 1 for unedited image frames, and calculates a selection probability, which is the probability that an image frame will be selected, for example, for each image frame.

[0031] The weight W is an evaluation index that takes into account, for example, the time spent editing by the user, the degree of match between the edited data and the heat map, etc. For example, the weight W can be increased so that the larger the weight W, the more likely it is to be selected.

[0032] The data priority selection unit 101 uses the selection probability to select the image frame of the segment obtained in Comparative Example 1. The weight of the data i edited by the user is calculated based on the time t ｅ and time t ｅ The maximum value of max(t ｅ ), and the region of interest A generated by the region of interest generating unit 103 ｉ and edited focus area E ｉ and the size s of the edited region of interest relative to the image area S. ｉ and the attribute r of the user who edited it, it can be calculated, for example, as in the following formula (1). In this case, the more gaps there are in the edited region of interest and the narrower the region, the higher the probability of selection. W = (t ｅ / max(t ｅ ))+((E ｉ -A ｉ ) / A ｉ ) + S / s ｉ +r (1)

[0033] As shown in equation (1), the data priority selection unit 101 can increase the probability of selecting an image frame that has been user edited among the image frames that make up each video pair compared to the probability of selecting the image frame that has not been user edited.

[0034] Furthermore, when there is user editing in the image frames that make up each moving image pair, the data priority selection unit 101 can increase the selection probability of the image frame as the time range in which the user editing has been performed becomes longer.

[0035] Furthermore, when there is user editing in the image frames that make up each moving image pair, the data priority selection unit 101 can increase the selection probability of the image frame as the time required for user editing increases.

[0036] Furthermore, when there is user editing in the image frames that make up each moving image pair, the data priority selection unit 101 can increase the selection probability of the image frame as the difference between before and after editing of the region of interest becomes greater.

[0037] Furthermore, when there is user editing in the image frames that make up each moving image pair, the data priority selection unit 101 can increase the selection probability of an image frame as the area of ​​the region of interest becomes smaller.

[0038] The model learning unit 102 inputs the video data selected by the data priority selection unit 101 into a convolutional neural network (CNN) and extracts features.

[0039] The region of interest generating unit 103 generates a region of interest using an architecture that has a branched Class Activation Mapping (CAM) structure such as an Attention branch network, and stores the generated result in the region of interest storage unit 104 .

[0040] The model learning unit 102 extracts features related to the region of interest by masking the CNN features with the region of interest generated by the region of interest generation unit 103. In FIG. 9, the data priority selection unit 101 selects a video pair (i.e., video #1 and video #2) from the dataset storage unit 110. ｉ and Video #2, Video P ｊ ) is selected. ｊ The skill level of the video ｉ (This is called "P ｉ <P ｊ "). In this case, the video P ｊThe skill score displayed on the screen (P ｊ ) is a video ｉ The skill score shown in ｉ ) should be judged to be greater than

[0041] In FIG. 9, the region of interest generating unit 103 generates a moving image P ｊ Skill score shown on the screen ａｔｔ (P ｊ ) = 0.1 is the video P ｉ Skill score shown on the screen ａｔｔ (P ｉ ) = 0.8 (i.e., the score for the region of interest is reversed relative to the score that should be), and the output from the FC layer ｊ Skill score shown on the screen ｒａｎｋ (P ｊ ) = 0.3 is the video P ｉ Skill score shown on the screen ｒａｎｋ (P ｉ ) is smaller than 0.6 (i.e., the score output from the FC layer is reversed relative to the score that should have been obtained).

[0042] The superiority / inferiority determination unit 106 converts the feature extraction results of the region of interest using a fully connected layer (FC layer) to extract superiority / inferiority determination information. The superiority / inferiority determination information is an evaluation result indicating which of the actions captured in video #1 and video #2 has a higher skill level. Based on correct answer data regarding the region of interest or superiority / inferiority determination provided in advance by the user, the superiority / inferiority determination unit 106 calculates the difference between the region of interest generated by the region of interest generation unit 103 or the superiority / inferiority determination result determined by the superiority / inferiority determination unit 106. Based on the calculated loss, the superiority / inferiority determination unit 106 updates the parameters of the CNN of the model learning unit 102, the CAM of the region of interest generation unit 103, and its own FC layer using backpropagation. The superiority / inferiority determination unit 106 checks in advance whether the learning convergence condition is met. If the condition is met, the learning is terminated. If not, the learning is repeated from multiple data selections.

[0043] The region of interest editing unit 105 acquires information about the region of interest from the region of interest storage unit 104 and visualizes it for the user. The region of interest editing unit 105 accepts input operations from the user and deletes or adds regions of interest. The region of interest editing unit 105 stores new regions of interest edited by the user in the region of interest storage unit 104 as learning data.

[0044] FIG. 6 is a diagram illustrating an example of the hardware configuration of a machine learning device 1 according to the first embodiment. The machine learning device 1 according to the first embodiment is a device that executes a learning process to generate a learning model by performing machine learning. The machine learning device 1 of FIG. 6 also functions as a skill assessment device that uses the learning model to infer the skill level of an action of an actor in an input video. The machine learning device 1 is a device that can implement the machine learning method according to the first embodiment. The machine learning device 1 is, for example, a computer, but may also be a computer system configured by cloud computing using a computer network. While FIG. 6 illustrates an example in which the machine learning device that generates the learning model and the skill assessment device that assesses the skill level of an action of an actor in a target video are provided in the same computer, these may also be provided in different computers.

[0045] The machine learning device 1 has a processor 3 such as a CPU (Central Processing Unit) and a storage device 2. The storage device 2 is composed of a semiconductor memory such as a RAM (Random Access Memory), a hard disk drive (HDD), a solid state drive (SSD), etc. The machine learning device 1 may also have a communication device that communicates with external devices. An input device 4 such as a mouse and keyboard, and a display device 5 having a display are connected to the machine learning device 1. The machine learning device 1 may also have a communication device that communicates with other devices.

[0046] Each function of the machine learning device 1 is realized by a processing circuit. The processing circuit is, for example, dedicated hardware. The processing circuit may be a processor 3 that executes a program (for example, a machine learning program according to an embodiment) stored in the storage device 2. The processor 3 may be a processing device, an arithmetic unit, a microprocessor, a microcomputer, or a DSP (Digital Signal Processor).

[0047] When the processing circuit is dedicated hardware, the processing circuit is, for example, an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0048] When the processing circuit is the processor 3, the machine learning method is executed by software, firmware, or a combination of software and firmware. The software and firmware are written as programs and stored in the storage device 2. The processor 3 can implement the machine learning method according to the first embodiment by reading and executing the programs stored in the storage device 2.

[0049] Note that the machine learning device 1 may be partially implemented by dedicated hardware and partially implemented by software or firmware. In this way, the processing circuit can implement each of the above-described functions by hardware, software, firmware, or any combination thereof.

[0050] A user uses a mouse or keyboard to register a learning dataset (video dataset) via the processor 3. The processor 3 reads out a machine learning program stored in the storage device 2 and performs learning or inference. Based on the learning dataset input by the user, the processor 3 performs machine learning and inference and stores the results in the storage device as learning results.

[0051] Processor 3 extracts multiple pieces of data from the video dataset. In Processor 3, a data priority selection unit 101 extracts portions of the multiple pieces of data to be compared. In Processor 3, a model learning unit 102, a superiority / inferiority determination unit 106, and a region of interest generation unit 103 compare the selected data and evaluate their superiority / inferiority, and perform learning to obtain the evaluation value previously assigned in the dataset, and register the region of interest generation results and the created model as the learning results. The display device 5 displays the region of interest generation results, and the user then annotates the data using the input device 4. The annotation results are registered in the dataset by a region of interest editing unit 105 in Processor 3 as information for new machine learning.

[0052] Fig. 7A is a diagram showing moving image segments S1, S2, and S3 selected from a moving image pair by the data selection unit in Comparative Example 2 shown in Fig. 3 and Fig. 4. Fig. 7B is a diagram showing portions (shaded areas) selected from each segment of a moving image pair by the data priority selection unit 101 in Embodiment 1.

[0053] FIG. 8 is an explanatory diagram illustrating the effect of the machine learning device 1 according to the first embodiment. As shown in FIG. 8 , the data priority selection unit 101 appropriately selects image frames that are paired data, thereby enabling consistency in comparison of superiority / inferiority comparison points and comparison of regions of interest when dealing with fine movements, such as in videos displaying skills. This is expected to improve the accuracy of skill assessment and provide an appropriate explanation of skill level. Furthermore, by comparing image frames edited by a user, it becomes easier to evaluate the skill level between image frames as intended by the user, thereby stabilizing learning. Furthermore, by comparing image frames edited by a user with unedited image frames, learning is performed taking into account regions of interest even in unedited regions. Therefore, similar regions of interest are generated for unedited regions, even if the user has not edited them.

[0054] FIG. 9 is an explanatory diagram illustrating the operation of the machine learning device 1 according to the first embodiment. FIG. 10 is a flowchart illustrating the learning operation by the machine learning device 1 according to the first embodiment. First, the data priority selection unit 101 selects a plurality of video pairs (video #1, P ｉ And video #2 is P ｊ Next, the data priority selection unit 101 acquires an edited data list indicating which image frames in the moving images that make up the moving image pair have been edited by the user from the region-of-interest storage unit 104 (step S102).

[0055] Next, the data priority selection unit 101 assigns weights of W to edited image frames and 1 to unedited image frames, and calculates a selection probability indicating the probability that an image frame will be selected. The weight W may be changed based on an evaluation index that takes into account the editing time by the user or the degree of match between the edited data and the heat map, etc. The data priority selection unit 101 selects image frames from segments divided like a conventional temporal segment network using the calculated selection probability (step S103).

[0056] The model learning unit 102 inputs the data selected by the data priority selection unit to the CNN and extracts image features (step S104).

[0057] The region of interest generating unit 103 generates a region of interest using an architecture that has a branched CAM structure therein, and stores the generated result in the region of interest storage unit 104 (step S105).

[0058] The model learning unit 102 extracts features related to the region of interest (step S106).

[0059] The superiority / inferiority determination unit 106 extracts superiority / inferiority determination information (step S107). The superiority / inferiority determination information is an evaluation result indicating which of the movements shown in video #1 and the movements shown in video #2 has a higher skill level.

[0060] Based on correct answer data relating to the region of interest or superiority judgment given in advance by the user, the superiority judgment unit 106 calculates the difference between the region of interest generated by the region of interest generation unit 103 or the superiority judgment result judged by the superiority judgment unit 106. Based on the calculated loss, the superiority judgment unit 106 updates the parameters of the CNN of the model learning unit 102, the CAM of the region of interest generation unit 103, and its own FC layer by backpropagation (step S108).

[0061] The superiority / inferiority determining unit 106 checks in advance whether the learning convergence condition is satisfied, and if the condition is satisfied, ends the learning, and if not, repeats the learning from a plurality of selected data (step S109).

[0062] 11 is a flowchart showing annotation by the machine learning device 1 according to the first embodiment. The region of interest editing unit 105 acquires information about the region of interest from the region of interest storage unit 104 and visualizes it for the user (presents it on the display screen 111) (step S151). Next, the region of interest editing unit 105 accepts an input operation from the user and edits the region of interest (step S152). The region of interest editing unit 105 stores the new region of interest edited by the user in the region of interest storage unit 104 as learning data (step S153).

[0063] As described above, according to the first embodiment, it is possible to stabilize the learning behavior when learning a learning model for inferring the skill level of the behavior of the subject of the action in a video.

[0064] <<4>> Second Embodiment Fig. 12 is a functional block diagram schematically illustrating the configuration of a machine learning device 1a according to a second embodiment. In Fig. 12, elements that are the same as or correspond to elements shown in Fig. 5 are assigned the same reference numerals as those shown in Fig. 5. The machine learning device 1a according to the second embodiment differs from the machine learning device 1 according to the first embodiment in that it includes a data priority selection unit (also referred to as a "paired data consecutive selection unit") 101a that selects consecutive image frames, and a time-series feature extraction unit 121 that extracts features of consecutive image frames.

[0065] The data priority selection unit 101a in the machine learning device 1a according to the second embodiment determines the selection probability of an image frame based on the characteristics in the time direction of the consecutive image frames that make up each moving image pair.

[0066] In the machine learning device 1a according to the second embodiment, a plurality of consecutive image frames are selected to increase the probability (hit rate) of selecting an appropriate video pair. In the comparative example 1, the video data constituting the video pair was divided into segments and image frames to be compared were randomly selected. However, in the machine learning device 1a according to the second embodiment, the data priority selection unit 101a selects a plurality of consecutive image frames. Furthermore, the machine learning device 1a further includes a time-series feature extraction unit 121 to be able to handle the features of consecutive image frames. Because the machine learning device 1a selects a plurality of consecutive image frames, the probability that an image frame edited with a region of interest will be included in any of the image frames is higher than when an image frame edited with a region of interest is selected randomly.

[0067] There are several methods for selecting multiple consecutive image frames: In the first method, a random location is specified and a predetermined number of consecutive image frames are selected, which increases the hit rate of the data being edited by the user compared to randomly selecting each frame one by one. An example of a hit rate simulation is shown in Table 1 below.

[0068]

[0069] In the machine learning device 1a according to the second embodiment, because there is a high possibility that skills are included in the vicinity of data that has been edited by a user, selection is performed based on a probability distribution centered on the data that has been edited by a user at that time. By preparing and selecting from multiple variances of normal distributions in advance, features of long time series and features of short time series are selected.

[0070] 13 is a flowchart showing the learning operation of the machine learning device 1a according to embodiment 2. The processes of steps S201, S203 to S205, and S207 to S209 in FIG. 13 are the same as the processes of steps S101, S104 to S106, and S107 to S109 in FIG.

[0071] In the second embodiment, the data priority selection unit 101b, which functions as a paired data consecutive selection unit, selects data of a plurality of consecutive image frames (step S202). By selecting data that is consecutive in the time direction in this manner, edited data of a region of interest annotated by a user is more likely to be included in the evaluation of superiority or inferiority.

[0072] In the second embodiment, the time-series feature extraction unit 121 performs convolution in the time direction on data of a plurality of consecutive image frames to extract features in the time direction from the moving image (step S206).

[0073] As described above, according to the second embodiment, it is possible to capture continuous, detailed movements such as those in skill videos. At the same time, the hit rate for the target area is higher than when each time point is selected randomly. This makes it possible to stabilize the learning behavior when training a learning model.

[0074] Other than the above, the second embodiment is the same as the first embodiment. The data priority selection unit 101b and the time-series feature extraction unit 121 in the second embodiment are also applicable to the first embodiment.

[0075] <<5>> Third Embodiment Figure 14 is a functional block diagram showing an outline of the configuration of a machine learning device 1b according to a third embodiment. In Figure 14, elements that are the same as or correspond to elements shown in Figure 5 are assigned the same reference numerals as those shown in Figure 5. The machine learning device 1b according to the third embodiment differs from the machine learning device 1 according to the first embodiment in that it includes a region-of-interest comparison unit 131 that calculates the similarity between image frames edited by a user, and in that a data priority selection unit 101b selects data based on the similarity.

[0076] In the second embodiment, a means for increasing the hit rate of image frames edited by a user is included, but the editing content is not taken into consideration, so video pairs with different content may be selected. For this reason, the machine learning device 1b according to the third embodiment includes a region of interest comparison unit 131, which compares the editing content in addition to comparing the regions of interest. The data priority selection unit 101b adjusts the selection probability so as to preferentially select video pairs with similar editing content. This makes it possible to compare similar regions of interest, which is expected to improve the accuracy of skill level evaluation.

[0077] In Example 1 of Embodiment 3, first, the region of interest comparison unit 131 calculates the similarity between image frames that have been edited by the user. The data priority selection unit 101b selects video data by prioritizing video pairs (increasing the selection probability) for image frames that have been edited by the user. However, if data that has not been edited by the user is included in the video pair, a low similarity value (e.g., "0.01") is assigned to the video pair, thereby lowering the selection probability. The model learning unit 102 performs learning on the selected video pair, and the region of interest generation unit 103 generates a region of interest. If the selected pair is not edited by the user, the region of interest comparison unit 131 calculates the similarity with other user-edited image frames or data already selected as a pair.

[0078] In Example 2 of Embodiment 3, the region of interest comparison unit 131 first calculates the similarity between image frames edited by the user. Next, the region of interest comparison unit 131 preferentially selects pairs that are dissimilar with a certain probability. For example, the dissimilarity is calculated using the formula "(dissimilarity) = 1.0 - (similarity)" and the selection probability is determined based on this. This is because selecting dissimilar video pairs results in a certain probability of including video pairs that have not been edited by the user, making it easier for regions of interest that the user did not intend to generate, leading to the discovery of new regions of interest. The processing from this point onwards is the same as the processing in Example 1 of Embodiment 3.

[0079] In Example 3 of Embodiment 3, the region of interest comparison unit 131 first forms clusters based on similarity, since it is difficult to maintain the similarity between all image frame pairs, and selects data based on the similarity between the clusters. In this case, the region of interest comparison unit 131 classifies image frames edited by the user as a cluster (with one data item) and calculates the similarity between the clusters. Furthermore, all image frames not edited by the user are considered to belong to the unedited cluster. Next, the region of interest comparison unit 131 prioritizes the selection of clusters with higher similarity. However, the similarity between the unedited cluster and other clusters is considered to be a predetermined low value. For example, the region of interest comparison unit 131 randomly selects data included in the two selected clusters A and B, respectively, to form pairs. Here, the region of interest comparison unit 131 may select data that is not similar to the representative point or other data within the clusters, rather than randomly. This selection method allows learning to proceed using representative points and points distant from the representative as input, which can also be expected to lead to the discovery of new regions of interest. Next, the model learning unit 102 performs learning on the selected pair, and the region of interest generation unit 103 generates a region of interest. Next, if the selected pair is not edited by the user, the region of interest comparison unit 131 calculates the similarity with representative data of other clusters and assigns the data to the cluster with the highest similarity. Next, the region of interest comparison unit 131 updates the representative data as the data with the highest similarity within the cluster.

[0080] 15 is a flowchart showing the learning operation of the machine learning device 1b according to embodiment 3. The processes of steps S301 and S305 to S310 in FIG. 15 are the same as the processes of steps S101 and S104 to S109 in FIG.

[0081] The attention area comparison unit 131 selects data of a plurality of video pairs (step S301). The attention area comparison unit 131 selects attention area images (Attention Map) from the data of a plurality of video pairs (step S302). The attention area comparison unit 131 calculates the similarity between the attention area images (step S303). As a method for calculating the similarity, IoU (Intersection over Union), which calculates the degree of overlap of attention area images, may be used. The attention area comparison unit 131 selects data of a plurality of video pairs at a time t ｋ From a certain time t ｋ＋１ and time s in video #2 ｌ From a certain time s ｌ＋１ The target area comparison unit 131 performs this for all intervals and normalizes the sum so that the sum becomes 1. The data priority selection unit 101b uses random numbers to select which interval time t ｋ ~t ｋ＋１ or s ｌ ~s ｌ＋１ The data priority selection unit 101b may determine whether to select the video #1 and video #2 as a pair data section. The data priority selection unit 101b may determine the section by weighting the similarity so that images edited by the user are more likely to be selected. Furthermore, although the above calculation of similarity is performed for all combinations of video #1 and video #2, one video may be selected randomly and then selected based on the similarity with the other video.

[0082] As described above, according to the third embodiment, it is possible to capture continuous, detailed movements such as those in skill videos. At the same time, the hit rate of the target area is higher than when each time point is selected randomly. This makes it possible to stabilize the learning behavior when training the learning model.

[0083] Other than the above, the third embodiment is the same as the first embodiment. The data priority selection unit 101b in the third embodiment is also applicable to the first or second embodiment.

[0084] <<6>> Fourth Embodiment Figure 16 is a functional block diagram that schematically shows the configuration of a machine learning device 1c according to a fourth embodiment. In Figure 16, elements that are the same as or correspond to elements shown in Figure 5 are assigned the same reference numerals as those shown in Figure 5. The machine learning device 1c according to the fourth embodiment differs from the machine learning device 1 according to the first embodiment shown in Figure 5 in that it includes a motion extraction unit 141 and a motion comparison unit 142.

[0085] In the first to third embodiments, the movements of the subject of the action in the video are not sufficiently considered. In order to train a learning model for determining skill level, it is very important to determine whether the movement is the same as an excellent movement. The machine learning device 1c according to the fourth embodiment includes a movement extraction unit 141 and a movement comparison unit 142, and prioritizes data with similar movements when selecting a video pair.

[0086] According to the machine learning device 1c of embodiment 4, data with similar behavior can be compared, which makes it easier for similar skill levels to be selected as evaluation targets, and is expected to improve the accuracy of skill level evaluation.

[0087] A first example of processing for determining the similarity of movements is processing using hand pose tracking. The first example of processing can be performed by the following steps 11 to 15.

[0088] (Process 11) Image frames at time t (t=0, Δt, 2Δt, 3Δt, ..., NΔt) of each video are extracted. Here, N is a positive integer. (Process 12) Motion vectors (i.e., Flow) are calculated from image frames from time t=mΔt to time t=(m+1)Δt. Here, m=0, 1, ..., N-1. (Process 13) All frames (N ｘ ) and all frames of video Y (N ｙ For each pair, the motion vector (Δx ｌ , Δy ｌ ) and calculate the cosine distance. ｘ ×N ｙ) are calculated. (Process 14) (Process 13) is repeated for all video pairs. (Process 15) Video pairs are selected so that the higher the similarity calculated in (Process 13) is, the higher the selection priority.

[0089] As a second example of processing for determining the similarity of actions, a process for reducing the amount of calculation using clustering, etc. can be considered. The second example of processing can be performed by the following steps (Process 21) to (Process 26).

[0090] (Process 21) The same process as (Process 11) in the first processing example is performed. (Process 22) The same process as (Process 12) in the first processing example is performed. (Process 23) Only the similarity between adjacent image frames in video X is calculated, and hierarchical clustering is performed. The number of clusters is determined by a user in advance, for example. (Process 24) In (Process 23), the similarity between the data contained in the cluster is calculated, and data that is similar on average to all data in the cluster is set as representative data. (Process 25) The similarity is calculated for representative data between the cluster group (Cx) of video X and the cluster group (Cy) of video Y generated in (Process 23) and (Process 24). (Process 26) A cluster is selected using the similarity between the clusters calculated in (Process 25), and data within the cluster is randomly selected to obtain pairs. After the clusters are obtained, the same processes as in the third embodiment can be applied.

[0091] In the fourth embodiment, user editing is not used, but it is also possible to weight the motion vectors by user editing, or to select data pairs using an index that combines the similarity of the third embodiment obtained by user editing with the similarity based on the motion vectors.

[0092] Alternatively, the similarity of a specific range of a video may be calculated using a distance index such as dynamic time warping (DTW). ｍｉｎ Δt to t = m ｍａｘThe section up to Δt is selected and segmented, the DTW distance between each segment is calculated, and clustering is performed. A selection probability is set so that data belonging to the same cluster is more likely to be selected as a pair. For example, data within clusters are more likely to be selected as a pair. In this case, pairs of data from different clusters are rarely selected.

[0093] 17 is a flowchart showing the learning operation of the machine learning device 1c according to embodiment 4. The processes of steps S401 and S405 to S410 in FIG. 17 are the same as the processes of steps S101 and S104 to S109 in FIG.

[0094] The motion extraction unit 141 selects data of a plurality of moving image pairs (step S401). The motion comparison unit 142 extracts motion from the data of a plurality of moving image pairs using a technique such as Optical Flow (step S402). The motion extraction unit 141 may extract the direction in which an area obtained by dividing an image into blocks moves and store the extracted motion vector as a feature vector. The motion comparison unit 142 calculates the similarity using the cosine distance or the like of the extracted motion feature vector (step S403). The motion comparison unit 142 calculates the similarity between the extracted motion feature vectors at a certain time t ｋ From a certain time t ｋ＋１ Until the time s of video #2 ｌ From a certain time s ｌ＋１ The motion comparator 142 performs this for all sections and normalizes the sum so that the sum is 1. The data priority selector 101c selects which section time t ｋ ~t ｋ＋１ or s ｌ ~s ｌ＋１ It is determined whether to select the pair data section (step S404).

[0095] As described above, according to the fourth embodiment, motion is extracted from data of a plurality of video pairs and used to train a learning model, thereby stabilizing the learning behavior.

[0096] Other than the above, the fourth embodiment is the same as the first embodiment. The motion extractor 141 and the motion comparator 142 in the fourth embodiment are applicable to any of the first to third embodiments.

[0097] <<7>> Fifth Embodiment Fig. 18 is a functional block diagram schematically illustrating the configuration of a machine learning device 1d according to a fifth embodiment. In Fig. 18, elements that are the same as or correspond to elements shown in Fig. 5 are assigned the same reference numerals as those shown in Fig. 5. The machine learning device 1d according to the fifth embodiment differs from the machine learning device 1 according to the first embodiment shown in Fig. 5 in that it includes a foreground extraction unit 151 and a region-of-interest comparison unit 152.

[0098] In the first to fourth embodiments, examples have been described in which a learning operation is performed on the entire video. However, when training a learning model for inferring the skill level of a subject's action, the results of analyzing the background of the video may become noise, which may reduce the accuracy of determining the skill level. For this reason, the machine learning device 1d according to the fifth embodiment includes a foreground extraction unit 151 that extracts a foreground from a video pair selected from the video dataset storage unit 110, and a region-of-interest comparison unit 152 that masks the background, which is an area other than the foreground, to set the foreground as a region of interest and calculates the similarity of the region of interest.

[0099] In this way, areas unrelated to skill level are masked, so video pairs that are more likely to be directly linked to skill level are selected, which is expected to improve the accuracy of skill level assessment and the interpretability of the assessment.

[0100] 19 is a flowchart showing the learning operation of the machine learning device 1d according to embodiment 5. The processes of steps S501 and S507 to S512 in FIG. 19 are the same as the processes of steps S101 and S104 to S109 in FIG.

[0101] First, the foreground extraction unit 151 selects a plurality of data from the video data set storage unit 110 and extracts the foreground (steps S501 and S502). The foreground can be extracted by, for example, setting an area in which no change occurs between the previous image frame and the current image frame as the background.

[0102] Next, the foreground extraction unit 151 acquires the region of interest image from the region of interest storage unit 104 (step S503), and performs masking processing on the foreground and region of interest image (step S504). The region of interest comparison unit 152 calculates the similarity between the masked region of interest images (step S505). The region of interest comparison unit 152 calculates the similarity between the masked region of interest images at a certain time t ｋ From a certain time t ｋ＋１ Until the time s of video #2 ｌ From a certain time s ｌ＋１ The target area comparison unit 152 performs this for the entire interval and normalizes the sum so that the sum is 1. The data priority selection unit 101d uses random numbers to select which interval time t ｋ ~t ｋ＋１ or s ｌ ~ s ｌ＋１ Determine whether to select as a pair data section.

[0103] As described above, according to the fifth embodiment, movements are extracted from data of a plurality of video pairs and used to train a learning model, thereby stabilizing the learning behavior.

[0104] Other than the above, the fifth embodiment is the same as the first embodiment. Furthermore, the foreground extraction unit 151 and the region-of-interest comparison unit 152 in the fifth embodiment can also be applied to any of the first to fourth embodiments.

[0105] 1, 1a to 1d Machine learning device, 2 Storage device, 3 Processor, 4 Input device, 5 Display device, 101, 101a to 101d Data priority selection unit, 102 Model learning unit, 103 Region of interest generation unit, 104 Region of interest storage unit, 105 Region of interest editing unit, 106 Superiority / inferiority determination unit, 110 Video data set storage unit, 111, 112 Display example, 121 Time series feature extraction unit, 131 Region of interest comparison unit, 141 Motion extraction unit, 142 Motion comparison unit, 151 Foreground extraction unit.

Claims

1. A machine learning device that learns a learning model for inferring the skill level of an action of a subject in a video, comprising: a data priority selection unit that selects a plurality of video pairs from a learning video dataset, and selects an image frame from each of the selected plurality of video pairs to be used for judging the superiority or inferiority of the skill level; a region of interest generation unit that generates a region of interest in the image frames to be used for judging the superiority or inferiority of the skill level; a superiority or inferiority judgment unit that uses the learning model to judge the superiority or inferiority of the skill level in the region of interest for each video pair; and a model learning unit that stores the learning model and updates the learning model based on the results of the judgment of the superiority or inferiority of the skill level, wherein the data priority selection unit selects the image frame to be used for judging the superiority or inferiority of the skill level using a selection probability determined based on one or more of user edits in the image frames that constitute each video pair, time-wise features in a plurality of consecutive image frames that constitute each video pair, and similarity between the image frames that constitute each video pair.

2. The machine learning device described in claim 1, characterized in that the data priority selection unit makes the selection probability of the image frames that have user editing among the image frames that constitute each video pair higher than the selection probability of the image frames that do not have user editing.

3. The machine learning device of claim 1 or 2, characterized in that, when there is user editing in the image frames that constitute each video pair, the data priority selection unit increases the selection probability of the image frame the longer the time range in which the user editing is performed.

4. The machine learning device described in any one of claims 1 to 3, characterized in that, when there is user editing in the image frames that constitute each video pair, the longer the time required for the user editing, the higher the selection probability of the image frame.

5. The machine learning device described in any one of claims 1 to 4, characterized in that when there is user editing in the image frames that constitute each video pair, the data priority selection unit increases the selection probability of the image frame the greater the difference between before and after editing of the area of ​​interest.

6. The machine learning device described in any one of claims 1 to 5, characterized in that, when there is user editing in the image frames that constitute each video pair, the data priority selection unit increases the selection probability of the image frame the smaller the area of ​​the region of interest.

7. A machine learning device as described in any one of claims 1 to 6, characterized in that the data priority selection unit determines the selection probability of the image frame based on the time-direction characteristics of multiple consecutive image frames that constitute each video pair.

8. A machine learning device as described in any one of claims 1 to 7, further comprising a region of interest comparison unit that calculates the similarity for the region of interest of each video pair, and the data priority selection unit increases the selection probability the higher the similarity between the regions of interest of the image frames that constitute each video pair.

9. A machine learning device as described in any one of claims 1 to 8, further comprising: a motion extraction unit that extracts motion in the region of interest of each video pair; and a motion comparison unit that calculates the similarity of the motion in the region of interest of each video pair, wherein the data priority selection unit increases the selection probability as the similarity of the motion becomes higher.

10. A machine learning device as described in any one of claims 1 to 9, further comprising: a foreground extraction unit that extracts the foreground of each video pair; and a region of interest comparison unit that calculates a similarity of the region of interest in the foreground, wherein the data priority selection unit increases the selection probability the higher the similarity of the region of interest in the foreground.

11. The machine learning device according to any one of claims 1 to 10, characterized in that the operating subject is a person or a mechanism that moves in conjunction with the movement of a part of a person's body.

12. A skill assessment device comprising: a video input; a learning model generated by a machine learning device according to any one of claims 1 to 11; and a skill level of the behavior of a subject in a target video determined by the learning model.

13. A machine learning method for training a learning model for inferring a skill level of an action of a subject in a video, comprising the steps of: selecting a plurality of video pairs from a learning video dataset; and selecting an image frame to be used for judging the superiority or inferiority of the skill level from each video pair constituting the selected plurality of video pairs; generating a region of interest in the image frames to be used for judging the superiority or inferiority of the skill level; for each video pair, judging the superiority or inferiority of the skill level in the region of interest using the learning model; storing the learning model and updating the learning model based on the result of the judgment of the superiority or inferiority of the skill level; wherein the step of selecting the image frame to be used for judging the superiority or inferiority of the skill level selects the image frame to be used for judging the superiority or inferiority of the skill level using a selection probability determined based on one or more of user editing in the image frames constituting each video pair, time-directional features in a plurality of consecutive image frames constituting each video pair, and similarity between the image frames constituting each video pair.

14. A machine learning program for causing a computer to learn a learning model for inferring the skill level of an action of an actor in a video, comprising the steps of: selecting a plurality of video pairs from a learning video dataset; and selecting an image frame to be used for judging the superiority or inferiority of the skill level from each video pair constituting the plurality of selected video pairs; generating a region of interest in the image frames to be used for judging the superiority or inferiority of the skill level; for each video pair, judging the superiority or inferiority of the skill level in the region of interest using the learning model; storing the learning model and updating the learning model based on the result of the judgment of the superiority or inferiority of the skill level; wherein the step of selecting the image frame to be used for judging the superiority or inferiority of the skill level selects the image frame to be used for judging the superiority or inferiority of the skill level using a selection probability determined based on one or more of user editing in the image frames constituting each video pair, time-directional features in a plurality of consecutive image frames constituting each video pair, and similarity between the image frames constituting each video pair.

Images