Learning system, worker action recognition system, and recognition model learning method

Abstract

To provide a recognition model capable of recognizing an action with a high risk even if a video is subjected to insufficient trimming.SOLUTION: A learning system includes: a learning video generating unit which couples, at a coupling point, two videos that are a main category video belonging to a main category contained in action categories, and a sub category video belonging to a sub category contained in the action categories but different from the main category, and performs trimming in such a way that the coupling point is included so as to generate a learning video; a learning unit that performs, using the learning video, learning on a recognition model which outputs the main category to the input learning video; and a performance evaluating unit that evaluates a prediction result by the recognition model so as to generate an evaluation result. The sub category is selected on the basis of a risk level of the main category and that of the sub category both in the action categories, and the evaluation result.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a learning system, a worker action recognition system, and a method for learning a recognition model. [Background technology]

[0002] The long tail is a fundamental problem often faced in recognition work using artificial intelligence. This problem is that there is very little training data for some categories compared to other categories. The same unevenness in training data is likely to occur in industrial worker activity recognition. In industry, some activities are more risky than others. For example, climbing onto a roof is much riskier than tightening screws, and if performed frequently, it can lead to an accident. As such, because some activities involve high risks, it is very difficult to collect the same amount of training data for all activities, resulting in a long tail dataset.

[0003] Creating an activity recognition model using such a long-tail dataset results in biased performance. Specifically, it shows very good performance for categories with sufficient training data and very poor performance for categories with little training data. This is because traditionally, recognition models are updated by assigning equal weight to all training samples, resulting in learning more from categories with more data. The simplest way to solve this problem is to collect more data for activities with less data. However, this is not practical because categories with less data typically represent high-risk activities, such as roof climbing, that cannot be repeated many times. There is a need for a system that can recognize such dangerous activities with high accuracy and provide support for preventing worker accidents.

[0004] Previous research has attempted to solve the long tail without collecting more data. Two common approaches to the long tail are data resampling and weighted loss. Data resampling involves sampling training samples with unequal sampling probabilities, generally with higher sampling probabilities for categories with fewer training samples. Weighted loss assigns a weight to each sample, which is used to increase or decrease the update from each sample. The weight is generally higher for categories with fewer training samples, with the goal of balancing the amount of update from all categories.

[0005] Patent Document 1 uses a weighted loss, and other methods using data resampling are also known. However, most of these methods have been proposed for image recognition tasks. Furthermore, a method that introduces a frame stacking technique to address long-tail video recognition problems is also known. In this method, two categories are randomly sampled, and frames are sampled and stacked from each category based on their training data frequency. Next, a recognition model is trained using soft labels to achieve balanced learning from the two categories. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] US Patent Application Publication No. 2019 / 0232964 Summary of the Invention [Problem to be solved by the invention]

[0007] Patent Document 1 and other known techniques assume that the provided video is properly trimmed, i.e., segmented in the time domain, during inference. However, in reality, it is quite conceivable that proper trimming is not performed. For example, if a video contains both high-risk actions and portions of low-risk actions, the recognition system is likely to recognize only the low-risk actions and ignore the risky actions. However, in such improperly trimmed videos, it is very important to recognize high-risk actions in order to prevent potential accidents in the industrial world. [Means for solving the problem]

[0008] A learning system according to a first aspect of the present invention comprises a learning video generation unit that combines two videos, a main category video that is a video belonging to a main category included in a behavioral category, and a sub-category video that is a video belonging to a sub-category that is included in the behavioral category and is different from the main category, at a connection point and trims the video to include the connection point to generate a learning video; a learning unit that uses the learning video to train a recognition model that outputs the main category for the input learning video; and a performance evaluation unit that evaluates the prediction results by the recognition model to generate evaluation results, and the sub-category is selected based on the risk levels of the main category and the sub-category in the behavioral category and the evaluation results. A worker action recognition system according to a second aspect of the present invention comprises a recognition model generated by the aforementioned learning system, a detector for detecting a worker, a tracker for tracking the worker, and an extractor for extracting a candidate area, which is an area in which the worker is present, using the detector and the tracker, and the recognition model receives as input an image of the candidate area. A worker action recognition system according to a third aspect of the present invention comprises a recognition model generated by the aforementioned learning system, a detector for detecting the worker's hands, a tracker for tracking the worker's hands, and an extractor for extracting a candidate area, which is an area where the worker is present, using the detector and the tracker, and the recognition model receives as input an image of the candidate area. A fourth aspect of the present invention provides a method for training a recognition model that outputs a category corresponding to an input video, and includes the steps of: combining two videos, a main category video that is a video belonging to a main category included in a behavioral category, and a sub-category video that is a video belonging to a sub-category included in the behavioral category but different from the main category, at a connection point, and trimming the videos to include the connection point to generate a training video; using the training video, training the recognition model to output the main category for the input training video; and evaluating the prediction results by the recognition model to generate an evaluation result, wherein the sub-category is selected based on the risk levels of the main category and the sub-category in the behavioral category and the evaluation result. [Effects of the Invention]

[0009] According to the present invention, it is possible to provide a recognition model that can recognize high-risk behavior even in videos that are insufficiently trimmed. [Brief explanation of the drawings]

[0010] [Figure 1] Configuration diagram of a learning system according to the first embodiment [Figure 2] Conceptual diagram explaining segmented video and composite video [Figure 3] FIG. 10 is a diagram showing an example of an action label group. [Figure 4] FIG. 10 is a diagram showing an example of a history of decisions made by a main category decision unit; [Figure 5] A diagram showing an example of statistical information [Figure 6] An example of risk information [Figure 7] A diagram showing an example of evaluation results [Figure 8] Learning system hardware configuration diagram [Figure 9] Flowchart showing the operation of the relative difficulty calculation unit [Figure 10] Flowchart showing the operation of the determination unit [Figure 11]Flowchart showing the operation of the learning video generation unit [Figure 12] A diagram showing a specific example of the operation of the learning video generation unit [Figure 13] Flowchart showing the operation of the performance evaluation unit [Figure 14] Configuration diagram of a learning system according to a second embodiment [Figure 15] Flowchart showing the operation of the learning video generation unit in the second embodiment [Figure 16] Configuration diagram of a learning system according to a third embodiment [Figure 17] 10 is a flowchart showing the operation of the main category determination unit in the third embodiment. [Figure 18] FIG. 13 is a diagram showing an example of gaze parameters in a modified example of the third embodiment. [Figure 19] Configuration diagram of a learning system according to the fourth embodiment [Figure 20] Diagram explaining GUI [Figure 21] Worker motion recognition system configuration diagram DETAILED DESCRIPTION OF THE INVENTION

[0011] -First embodiment-

[0012] A first embodiment of a learning system will be described below with reference to FIGS.

[0013] FIG. 1 is an overall configuration diagram of learning system 1. Learning system 1 has, as its functions, a main category determination unit 103, a sub-category determination unit 106, a learning video generation unit 107, a performance evaluation unit 113, a learning unit 114, and a risk information generation unit 118. Learning system 1 also has information such as a segmented video group 101, an action label group 102, statistical information 119, risk information 104, a ratio parameter 105, and an evaluation result 117. Before describing the elements that make up learning system 1, the classification of videos in this embodiment will be explained.

[0014] FIG. 2 is a conceptual diagram illustrating segmented videos and composite videos in this embodiment. Video V100 is a 40-second video, i.e., a video clip. Video V100 includes multiple scenes, such as a worker walking scene from 0 to 10 seconds, a scene of tightening screws from 10 to 20 seconds, a scene of working on a roof from 20 to 30 seconds, and a scene of creating a record from 30 to 40 seconds. Each scene includes only one type of human action. In this embodiment, human actions are classified as "action labels." Action labels include, for example, walking, running, tightening screws, hammering, climbing onto the roof, and creating a record.

[0015] In this embodiment, this video V100 is trimmed to include only one scene, i.e., extracted in the time domain, and is referred to as a "divided video." In contrast, a video that is incompletely divided and trimmed to include multiple scenes is referred to as a "composite video." That is, a composite video includes multiple actions by a person. For example, video V101, which is obtained by extracting the section from 0 to 10 seconds of video V100, video V102, which is obtained by extracting the section from 10 to 20 seconds of video V100, video V103, which is obtained by extracting the section from 20 to 30 seconds of video V100, and video V104, which is obtained by extracting the section from 30 to 40 seconds of video V100, are all divided videos. In contrast, video V111, which is obtained by extracting the section from 0 to 13 seconds of video V100, video V112, which is obtained by extracting the section from 5 to 20 seconds of video V100, and video V113, which is obtained by extracting the section from 20 to 45 seconds of video V100, are all composite videos. Note that in this embodiment, trimming a video refers to extracting a video, i.e., a video clip, in the time domain, as shown in FIG. 2. Returning to FIG. 1, we will continue the explanation.

[0016] The divided moving image group 101 is a collection of divided moving images created in advance. The action label group 102 is a collection of action labels created in advance and corresponding to each moving image included in the divided moving image group 101.

[0017] The main category determination unit 103 determines the main category of a video to be included in a composite video generated by the learning video generation unit 107. The main category determination unit 103 may determine the main category by referring to the action label group 102, or may determine the main category by referring to the risk information 104. The main category determination unit 103 outputs information about the determined main category to the subcategory determination unit 106, the learning video generation unit 107, the loss calculation unit 111, and the performance evaluation unit 113. However, for convenience of illustration, arrows from the main category determination unit 103 to the loss calculation unit 111 and the performance evaluation unit 113 are not shown in FIG. 1. The detailed operation of the main category determination unit 103 will be described later.

[0018] The sub-category determination unit 106 determines the sub-category of the video to be included in the training video generated by the training video generation unit 107. The sub-category determination unit 106 outputs the determined sub-category to the training video generation unit 107. The sub-category determination unit 106 receives as input the main category determined by the main category determination unit 103, the ratio parameter 105, the risk information 104, and the performance of the recognition model 108 output by the performance evaluation unit 113. The training video generation unit 107 generates a training video, which is a composite video, based on the main category determined by the main category determination unit 103 and the sub-category determined by the sub-category determination unit 106. The training video generation unit 107 outputs the generated training video to the recognition model 108.

[0019] The learning unit 114 includes a recognition model 108, a prediction result 110, a loss calculation unit 111, an update unit 112, and model parameters 109. The learning unit 114 updates the model parameters 109 of the recognition model 108 using training videos generated by a training video generation unit 107. The training video generation unit 107 inputs the training video to the learning unit 114, but since the training process of a neural network that outputs labels of input videos is widely known, the configuration and processing of the learning unit 114 will be explained briefly.

[0020] The recognition model 108 is any video recognition model, such as Transformers, LSTM (Long Short Term Memory), or 3D-CNN (3D Convolutional Neural Network). The model parameters 109 are the weights and biases of the model. When a training video is input from the training video generation unit 107, the recognition model 108 outputs a prediction result 110. This prediction result 110 is compared with the correct category information of the training video, i.e., the main category output by the main category determination unit 103, by both the loss calculation unit 111 and the performance evaluation unit 113.

[0021] The loss calculation unit 111 calculates the loss using, for example, a cross-entropy loss function. However, the loss calculation unit 111 may also use other loss functions such as focus loss, mean squared error loss, and binary cross-entropy loss. The update unit 112 updates the model parameters 109 using the loss calculated by the loss calculation unit 111. The update unit 112 may use, for example, a backpropagation method. The performance evaluation unit 113 calculates the performance of the recognition model 108 using the prediction result 110 and feeds it back to the subcategory determination unit 106. The performance evaluation unit 113 calculates the performance of the recognition model 108 for a predetermined number of batches or a predetermined number of epochs. The learning process by the learning unit 114 is repeated until the loss reaches a minimum value or the number of iterations reaches a maximum value.

[0022] Fig. 3 is a diagram showing an example of the action label group 102. The action label group 102 is made up of multiple records, and each record has fields for an identifier 201 and an action label 202. "VC001" and the like written in the identifier 201 column are identifiers of videos included in the divided video group 101. "A1", "A2", and the like written in the action label 202 column are action labels. In the example shown in Fig. 3, it is shown that the action label of the video "VC001" is "A1".

[0023] Fig. 4 is a diagram showing an example of the history of determinations made by the main category determination unit 103. The example shown in Fig. 4 shows that the first determination made by the main category determination unit 103 determined the main category to be "A4," the second determination determined "A2," and the third determination determined "A7." Note that this information may be transmitted each time the main category determination unit 103 makes a determination, or may be transmitted collectively after multiple determinations have been made.

[0024] FIG. 5 is a diagram showing an example of statistical information 119. The statistical information 119 may be statistics related to videos included in the segmented video group 101, or may be statistics for a predetermined period, for example, the past year, regardless of whether the video is included in the segmented video group 101. The statistical information 119 is made up of multiple records, and each record has fields for action label 1191, number of accidents 1192, and total number 1193. The action label 1191 has the same meaning as the action label 1022. The number of accidents 1192 is the number of accidents that occurred in the work corresponding to the action label 1191 of the same record. The total number 1193 is the total number of times the work corresponding to the action label 1191 of the same record was performed. That is, in the statistical information 119, the number of accidents 1192 and the total number 1193 are actually entered as numerical values.

[0025] Since statistical information 119 is used for a relative evaluation of the risk between action labels 1191, various standards may be used as long as the standard is consistent within one statistical information 119. For example, total number 1193 may be the number of videos in which an action corresponding to action label 1191 is captured, or the number of times an action corresponding to action label 1191 is performed regardless of whether it is captured or not. Furthermore, total number 1193 may be information indicating the length of time that an action corresponding to action label 1191 is performed, for example, a number in "minutes" indicating the total time that it is performed.

[0026] FIG. 6 is a diagram showing an example of risk information 104. The risk information 104 is composed of multiple records, each of which has fields for an action label 1041 and a risk index 1042. The action label 1041 is the same as the action label 202, etc. The risk index 402 is a numerical value indicating the magnitude of risk of an action represented by the action label 401 in the same record. The risk index 402 may be defined and calculated in any manner as long as it allows comparison between action labels. For example, the risk information generation unit 118 may calculate the risk index 402 using the statistical information 119, or a human may determine the risk index 402 based on the results of a survey. For example, the risk information generation unit 118 may set the risk index 402 to a value between 0 and 1 obtained by dividing the number of accidents 501 in the statistical information 119 by the total number 502 for each action label.

[0027] 7 is a diagram showing an example of the evaluation result 117 by the performance evaluation unit 113. The evaluation result 117 is composed of multiple records, and each record has a field for an action label and performance. In the example shown in FIG. 7, classification accuracy is used as the performance, but various other metrics such as accuracy, f-measure, error, etc. may also be used.

[0028] FIG. 8 is a hardware configuration diagram of the learning system 1. The learning system is composed of one or more arithmetic units 40. The arithmetic unit 40 includes a CPU 41, which is a central processing unit; a ROM 42, which is a read-only storage device; a RAM 43, which is a read / write storage device; an input / output device 44, which is a user interface; and a communication device 45. The CPU 41 performs the various calculations described above by expanding the programs stored in the ROM 42 into the RAM 43 and executing them. That is, the main category determination unit 103, the sub-category determination unit 106, the learning video generation unit 107, the performance evaluation unit 113, the loss calculation unit 111, the update unit 112, and the risk information generation unit 118 are realized by the CPU 41.

[0029] The divided video group 101, the action label group 102, the statistical information 119, the risk information 104, the ratio parameter 105, and the evaluation result 117 may be stored in the RAM 43 or in a non-volatile storage device (not shown). The communication device 45 is used for communication with other arithmetic devices 40. When the learning system 1 is configured with a single arithmetic device 40, the arithmetic device 40 does not need to be equipped with the communication device 45.

[0030] The arithmetic device 40 may be realized by a field programmable gate array (FPGA), which is a rewritable logic circuit, or an application specific integrated circuit (ASIC), which is an integrated circuit for a specific application, instead of the combination of the CPU 41, the ROM 42, and the RAM 43. Furthermore, the arithmetic device 40 may be realized by a combination of different configurations, for example, a combination of the CPU 41, the ROM 42, the RAM 43, and the FPGA, instead of the combination of the CPU 41, the ROM 42, and the RAM 43.

[0031] The main category determination unit 103 operates in the following two ways. The first operation is determining the correct frequency. In this case, the main category determination unit 103 randomly extracts one regular divided video from the divided video group 101, and reads out the action label corresponding to that regular divided video from the action label group 102. In this case, since videos are extracted randomly from the divided video group 101, the proportion of categories of videos included in the divided video group 101 is directly proportional to the proportion of extracted categories. The more videos a category contains in the divided video group 101, the more likely it is to be selected. In other words, in this case, it can be said that categories are selected with a directly proportional frequency.

[0032] The second operation is inverse frequency determination. In this case, the main category determination unit 103 first reads the entire action label group 102 and calculates the presence ratio of action labels for videos included in the divided video group 101. The main category determination unit 103 then selects an action label with a probability corresponding to the inverse of the presence ratio. Therefore, in inverse frequency determination, categories with fewer videos included in the divided video group 101 are more likely to be selected. Inverse frequency determination is effective for long-tail recognition problems.

[0033] 9 is a flowchart showing the operation of the relative difficulty calculation unit 703. First, in step S301, the relative difficulty calculation unit 703 reads necessary information. The information read in this step is the risk information 104, the main category determined by the main category determination unit 103, the evaluation result 117 calculated by the performance evaluation unit 113, and the ratio parameter 105. In the following step S302, the relative difficulty calculation unit 703 initializes a variable i to 1 and proceeds to step S303. This variable i indicates an action category, and changing the value of the variable i changes the action category indicated by the action label Ai. In step S303, the relative difficulty calculation unit 703 calculates the relative difficulty d(n, i) between the action label An and the action label Ai. The relative difficulty d(n, i) is calculated using the following equation 1.

[0034] d(n, i)=POW(Rn / Ri, γ)*POW(ACCi / ACCn, 1-γ) (Formula 1)

[0035] In Equation 1, the function POW indicates the exponentiation of the second argument to the first argument; for example, POW(10, 2) indicates the square of 10, which is 100. In Equation 1, Rn is the risk index of action label An, Ri is the risk index of action label Ai, ACCi is the evaluation value of action label Ai, and ACCn is the evaluation value of action label An. γ is a ratio parameter 105, which is a preset value between 0 and 1.

[0036] Note that action label An is the action label of the main category, and action label Ai is the action label of the category corresponding to variable i. If the risk index of action label An is greater than the risk index of action label Ai, Rn / Ri is greater than 1. ACCi / ACCn is the bias of category n relative to category i in model performance. As γ increases, the weight of the risk ratio at relative difficulty d increases, and as γ decreases, the weight of the risk ratio at relative difficulty d decreases.

[0037] In the following step S304, the relative difficulty calculation unit 703 increments the variable i, i.e., increases the value of the variable i by "1", and proceeds to step S305. In step S305, the relative difficulty calculation unit 703 determines whether the value of the variable i exceeds the total number C of action labels. If the relative difficulty calculation unit 703 determines that the value of the variable i exceeds the total number C of action labels, the process proceeds to step S306, and if it determines that the value of the variable i does not exceed the total number C of action labels, the process returns to step S303. In step S306, the relative difficulty calculation unit 703 outputs the relative difficulty d calculated in step S303 up to this point to the determination unit 704, and ends the processing shown in FIG. 9. In steps S302 to S305, the variable i is changed from 1 to C to calculate the relative difficulty, so the relative difficulty d output in step S306 is d(n,1), d(n,2), d(n,3), . . . , d(n,C).

[0038] 10 is a flowchart showing the operation of the determination unit 704. First, in step S311, the determination unit 704 reads the relative difficulty d calculated by the relative difficulty calculation unit 703. In the following step S312, the determination unit 704 identifies the category number of the relative difficulty having the largest value from among the multiple read relative difficulty levels d. In the following step S313, the determination unit 704 outputs the category number identified in step S312 as the number of the subcategory. In the following step S314, the determination unit 704 outputs the largest relative difficulty and ends the processing shown in FIG.

[0039] FIG. 11 is a flowchart showing the operation of the learning video generation unit 107. The processing of FIG. 11 will be described below with reference to FIG. 12, which shows a specific example of the operation of the learning video generation unit 107. The learning video generation unit 107 first reads necessary information in step S321. The information read in this step is the main category and sub-category. In the following step S322, the learning video generation unit 107 randomly selects one video of the main category from the divided video group 101. However, if step S322 has been executed before, the processing of this step may be omitted and a video of the same main category as the previous time may be selected.

[0040] In the next step S323, the learning video generation unit 107 randomly selects one video of the sub-category from the divided video group 101. In the next step S323, the learning video generation unit 107 simply combines the two videos selected in steps S322 and S323. The simple combination here refers to the process of joining the beginning of one video to the end of the other video to create a single video. This will be explained in detail with reference to FIG. 12.

[0041] In FIG. 12, the symbol Vp indicates a main category video selected in step S322. The symbol Vs indicates a sub-category video selected in step S323. For the following explanation, the length of the main category video Vp is referred to as Tp, and the length of the sub-category video Vs is referred to as Ts. In step S324, the sub-category video Vs is placed after the main category video Vp, or the main category video Vp is placed after the sub-category video Vs. In either case, the length of the simply combined videos is Tp+Ts. Whether the main category video Vp or the sub-category video Vs is placed after the other is determined randomly.

[0042] In the following step S325, the learning video generation unit 107 identifies the switching timing Tw. The switching timing Tw is the timing at which the main category video Vp and the sub-category video Vs switch in the simply combined video Vc. The switching timing Tw can be indicated, for example, by the time from the beginning of the simply combined video Vc or the time from the end of the simply combined video Vc. In this embodiment, the switching timing Tw is indicated by the time from the beginning of the simply combined video Vc. When the sub-category video Vs is arranged after the main category video Vp to form the simply combined video Vc, the switching timing Tw is Tp, which is the length of the main category video Vp. When the main category video Vp is arranged after the sub-category video Vs to form the simply combined video Vc, the switching timing Tw is Ts, which is the length of the sub-category video Vs.

[0043] In the following step S326, the learning video generation unit 107 determines a front length L1, which is the length of the video before the switch, and a back length L2, which is the length of the video after the switch. The front length L1 and the back length L2 may be determined randomly or may be specified in advance by the operator. Furthermore, instead of directly determining or specifying the front length L1 and the back length L2, the sum of the front length L1 and the back length L2, or the ratio of the front length L1 to the back length L2, may be determined or specified. In the following step S326, the learning video generation unit 107 extracts the video from time "Tw-L1" ​​to time "Tw+L2" in the simply combined video Vc as a learning video, and then ends the processing in FIG. 11. As shown in FIG. 12, the learning video VT has a length of L1+L2, and is a video in which the main category video Vp and the sub-category video Vs switch at time L1.

[0044] The processing of the learning video generation unit 107 described above can be summarized as follows: The learning video generation unit 107 generates a learning video by joining two videos, a main category video Vp and a sub-category video Vs, which is a video belonging to a sub-category different from the main category, at a joining point at the end of one of the videos, and trimming it to include the joining point.

[0045] 13 is a flowchart showing the operation of the performance evaluation unit 113. In step S331, the performance evaluation unit 113 reads the prediction result 110 by the recognition model 108. As mentioned above, the performance evaluation unit 113 is given in advance information on the main category of each training video VT. In the following step S332, the performance evaluation unit 113 initializes a variable i to 1. This variable i indicates an action category, and changing the value of the variable i changes the action category indicated by the action label Ai.

[0046] In the following step S333, the performance evaluation unit 113 evaluates the prediction result 110. Specifically, the performance evaluation unit 113 determines whether or not the prediction result 110 matches the action label Ai for the training video VT for which all main categories for which prediction by the recognition model 108 has been completed are Ai. In the following step S334, the performance evaluation unit 113 calculates an evaluation value ACCi for the action label Ai of the recognition model 108. The evaluation value ACCi takes a value between 0 and 1 and is calculated using the following equation 2.

[0047] ACCi = GOOD_VTi / ALL_VTi (Formula 2)

[0048] where GOOD_VTi is the number of training videos whose main category is Ai that the recognition model 108 correctly determined to be category Ai, and ALL_VTi is the number of training videos whose main category is Ai. In the following step S335, the performance evaluation unit 113 increments the variable i, i.e., increases the value of the variable i by "1", and proceeds to step S336. In step S336, the performance evaluation unit 113 determines whether the value of the variable i exceeds the total number C of action labels. If the performance evaluation unit 113 determines that the value of the variable i exceeds the total number C of action labels, it proceeds to step S337, and if it determines that the value of the variable i does not exceed the total number C of action labels, it returns to step S333. In step S337, the performance evaluation unit 113 outputs the total "C" evaluation values ​​calculated in step S334, i.e., ACC1, ACC2, . . . , ACCC, as the evaluation result 117, and ends the processing shown in FIG. 13.

[0049] According to the first embodiment described above, the following advantageous effects can be obtained. (1) The learning system 1 includes a learning video generation unit 107 that generates a learning video by joining two videos, a main category video that is a video belonging to a main category included in an action category, and a secondary category video that is a video belonging to a secondary category that is included in an action category but different from the main category, at a joining point at the end of one of the videos and trimming the video to include the joining point; a learning unit 114 that uses the learning video to train a recognition model 108 that outputs a main category for an input learning video; and a performance evaluation unit 113 that evaluates a prediction result 110 by the recognition model 108 and generates an evaluation result 117. The secondary category is selected based on the risk levels of the main category and secondary category in the action category and the evaluation result 117. Therefore, a recognition model 108 that can recognize high-risk behaviors can be provided even in videos that are insufficiently trimmed.

[0050] (2) The learning system 1 includes a sub-category determination unit 106 that determines, as a sub-category, the behavior category that most often results in erroneous determination of the main category determined by the recognition model 108. This improves the accuracy of the recognition model 10.

[0051] (3) The learning system 1 includes a main category determination unit 103 that determines an activity category as a main category more frequently as the risk in the activity category increases. This reduces errors in high-risk activity categories made by the recognition model 108.

[0052] (4) The subcategory determination unit 106 determines a subcategory based on the risk information 104 indicating the risk of each category included in the behavior category and the evaluation result 117, as shown in Equation 1.

[0053] (5) The learning system 1 includes a risk information generation unit 118 that calculates the risk information 104 based on past statistics. Therefore, the user does not need to generate the risk information 104 manually.

[0054] (6) The subcategory determination unit 106 determines a subcategory based on the risk information 104, the evaluation result 117, and a ratio parameter 105 represented by a variable γ. The ratio parameter 105 determines the ratio of the influence of the risk information 104 and the evaluation result 117 on the determination of the subcategory, as shown in Equation 1. Therefore, by appropriately setting the value of the ratio parameter 105, it is possible to achieve a balance between risk and evaluation.

[0055] (7) The performance evaluation unit 113 evaluates the performance of the recognition model 108 for each predetermined number of batches.

[0056] (8) A learning method of the recognition model 108 is a learning method of the recognition model 108 that outputs a category corresponding to an input video, and includes: combining two videos, a main category video that is a video belonging to a main category included in a behavioral category, and a sub-category video that is a video belonging to a sub-category that is included in a behavioral category and different from the main category, at a connection point, trimming the video to include the connection point, and generating a learning video; using the learning video, training the recognition model 108 to output a main category for the input learning video; and evaluating the prediction result 110 by the recognition model 108 to generate an evaluation result 117, where the sub-category is selected based on the risk of the main category and the sub-category in the behavioral category and the evaluation result 117.

[0057] --Second embodiment-- A second embodiment of the learning system will be described with reference to Figures 14 and 15. In the following description, the same components as in the first embodiment are given the same reference numerals, and differences will be mainly described. Points that are not particularly described are the same as in the first embodiment. In this embodiment, the main difference from the first embodiment is the processing of the learning video generation unit.

[0058] 14 is a configuration diagram of a learning system 1A in the second embodiment. Learning system 1A includes a learning video generation unit 107A instead of learning video generation unit 107 in the first embodiment. The second embodiment differs from the first embodiment only in the operation of learning video generation unit 107A.

[0059] Fig. 15 is a flowchart showing the operation of the learning video generation unit 107A in the second embodiment. Fig. 15 differs from Fig. 11 in that steps S323A and S323B are added between steps S323 and S324. The differences between Fig. 15 and Fig. 11 will be explained below. Points that are not particularly explained are the same as those in the first embodiment.

[0060] In step S323A, the learning video generation unit 107A determines whether the relative difficulty d between the main category and the sub-category is greater than a predetermined threshold. This relative difficulty is calculated by the relative difficulty calculation unit 703. If the learning video generation unit 107A determines that the relative difficulty d is greater than the predetermined threshold THRE, the process proceeds to step S324, where it performs the same processing as in the first embodiment. If the learning video generation unit 107A determines that the relative difficulty d is equal to or less than the predetermined threshold THRE, the process proceeds to step S323B. In step S323B, the learning video generation unit 107A outputs the video of the main category selected in step S322 as the learning video, and the processing shown in FIG. 15 ends.

[0061] By increasing the value of the threshold THRE, the amount of calculation and time required for training can be reduced. However, if the value of the threshold THRE is too large, the number of training videos that combine videos from multiple categories will be too small, reducing the robustness of the recognition model 108 against imperfect trimming. Therefore, the operator must carefully set the value of the threshold THRE.

[0062] According to the second embodiment described above, the following advantageous effects can be obtained. (9) The learning video generation unit 107 generates learning videos when the recognition performance of the recognition model 108 determines that the proportion of learning videos of the main category as sub-categories is higher than a predetermined threshold, i.e., when a positive judgment is made in S323A of Fig. 15. Therefore, when the performance of the recognition model 108 is sufficient, the processing load can be reduced by omitting the generation of learning videos.

[0063] -Third embodiment- A third embodiment of a learning system will be described with reference to Figures 16 to 18. In the following description, the same components as in the first embodiment are designated by the same reference numerals, and differences will be mainly described. Points that are not particularly described are the same as in the first embodiment. In this embodiment, the main difference from the first embodiment is the operation of the main category determination unit.

[0064] During training, videos with high-risk action labels are sampled more frequently than videos with low-risk action labels because risky actions are more difficult for the model to learn due to data limitations. Training risky actions more frequently allows the recognition model 108 to learn a balanced set of risky and non-risky actions, which is desirable because it makes the model more robust to insufficient trimming of risky categories.

[0065] 16 is a configuration diagram of a learning system 1B according to the third embodiment. In this embodiment, the operation of the main category determination unit 103A differs from that of the first embodiment. In addition, risk information 104 is provided not only to the subcategory determination unit 106 but also to the main category determination unit 103A. The main category determination unit 103A reads not only the action label group 102 but also the risk information 104 and gaze parameters 1801.

[0066] 17 is a flowchart showing the operation of the main category determination unit 103A in the third embodiment. First, the main category determination unit 103A reads necessary information in step S351. In the following step S352, the main category determination unit 103A calculates the video frequency fi of each category using the action label group 102. The video frequency fi is the ratio of videos in category Ai included in the divided video group 101, and is the value obtained by dividing the number of videos in category Ai by the total number of videos included in the divided video group 101.

[0067] Specifically, the main category determination unit 103A calculates, for all categories, f1, which is the value obtained by dividing the number of moving images in category A1 by the total number of moving images included in the divided moving image group 101, f2, which is the value obtained by dividing the number of moving images in category A2 by the total number of moving images included in the divided moving image group 101, etc. In the following step S353, the main category determination unit 103A calculates the sampling index SIi for each category. The sampling index SIi is found by the following equation 3.

[0068] SIi = POW(Ri, λ)*POW(1 / fi, 1-λ) (Equation 3)

[0069] where Ri in Equation 3 is the risk index of category Ai, λ is the gaze parameter 1801, and fi is the video frequency fi of category Ai. In the following step S354, the main category determination unit 103A calculates the index sum SI_SUM, which is the sum of the sampling indexes SIi of each category. Since the index sum SI_SUM is the sum of the sampling indexes SIi of all categories, it can be calculated using the following Equation 4.

[0070] SI_SUM = SI1+SI2+···+SIC···(Formula 4)

[0071] In the next step S355, the main category determination unit 103A calculates the normalized sampling index NSIi for each category. The normalized sampling index NIi is a value obtained by dividing each sampling index SIi by the sum of the indices SI_SUM, and can be calculated using the following equation 5.

[0072] NSIi = SIi / SI_SUM (Equation 5)

[0073] In the following step S356, the main category determination unit 103A determines the main category based on the normalized sampling index calculated in step S355, and the process shown in FIG. 17 ends.

[0074] According to the above-described third embodiment, the following advantageous effects can be obtained. (10) The main category determination unit 103A determines the main category based on the risk of each category included in the behavior category, the frequency of existence of each behavior category in the divided moving image group 101, which is the population of main category videos, and the variable λ, which is the gaze parameter 1801. The gaze parameter 1801 determines the ratio of the influence on the determination of the main category between the risk of each category included in the behavior category and the frequency of existence of each behavior category in the divided moving image group 101. Therefore, the main category can be determined taking into account the risk of each category.

[0075] (Modification of the third embodiment) In the third embodiment, the gaze parameter 1801 represented by λ in Equation 3 is a single value. However, the gaze parameter 1801 may be set to a different value for each category.

[0076] 18 is a diagram showing gaze parameters 1801 in this modified example. As shown in this diagram, in this modified example, the value of the gaze parameters 1801 is set for each category.

[0077] This modified example provides the following advantages. (11) As shown in Fig. 18, the gaze parameter 1801 is set for each category included in the behavior category, so that the optimum value of the gaze parameter 1801 can be set for each category.

[0078] --Fourth embodiment-- A fourth embodiment of the learning system will be described with reference to Figure 19. In the following description, the same components as those in the first embodiment are given the same reference numerals, and differences will be mainly described. Points that are not specifically described are the same as those in the first embodiment. This embodiment differs from the first embodiment mainly in that it includes a video for evaluation.

[0079] FIG. 19 is a configuration diagram of a learning system 1C according to the fourth embodiment. In addition to the configuration according to the first embodiment, the learning system 1C includes an evaluation moving image group 2201, an evaluation label group 2202, a recognition model 2203, and an evaluation prediction result 2204. In the first embodiment, the performance evaluation unit 113 generated the evaluation result 117 using the prediction result 110 output by the recognition model 108. In this embodiment, the evaluation result 117 is generated using the evaluation prediction result 2204. The evaluation moving image group 2201 is a composite moving image created in advance. The evaluation moving image group 2201 is not particularly limited as long as it is a collection of composite moving images, but it is not permitted to be completely identical to the collection of training moving images created by the training moving image generation unit 107 and input to the recognition model 108. The evaluation label group 2202 is an action label of the main category of each moving image included in the evaluation moving image group 2201.

[0080] In this embodiment, the learning videos used to generate the recognition model 108 are different from the videos included in the evaluation video set 2201, so the model can be evaluated using videos different from the videos used to generate the model.

[0081] According to the above-described fourth embodiment, the following advantageous effects can be obtained. (12) The performance evaluation unit 113 evaluates the evaluation prediction results 2204 obtained by inputting videos that are different from the training videos used for training the recognition model 108, i.e., videos included in the evaluation video set 2201. This prevents overlearning.

[0082] (GUI) With reference to Fig. 20, a description will be given of a GUI (Graphical User Interface) relating to the above-mentioned first to fourth embodiments. Fig. 20 shows the initial state of the GUI. To start the machine learning process, it is necessary to input the necessary information into the GUI. The information necessary to start the machine learning process can be divided into five parts: first, basic information, second, risk value, third, category determination method, fourth, main category video determination method, and fifth, rating frequency. The GUI accepts input for these five parts in sequence.

[0083] The first thing to be input is basic information 2301, which includes the number of epochs 2302, storage location of training data 2303, and storage location of validation data 2304. The number of epochs 2302 specifies the stopping point of repeated training of the recognition model 108. The locations where the respective data sets can be accessed are input in the codes 2303 and 2304. The second setting of the risk value has two options: manual input 2401 and automatic calculation 2402. If manual input 2401 is selected, a dialog box for manual input is displayed. If automatic calculation 2402 is selected, a dialog box prompting the user to input an input field for data to be used in automatic calculation, for example, the storage location of the statistical information 119, is displayed.

[0084] In the third category determination method, one of random 2501, frequency criterion 2502, and risk criterion 2503 is selected. When random 2501 is selected, the main category determination unit 103 performs the first operation, i.e., positive frequency determination. When frequency criterion 2502 is selected, the main category determination unit 103 performs the second operation, i.e., inverse frequency determination. When risk criterion 2503 is selected, a dialog box for setting the value of a gaze parameter 1801 represented by the variable λ is displayed. This dialog box may accept numerical input, or may display a slider for setting the value. Once the value of the gaze parameter 1801 is set, the main category determination unit 103 determines the main category using the method of the third embodiment.

[0085] When determining the fourth main category video, either normal 2601 or selective 2602 is selected. If normal 2601 is selected, a dialog is displayed for setting the value of ratio parameter 105 represented by variable γ. This dialog may accept input of a numerical value, or may display a slider for setting a value. It may also accept input of values ​​for L1 and L2 shown in FIG. 12. If selective 2602 is selected, a dialog is displayed for setting the value of ratio parameter 105 represented by variable γ, the values ​​of L1 and L2, and the value of threshold THRE.

[0086] In the fifth evaluation frequency determination, either epoch 2701 or batch 2702 is selected. Furthermore, count 2703 is also input. In the example shown in Fig. 20, epoch 2701 is selected and "1" is input as count 2703, so that evaluation by performance evaluation unit 113 is performed for each epoch, i.e., evaluation result 117 is output. Once all five items have been input, learning of recognition model 108 begins.

[0087] (Worker motion recognition system) A worker action recognition system that uses the recognition model 108 generated in the first to fourth embodiments described above will be described with reference to Fig. 21. Fig. 21 is a configuration diagram of a worker action recognition system 3000. The worker action recognition system 3000 includes a video acquisition unit 3001, a detector 3002 that detects a worker, a tracker 3003 that tracks the worker, an extractor 3004 that uses the detector 3002 and the tracker 3003 to extract a candidate area where the worker is present, the recognition model 108, and an output unit 3005.

[0088] The video acquisition unit 3001 may be a communication interface or storage medium reader that acquires captured video files, or a video capture device including an image sensor, such as a web camera. The detector 3002, tracker 3003, and extractor 3004 may have known configurations, and may be realized, for example, by a CPU 41, a ROM 42, and a RAM 43, as in the arithmetic unit 40 shown in Figure 8. The output unit 3005 is a communication interface, a video output device such as an LCD display, or a display adapter that outputs a video signal to the video output device.

[0089] The video acquired by video acquisition unit 3001 is transmitted to detector 3002. Detector 3002 detects a worker, i.e., a human, from the input video and outputs the position information to tracker 3003. Tracker 3003 uses the position information output from detector 3002 as an initial value to track the position of the worker in the video. Extractor 3004 uses the output of detector 3002 and the position of the worker tracked by tracker 3003 to extract a spatial region of the video and output it to recognition model 108. Recognition model 108 performs recognition processing on the spatial region of the video extracted by extractor 3004 and outputs an action category. Output unit 3005 outputs the information on the action category output by recognition model 108.

[0090] Note that the detector 3002 and the tracker 3003 may detect and track only the worker's hands instead of detecting and tracking the worker's entire body. In this case, the extractor 3004 extracts a region including the worker's entire body based on the position of the tracked hands. In other words, even in this case, the output of the extractor 3004 does not change.

[0091] The worker action recognition system 3000 described above has the following advantages. (13) The worker action recognition system 3000 includes a recognition model 108 generated by the learning system 1, a detector 3002 that detects a worker, a tracker 3003 that tracks the worker, and an extractor 3004 that extracts a candidate area where the worker is present using the detector 3002 and the tracker 3003. The recognition model 108 receives an image of the candidate area. Therefore, the worker action recognition system 3000 can recognize high-risk behavior.

[0092] In each of the above-described embodiments and modifications, the functional block configurations are merely examples. Some functional configurations shown as separate functional blocks may be configured as an integrated unit, or a configuration shown in a single functional block diagram may be divided into two or more functions. Furthermore, some of the functions of each functional block may be provided by other functional blocks.

[0093] In the above-described embodiments and modifications, the program is stored in ROM 42 (not shown), but the program may be stored in a non-volatile storage device (not shown). Furthermore, the arithmetic device 40 may be provided with an input / output interface (not shown), and the program may be loaded from another device as needed via the input / output interface and a medium available to the arithmetic device 40. Here, the medium refers to, for example, a storage medium detachable from the input / output interface, or a communication medium, i.e., a wired, wireless, or optical network, or a carrier wave or digital signal propagating through the network. Furthermore, some or all of the functions realized by the program may be realized by a hardware circuit or FPGA.

[0094] The above-described embodiments and modifications may be combined with each other. Although various embodiments and modifications have been described above, the present invention is not limited to these. Other embodiments conceivable within the scope of the technical concept of the present invention are also included within the scope of the present invention. [Explanation of symbols]

[0095] 1, 1A, 1B, 1C: Learning System 101: Divided video group 102: Action Label Group 103, 103A: Main category determination unit 104: Risk Information 106: Subcategory determination unit 107, 107A: Learning video generation unit 108: Recognition model 113: Performance Evaluation Department 114: Learning Department 117: Evaluation results 118: Risk information generation unit 119:Statistics information 703: Relative difficulty calculation unit

Claims

1. a learning video generation unit that combines two videos, a main category video that is a video that belongs to a main category included in a behavior category, and a sub-category video that is a video that is a video that is included in the behavior category and belongs to a sub-category that is different from the main category, at a connection point, and trims the videos to include the connection point, thereby generating a learning video; a learning unit that uses the learning video to learn a recognition model that outputs the main category for the input learning video; a performance evaluation unit that evaluates a prediction result by the recognition model and generates an evaluation result; The sub-category is selected based on the risk levels of the main category and the sub-category in the behavior category and the evaluation results.

2. 2. The learning system according to claim 1, The learning system further comprises a subcategory determination unit that determines, as the subcategory, the behavior category that the recognition model most often erroneously determines as the determined main category.

3. 2. The learning system according to claim 1, The learning system wherein the learning video generation unit generates the learning videos when the recognition performance of the recognition model determines that the learning videos of the main category are the sub-category at a rate higher than a predetermined threshold.

4. 2. The learning system according to claim 1, The learning system further comprises a main category determination unit that determines the behavior category as a main category with a higher frequency as the risk in the behavior category increases.

5. 3. The learning system according to claim 2, The subcategory determination unit is a learning system that determines the subcategory based on risk information indicating the risk of each category included in the behavior category and the evaluation result.

6. 5. The learning system according to claim 4, the main category determination unit determines the main category based on a risk of each category included in the behavior category, a frequency of existence of each category of the behavior category in a divided video group that is a population of the main category videos, and a gaze parameter; A learning system in which the attention parameter determines the ratio of the influence that the risk of each category included in the behavioral category and the frequency of occurrence of each category of the behavioral category in the divided video group have on determining the main category.

7. 6. The learning system according to claim 5, The learning system further comprises a risk information generation unit that calculates the risk information based on past statistics.

8. 6. The learning system according to claim 5, the sub-category determination unit determines the sub-category based on the risk information, the evaluation result, and a ratio parameter; A learning system in which the ratio parameter determines the ratio of influence that the risk information and the evaluation result have on determining the subcategory.

9. 2. The learning system according to claim 1, The performance evaluation unit is a learning system that evaluates prediction results obtained by inputting a video that is different from the learning video used for learning by the recognition model.

10. 2. The learning system according to claim 1, The performance evaluation unit is a learning system that evaluates the performance of the recognition model for each predetermined number of batches.

11. 7. The learning system according to claim 6, A learning system in which the gaze parameters are set for each category included in the behavior category.

12. A recognition model generated by the learning system according to claim 1; a detector for detecting a worker; a tracker for tracking the worker; an extractor that extracts a candidate area where the worker is present using the detector and the tracker; The recognition model is a worker action recognition system that inputs an image of the candidate area.

13. A recognition model generated by the learning system according to claim 1; a detector for detecting the worker's hand; a tracker for tracking the worker's hand; an extractor that extracts a candidate area where the worker is present using the detector and the tracker; The recognition model is a worker action recognition system that inputs an image of the candidate area.

14. A method for learning a recognition model that outputs a category corresponding to an input video, A main category video is a video that belongs to a main category included in a behavior category, and a sub-category video is a video that belongs to a sub-category that is included in the behavior category and is different from the main category. Two videos are joined at a joining point, and the two videos are trimmed to include the joining point, thereby generating a learning video. training the recognition model using the training video so as to output the main category for the input training video; evaluating a prediction result by the recognition model to generate an evaluation result; A method for learning a recognition model, wherein the sub-category is selected based on the risk levels of the main category and the sub-category in the behavior category and the evaluation result.

Images