Learning device, inference device, program, learning method, and inference method

Abstract

A learning device (110) comprises: a state class training data generation unit (117) that generates state class training data by imparting a state label indicating a predetermined state to an appearance feature amount group indicating, in time series, a plurality of appearance feature amounts including appearance feature amounts indicating features of an appearance of one target calculated from each of a plurality of frames of a video; an asymmetric class training data generation unit (118) that processes one or more appearance feature amounts included in the plurality of appearance feature amounts, thereby changing a portion of the plurality of features indicated in time series by the plurality of appearance feature amount, generating an asymmetric appearance feature quantity group, and imparting an asymmetric label indicating asymmetry to the asymmetric appearance feature quantity group, and thus generating asymmetric class training data; and a learning unit (119) that receives an input of the appearance feature quantity group acquired from the video and generates a learning model for inferring a state or asymmetry corresponding to the input by carrying out training by using the state class training data and the asymmetric class training data.

Description

Learning device, inference device, program, learning method, and inference method

[0001] The present disclosure relates to a learning device, an inference device, a program, a learning method, and an inference method.

[0002] Techniques for identifying the state of a person, animal, robot, or the like from a video have been used for some time. For example, when identifying a person's state from a video, if the video contains a person other than the person whose state is being identified, the state cannot be accurately identified. Therefore, in Patent Document 1, it is detected whether or not people are overlapping in the video.

[0003] Japanese Patent Application Laid-Open No. 2018-073385

[0004] However, while conventional technology can detect whether people are overlapping, when detecting a person's state from video, if there are multiple people in one frame and the appearances of the people in multiple frames are used as features, the detection results will contain a mixture of appearance features of different people in each frame, making it often impossible to accurately identify the state of a single person.

[0005] Therefore, one or more aspects of the present disclosure aim to enable accurate detection of the state of a single object from an image.

[0006] A learning device according to a first aspect of the present disclosure is characterized by comprising: a state class learning data generation unit that generates state class learning data by assigning a state label, which is a label indicating a predetermined state, to an appearance feature group that indicates, in time series, a plurality of appearance features, the plurality of appearance features being calculated from each of a plurality of frames of a video capturing an image of the object in a predetermined state; an asymmetric class learning data generation unit that processes one or more appearance features included in the plurality of appearance features to change a portion of the plurality of features that are indicated in time series by the plurality of appearance features, to generate an asymmetric appearance feature group, and assigns an asymmetric label indicating asymmetricity to the asymmetric appearance feature group, thereby generating asymmetric class learning data; and a learning unit that receives input of an appearance feature group acquired from a video, by learning using the state class learning data and the asymmetric class learning data, and generates a learning model for inferring the state or asymmetricity corresponding to the input.

[0007] A learning device according to a second aspect of the present disclosure is characterized by comprising: a state class learning data generation unit that generates state class learning data by assigning a state label, which is a label indicating a predetermined state, to a group of appearance features that indicates, in time series, a plurality of appearance features calculated from each of a plurality of frames of a video capturing a predetermined state of the object, the appearance feature being composed of appearance features that indicate characteristics in the appearance of the single object; an asymmetric class learning data generation unit that generates asymmetric class learning data by assigning an asymmetric label, which indicates asymmetric, to a group of appearance features that indicates, in time series, a plurality of appearance features that include appearance features that indicate characteristics in the appearance of a first object and appearance features that indicate characteristics in the appearance of a second object, calculated from the plurality of frames of a video comprising a plurality of frames; and a learning unit that receives input of a group of appearance features acquired from a video by learning using the state class learning data and the asymmetric class learning data, and generates a learning model for inferring a state or asymmetric corresponding to the input.

[0008] The inference device according to the first aspect of the present disclosure includes a time-series appearance feature calculation unit that calculates inferred appearance features, which are appearance features of an object, from each of a plurality of frames of an inference video to generate a group of inferred appearance features that indicate a plurality of inferred appearance features in a time series; state class learning data that is generated by assigning state labels that indicate the predetermined state to a group of appearance features that indicate a plurality of appearance features that indicate a feature in the appearance of a single object, the group of appearance features being calculated from each of a plurality of frames of a video capturing a predetermined state of the object; and and an inference unit that processes one or more appearance features included in the appearance features to change some of the features indicated in time series by the plurality of appearance features to generate an asymmetric appearance feature group, and learns using asymmetry class training data generated by assigning an asymmetry label indicating asymmetry to the asymmetric appearance feature group, thereby receiving an input of an appearance feature group acquired from a video, and inputting the inferred appearance feature group into a learning model that has been generated to infer a state or asymmetry corresponding to the input.

[0009] An inference device according to a second aspect of the present disclosure includes a time-series appearance feature calculation unit that calculates inferred appearance features, which are appearance features of an object, from each of a plurality of frames of an inference video to generate an inferred appearance feature group that indicates a plurality of inferred appearance features in a time series; state class learning data that is generated by assigning state labels that indicate the predetermined state to the appearance feature group that indicates a plurality of appearance features that are calculated from each of a plurality of frames of a video capturing an object in a predetermined state; and and an inference unit that receives an input of a group of appearance features acquired from a video, and inputs the inferred appearance feature group into a learning model that has been generated so as to infer a state or asymmetry corresponding to the input, by learning using asymmetry class learning data that has been generated by assigning an asymmetry label indicating asymmetry to a group of appearance features that indicates, in time series, a plurality of appearance features that include appearance features that indicate features in the appearance of a first object and appearance features that indicate features in the appearance of a second object, which are calculated from the plurality of frames of the video.

[0010] A program according to a first aspect of the present disclosure causes a computer to function as: a state class learning data generation unit that generates state class learning data by assigning a state label, which is a label indicating a predetermined state, to an appearance feature group that indicates, in time series, a plurality of appearance features composed of appearance features that indicate features in the appearance of a single object, the appearance feature being calculated from each of a plurality of frames of a video capturing an image of the single object in a predetermined state; an asymmetry class learning data generation unit that processes one or more appearance features included in the plurality of appearance features to change a portion of the plurality of features that are indicated in time series by the plurality of appearance features, to generate an asymmetric appearance feature group, and assigns an asymmetry label, indicating asymmetry, to the asymmetric appearance feature group, thereby generating asymmetric class learning data; and a learning unit that receives input of an appearance feature group acquired from a video, and generates a learning model for inferring a state or asymmetry corresponding to the input, by learning using the state class learning data and the asymmetry class learning data.

[0011] A program according to a second aspect of the present disclosure causes a computer to function as: a state class learning data generation unit that generates state class learning data by assigning a state label, which is a label indicating a predetermined state, to a group of appearance features that indicates, in time series, a plurality of appearance features calculated from each of a plurality of frames of a video capturing a predetermined state of the single object, the appearance feature being composed of appearance features that indicate characteristics in the appearance of the single object; an asymmetric class learning data generation unit that generates asymmetric class learning data by assigning an asymmetric label, which indicates asymmetric, to a group of appearance features that indicates, in time series, a plurality of appearance features that include appearance features that indicate characteristics in the appearance of a first object and appearance features that indicate characteristics in the appearance of a second object, calculated from the plurality of frames of a video comprising the plurality of frames; and a learning unit that receives input of a group of appearance features acquired from a video and generates a learning model for inferring a state or asymmetric corresponding to the input by learning using the state class learning data and the asymmetric class learning data.

[0012] A program according to a third aspect of the present disclosure includes a time-series appearance feature calculation unit that calculates inferred appearance features that are appearance features of an object from each of a plurality of frames of an inferred video to generate an inferred appearance feature group that indicates a plurality of inferred appearance features in a time series; state class learning data that is generated by assigning state labels that indicate a predetermined state to a plurality of appearance feature groups that indicate a plurality of appearance features that are calculated from each of a plurality of frames of a video capturing an object in a predetermined state; and The system is characterized in that one or more appearance features included in a plurality of appearance features are processed to change some of the features indicated in time series by the plurality of appearance features to generate a group of asymmetric appearance features, and learning is performed using asymmetry class training data generated by assigning an asymmetry label indicating asymmetry to the group of asymmetric appearance features, thereby receiving an input of a group of appearance features acquired from a video, and inputting the inferred group of appearance features into a learning model generated to infer a state or asymmetry corresponding to the input, thereby functioning as an inference unit that infers a state or asymmetry corresponding to the group of inferred appearance features.

[0013] A program according to a fourth aspect of the present disclosure includes a time-series appearance feature calculation unit that calculates inferred appearance features that are appearance features of an object from each of a plurality of frames of an inferred video to generate an inferred appearance feature group that indicates a plurality of inferred appearance features in a time series; state class learning data that is generated by assigning state labels that indicate the predetermined state to a plurality of appearance feature groups that indicate a plurality of appearance features that are calculated from each of a plurality of frames of a video capturing an object in a predetermined state; and The learning model is characterized in that it receives an input of a group of appearance features acquired from a video, and functions as an inference unit that infers a state or asymmetry corresponding to the inferred group of appearance features by inputting the inferred group of appearance features into a learning model generated so as to receive an input of a group of appearance features acquired from the video, and infer a state or asymmetry corresponding to the input, by applying an asymmetry label indicating asymmetry to a group of appearance features that indicates a time series of a plurality of appearance features, the group of appearance features including appearance features that indicate features in the appearance of a first object and appearance features that indicate features in the appearance of a second object, calculated from the plurality of frames of a video consisting of a video.

[0014] A learning method according to a first aspect of the present disclosure is characterized in that it generates state class learning data by assigning a state label, which is a label indicating a predetermined state, to an appearance feature group that indicates, in time series, a plurality of appearance features that are composed of appearance features that indicate features in the appearance of a single object, calculated from each of a plurality of frames of a video capturing the object in a predetermined state; it processes one or more appearance features included in the plurality of appearance features to change some of the plurality of features that are indicated in time series by the plurality of appearance features, thereby generating an asymmetric appearance feature group; it assigns an asymmetric label indicating asymmetry to the asymmetric appearance feature group, thereby generating asymmetric class learning data; and it receives input of an appearance feature group acquired from a video, and generates a learning model for inferring a state or asymmetry corresponding to the input, by learning using the state class learning data and the asymmetric class learning data.

[0015] A learning method according to a second aspect of the present disclosure generates state class learning data by assigning a state label, which is a label indicating a predetermined state, to a group of appearance features that indicates, in time series, a plurality of appearance features calculated from each of a plurality of frames of a video capturing a predetermined state of the single object, the appearance feature being composed of appearance features indicating characteristics in the appearance of the single object; generates asymmetric class learning data by assigning an asymmetric label, which indicates asymmetric, to a group of appearance features that indicates, in time series, a plurality of appearance features that include appearance features that indicate characteristics in the appearance of a first object and appearance features that indicate characteristics in the appearance of a second object, calculated from the plurality of frames of a video comprising a plurality of frames; and generates a learning model for inferring a state or asymmetric corresponding to the input by learning using the state class learning data and the asymmetric class learning data.

[0016] The inference method according to the first aspect of the present disclosure calculates inferred appearance features, which are appearance features of an object, from each of a plurality of frames of an inference video to generate a group of inferred appearance features indicating a plurality of inferred appearance features in time series, and assigns state labels, which are labels indicating the predetermined state, to the group of appearance features indicating a plurality of appearance features in time series, which are calculated from each of a plurality of frames of a video capturing a predetermined state of the object, and generates state class learning data and the plurality of appearance features. The method is characterized in that one or more appearance features included in the characteristics are processed to change some of the multiple features shown in time series by the multiple appearance features to generate an asymmetric appearance feature group, and learning is performed using asymmetry class training data generated by assigning an asymmetry label indicating asymmetry to the asymmetric appearance feature group, thereby receiving an input of an appearance feature group acquired from a video, and inputting the inferred appearance feature group into a learning model generated to infer the state or asymmetry corresponding to the input, thereby inferring the state or asymmetry corresponding to the inferred appearance feature group.

[0017] An inference method according to a second aspect of the present disclosure generates a group of inferred appearance features that indicate a time series of the plurality of inferred appearance features by calculating inferred appearance features that are appearance features of an object from each of a plurality of frames of an inferred video, and generates state class learning data that is generated by assigning state labels that indicate the predetermined state to the group of appearance features that indicate a time series of the plurality of appearance features that are calculated from each of a plurality of frames of a video capturing a predetermined state of the object, and The learning model is characterized in that it receives an input of a group of appearance features acquired from a video, and infers a state or asymmetry corresponding to the input by inputting the inferred group of appearance features into a learning model generated so as to infer the state or asymmetry corresponding to the input.

[0018] According to one or more aspects of the present disclosure, the state of a single object can be accurately detected from an image.

[0019] FIG. 1 is a block diagram schematically showing the configuration of a learning detection system according to embodiments 1 to 7. FIG. 2 is a block diagram schematically showing the configuration of a learning device according to embodiments 1 to 6. FIG. 3 is a schematic diagram for explaining a first example of generating asymmetric class learning data. FIG. 4 is a block diagram schematically showing the configuration of a PC. FIG. 5 is a block diagram schematically showing the configuration of a state detection device according to embodiments 1 to 6. FIG. 6 is a flowchart showing the operation of the learning device according to embodiment 1. FIG. 7 is a flowchart showing the operation of the state detection device according to embodiment 1. FIG. 8 is a schematic diagram for explaining a second example of generating asymmetric class learning data. FIG. 9 is a schematic diagram for explaining a third example of generating asymmetric class learning data. FIG. 10 is a schematic diagram for explaining a fourth example of generating asymmetric class learning data. FIG. 11 is a schematic diagram for explaining a fifth example of generating asymmetric class learning data. FIG. 12 is a schematic diagram for explaining a sixth example of generating asymmetric class learning data. FIG. 13 is a block diagram schematically showing the configuration of a learning device according to embodiment 7. FIG. 14 is a block diagram schematically showing the configuration of a state detection device according to embodiment 7.

[0020] 1 is a block diagram showing a schematic configuration of a learning detection system 100 according to embodiment 1. The learning detection system 100 includes a learning device 110 and a state detection device 130 serving as an inference device. Here, the learning device 110 and the state detection device 130 are connected to a network 101 such as the Internet, and can transmit and receive data via the network 101.

[0021] 2 is a block diagram showing a schematic configuration of the learning device 110. The learning device 110 includes a communication unit 111, a time-series appearance feature calculation unit 112, a state class learning data generation unit 117, an asymmetric class learning data generation unit 118, a learning unit 119, and a learning model storage unit 120.

[0022] The communication unit 111 transmits and receives data via the network 101. For example, the communication unit 111 transmits the learning model learned by the learning device 110 to the state detection device 130 via the network 101.

[0023] The time-series appearance feature calculation unit 112 calculates appearance features that indicate the characteristics of a person's appearance from each of multiple frames of video represented by video data, thereby generating a group of appearance features that indicate multiple appearance features in time series. Here, a frame refers to one still image that forms the basis of a video. In this embodiment, the subject for which appearance features are calculated is a human, but the subject may also be other animals such as livestock, wild animals, insects, or movable robots. The time-series appearance feature calculation unit 112 includes a video acquisition unit 113, a time-series image group extraction unit 114, and an appearance feature calculation unit 115.

[0024] Video acquisition unit 113 acquires video data. For example, video acquisition unit 113 may acquire video data from a camera, which is an imaging device (not shown), via communication unit 111, or may acquire video data from a camera connected to learning device 110 via a connection I / F (Interface) (not shown), or may acquire the video data by reading out video data stored in a storage unit (not shown).

[0025] The video data used by the learning device 110 is assumed to capture an image of a person in a predetermined state. The acquired video data is provided to the time-series image group extraction unit 114 and the asymmetric class learning data generation unit 118.

[0026] The time-series image group extraction unit 114 extracts a predetermined number of frames for a predetermined time from the video represented by the video data as a time-series image group. The time-series image group is a time series of a plurality of frames. Here, the time-series image group extraction unit 114 sequentially extracts the time-series image group from the video, and the extracted time-series image group is provided to the appearance feature amount calculation unit 115.

[0027] The appearance feature calculation unit 115 calculates appearance features from each frame included in the time-series image group to generate an appearance feature group, which is a time series of multiple appearance features. For example, the appearance feature calculation unit 115 may calculate skeletal information indicating the positions of the joints of one person included in the frames as the appearance feature. Note that the appearance feature calculation may use a known technique such as OpenPose, PoseNet, or Movenet that extracts skeletal information from frames. The generated appearance feature group is provided to the state class training data generation unit 117 and the asymmetric class training data generation unit 118.

[0028] The state class training data generator 117 generates state class training data by assigning state labels to appearance feature groups, each state label indicating a state corresponding to the appearance feature group. In the video data used by the learning device 110, a single person to be trained is captured in states corresponding to the labels, and therefore the appearance feature groups indicate predetermined states of the single person in correct chronological order.

[0029] Here, the predetermined states are assumed to be a standing state where a person is standing, a falling state where a person is falling, and a lying state where a person is lying. In other words, the state class training data generation unit 117 generates state class training data by labeling a group of appearance features with a standing state, a falling state, or a lying state. The state class training data is provided to the training unit 119.

[0030] The asymmetric class training data generation unit 118 processes one or more appearance features included in the plurality of appearance features constituting the appearance feature group, thereby changing some of the plurality of features represented in time series by the plurality of appearance features, thereby generating an asymmetric appearance feature group. The asymmetric class training data generation unit 118 then generates asymmetric class training data by assigning an asymmetric label indicating asymmetry to the asymmetric appearance feature group. Here, asymmetric means that some of the plurality of features represented by the plurality of appearance features included in the appearance feature group are inconsistent with the remaining features excluding that part.

[0031] The asymmetric class learning data generation unit 118 processes one or more appearance features included in the appearance feature group so that the features indicated in time series by the appearance feature group differ from the features indicated in time series by the video.

[0032] Here, the asymmetric class training data generation unit 118 generates an asymmetric appearance feature group by replacing one or more appearance features included in a certain appearance feature group with one or more appearance features included in another appearance feature group. The asymmetric class training data is provided to the training unit 119.

[0033] 3 is a schematic diagram illustrating a first example of generating asymmetric class training data. In the first embodiment, the asymmetric class training data generation unit 118 generates an asymmetric appearance feature group by replacing two appearance features 102a and 102b included in one appearance feature group 102 with two appearance features 103a and 103b included in another appearance feature group 103. Note that, although the two appearance features 102a and 102b are replaced with the two appearance features 103a and 103b here, the first embodiment is not limited to this example. For example, each of the two appearance features 102a and 102b may be replaced with one appearance feature 103a.

[0034] Returning to Fig. 2, the learning unit 119 performs learning using the state class learning data and the asymmetric class learning data to generate a learning model that receives an input of an appearance feature group acquired from a video and infers a state or asymmetric corresponding to the input. The learning model here is a model that classifies an appearance feature group into one of a plurality of classes, including an upright class, a falling class, and a lying class as state classes, and further an asymmetric class. The asymmetric class is a class that classifies an appearance feature group that includes an appearance feature that is inconsistent with other appearance features.

[0035] In general, in learning-based classifiers, especially in the learning and inference of deep learning models, adding classes hardly increases the amount of computation. This is because the number of convolutional layers and hidden layers in the network remains the same, so the amount of computation in those layers remains unchanged. The only difference is that the number of connections between the output layer and its previous stage increases, but this is only a small difference compared to the overall amount of computation, so in most cases, it can be considered that the amount of computation does not change substantially.

[0036] The learning model storage unit 120 stores the learning model generated by the learning unit 119. The learning model stored in the learning model storage unit 120 is sent to the state detection device 130 via the communication unit 111.

[0037] The learning device 110 described above can be realized by, for example, a computer such as the PC 10 shown in Fig. 4. The PC 10 includes a storage 11 such as a hard disk drive (HDD) and a solid state drive (SSD), a memory 12, a processor 13 such as a central processing unit (CPU), a communication interface (I / F) 14 such as a network interface card (NIC), an input interface 15 such as a keyboard and a mouse, and a display 16.

[0038] For example, the learning model storage unit 120 can be realized by the storage 11 or the memory 12. The time-series appearance feature calculation unit 112, the state class learning data generation unit 117, the asymmetric class learning data generation unit 118, and the learning unit 119 can be realized by the processor 13 loading a program stored in the storage 11 into the memory 12 and executing the program. The communication unit 111 can be realized by the communication I / F 14.

[0039] The program may be downloaded to the storage 11 from a recording medium (not shown) via a reader / writer (not shown) or from the network 101 via the communication I / F 14, and then loaded onto the memory 12 and executed by the processor 13. Alternatively, the program may be directly loaded onto the memory 12 from a recording medium via the reader / writer or from the network via the communication I / F 14, and then executed by the processor 13. In other words, the program may be provided by a program product such as a recording medium.

[0040] 5 is a block diagram showing a schematic configuration of state detection device 130. State detection device 130 includes a communication unit 131, a time-series appearance feature calculation unit 132, a learning model storage unit 137, an inference unit 138, and an alarm generation unit 139.

[0041] The communication unit 131 transmits and receives data via the network 101. For example, the communication unit 131 receives, via the network 101, a learning model trained by the learning device 110.

[0042] The time-series appearance feature calculation unit 132 calculates appearance feature values ​​from multiple frames of a predetermined period in the video represented by the video data, and generates a group of appearance feature values. The video data handled by the state detection device 130 is also referred to as inferred video data, and the video represented by the inferred video data is also referred to as inferred video. The appearance feature values ​​calculated here are also referred to as inferred appearance feature values, and the group of appearance feature values ​​generated here are also referred to as a group of inferred appearance feature values. The time-series appearance feature calculation unit 132 includes a video acquisition unit 133, a time-series image group extraction unit 134, and an appearance feature calculation unit 135.

[0043] The video acquisition unit 133 acquires video data. For example, the video acquisition unit 133 may acquire video data from a camera, which is an imaging device (not shown), via the communication unit 131, or may acquire video data from a camera connected to the state detection device 130 via a connection I / F (Interface) (not shown), or may acquire the video data by reading out video data stored in a storage unit (not shown). The acquired video data is provided to the time-series image group extraction unit 134.

[0044] The time-series image group extraction unit 134 extracts a predetermined time or a predetermined number of frames from the video represented by the video data as a time-series image group. The time-series image group is a time series of a plurality of frames. Here, the time-series image group extraction unit 134 sequentially extracts the time-series image group from the video. The extracted time-series image group is provided to the appearance feature amount calculation unit 135.

[0045] The appearance feature calculation unit 135 calculates the appearance feature of the target from each frame included in the time-series image group, thereby generating an appearance feature group, which is a time series of multiple appearance feature groups. For example, the appearance feature calculation unit 135 may calculate, as an appearance feature group, skeletal information indicating the positions of the joints of one person, who is the target included in the frames. The generated appearance feature group is provided to the inference unit 138.

[0046] The learning model storage unit 137 stores the learning model learned by the learning device 110. Then, the learning model storage unit 137 provides the learning model to the inference unit 138.

[0047] The inference unit 138 inputs a group of appearance features into a learning model, thereby inferring a state or asymmetry corresponding to the group of appearance features. For example, the learning model outputs the likelihoods of standing, falling, lying down, and asymmetry for the input group of appearance features so that the total sum is "1." This allows the inference unit 138 to infer the state or asymmetry with the highest likelihood as the state or asymmetry for that group of appearance features. The inference unit 138 notifies the reporting unit 139 of the inference result.

[0048] The alarm issuing unit 139 uses the result of inference by the inference unit 138 to determine whether to issue an alarm. Specifically, the alarm issuing unit 139 determines to issue an alarm when the inference result from the inference unit 138 is a predetermined combination without including a predetermined number of asymmetric inferences. For example, the alarm issuing unit 139 determines to issue a fall alarm when the person remains standing for 0.5 seconds or more, falls for 0.5 seconds or more, and remains lying down for 5 seconds or more, and the asymmetric period is 0.5 seconds or less. In other words, the alarm issuing unit 139 determines to issue an alarm when it determines, based on the result of inference by the inference unit 138, that a person included in the inference video has fallen.

[0049] The alarm unit 139 may output a predetermined sound from a speaker serving as an output unit (not shown) as an alarm, or may transmit data indicating the alarm to a predetermined device (not shown) via the communication unit 131.

[0050] The state detection device 130 described above can also be realized by a computer such as the PC 10 shown in Fig. 4. For example, the learning model storage unit 137 can be realized by the storage 11 or the memory 12. The time-series appearance feature calculation unit 132, the inference unit 138, and the alarm issuing unit 139 can be realized by the processor 13 loading a program stored in the storage 11 into the memory 12 and executing the program. The communication unit 131 can be realized by the communication I / F 14.

[0051] 6 is a flowchart showing the operation of the learning device 110 in embodiment 1. First, the video acquisition unit 113 acquires video data (S10). The video data here shows a person to be studied in a predetermined state to be studied. The acquired video data is provided to the time-series image group extraction unit 114.

[0052] Next, the time-series image group extraction unit 114 sequentially extracts a predetermined number of frames for a predetermined time from the video represented by the video data as a time-series image group (S11). The extracted time-series image group is provided to the appearance feature amount calculation unit 115.

[0053] Next, the appearance feature calculation unit 115 calculates appearance features from each frame included in the time-series image group to generate an appearance feature group, which is a time series of multiple appearance features (S12). Here, the appearance feature calculation unit 115 calculates skeletal information indicating the positions of the joints of one person included in the frames as appearance features. The generated appearance feature group is provided to the state class training data generation unit 117 and the asymmetric class training data generation unit 118.

[0054] The state class training data generation unit 117 generates state class training data by assigning state labels indicating states corresponding to the appearance feature groups to the appearance feature groups (S13). Here, for example, a user of the training device 110 may specify state labels corresponding to the appearance feature groups via an input unit (not shown). The state class training data generated in this manner is provided to the training unit 119.

[0055] The asymmetric class training data generation unit 118 processes one or more appearance features included in the appearance feature group so that the features indicated in time series by the appearance feature group are different, thereby generating an asymmetric appearance feature group, and attaches an asymmetric label to the asymmetric appearance feature group to generate asymmetric class training data (S14). The asymmetric class training data generated in this manner is provided to the training unit 119.

[0056] Next, the learning unit 119 performs learning using the state class learning data and the asymmetric class learning data to generate a learning model (S15). The learning model here is a model that classifies the group of appearance features into one of a plurality of classes including a state class and an asymmetric class.

[0057] Then, the learning model storage unit 120 stores the learning model generated by the learning unit 119 (S16). The learning model stored in the learning model storage unit 120 is sent to the state detection device 130 via the communication unit 111.

[0058] 7 is a flowchart showing the operation of the state detection device 130 in embodiment 1. Here, it is assumed that the communication unit 131 receives the learning model learned by the learning device 110 via the network 101, and that the learning model is stored in the learning model storage unit 137.

[0059] First, the video acquisition unit 133 acquires video data (S20). The acquired video data is provided to the time-series image group extraction unit 134.

[0060] Next, the time-series image group extraction unit 134 extracts a predetermined number of frames from the video represented by the video data as a time-series image group (S21). The extracted time-series image group is provided to the appearance feature amount calculation unit 135.

[0061] Next, the appearance feature calculation unit 135 calculates the appearance feature of the target from each frame included in the time-series image group, thereby generating an appearance feature group, which is a time series of multiple appearance feature groups (S22). Here, the appearance feature calculation unit 135 targets one person included in the frames and calculates skeletal information indicating the positions of the target's joints as appearance feature groups. The generated appearance feature group is provided to the inference unit 138.

[0062] Next, the inference unit 138 inputs the learning model from the learning model storage unit 137. The inference unit 138 inputs the group of appearance features into the learning model, and thereby infers the state or asymmetry corresponding to the group of appearance features (S23). Then, the inference unit 138 notifies the reporting unit 139 of the inference result.

[0063] The alarm issuing unit 139 determines whether an alarm is necessary based on whether the inference result from the inference unit 138 is a predetermined combination without including a predetermined number of asymmetric inferences (S24). If an alarm is necessary (Yes in S24), the process proceeds to step S25, and if an alarm is not necessary (No in S24), the process returns to step S21. Then, in step S25, the alarm issuing unit 139 issues an alarm.

[0064] When calculating appearance features such as skeletal information from video data, if the video data contains a person other than the person being inferred or an object that is difficult to distinguish from a person, such as a doll, the appearance features may be calculated from the person or object other than the person being inferred at some time (frame). In embodiment 1, even in such cases, when training the learning model, training is performed so that a group of appearance features including appearance features calculated from a person or object other than the person being inferred is classified into an asymmetric class. Therefore, by using such a learning model, the state of the person being inferred can be accurately inferred by classifying a group of appearance features including appearance features calculated from a person or object other than the person being inferred into an asymmetric class and not classifying them into a state class.

[0065] 1, a learning detection system 200 according to the second embodiment includes a learning device 210 and a state detection device 130. The state detection device 130 of the learning detection system 200 according to the second embodiment is similar to the state detection device 130 of the learning detection system 100 according to the first embodiment.

[0066] As shown in FIG. 2 , the learning device 210 includes a communication unit 111, a time-series appearance feature calculation unit 112, a state class learning data generation unit 117, an asymmetric class learning data generation unit 218, a learning unit 119, and a learning model storage unit 120.

[0067] The communication unit 111, the time-series appearance feature calculation unit 112, the state class learning data generation unit 117, the learning unit 119, and the learning model storage unit 120 of the learning device 210 in embodiment 2 are similar to the communication unit 111, the time-series appearance feature calculation unit 112, the state class learning data generation unit 117, the learning unit 119, and the learning model storage unit 120 of the learning device 110 in embodiment 1.

[0068] The asymmetric class training data generation unit 218 processes one or more appearance features included in the plurality of appearance features constituting the appearance feature group, thereby changing some of the plurality of features indicated in time series by the plurality of appearance features, thereby generating an asymmetric appearance feature group. The asymmetric class training data generation unit 218 then generates asymmetric class training data by assigning an asymmetric label to the asymmetric appearance feature group.

[0069] In the second embodiment, the asymmetric class training data generating unit 218 generates an asymmetric appearance feature group so as to include one or more appearance features extracted from a plurality of appearance feature groups. The asymmetric class training data is provided to the training unit 119.

[0070] 8 is a schematic diagram illustrating a second example of generating asymmetric class training data. In the second embodiment, the asymmetric class training data generating unit 218 generates an asymmetric appearance feature group by replacing four appearance features included in a certain appearance feature group 202 with two appearance features 203 a and 203 b included in another appearance feature group 203 and two appearance features 204 a and 204 b included in yet another appearance feature group 204.

[0071] As described above, according to the second embodiment, three appearance feature sets 202, 203, and 204 are used to generate an appearance feature set that can be easily classified into an asymmetric class, and learning can be performed.

[0072] 1, a learning detection system 300 according to the third embodiment includes a learning device 310 and a state detection device 130. The state detection device 130 of the learning detection system 300 according to the third embodiment is similar to the state detection device 130 of the learning detection system 100 according to the first embodiment.

[0073] As shown in FIG. 2 , the learning device 310 includes a communication unit 111, a time-series appearance feature calculation unit 112, a state class learning data generation unit 117, an asymmetric class learning data generation unit 318, a learning unit 119, and a learning model storage unit 120.

[0074] The communication unit 111, the time-series appearance feature calculation unit 112, the state class learning data generation unit 117, the learning unit 119, and the learning model storage unit 120 of the learning device 310 in embodiment 3 are similar to the communication unit 111, the time-series appearance feature calculation unit 112, the state class learning data generation unit 117, the learning unit 119, and the learning model storage unit 120 of the learning device 110 in embodiment 1.

[0075] The asymmetric class training data generation unit 318 processes one or more appearance features included in the plurality of appearance features constituting the appearance feature group, thereby changing some of the plurality of features indicated in time series by the plurality of appearance features, thereby generating an asymmetric appearance feature group. The asymmetric class training data generation unit 318 then generates asymmetric class training data by assigning an asymmetric label to the asymmetric appearance feature group.

[0076] In the third embodiment, the asymmetric class training data generation unit 318 generates an asymmetric appearance feature group such that one or more first appearance features and one or more second appearance features extracted from one appearance feature group are interchanged in the time series of that one appearance feature group. In other words, the asymmetric class training data generation unit 318 generates an asymmetric appearance feature group by interchangeing the order in time series of the first appearance feature included in the appearance feature group and the second appearance feature included in that appearance feature group. The asymmetric class training data is provided to the training unit 119.

[0077] 9 is a schematic diagram illustrating a third example of generating asymmetric class training data. In the third embodiment, the asymmetric class training data generator 318 generates an asymmetric appearance feature group by swapping the time series order of two first appearance feature values ​​303a and 303b and two second appearance feature value groups 303c and 303d included in the appearance feature group 303, and then swapping these appearance feature values ​​303a to 303d with four appearance feature values ​​included in another appearance feature group 302.

[0078] In the example of FIG. 9, two appearance feature groups 302 and 303 are used, but within one appearance feature group 303, the chronological order of the first appearance feature groups 303a and 303b and the second appearance feature group 303c and 303d may be interchanged to generate an asymmetric appearance feature group.

[0079] As described above, according to the third embodiment, at least one group of appearance features can be used to generate a group of appearance features that can be easily classified into an asymmetric class, and learning can be performed.

[0080] 1, a learning detection system 400 according to the fourth embodiment includes a learning device 410 and a state detection device 130. The state detection device 130 of the learning detection system 400 according to the fourth embodiment is similar to the state detection device 130 of the learning detection system 100 according to the first embodiment.

[0081] As shown in FIG. 2 , the learning device 410 includes a communication unit 111, a time-series appearance feature calculation unit 112, a state class learning data generation unit 117, an asymmetric class learning data generation unit 418, a learning unit 119, and a learning model storage unit 120.

[0082] The communication unit 111, the time-series appearance feature calculation unit 112, the state class learning data generation unit 117, the learning unit 119, and the learning model storage unit 120 of the learning device 410 in embodiment 4 are similar to the communication unit 111, the time-series appearance feature calculation unit 112, the state class learning data generation unit 117, the learning unit 119, and the learning model storage unit 120 of the learning device 110 in embodiment 1.

[0083] The asymmetric class training data generation unit 418 processes one or more appearance features included in the plurality of appearance features constituting the appearance feature group, thereby changing some of the plurality of features indicated in time series by the plurality of appearance features, thereby generating an asymmetric appearance feature group. The asymmetric class training data generation unit 418 then generates asymmetric class training data by assigning an asymmetric label to the asymmetric appearance feature group.

[0084] In the fourth embodiment, the asymmetric class training data generation unit 418 generates an asymmetric appearance feature group so as to include transformed appearance features that are obtained by transforming one or more appearance features extracted from the appearance feature group so as to change the features represented by the one or more appearance features. In other words, the asymmetric class training data generation unit 418 generates an asymmetric appearance feature group by performing an affine transformation on one or more appearance features included in the appearance feature group. The asymmetric class training data is provided to the training unit 119.

[0085] 10 is a schematic diagram illustrating a fourth example of generating asymmetric class training data. In the fourth embodiment, asymmetric class training data generator 418 generates asymmetric appearance feature group 402# by replacing appearance feature 402a, 402b with appearance feature 402a#, 402b# generated by performing affine transformation on two appearance feature 402a, 402b included in a certain appearance feature group 402.

[0086] Here, the affine transformation is a transformation for enlarging, reducing, translating, rotating, flipping, skewing, etc. the appearance of the corresponding person.

[0087] 10 uses one appearance feature amount group 402, but an appearance feature amount obtained by affine transforming an appearance feature amount extracted from one appearance feature amount group 402 may be replaced with an appearance feature amount from another appearance feature amount group. Furthermore, the affine transform performed on appearance feature amount 402a may be different from the affine transform performed on appearance feature amount 402b.

[0088] As described above, according to the fourth embodiment, at least one appearance feature group 402 is used to generate an appearance feature group that can be easily classified into an asymmetric class, and learning can be performed.

[0089] 1, a learning detection system 500 according to the fifth embodiment includes a learning device 510 and a state detection device 130. The state detection device 130 of the learning detection system 500 according to the fifth embodiment is similar to the state detection device 130 of the learning detection system 100 according to the first embodiment.

[0090] As shown in FIG. 2 , the learning device 510 includes a communication unit 111, a time-series appearance feature calculation unit 112, a state class learning data generation unit 117, an asymmetric class learning data generation unit 518, a learning unit 119, and a learning model storage unit 120.

[0091] The communication unit 111, the time-series appearance feature calculation unit 112, the state class learning data generation unit 117, the learning unit 119, and the learning model storage unit 120 of the learning device 510 in embodiment 5 are similar to the communication unit 111, the time-series appearance feature calculation unit 112, the state class learning data generation unit 117, the learning unit 119, and the learning model storage unit 120 of the learning device 110 in embodiment 1.

[0092] The asymmetric class training data generation unit 518 processes one or more appearance features included in the plurality of appearance features constituting the appearance feature group, thereby changing some of the plurality of features indicated in time series by the plurality of appearance features, thereby generating an asymmetric appearance feature group. The asymmetric class training data generation unit 518 then generates asymmetric class training data by assigning an asymmetric label to the asymmetric appearance feature group.

[0093] In the fifth embodiment, the asymmetric class training data generation unit 518 generates an asymmetric appearance feature group so as to include transformed appearance features obtained by transforming a plurality of features represented by a plurality of appearance features extracted from the appearance feature group so as to change the plurality of features. In other words, the asymmetric class training data generation unit 518 generates an asymmetric appearance feature group by performing a first affine transformation on a first appearance feature included in the appearance feature group, and performing a second affine transformation different from the first affine transformation on a second appearance feature included in the appearance feature group and different from the first appearance feature. The asymmetric class training data is provided to the training unit 119.

[0094] 11 is a schematic diagram illustrating a fifth example of generating asymmetric class training data. In the fifth embodiment, the asymmetric class training data generator 518 generates appearance features 502a# and 502b# by applying a first affine transformation to two appearance features 502a and 502b included in a certain appearance feature group 502. The asymmetric class training data generator 518 also generates appearance features 502c# and 502d# by applying a second affine transformation to two appearance features 502c and 502d included in the same appearance feature group 502. The asymmetric class training data generation unit 518 then replaces the converted appearance features 502c# and 502d# with the appearance features 502a and 502b, and replaces the converted appearance features 502a# and 502b# with the appearance features 502c and 502d, thereby generating an asymmetric appearance feature group 502#. Note that the converted appearance features 502c# and 502d# may be replaced with the appearance features 502c and 502d, and the converted appearance features 502a# and 502b# may be replaced with the appearance features 502a and 502b.

[0095] In the example of Figure 11, one appearance feature group 502 is used, but appearance features extracted from one appearance feature group 502 and subjected to affine transformation may be replaced with appearance features from another appearance feature group.

[0096] As described above, according to the fifth embodiment, at least one appearance feature group 502 is used to generate an appearance feature group that can be easily classified into an asymmetric class, and learning can be performed.

[0097] 1, a learning detection system 600 according to the sixth embodiment includes a learning device 610 and a state detection device 130. The state detection device 130 of the learning detection system 600 according to the sixth embodiment is similar to the state detection device 130 of the learning detection system 100 according to the first embodiment.

[0098] As shown in FIG. 2 , the learning device 610 includes a communication unit 111, a time-series appearance feature calculation unit 112, a state class learning data generation unit 117, an asymmetric class learning data generation unit 618, a learning unit 119, and a learning model storage unit 120.

[0099] The communication unit 111, the time-series appearance feature calculation unit 112, the state class learning data generation unit 117, the learning unit 119, and the learning model storage unit 120 of the learning device 610 in embodiment 6 are similar to the communication unit 111, the time-series appearance feature calculation unit 112, the state class learning data generation unit 117, the learning unit 119, and the learning model storage unit 120 of the learning device 110 in embodiment 1.

[0100] The asymmetric class training data generation unit 618 processes one or more appearance features included in the plurality of appearance features constituting the appearance feature group, thereby changing some of the plurality of features indicated in time series by the plurality of appearance features, thereby generating an asymmetric appearance feature group. The asymmetric class training data generation unit 618 then generates asymmetric class training data by assigning an asymmetric label to the asymmetric appearance feature group.

[0101] In the sixth embodiment, the asymmetric class training data generation unit 618 generates an asymmetric appearance feature group to include transformed appearance features obtained by transforming a plurality of features represented by a plurality of appearance features extracted from the appearance feature group. In other words, the asymmetric class training data generation unit 618 generates a first transformed appearance feature by performing a first affine transformation on a first appearance feature included in a first appearance feature group different from the appearance feature group, and generates a second transformed appearance feature by performing a second affine transformation different from the first affine transformation on a second appearance feature included in the first appearance feature group and a second appearance feature group different from the first appearance feature group, and generates a second transformed appearance feature by replacing two appearance features included in the appearance feature group with the first transformed appearance feature and the second appearance feature. The asymmetric class training data generation unit 618 generates an asymmetric appearance feature group.

[0102] 12 is a schematic diagram illustrating a sixth example of generating asymmetric class training data. In the sixth embodiment, asymmetric class training data generation unit 618 generates appearance features 602a# and 602b# by performing a first affine transformation on two appearance features 602a and 602b included in first appearance feature group 602. Also, asymmetric class training data generation unit 618 generates appearance features 603a# and 603b# by performing a second affine transformation on two appearance features 603a and 603b included in second appearance feature group 603. Then, asymmetric class learning data generation unit 618 replaces two appearance features included in the target appearance feature group with appearance features 602a# and 602b#, and replaces another two appearance features included in the target appearance feature group with appearance features 603a# and 603b#, thereby generating asymmetric appearance feature group 604#.

[0103] As described above, according to the sixth embodiment, at least three appearance feature sets are used to generate an appearance feature set that can be easily classified into an asymmetric class, and learning can be performed.

[0104] Seventh Embodiment As shown in Fig. 1, a learning detection system 700 according to a seventh embodiment includes a learning device 710 and a state detection device 730 as an inference device.

[0105] 13 is a block diagram showing a schematic configuration of a learning device 710 according to embodiment 7. The learning device 710 includes a communication unit 111, a time-series appearance feature calculation unit 112, a state class training data generation unit 117, an asymmetric class training data generation unit 118, a learning unit 119, a learning model storage unit 120, a relearning unit 721, a display unit 722, and an input unit 723.

[0106] The communication unit 111, the time-series appearance feature calculation unit 112, the state class learning data generation unit 117, the asymmetric class learning data generation unit 118, the learning unit 119, and the learning model storage unit 120 of the learning device 710 in embodiment 7 are similar to the communication unit 111, the time-series appearance feature calculation unit 112, the state class learning data generation unit 117, the asymmetric class learning data generation unit 118, the learning unit 119, and the learning model storage unit 120 of the learning device 110 in embodiment 1.

[0107] When a judgment using the results of inference by the learning model is incorrect, the re-learning unit 721 re-learns the learning model using re-learning learning data that includes at least data generated by assigning asymmetric labels to one or more appearance feature groups included in the appearance feature group used in the inference.

[0108] Specifically, the relearning unit 721 uses the set of appearance feature values ​​for relearning from the state detection device 730 to re-learn the learning model stored in the learning model storage unit 120. For example, the relearning unit 721 generates learning data for relearning by displaying the set of appearance feature values ​​for relearning on the display unit 722 and receiving input of correct labels for each appearance feature value included in the set of appearance feature values ​​for relearning from the user of the learning device 710 via the input unit 723. The relearning unit 721 then re-learns the learning model using the generated learning data for relearning.

[0109] As another example, when the conditions for issuing an alert are met (for example, when the upright position lasts for 0.5 seconds or more, the falling position lasts for 0.5 seconds or more, and the lying position lasts for 5 seconds or more, and the asymmetry during that period is less than 0.5 seconds), the relearning unit 721 displays this fact on the display unit 722, and when it receives input from the user via the input unit 723 that there was no fall, it adds a label of "asymmetry" to the group of appearance features in the issuing sequence for which the result of inference by the learning model was "falling," and generates learning data for relearning.

[0110] This is because lying down and standing are state labels that appear relatively frequently, so there is concern that relearning may have a negative effect, whereas falling is less likely to occur, as it appears less frequently.

[0111] In other words, when a judgment using the results of inference by the learning model is incorrect, the re-learning unit 721 generates learning data for re-learning by adding an "asymmetric" label to one or more groups of appearance features that were inferred to represent a specific state (here, falling) from among the groups of appearance features used in the incorrect judgment.

[0112] The re-learning unit 721 may re-learn the learning model using only the appearance feature groups to which the asymmetric label has been input as training data for re-learning.

[0113] Here, the re-learning unit 721 may perform re-learning when it receives a set of appearance feature values ​​for re-learning via the communication unit 111, or may store the sets of appearance feature values ​​for re-learning in a memory unit (not shown) and perform re-learning when a predetermined number of sets of appearance feature values ​​for re-learning have been stored, or when sets of appearance feature values ​​for re-learning have been stored for a predetermined period of time.

[0114] The retrained learning model is sent to the state detection device 230 via the communication unit 111. The display unit 722 displays various screens. The input unit 723 accepts input.

[0115] The learning device 710 described above can also be realized by a computer such as the PC 10 shown in Fig. 4. For example, the relearning unit 721 can also be realized by the processor 13 loading a program stored in the storage 11 into the memory 12 and executing the program. The display unit 722 can be realized by the display 16. The input unit 723 can be realized by the input I / F 15.

[0116] 14 is a block diagram showing a schematic configuration of a state detection device 730 according to embodiment 7. The state detection device 730 includes a communication unit 131, a time-series appearance feature calculation unit 132, a learning model storage unit 137, an inference unit 738, an alarm generation unit 739, an accumulation unit 740, a relearning determination unit 741, and an input unit 742.

[0117] The communication unit 131, the time-series appearance feature calculation unit 132, and the learning model storage unit 137 of the state detection device 730 in embodiment 7 are similar to the communication unit 131, the time-series appearance feature calculation unit 132, and the learning model storage unit 137 of the state detection device 130 in embodiment 1.

[0118] Inference unit 738 performs inference in the same manner as inference unit 138 in the first embodiment, and provides the inference result and the set of appearance feature amounts input for performing the inference to alarm unit 739.

[0119] Alert issuing unit 739 determines whether to issue an alert, as in embodiment 1, and issues an alert if it determines that an alert should be issued. If alert issuing unit 739 determines that an alert should be issued, it stores in storage unit 740 the set of appearance feature amounts that were input to infer the inference result used in that determination.

[0120] The relearning determination unit 741 determines whether or not to perform relearning using the set of appearance feature values ​​stored in the storage unit 740. For example, if an alarm issued in accordance with a set of appearance feature values ​​stored in the storage unit 740 is a false alarm, the relearning determination unit 741 may determine to perform relearning using the corresponding set of appearance feature values. If the relearning determination unit 741 determines to perform relearning, it transmits the corresponding set of appearance feature values ​​to the learning device 710 via the communication unit 131.

[0121] Specifically, after the alarm issuing unit 739 issues an alarm, the re-learning judgment unit 741 receives input of true or false for the alarm from the user of the status detection device 730 via the input unit 742, and determines whether the alarm was a false alarm.

[0122] In addition, when the re-learning judgment unit 741 receives the re-learned learning model via the communication unit 131, it deletes the learning model stored in the learning model memory unit 137 and stores the re-learned learning model in the learning model memory unit 137.

[0123] The input unit 742 accepts input.

[0124] The above-described state detection device 730 can also be realized by, for example, a computer such as the PC 10 shown in Fig. 4. For example, the accumulation unit 740 can also be realized by the storage 11 or the memory 12. The relearning determination unit 741 can also be realized by loading a program stored in the storage 11 into the memory 12 and executing that program. The input unit 742 can be realized by the input I / F 15.

[0125] As described above, according to the seventh embodiment, when a false alarm occurs, the learning model can be efficiently retrained.

[0126] In the seventh embodiment described above, relearning is performed by the learning device 710, but the seventh embodiment is not limited to this example. For example, the state detection device 730 may further include a relearning unit and a display unit that perform the same functions as the relearning unit 721 and the display unit 722, and the input unit 742 may be made to perform the same function as the input unit 723, thereby enabling the state detection device 730 to relearn the learning model stored in the learning model storage unit 137.

[0127] Furthermore, in the seventh embodiment described above, the learning model trained using the state class learning data and asymmetric class learning data generated by the learning device 710 is retrained using the retraining learning data generated by the state detection device 730, but the seventh embodiment is not limited to this example. For example, the learning device 710 may generate a learning model using the state class learning data, and then use the retraining learning data generated using the results of inference using the learning model as asymmetric class learning data to further train the learning model. In other words, the learning device may include: a state class learning data generation unit that generates state class learning data by assigning a state label, which is a label indicating a predetermined state, to a group of appearance features that indicates, in time series, a plurality of appearance features calculated from each of a plurality of frames of a video capturing a predetermined state of the target; an asymmetric class learning data generation unit that generates asymmetric class learning data by assigning an asymmetric label, which indicates asymmetric, to a group of appearance features that indicates, in time series, a plurality of appearance features including appearance features that indicate features in the appearance of a first target and appearance features that indicate features in the appearance of a second target, calculated from a plurality of frames of a video comprising a plurality of frames in which a first target is captured in at least one frame and a second target is captured in at least one frame; and a learning unit that receives input of a group of appearance features acquired from a video and generates a learning model for inferring a state or asymmetric corresponding to the input by learning using the state class learning data and the asymmetric class learning data. Note that the appearance features of the first object and the appearance features of the second object included in a single appearance feature group are calculated from different frames. For example, when inferring the state of the first object, the appearance feature calculation unit 135 may include the appearance features of the second object in the appearance feature group of the first object. In such cases, an error may occur in the judgment using the inference results. By adding an asymmetric label to the appearance feature group obtained in such cases and generating asymmetric class training data, accurate inference can be performed based on actual operation.

[0128] 100, 200, 300, 400, 500, 600, 700 Learning detection system, 110, 210, 310, 410, 510, 610, 710 Learning device, 111 Communication unit, 112 Time-series appearance feature calculation unit, 113 Video acquisition unit, 114 Time-series image group extraction unit, 115 Appearance feature calculation unit, 117 State class learning data generation unit, 118, 218, 318, 418, 518, 618 Asymmetric class learning data generation unit, 119 Learning unit, 120 Learning model storage unit, 721 Re-learning unit, 722 Display unit, 723 Input unit, 130, 730 State detection device, 131 Communication unit, 132 Time-series appearance feature calculation unit, 133 Image acquisition unit, 134 time-series image group extraction unit, 135 appearance feature calculation unit, 137 learning model storage unit, 138 inference unit, 139, 739 alarm generation unit, 740 accumulation unit, 741 relearning judgment unit, 742 input unit.

Claims

1. A learning device comprising: a state class learning data generation unit that generates state class learning data by assigning state labels that indicate a predetermined state to a group of appearance features that indicate, in time series, a plurality of appearance features that are calculated from each of a plurality of frames of video that captures a predetermined state of an object; an asymmetric class learning data generation unit that processes one or more appearance features included in the plurality of appearance features to change a portion of the plurality of features that are indicated in time series by the plurality of appearance features, to generate a group of asymmetric appearance features, and assigns an asymmetric label that indicates asymmetricity to the group of asymmetric appearance features, to generate asymmetric class learning data; and a learning unit that receives input of a group of appearance features acquired from a video, and generates a learning model for inferring a state or asymmetricity corresponding to the input, by learning using the state class learning data and the asymmetric class learning data.

2. The learning device according to claim 1, wherein the asymmetric class training data generation unit generates the asymmetric appearance feature group by replacing the one or more appearance feature values ​​with one or more appearance feature values ​​included in another appearance feature group.

3. The learning device according to claim 1, wherein the asymmetric class training data generation unit generates the asymmetric appearance feature group by switching the order in time series of a first appearance feature included in the appearance feature group and a second appearance feature included in the appearance feature group.

4. The learning device according to claim 1, wherein the asymmetric class learning data generation unit generates the asymmetric appearance feature group by performing an affine transformation on the one or more appearance features.

5. The learning device according to claim 1, wherein the asymmetric class training data generation unit generates the asymmetric appearance feature group by performing a first affine transformation on a first appearance feature included in the appearance feature group, and performing a second affine transformation on a second appearance feature included in the appearance feature group and different from the first appearance feature, the second affine transformation being different from the first affine transformation.

6. The learning device according to claim 5, wherein the asymmetric class training data generation unit switches the order in time series of the first appearance feature that has undergone the first affine transformation and the second appearance feature that has undergone the second affine transformation in the asymmetric appearance feature group.

7. The learning device according to claim 1, wherein the asymmetric class training data generation unit generates transformed appearance features by performing an affine transformation on appearance features included in an appearance feature group other than the appearance feature group, and generates the asymmetric appearance feature group by replacing one appearance feature included in the appearance feature group with the transformed appearance feature.

8. The learning device according to claim 1, wherein the asymmetric class training data generation unit generates a first transformed appearance feature by performing a first affine transformation on a first appearance feature included in a first appearance feature group different from the appearance feature group, generates a second transformed appearance feature by performing a second affine transformation different from the first affine transformation on a second appearance feature included in a second appearance feature group different from the appearance feature group and the first appearance feature group, and generates the asymmetric appearance feature group by replacing two appearance features included in the appearance feature group with the first transformed appearance feature and the second appearance feature.

9. The learning device according to any one of claims 1 to 8, further comprising a re-learning unit that, when a judgment using the results of inference by the learning model is incorrect, re-learns the learning model using re-learning training data that includes at least data generated by assigning the asymmetric label to one or more groups of appearance features included in the group of appearance features used in the inference.

10. The learning device according to claim 9, wherein the one or more groups of appearance features are inferred to represent a specific state.

11. A learning device comprising: a state class learning data generation unit that generates state class learning data by assigning state labels, which are labels indicating a predetermined state, to a group of appearance features that indicate, in time series, a plurality of appearance features calculated from each of a plurality of frames of video capturing a predetermined state of the object; an asymmetric class learning data generation unit that generates asymmetric class learning data by assigning an asymmetric label, which indicates asymmetric, to a group of appearance features that indicate, in time series, a plurality of appearance features that include appearance features that indicate features in the appearance of a first object and appearance features that indicate features in the appearance of a second object, calculated from the plurality of frames of video comprising the plurality of frames; and a learning unit that receives input of a group of appearance features acquired from video by learning using the state class learning data and the asymmetric class learning data, and generates a learning model for inferring a state or asymmetric corresponding to the input.

12. A time-series appearance feature calculation unit that calculates inferred appearance features, which are appearance features of a target, from each of a plurality of frames of an inferred video to generate an inferred appearance feature group that indicates a plurality of inferred appearance features in a time series; an inference unit that receives input of a group of appearance features acquired from a video and infers a state or asymmetry corresponding to the group of inferred appearance features by inputting the group of inferred appearance features into a learning model that is generated to infer a state or asymmetry corresponding to the input, by learning using state class learning data generated by assigning state labels that are labels that indicate a predetermined state to a group of appearance features that indicate, in time series, a plurality of appearance features that are calculated from each of a plurality of frames of a video capturing the object in a predetermined state, the plurality of appearance features being the appearance features that indicate features in the appearance of the object; and an inference unit that receives input of a group of appearance features acquired from a video and infers a state or asymmetry corresponding to the input of the inferred appearance feature group by learning using asymmetry class learning data generated by processing one or more appearance features included in the plurality of appearance features to change some of the plurality of features that are indicated in time series by the plurality of appearance features, and assigning an asymmetry label that indicates asymmetry to the group of asymmetric appearance features.

13. The inference device according to claim 12, further comprising an alarm issuing unit that uses the results of the inference made by said inference unit to determine whether or not to issue an alarm.

14. The inference device described in claim 13, characterized in that the alarm issuing unit decides to issue the alarm when it determines based on the result that a person included in the inference video has fallen.

15. An inference device as described in claim 13 or 14, further comprising a re-learning unit that, when the decision to issue an alert is incorrect, re-learns the learning model using re-learning training data that includes at least data generated by assigning the asymmetric label to one or more inferred appearance features included in the group of inferred appearance features used in the inference.

16. A time-series appearance feature calculation unit that calculates inferred appearance features, which are appearance features of a target, from each of a plurality of frames of an inferred video to generate an inferred appearance feature group that indicates a plurality of inferred appearance features in a time series; an inference unit that receives an input of a group of appearance features acquired from a video and infers a state or asymmetry corresponding to the group of inferred appearance features by inputting the group of inferred appearance features into a learning model that is generated to infer a state or asymmetry corresponding to the input, by learning using state class learning data generated by assigning state labels, which are labels indicating a predetermined state, to a group of appearance features that indicate, in time series, a plurality of appearance features that are calculated from each of a plurality of frames of a video that captures a predetermined state of the object, the plurality of appearance features being appearance features that indicate features in the appearance of the single object, and asymmetry class learning data generated by assigning an asymmetry label, which indicates asymmetry, to a group of appearance features that indicate, in time series, a plurality of appearance features that include appearance features that indicate features in the appearance of a first object and appearance features that indicate features in the appearance of a second object, the plurality of frames of a video that consists of a plurality of frames.

17. A program causing a computer to function as: a state class learning data generation unit that generates state class learning data by assigning state labels that indicate predetermined states to a group of appearance features that indicate, in time series, a plurality of appearance features that are calculated from each of a plurality of frames of video capturing an object in a predetermined state of the object; an asymmetric class learning data generation unit that processes one or more appearance features included in the plurality of appearance features to change a portion of the plurality of features that are indicated in time series by the plurality of appearance features, to generate a group of asymmetric appearance features, and assigns an asymmetric label that indicates asymmetricity to the group of asymmetric appearance features, thereby generating asymmetric class learning data; and a learning unit that receives input of a group of appearance features acquired from video and generates a learning model for inferring the state or asymmetricity corresponding to the input by learning using the state class learning data and the asymmetric class learning data.

18. A program causing a computer to function as: a state class learning data generation unit that generates state class learning data by assigning state labels, which are labels indicating a predetermined state, to a group of appearance features that indicate, in time series, a plurality of appearance features calculated from each of a plurality of frames of video capturing a predetermined state of the object, the appearance features being composed of appearance features that indicate characteristics in the appearance of the object; an asymmetric class learning data generation unit that generates asymmetric class learning data by assigning an asymmetric label, which indicates asymmetric, to a group of appearance features that indicate, in time series, a plurality of appearance features that include appearance features that indicate characteristics in the appearance of a first object and appearance features that indicate characteristics in the appearance of a second object, calculated from the plurality of frames of video comprising the plurality of frames; and a learning unit that receives input of a group of appearance features acquired from video and generates a learning model for inferring a state or asymmetric corresponding to the input by learning using the state class learning data and the asymmetric class learning data.

19. A computer including a time-series appearance feature calculation unit that calculates inferred appearance features, which are appearance features of a target, from each of a plurality of frames of an inferred video to generate an inferred appearance feature group that indicates a plurality of inferred appearance features in a time series; a learning model generated to infer a state or asymmetry corresponding to an input of an appearance feature group acquired from a video, the learning model being configured to infer a state or asymmetry corresponding to the input, by inputting the inferred appearance feature group into the inferred appearance feature group, the inferred appearance feature group being input by learning using state class learning data generated by assigning state labels that are labels that indicate a predetermined state to a group of appearance features that indicate, in a time series, a plurality of appearance features that are calculated from each of a plurality of frames of a video capturing the object in the predetermined state, and processing one or more appearance features included in the plurality of appearance features to change some of the plurality of features that are indicated in time series by the plurality of appearance features, and 20. A computer including a time-series appearance feature calculation unit that calculates inferred appearance features, which are appearance features of a target, from each of a plurality of frames of an inferred video to generate an inferred appearance feature group that indicates a plurality of inferred appearance features in a time series; a learning model generated to infer a state or asymmetry corresponding to an input of a group of appearance features acquired from a video, the learning model receiving the input of the group of appearance features acquired from the video, and inputting the inferred appearance feature group into the learning model, the learning model functioning as an inference unit that infers a state or asymmetry corresponding to the input, by learning using state class learning data generated by assigning state labels, which are labels indicating a predetermined state, to a group of appearance features that indicate, in time series, a plurality of appearance features, the plurality of appearance features being calculated from each of a plurality of frames of a video capturing the single object in a predetermined state, the plurality of appearance features being appearance features indicating features in the appearance of the single object, and asymmetry class learning data generated by assigning an asymmetry label, which indicates asymmetry, to a group of appearance features that indicate, in time series, a plurality of appearance features, the plurality of appearance features being calculated from the plurality of frames of a video comprising a plurality of frames.

21. A learning method comprising: generating state class training data by assigning state labels that indicate a predetermined state to a group of appearance features that indicate, in time series, a plurality of appearance features that are calculated from each of a plurality of frames of video capturing an object in a predetermined state of the object; processing one or more appearance features included in the plurality of appearance features to change a portion of the plurality of features that are indicated in time series by the plurality of appearance features to generate a group of asymmetric appearance features; assigning an asymmetric label that indicates asymmetricity to the group of asymmetric appearance features to generate asymmetric class training data; and learning using the state class training data and the asymmetric class training data to generate a learning model that receives input of a group of appearance features acquired from the video and infers the state or asymmetricity corresponding to the input.

22. A learning method comprising: generating state class training data by assigning state labels, which are labels indicating a predetermined state, to a group of appearance features, which are calculated from each of a plurality of frames of video capturing a predetermined state of a single object and which represent, in time series, a plurality of appearance features comprising appearance features indicating characteristics in the appearance of the single object; generating asymmetric class training data by assigning asymmetric labels, which indicate asymmetricity, to a group of appearance features, which are calculated from the plurality of frames of video capturing a plurality of frames and represent, in time series, a plurality of appearance features including appearance features indicating characteristics in the appearance of a first object and appearance features indicating characteristics in the appearance of a second object; and training using the state class training data and the asymmetric class training data to receive input of a group of appearance features acquired from the video and generate a learning model for inferring a state or asymmetricity corresponding to the input.

23. Calculating inferred appearance features, which are appearance features of an object, from each of a plurality of frames of an inferred video to generate a group of inferred appearance features that indicate a plurality of inferred appearance features in a time series; generating state class learning data by assigning state labels that indicate a predetermined state to a group of appearance features that indicate a plurality of appearance features in a time series, the group being composed of appearance features that indicate features in the appearance of a single object, calculated from each of a plurality of frames of a video capturing a predetermined state of the object, and processing one or more appearance features included in the plurality of appearance features to generate a group of asymmetric appearance features by changing a portion of the plurality of features that are indicated in time series by the plurality of appearance features; and learning using asymmetric class learning data generated by assigning an asymmetric label that indicates asymmetricity to the group of asymmetric appearance features, thereby receiving input of a group of appearance features acquired from a video and inputting the inferred appearance feature group into a learning model that is generated to infer a state or asymmetry corresponding to the input. An inference method characterized by:

24. Calculating inferred appearance features, which are appearance features of an object, from each of a plurality of frames of an inferred video to generate a group of inferred appearance features that indicate a plurality of inferred appearance features in a time series; learning using state class learning data generated by assigning state labels, which are labels indicating a predetermined state, to a group of appearance features that indicate a plurality of appearance features in a time series, which are calculated from each of a plurality of frames of a video capturing a predetermined state of the object; and asymmetric class learning data generated by assigning asymmetric labels, which indicate asymmetric, to a group of appearance features that indicate a plurality of appearance features in a time series, which include appearance features that indicate features in the appearance of a first object and appearance features that indicate features in the appearance of a second object, calculated from the plurality of frames of a video consisting of a plurality of frames; receiving input of a group of appearance features acquired from a video and inputting the inferred appearance feature group into a learning model generated to infer a state or asymmetric corresponding to the input, thereby inferring a state or asymmetric corresponding to the inferred appearance feature group. An inference method characterized by: